EdgeKit Bot Analytics & Blocking
Every AI crawler on terms you set.
EdgeKit sorts every page request to your titles into people, search engines and AI crawlers, with what each AI company takes and the readers it sends back. Then a rule for each crawler: allow it, block it or charge it per page, on your own domain.
- AI crawls
- 395,300
- AI referrals
- 2,140
- Crawl-to-referral
- 185:1
The choice today
AI companies read your archive every day. The usual answers leave you guessing.
Ask in robots.txt
A request, not a lock. A crawler that ignores it keeps reading, and nothing shows you which ones do.
Switch AI crawlers off at the CDN
The switch gives a crawler the whole article or a refusal. There is no answer in between for the AI search engine you want citing you.
Look in your web analytics
Crawlers don't run your tracking script, so they never show up there. Your archive is read, and the dashboard stays quiet.
What it does
Count every crawler, then decide each one
The count is what you bring to a licensing conversation. The rules are what you run until one is signed.
Traffic
Every request, people and machines alike
Every page request to your titles, counted by who sent it: your signed-in readers, likely people, crawlers that prove who they are, and automation that does not. Each crawler is named with the company behind it and what it does: training a model, answering in AI search, fetching a page for one person's question, or indexing for search.
- Classify by your CDN's verdict or by EdgeKit's own bot protection, with one switch. The two sit side by side and are never merged.
- Signed-in people come from your own EdgeKit sign-in, so they count the same under either verdict.
- Filter by title, traffic, category, crawler or operator, over 24 hours, 7, 30 or 90 days.
- AI training214,600
- AI search128,300
- AI user fetch52,400
- Other bots1,152,000
- Likely human286,500
- Signed-in people61,200
Two verdicts, side by sideSwitch to your CDN and the same requests are classified by its verdict. Signed-in people read the same under both.
Referrals
Who takes, and who sends readers back
Readers who arrive from ChatGPT, Perplexity, Claude and other AI answer engines are counted by operator and source, beside that operator's crawls. The crawl-to-referral ratio says what each one's crawling earns you.
- Bandwidth per operator, and the status codes AI crawlers were answered over time.
- The most crawled pages, title by title, with CSV.
- What EdgeKit's decision answered each operator: allowed or denied.
Rules
Allow, block or charge, crawler by crawler
Rules sit in EdgeKit's decision editor, the one EdgeKit Paywall uses, and match top down on the crawler, its operator, its category and whether it is verified. Allow serves the page. Block answers 403 Forbidden. Charge answers 402 Payment Required with your price per page.
- A rule names a verified crawler. A request that only borrows its name is caught by a rule of its own.
- A crawler you hold a licence with gets its own allow, above the rest.
- Publish a change and it reaches your domain in 60 seconds, as a paywall change does.
Which crawlers get the pageChecked on every request
- Category is Search engineVerifiedAllow
- Category is AI searchVerifiedAllow
- Category is AI user fetchVerifiedAllow
- Crawler is BytespiderBlock
- Category is AI trainingVerifiedCharge$0.002 a page
- Claims a crawlerNot verifiedBlock
- OtherwisePeople and every other requestYour paywall decides
- Allow
- The page, served from your origin
- Charge
- 402 Payment Required, until the price is accepted
- Block
- 403 Forbidden
Enforcement
Decided on your own domain
EdgeKit runs on your own domain, in front of your titles, and applies your published crawler rules on every page request. A blocked crawl is refused at your domain. Where your servers ask EdgeKit's decision themselves, the same rules come back as the outcome.
- Verified search engines get the full page, so search keeps reading.
- A crawler no rule names is decided like an anonymous reader: on a page where EdgeKit knows which part is paywalled, it gets the preview.
- People are untouched: their requests go to your paywall's decision as before.
Request
GPTBotVerified crawler/markets/tariff-rulingAnswer
Allow200The page, from your originCharge402 Payment RequiredYour price, until it is acceptedBlock403 ForbiddenRefused at your domainYour servers ask EdgeKit?The same rules come back as the decision's outcome, for your servers to enforce.
Your data
One row per request, and nobody's identity in it
Every page request is one row in a dataset of your organization's own, kept 90 days: the title, the page, the status, the size, the crawler it claims and each verdict. No row holds an IP address, reader account, cookie or device ID, so the numbers can be shared without sharing anyone.
- Signed in is a yes or no on the row, never whose.
- Never pooled with another organization's traffic.
- Time
- 09:41:07 UTC
- Title · page
- The Northern Post · /markets/tariff-ruling
- Status · size
- 402 · 1.2 kB
- Crawler
- GPTBot · OpenAI · AI training
- CDN
- Verified bot
- Bot protection
- Crawler: GPTBot
- Rule
- Charge
Never stored
IP addressReader accountCookie or device IDIllustrativeSigned in: yes or no, never whose
Getting live
From the first count to the first rule in four steps
Your CMS, your CDN and your robots.txt stay as they are.
White-glove onboarding: we set it up with you, start to finish.
- 01
Put EdgeKit in front
Your developer puts EdgeKit in front of your titles, on your own domain, and we set it up with them. It fails open: if EdgeKit cannot answer, the page is served as it is today.
- 02
Read a month of crawling
The Bots pages fill as requests arrive: who crawls, what they take and what they send back, operator by operator.
- 03
Write the rules
Start where the numbers point: allow the engines that send readers back, then charge or block the crawlers that only take.
- 04
Publish
Rules reach your domain in 60 seconds, as paywall changes do. Change one the day a licence is signed.
Beside your stack
Adds the count and the rules. Replaces nothing.
Your CDN
Keeps serving your titles. EdgeKit reads its bot verdict and never overrides it.
Your robots.txt
Stays the request it is. A rule is the lock for the crawlers that ignore it.
Your licences
A licensed crawler gets an allow rule, and a licensing service's fetch addresses get the full page.
What you bring
It works inside what you already run.
- Your CDN in front of your titlesEdgeKit runs at your CDN, and the verdict on which crawlers are genuine is your CDN's. We check yours on the demo.
- EdgeKit sign-in, for the people countSigned-in readers are counted from your EdgeKit sign-in. Without it, people are counted as likely human and nobody as signed in.