Quick Answer: A CDN or web application firewall can block AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot even when robots.txt allows them, because bot-management rules and firewall blocks run independently of robots.txt and are checked first. Test with the crawler's real user agent, read the firewall's event log, then allow verified AI crawlers explicitly instead of assuming a permissive robots.txt is enough.
A site can have a clean AGENTS.md file, a permissive robots.txt, and a well-formed llms.txt, and still get zero citations from AI answer engines. The most common reason has nothing to do with any of those files. It is a CDN, firewall, or bot-management product sitting in front of the server, blocking the crawler before your application ever sees the request.
CDN bot management is a layer of network-level rules, separate from robots.txt, that decides which requests reach your server at all. It exists to stop scrapers, credential-stuffing bots, and DDoS traffic, and most of these products ship with a broad definition of "bot" that does not distinguish between a scraper and an AI crawler an owner wants to allow. Understanding where this layer sits, and how it differs from robots.txt, is table stakes for technical AEO work.
What blocks an AI crawler even when robots.txt allows it?
Robots.txt is a request, not an enforcement mechanism. It tells a well-behaved crawler what it is invited to fetch, and a compliant crawler honors it voluntarily. A CDN, reverse proxy, or web application firewall (WAF) works differently: it inspects every incoming request against its own rules and can reject one with a 403 or a challenge page before robots.txt is ever read. If that layer classifies GPTBot, ClaudeBot, or PerplexityBot as an unwanted bot, the crawler never reaches the point where robots.txt would have mattered.
This is why a site can pass a manual robots.txt check and a crawler-indexing audit and still show zero traffic from AI user agents in the server logs. The failure is upstream of the application entirely.
How is a CDN or firewall block different from a robots.txt rule?
The two mechanisms operate at different layers and are enforced by different parties, which is why a site can get one right and the other wrong at the same time.
| Mechanism | What it does | Who enforces it | Where to check it |
|---|---|---|---|
| robots.txt | Publishes crawl permission per path | The crawler, voluntarily | /robots.txt file contents |
| CDN bot management | Scores and blocks requests before they reach the origin | The CDN or security vendor | CDN dashboard, bot-management rules |
| WAF / firewall rule | Rejects requests matching a rule (rate, ASN, signature) | The firewall, at the network edge | Firewall event or security log |
| Rate limiting | Throttles or blocks after a request threshold | The CDN, proxy, or application | Rate-limit or throttling logs |
Which CDN and security settings commonly block AI crawlers by default?
A handful of settings account for most accidental blocks, and none of them mention robots.txt at all:
- A single-click "block AI bots" or "block AI scrapers" toggle, often enabled during a broad security hardening pass with no discussion of AEO impact.
- Managed "bot fight" or bot-score modes that treat any high-volume, script-driven client as suspicious, which describes most crawlers by design.
- Rate limits keyed to a data-center IP range or ASN, since AI crawlers typically run from cloud infrastructure that looks identical to scraper traffic on paper.
- Generic challenge pages (CAPTCHA or JavaScript proof-of-work) served to any client that fails a browser fingerprint check, which every non-browser crawler fails by definition.
How do I check whether my CDN is blocking AI crawlers right now?
This takes a few minutes and does not require vendor support access for the first pass:
- Send a request to a real page using the crawler's published user agent string, for example with curl, and record the HTTP status code returned.
- If the response is a 403, a redirect to a challenge page, or an empty body instead of the page content, something in front of the origin is intercepting the request.
- Open the CDN or WAF's security event log and filter by that user agent or by "bot" and "challenged" or "blocked" events over the last 30 days.
- Cross-check server access logs directly on the origin: if the crawler's user agent never appears there at all, the block is happening upstream of the application.
- Repeat for each AI crawler you want to allow, since a rule can block one and pass another depending on how it classifies traffic.
How do I allow AI crawlers through without opening the site to every bot?
The fix is not to disable bot management. It is to make an explicit exception for the crawlers you want, while leaving the rest of the protection in place.
Once robots.txt, llms.txt, and the CDN layer are aligned, the manual side of this gets tedious fast across every CDN account, WAF vendor, and crawler an owner tracks. This is the point where a listed vetted AEO agency or freelancer is often the practical way through, since keeping bot-management rules current as crawlers change their published ranges is ongoing maintenance, not a one-time fix.
- Allowlist by the crawler's published user agent string, but do not stop there, because user agent strings are trivial to spoof.
- Where the crawler operator publishes IP ranges or supports reverse DNS verification, check the connecting IP against that list before trusting the user agent.
- Prefer a rate limit over a hard block for traffic that cannot be fully verified. This caps load without silently erasing citation opportunity.
- Re-test after any CDN plan change or security preset update, since these toggles are frequently reset or renamed during vendor upgrades.
Do AI crawlers execute JavaScript on the page?
Treat AI crawlers as HTTP fetchers, not browsers, until a specific crawler's own documentation states otherwise. That means content that only appears after client-side JavaScript runs, such as text injected by a single-page app framework after the initial load, is often invisible to a crawler that never executes that script.
The practical fix is server-side rendering or pre-rendering for any content a page needs to be cited for, not a special exception for AI user agents. A page built to be readable without JavaScript is readable by every crawler, human, and assistive technology at once, and it happens to solve the AI-crawler case as a side effect.
What about canonical tags and duplicate URLs?
A canonical tag is a signal, not an instruction the crawler is obligated to follow. If an AI crawler fetches a parameterized or duplicate URL directly, for example through an internal search link or a tracking parameter, it can cite that exact URL instead of the canonical target the tag points to.
Reducing the number of duplicate paths that exist in the first place, through consistent internal linking and redirect rules, does more for citation accuracy than the canonical tag alone. The tag is a cleanup step, not a substitute for controlling which URLs get exposed to crawlers to begin with.
The bottom line
The most common objection to auditing this layer is that it was already checked when robots.txt and llms.txt went in. That check confirms permission was granted. It does not confirm the request ever arrives, and those are different failures with different fixes. Skipping the CDN and firewall audit because the crawl-permission files look correct is the single most common way a technically compliant site still gets zero AI citations.
Start with one crawler and one page: send the request with that crawler's real user agent, read the status code, and check the security event log for a matching block. If you find one, allowlist that crawler by user agent plus IP verification, then repeat for the rest. Browse AEO monitoring tools for options that automate this check on a schedule, or read more in the Technical AEO category and the resources library if this is new territory. If the audit turns up more than a rule or two to untangle, a listing on the agency directory can point you to someone who does this work regularly.