What AI crawlers actually do — and what their traffic doesn't prove

One provider, several bots

The major AI providers no longer run a single crawler. They split the work into separate user agents with different purposes — and different rules. Broadly there are three jobs:

1. Training — collecting content that may be used to train models.
2. Search indexing — building an index that the provider's search or answer features retrieve from.
3. User-initiated fetching — visiting a page because a person asked the assistant something just now.

Here's how the providers' own documentation describes their bots (as of October 2026 — these change, so check the sources):

Two details catch people out. First, Google's AI Overviews and AI Mode use ordinary Googlebot crawling. Google-Extended controls whether content is used for Gemini training and grounding in other Google systems; it does not remove you from Search or its AI features. Second, user-initiated fetchers act on behalf of a person, and providers say robots.txt may not apply to them in the same way.

What robots.txt can and can't do

The Robots Exclusion Protocol is a standard (RFC 9309), but compliance is voluntary. It controls crawling, not indexing: a blocked URL can still appear in results if other pages link to it. And because each provider separates its bots, blocking a training crawler doesn't block that provider's search crawler — or the reverse.

That makes crawler access a business decision, made bot by bot. A publisher might block training crawlers while allowing search crawlers so it can still be cited. A software company might allow everything. Either is defensible; doing it by accident isn't.

Verify before you count

User-agent strings are trivially spoofed. Before you report any bot's traffic, verify it: OpenAI, Perplexity and Anthropic all publish IP lists for their bots; Google supports reverse-DNS verification; Cloudflare classifies verified bots. Those lists are for verifying traffic, not for opting out — Anthropic warns that blocking its IP addresses may not reliably opt you out, because it stops its crawler reading your robots.txt. Unverified bot traffic belongs in an unclassified automation bucket, not in your AI visibility report.

What crawler traffic doesn't prove

A crawler request proves one thing: a bot could fetch the page. It says nothing reliable about any of these, which are separate phenomena:

1. That the content was indexed or retained.
2. That it's retrieved when someone asks a relevant question.
3. That the system represents your business accurately.
4. That your page is cited or your brand mentioned.
5. That any person clicked through.
6. That an agent interacted with your service.
7. That anything converted.

Training crawlers in particular can fetch heavily from sites that are never cited anywhere. Rising crawler volume can just as easily mean a provider re-crawled its index as that you've become more relevant.

What to measure instead

• Google's AI surfaces: Search Console's Generative AI performance report and its overall Performance data.
• Copilot and Bing: Bing Webmaster Tools' AI Performance report, which shows citations and grounding queries — but not clicks.
• Assistant referrals: a segment in your analytics for known assistant referrers and assistant-added UTM parameters. Expect undercounting.
• Answer accuracy: a fixed set of prompts, run on a schedule, scored against a fact sheet. Report it as a sample, with dates.
• Outcomes: qualified leads and conversions, by source.

Keep each of these in its own bucket. The moment crawler hits, referrals and conversions are summed into one “AI visibility score”, the number stops meaning anything.

Sources

1. OpenAI — Overview of OpenAI crawlers
2. Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?
3. Perplexity — Perplexity crawlers
4. Google — AI features and your website
5. Google — Optimizing your website for generative AI features on Google Search
6. Bing Webmaster Tools — AI Performance
7. IETF — RFC 9309: Robots Exclusion Protocol