Three kinds of access — one crucial distinction
Before you block a single bot, it pays to look at the intent behind it. AI crawlers fall into three categories, and confusing them is the most common mistake when steering AI bots:
- Training crawlers — collect text to train language models from it. Examples:
GPTBot,ClaudeBot,CCBot. Block them and your content does not flow into the next model. - Search / retrieval crawlers — build an index for answer engines. Examples:
OAI-SearchBot,Claude-SearchBot,PerplexityBot. Block them and you disappear from AI search results. - User-triggered fetches — pull a page because a human is asking for it right now. Examples:
ChatGPT-User,Claude-User,Perplexity-User. Block them and an assistant cannot open your page live for a user.
Blocking one is not blocking the other. Anyone who only locks out GPTBot keeps OpenAI's training crawler away — but OAI-SearchBot and ChatGPT-User keep running until you name them separately.
The list of AI bots 2026 by provider
This table maps the most important AI user agents to their provider and their function. It is the quick reference when you write a robots.txt for AI bots or interpret server logs.
| Provider | User agent | Function |
|---|---|---|
| OpenAI | GPTBot | Training |
OAI-SearchBot | Search | |
ChatGPT-User | User fetch | |
| Anthropic | ClaudeBot | Training |
Claude-SearchBot | Search | |
Claude-User | User fetch | |
| Perplexity | PerplexityBot | Index |
Perplexity-User | User fetch | |
Googlebot | Search | |
Google-Extended | steers AI training separately | |
| Microsoft | bingbot | Search |
| Apple | Applebot | Search |
Applebot-Extended | Training opt-out | |
| Amazon | Amazonbot | Index |
| Common Crawl | CCBot | Dataset (often used for training) |
| Meta | Meta-ExternalAgent | AI / fetch |
| ByteDance | Bytespider | Training / index |
| Cohere | cohere-ai | AI / fetch |
| Mistral | MistralAI-User | User fetch |
| DuckDuckGo | DuckAssistBot | Assistant fetch |
Two special cases deserve a look. Google-Extended is not a standalone crawler but a pure robots.txt token: it decides whether Google's AI products use your content for training — while Googlebot keeps feeding classic search. Analogously, Applebot-Extended is an opt-out signal for Apple's AI training, separate from Applebot, which serves search. CCBot belongs to Common Crawl, which publishes its datasets freely — they are used by many third parties for training.
robots.txt: allow or block precisely
The robots.txt steers crawlers per user agent. Each block names a bot and, with Allow or Disallow, opens a path to it or denies it. This is what a precise allow for maximum AI visibility looks like:
And this is how you block only the training crawlers precisely, while search and user fetches stay unobstructed — useful when you want to appear in AI search results but not flow into training:
- Decide per category — training, search, user fetch — whether you want it.
- Name each affected user agent individually. A blocked training crawler does not block a search crawler.
- For AI training control at Google and Apple, use the tokens
Google-ExtendedandApplebot-Extended, not the search bots themselves. - Check your server logs: which user agents actually come by? Only then do you know what to regulate.
Honestly: robots.txt is a request, not a constraint
The robots.txt has no technical enforcement. It is a polite notice that reputable providers follow — voluntarily. Anyone who ignores it is not stopped by it. Cloudflare reported in 2024 that Perplexity used undeclared crawlers that did not identify themselves as PerplexityBot, in order to bypass blocks. Anyone who needs hard control over access has to work one level deeper: at the server, firewall or WAF level, where rules are enforced rather than requested.
Block or allow — the strategic question
There is no universally right answer, only two clear stances. Anyone seeking AI visibility — to be cited when someone asks ChatGPT, Claude or Perplexity — allows the crawlers deliberately and makes their own content machine-readable. Anyone who wants to keep control over training use blocks individual training crawlers deliberately and accepts lower presence in AI answers in return. Both are legitimate; what matters is that the choice is made consciously — per category, not across the board.
Frequently asked questions
What is an AI crawler?
An AI crawler is an automated bot from an AI provider that fetches web pages — either to train models, to build a search index, or to answer a specific user query in real time. Every crawler identifies itself through a user agent, such as GPTBot, ClaudeBot or PerplexityBot.
What is the difference between training crawlers and search crawlers?
Training crawlers like GPTBot or ClaudeBot collect text to train models from it. Search crawlers like OAI-SearchBot or Claude-SearchBot build an index for answer engines. User-triggered fetches like ChatGPT-User or Perplexity-User pull a page because a human just asked for it. Blocking one does not automatically block the other.
How do I block or allow AI crawlers in robots.txt?
In robots.txt you steer each bot per user agent with Allow or Disallow. A block “User-agent: GPTBot” followed by “Disallow: /” keeps OpenAI's training crawler out, but leaves search and user crawlers untouched as long as you do not name them as well.
Does every AI crawler obey robots.txt?
No. robots.txt is a request, not a technical constraint. Reputable providers respect it, but Cloudflare reported in 2024 that Perplexity used undeclared crawlers to bypass blocks. Anyone who needs hard control has to work at the server or firewall level.
What does Google-Extended do?
Google-Extended is not a crawler of its own but a separate robots.txt token. It controls whether Google's AI products may use your content for training — independently of the fact that Googlebot keeps indexing your pages for classic search.
Should I block or allow AI crawlers?
This is a strategic decision. Anyone who wants AI visibility — to be cited in ChatGPT, Claude or Perplexity — allows the crawlers deliberately. Anyone who wants to keep control over training use blocks individual training crawlers deliberately. KAMINSKI+ deliberately welcomes every known AI crawler.
How KAMINSKI+ leads by example
This site is built machine-first, and it holds to its own recommendation. Our /robots.txt welcomes every known AI crawler by name — GPTBot, ClaudeBot, PerplexityBot and the entire list above stand there with Allow: /. What that means in detail, which rights machines have here and how they take in the content fastest, is set out under AI Usage Rights and in the Guide for Machines. No contradiction between claim and implementation: anyone who writes about crawler control should disclose their own.