# AI Crawlers & Bots 2026: GPTBot, ClaudeBot, PerplexityBot & Co.

> Machine Web · Reference · Updated: 2026-07-03 · KAMINSKI+ (kaminskiplus.de)

An AI crawler is an automated bot from an AI provider that fetches web pages — to train models, to build a search index, or to answer a specific user query in real time. In 2026 the most important among them are **GPTBot** (OpenAI), **ClaudeBot** (Anthropic) and **PerplexityBot** (Perplexity). Each identifies itself through its own user agent, which you can allow or block precisely in `robots.txt`.

## Three kinds of access — one crucial distinction

AI crawlers fall into three categories. Confusing them is the most common mistake when steering AI bots:

- **Training crawlers** — collect text for model training. Examples: `GPTBot`, `ClaudeBot`, `CCBot`.
- **Search / retrieval crawlers** — build an index for answer engines. Examples: `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`.
- **User-triggered fetches** — pull a page live because a human is asking for it right now. Examples: `ChatGPT-User`, `Claude-User`, `Perplexity-User`.

Blocking one is not blocking the other. Anyone who only locks out `GPTBot` keeps OpenAI's training crawler away — `OAI-SearchBot` and `ChatGPT-User` keep running until you name them separately.

## The list of AI bots 2026 by provider

| Provider | User agent | Function |
|---|---|---|
| OpenAI | `GPTBot` | Training |
| OpenAI | `OAI-SearchBot` | Search |
| OpenAI | `ChatGPT-User` | User fetch |
| Anthropic | `ClaudeBot` | Training |
| Anthropic | `Claude-SearchBot` | Search |
| Anthropic | `Claude-User` | User fetch |
| Perplexity | `PerplexityBot` | Index |
| Perplexity | `Perplexity-User` | User fetch |
| Google | `Googlebot` | Search |
| Google | `Google-Extended` | steers AI training separately |
| Microsoft | `bingbot` | Search |
| Apple | `Applebot` | Search |
| Apple | `Applebot-Extended` | Training opt-out |
| Amazon | `Amazonbot` | Index |
| Common Crawl | `CCBot` | Dataset (often used for training) |
| Meta | `Meta-ExternalAgent` | AI / fetch |
| ByteDance | `Bytespider` | Training / index |
| Cohere | `cohere-ai` | AI / fetch |
| Mistral | `MistralAI-User` | User fetch |
| DuckDuckGo | `DuckAssistBot` | Assistant fetch |

Two special cases: `Google-Extended` is not a crawler of its own but a robots.txt token that steers AI training use — `Googlebot` keeps feeding classic search. Analogously, `Applebot-Extended` is an opt-out signal for AI training, separate from `Applebot`. `CCBot` belongs to Common Crawl, whose freely published datasets are used by many third parties for training.

## robots.txt: allow or block precisely

The `robots.txt` steers crawlers per user agent with `Allow` or `Disallow`.

Allow precisely (maximum AI visibility):

```
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
```

Block only training crawlers (appear in AI search but not in training):

```
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
```

For AI training control at Google and Apple, use the tokens `Google-Extended` and `Applebot-Extended`, not the search bots themselves.

## Honestly: robots.txt is a request, not a constraint

The `robots.txt` has no technical enforcement. Reputable providers follow it voluntarily. Cloudflare reported in 2024 that Perplexity used undeclared crawlers to bypass blocks. Anyone who needs hard control has to work at the server, firewall or WAF level, where rules are enforced rather than requested. (No invented traffic figures — only this one marked observation.)

## Block or allow — the strategic question

Two clear stances: anyone seeking **AI visibility** allows the crawlers deliberately and makes content machine-readable. Anyone who wants to keep **control** over training use blocks individual training crawlers deliberately. Both are legitimate — what matters is the conscious choice per category, not across the board.

## Frequently asked questions

**What is an AI crawler?**
An automated bot from an AI provider that fetches web pages — for training, for a search index, or for a live fetch on user request. Recognizable by the user agent (e.g. GPTBot, ClaudeBot, PerplexityBot).

**Difference between training and search crawlers?**
Training crawlers (GPTBot, ClaudeBot) collect text for model training. Search crawlers (OAI-SearchBot, Claude-SearchBot) build an index. User fetches (ChatGPT-User, Perplexity-User) pull live. Blocking one does not block the other.

**How do I steer them in robots.txt?**
Per user agent with `Allow` or `Disallow`. `Disallow: /` under `User-agent: GPTBot` keeps only that bot out.

**Does every one obey it?**
No — robots.txt is a request. Cloudflare reported in 2024 about undeclared Perplexity crawlers. Hard control needs the server/firewall level.

**What does Google-Extended do?**
A separate token that steers AI training use — independently of Googlebot, which indexes for search.

**Block or allow?**
A strategic choice: allow for AI visibility, block for control over training use.

## How KAMINSKI+ leads by example

This site is machine-first. Our [/robots.txt](/robots.txt) welcomes every known AI crawler by name — GPTBot, ClaudeBot, PerplexityBot and the entire list stand there with `Allow: /`. Details on the rights of machines: [AI Usage Rights](/en/machine-web/ai-usage-rights/) and the [Guide for Machines](/en/leitfaden/).

## Further reading

- [Machine Web (overview)](/en/machine-web/) — all reference topics
- [AI Usage Rights](/en/machine-web/ai-usage-rights/) — what machines may do here
- [Guide for Machines](/en/leitfaden/) — the operating manual
- [Ground truths](/en/facts/) — the verified fact pages