# AI usage rights: allow or block AI training on your website (ai.txt, robots.txt & co.)

> Machine Web · Reference · Updated: 2026-07-03 · KAMINSKI+ (kaminskiplus.de)

AI usage rights govern whether and how AI systems may fetch, index and use your website for training. You steer this through several stacked signals: the `robots.txt` per user agent, the separate training tokens `Google-Extended` and `Applebot-Extended`, the emerging `ai.txt`, the legal TDM reservation of rights, plus meta tags and HTTP headers. **Important:** none of these is a technical enforcement — they are requests and legal declarations that reputable providers honor. (Orientation, not legal advice.)

## Answer first: Can AI train on my website?

By default, yes. If your page is publicly reachable, an AI provider can fetch it as long as you don't actively object. So the question is not "whether" but "which signal you set". Five tools act on different levels and complement each other:

| Tool | Effect |
|---|---|
| `robots.txt` | access per user agent — standard, but only a request |
| `Google-Extended` / `Applebot-Extended` | steer AI training separately from search |
| `ai.txt` | machine-readable usage policy — permissive or restrictive |
| TDM reservation of rights | legal opt-out against text and data mining (EU among others) |
| Meta tags / HTTP headers | `X-Robots-Tag`, `noai`/`noimageai` — where supported |

## 1. robots.txt — per user agent

Steers crawlers per user agent with `Allow` or `Disallow`.

Block AI training:

```
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
```

Allow AI crawlers (max. visibility):

```
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
```

The `robots.txt` is a request, not a technical enforcement. Hard control needs the server/firewall/WAF level.

## 2. Google-Extended & Applebot-Extended

Google and Apple deliberately separate AI training from normal search via their own tokens. This is how you do a Google-Extended opt out without disappearing from classic search:

```
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
```

`Google-Extended` is not a crawler of its own but a robots.txt token; `Googlebot` continues to index unaffected by it. `Applebot-Extended` is the Apple counterpart to `Applebot`.

## 3. ai.txt — machine-readable usage policy

The `robots.txt` only governs access, not use. This gap is filled by the emerging `ai.txt` (usually at `/.well-known/ai.txt`): you name individual kinds of use — training, indexing, quoting, storing, inference — and allow or forbid them, **permissive** or **restrictive**. A young standard, not yet universally supported: a clear declaration of intent, not an enforceable contract. It complements other signals, it doesn't replace them.

## 4. TDM reservation of rights — legal opt-out

TDM stands for text and data mining — the automated analysis that underlies AI training. In the EU context, rights holders can reserve it for themselves; a reservation, ideally machine-readable, is a legal opt-out. The sharpest of the instruments named here, because it declares a legal position instead of merely asking for good behavior. **Orientation, not legal advice** — the effect depends on jurisdiction, type of work and design.

## 5. Meta tags & HTTP headers — noai / X-Robots-Tag

Pinpoint per page instead of centrally:

```
<meta name="robots" content="noai, noimageai">
X-Robots-Tag: noai, noimageai
```

`noai`/`noimageai` signal the wish not to use text or images, respectively, for AI training — but are not evaluated by every provider. Not a universally established standard like `noindex`. Again the rule applies: a request, not an enforcement.

## The debate: opt-out vs. opt-in, visibility vs. control

Today's web is mostly **opt-out**: usable until someone objects. Some rights holders demand **opt-in** — use only after consent. The conflict touches copyright directly and is open both legally and politically. For operators, a trade-off remains: **visibility** (allow, presence in AI answers) versus **control** (block, less AI visibility). No answer right for everyone — only the deliberate choice.

## Practice for both camps

**Allow (visibility):** `Allow: /` per crawler (incl. allow GPTBot, ClaudeBot, PerplexityBot); permissive `ai.txt`; content machine-readable via `llms.txt`; leave out `noai`, leave the Extended tokens open.

**Block (control):** block training crawlers specifically; set `Google-Extended` and `Applebot-Extended` to `Disallow`; restrictive `ai.txt`, where supported `noai`/`noimageai`, plus a TDM reservation if applicable; hard enforcement only via server/firewall/WAF rules.

## Frequently asked questions

**Can AI train on my website?**
By default yes, as long as you don't actively object. Limit it via robots.txt blocks of the training crawlers, Google-Extended/Applebot-Extended on Disallow and — in the EU context — a TDM reservation of rights. None is a technical enforcement.

**How do I block AI training in robots.txt?**
Name each training crawler individually with `Disallow: /` (GPTBot, ClaudeBot, CCBot …). For Google/Apple instead the tokens Google-Extended and Applebot-Extended.

**How do I do a Google-Extended opt out?**
`User-agent: Google-Extended` with `Disallow: /`. Removes content from Google's AI training, while Googlebot keeps indexing for search. Applebot-Extended follows the same pattern.

**How do I explicitly allow GPTBot?**
An `Allow: /` block per user agent (GPTBot, ClaudeBot, PerplexityBot …), complemented by a permissive ai.txt and a machine-readable llms.txt.

**What is ai.txt and the TDM reservation of rights?**
ai.txt: machine-readable usage policy (permissive/restrictive) for training, indexing, quoting, storing. TDM reservation: legal opt-out against automated analysis (EU among others). Both orientation, not legal advice.

**Does every AI honor it?**
No — robots.txt, ai.txt, meta tags and the TDM reservation are requests or declarations, not an enforcement. Hard enforcement needs the server/firewall/WAF level.

## How KAMINSKI+ lives this out

This site makes its choice openly: allow. Our [/.well-known/ai.txt](/.well-known/ai.txt) declares an explicitly permissive usage policy — training, indexing, quoting and storing are wanted here. In keeping with that, our [/robots.txt](/robots.txt) welcomes every known AI crawler with `Allow: /`; who does what is listed under [AI crawlers & bots](/en/machine-web/ai-crawlers/).

## Further reading

- [Machine Web (overview)](/en/machine-web/) — all reference topics
- [AI crawlers & bots](/en/machine-web/ai-crawlers/) — which bot trains, searches, fetches
- [llms.txt](/en/machine-web/llms-txt/) — make content machine-readable
- [Guide for machines](/en/leitfaden/) — the operating manual