AI usage rights: allow or block AI training on your website (ai.txt, robots.txt & co.)

Answer first: Can AI train on my website?

By default, yes. If your page is publicly reachable, an AI provider can fetch it as long as you don't actively object. So the real question is not "whether it's possible" but "which signal you set". You have five tools that act on different levels — from the simple crawler request to the legal reservation. They don't exclude each other, they complement each other. This page is orientation, not legal advice; when in doubt, check the respective source or seek legal counsel.

1. robots.txt — the standard tool per user agent

The robots.txt is the established place where you grant or deny crawlers access to paths. You control each bot individually via its user agent with Allow or Disallow. Anyone who wants to block AI training names the training crawlers and sets them to Disallow:

Anyone seeking maximum AI visibility instead allows them expressly — allow GPTBot and siblings:

The robots.txt is a request, not a technical enforcement. It has no compelling effect — it is a polite notice that reputable providers honor voluntarily. Anyone who needs hard control must work a layer deeper: server, firewall or WAF rules that actually block access instead of merely requesting it.

2. Google-Extended & Applebot-Extended — steer AI training separately

The most common misconception: you block Googlebot and believe you've thereby stopped AI training. That's not true. Google and Apple deliberately separate AI training from normal search via their own tokens. This is how you do a Google-Extended opt out without disappearing from classic search:

• Google-Extended — not a crawler of its own, but a pure robots.txt token. It decides whether Google's AI products use your content for training. Googlebot continues to index your pages for search, unaffected by it.
• Applebot-Extended — the counterpart at Apple: an opt-out signal for AI training, separate from Applebot, which serves search.

Note: training control and search indexing are two different levers. You can stay visible in search and still object to training — or the other way around.

3. ai.txt — the machine-readable usage policy

The robots.txt only knows "may fetch / may not fetch". It says nothing about what may happen with the fetched content. This is exactly the gap the emerging ai.txt fills: a file (usually at /.well-known/ai.txt) in which you spell out your AI usage policy — permissive or restrictive. Instead of just governing access, you name individual kinds of use: training, indexing, quoting, storing, inference.

1. Decide whether your policy is fundamentally permissive or forbidding.
2. Allow or forbid the kinds of use individually — e.g. training yes, quoting yes, storing no.
3. Add conditions: desired attribution, contact, references to llms.txt and structured data.
4. Understand the limit: ai.txt is a young standard, not yet universally supported — a clear declaration of intent, not an enforceable contract.

Because the format is young, it should complement other signals, not replace them. A restrictive ai.txt without accompanying robots.txt blocks is weaker than both together.

4. TDM reservation of rights — the legal opt-out

While robots.txt and ai.txt are technical or conventional signals, the TDM reservation of rights is a legal declaration. TDM stands for text and data mining — the automated analysis of content that underlies AI training. In the EU context, rights holders can expressly reserve this analysis; such a reservation, ideally stored in machine-readable form, is an opt-out against the commercial TDM use of their works.

In the debate around AI and copyright, the TDM reservation is the sharpest of the instruments named here, because it doesn't merely ask for technical good behavior but declares a legal position. It does not, however, change the fact that technical fetchability remains as long as no access barrier is in place.

5. Meta tags & HTTP headers — noai, noimageai, X-Robots-Tag

Some signals can be set per page instead of centrally — via a meta tag in the <head> or the HTTP header X-Robots-Tag. Where supported, noai and noimageai signal the wish not to use text or images, respectively, for AI training:

These signals are pinpoint — ideal if you only want to protect individual pages or image areas. Their downside: they are not evaluated by every provider. noai is not a universally established standard like the classic noindex. As with all other signals: a request, not an enforcement.

The debate: opt-out or opt-in, visibility or control

Behind the technology lies an unresolved conflict. Today's web mostly runs on the opt-out principle: content is usable for AI until someone objects. Some rights holders demand opt-in — use only after express consent. This dispute touches copyright directly and is open both legally and politically.

For operators, it comes down to a trade-off: visibility versus control. Whoever allows gains presence in AI answers and gives up a piece of control over the use. Whoever blocks keeps control and forgoes part of the AI visibility. There is no answer right for everyone — only the deliberate choice.

Practice for both camps

You want to allow (maximize AI visibility)

• Set Allow: / per training and search crawler in the robots.txt — including allow GPTBot, ClaudeBot, PerplexityBot.
• Store an explicitly permissive ai.txt and make content machine-readable via llms.txt.
• Forgo noai signals and leave Google-Extended/Applebot-Extended open.

You want to block (keep control)

• Block the training crawlers specifically in the robots.txt (GPTBot, ClaudeBot, CCBot …).
• Set Google-Extended and Applebot-Extended to Disallow — the actual training opt-out at Google and Apple.
• Add a restrictive ai.txt, where supported noai/noimageai, and consider a TDM reservation of rights.
• For hard enforcement: server/firewall/WAF rules — because all the above signals are requests, not an enforcement.

Frequently asked questions

How KAMINSKI+ lives this out

This site is built machine-first and makes its choice openly: allow. Our /.well-known/ai.txt declares an explicitly permissive usage policy — training, indexing, quoting and storing are wanted here, not merely tolerated. In keeping with that, our /robots.txt welcomes every known AI crawler by name with Allow: /; which bot does what is listed under AI crawlers & bots. Anyone who writes about AI usage rights should disclose their own — here you can read them at the source.