Should I block AI crawlers in robots.txt?
It depends on whether your content is the product. If you sell access to an archive — a publisher, a research house, a paid database — blocking is a defensible commercial decision. If you sell goods or services and use content to be found, blocking removes you from the answers your buyers read while your competitors stay in them.
The question is usually asked in the wrong frame, as a moral one about whether models should be allowed to read your work. That argument is real, but it is not the one most businesses are actually making. For a company that sells machinery, or software, or consulting, the content was never the asset — it was the advertisement. Blocking the crawler is closer to refusing to be listed in the directory than to protecting intellectual property.
The distinction people miss
Does blocking training crawlers also remove me from live answers?
Often, yes — and this is the distinction people miss. Some agents fetch pages for training, others fetch them live during a conversation to answer a question right now. Blocking the second kind removes you from answers immediately, which is almost never what a business intends.
A publisher who wants to withhold their archive from training but still appear in assistant answers needs to separate the two, agent by agent. That is a legitimate middle position and it is the one most media companies eventually land on. It is also fiddly, changes as vendors rename their agents, and is worth revisiting a couple of times a year.
What blocking actually costs
Not a traffic number — an absence. When an assistant is asked to recommend suppliers in your category, it reads what it can reach. If it cannot reach you, it does not report a gap; it simply names the three companies it could read. The buyer never learns you exist, and you never see a metric move, because nothing happened.
That invisibility is the whole difficulty. A blocked site looks exactly like a site nobody asked about. This is why we measure answer-share directly rather than inferring it from analytics — it is the only way to see a loss that leaves no trace.
If you decide to allow them
Which AI crawlers should I name in robots.txt?
At minimum GPTBot, OAI-SearchBot and ChatGPT-User for OpenAI; ClaudeBot and Claude-User for Anthropic; PerplexityBot; Google-Extended; Applebot-Extended; CCBot; and Meta-ExternalAgent. Several of these evaluate their own user-agent block first and ignore the wildcard, so a bare "User-agent: *" rule is not reliably sufficient.
Name them explicitly. It costs ten lines and removes an entire class of silent failure — the site that believes it is open because of a wildcard, while a crawler that only reads its own block walks away. You can see how this site does it at siolfur.in/robots.txt.
Then check that the allow actually works, from the crawler's side rather than yours. Firewalls, bot-protection services and CDN rules block plenty of agents that robots.txt cheerfully permits, and nobody finds out until they look.
The short version
- Content is your product → blocking is defensible; separate training agents from live-retrieval agents.
- Content sells your product → allow them, name them explicitly, and verify the allow works.
- Undecided → allow. The reversible mistake is letting them in; the expensive one is a year of silence you cannot see.