robots.txt for AI crawlers on WordPress: the fifteen user-agents, what each one does, and what we chose

Most advice on “blocking AI in robots.txt” treats every AI user-agent as the same thing. They are not. Some copy your pages to train future models. Some build the index that decides whether an assistant can cite and link to you. Some are not crawlers at all but tokens that tell a company what it may do with pages it already fetched. And a few only open a page because a human asked, and their operators say robots.txt does not bind them. Blocking the wrong category removes you from AI answers without protecting anything. This is the list we work from, read from each operator’s documentation on 30 September 2026, and the file we publish at postiada.com/robots.txt.

Which AI user-agents exist, and who runs them?

A few rows deserve a note, because the names mislead.

• OpenAI documents three: GPTBot collects training data; OAI-SearchBot indexes for ChatGPT search; ChatGPT-User fetches a page when a user’s action requires it. In December 2025 OpenAI revised the docs to say that because those fetches are user-initiated, robots.txt rules “may not apply” to ChatGPT-User, and to separate OAI-SearchBot from training entirely (summary of the revision).
• Anthropic mirrors that split: ClaudeBot for training, Claude-SearchBot for search quality, Claude-User for user requests — and states that all three respect robots.txt (Anthropic’s crawler page). Older tokens such as anthropic-ai and Claude-Web still appear in many robots.txt files, including ours; they are harmless to keep.
• Perplexity: PerplexityBot indexes for its search results and is “not used to crawl content for AI foundation models”; Perplexity-User visits a page to answer a specific question and, in the operator’s words, “generally ignores robots.txt rules” (Perplexity crawler docs).
• Google-Extended is not a crawler. Googlebot fetches the page; the token only controls whether the content may be used to train and ground Gemini models. Disallowing it does not change crawling, indexing, ranking, AI Overviews or AI Mode — those are Search features fed by the ordinary index.
• Applebot-Extended is the same idea. Applebot powers Siri, Spotlight and Safari suggestions; Applebot-Extended “does not crawl webpages” and only decides whether Applebot’s data may train Apple’s foundation models (About Applebot).
• Meta publishes four: facebookexternalhit for link previews, Meta-WebIndexer for Meta AI search quality, Meta-ExternalAgent for training and indexing, and Meta-ExternalFetcher for user-requested links, which “may bypass robots.txt” (Meta web crawlers).
• DuckAssistBot fetches pages in real time for DuckDuckGo’s AI-assisted answers, states the data is not used for training, and takes up to 72 hours to honour a change. Amazonbot “may be used to train Amazon AI models” and respects robots.txt. CCBot is Common Crawl, the public corpus many models have been trained on.

Why are these three different decisions?

Training crawlers take your text into the next model. Blocking them has no effect on search today and no effect on whether assistants can cite your live pages; it only removes your text from the training set of future models. Publishers with paid content have good reasons to block; a business that wants to be described accurately by next year’s models has good reasons not to.

Search and answer indexers are the ones that matter for being found. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer and DuckAssistBot build the indexes their assistants cite from. Disallow these and you have opted out of AI answers on those platforms — the assistant may still know your name from training data, but it cannot link to you.

User-triggered fetchers are best thought of as browsers. When a person asks ChatGPT to summarise your pricing page, ChatGPT-User opens it once. Several operators say robots.txt does not bind these fetches, so there is little point writing rules for them; if you do not want a page read on request, do not publish it.

What does our robots.txt look like?

We allow all of them, explicitly. The file is managed from Rank Math’s robots.txt editor (General Settings → Edit robots.txt), which is the only place WordPress users should edit it — a physical file in the web root would silently override the plugin’s.

# postiada.com robots.txt
# Search engines and AI assistants are welcome to crawl the public site.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /?s=
Disallow: /search/

# AI crawlers, assistants and usage tokens — allowed explicitly.
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: meta-externalagent
Allow: /
User-agent: CCBot
Allow: /
# … plus Claude-Web, anthropic-ai, Perplexity-User, Applebot, Bingbot, DuckAssistBot, Amazonbot

Sitemap: https://postiada.com/sitemap_index.xml

Two things about this file are deliberate. The explicit Allow: / blocks are technically redundant — the * group already permits everything except the admin, login and search paths — but a named group is a visible statement of intent and it protects the decision from a future plugin default that might add a blanket disallow. And the site-wide rules disallow internal search results because those pages are infinite, thin and never worth an index slot for any crawler, AI or not.

Why allow training crawlers? Postiada is a product whose whole pitch is that specific, sourced, checkable text gets cited by assistants. A model that has read our pages describes us more accurately than one that has read only what others wrote about us. That is our trade-off; a newspaper’s is different, and a robots.txt should follow the business, not a template.

How do you edit robots.txt on WordPress safely?

1. Check for a physical file first. If /robots.txt exists on disk, WordPress and its plugins cannot serve theirs. Delete or move it before editing in the plugin.
2. Edit in one place. Rank Math: General Settings → Edit robots.txt. Yoast: Tools → File editor. Do not maintain both.
3. Keep the sitemap line. Every crawler in the table above reads it.
4. Verify from outside. Fetch https://yourdomain/robots.txt in a private window and confirm the change is live; a CDN may cache the old version for a while. Crawlers cache it too — Meta says up to 24 hours, DuckDuckGo up to 72.
5. Do not rely on robots.txt for privacy. It is a request, honoured by well-behaved crawlers. Anything that must not be read needs authentication, not a disallow line.

Whether an assistant can find you is only half the job; whether it can quote you accurately is the other half. The llms.txt article covers the file that describes your site to agents, and this one covers how the pages themselves should be written. Both are things Postiada sets up for the sites it publishes on.