Here is a list of AI crawlers from OpenAI, Anthropic, Perplexity and Google, ranked by how much you lose if you block them. Search bots sit at the top, because blocking them can take you out of AI answers; training bots sit at the bottom, because blocking them mostly says "do not train on this." Before you change anything, run your site through the seogeoaeo.ai robots.txt and AI crawler check to see which of these you already allow.

How the ranking works

Each vendor now runs separate bots for separate jobs, and each one reads its own line in robots.txt. That split matters. One bot gathers pages for model training, another builds a search index, and a third fetches a page only when a person asks. The ranking below puts the bots whose loss hurts reach first.

1. OAI-SearchBot (OpenAI)

This is the bot behind ChatGPT search. OpenAI says sites that opt out will not be shown in ChatGPT search answers, though they can still appear as navigational links. If you want ChatGPT to cite you, this is the one to keep open.

2. PerplexityBot (Perplexity)

PerplexityBot surfaces and links websites in Perplexity's search results. Perplexity states it is not used to crawl content for AI foundation models, so blocking it buys you little and costs you citations.

3. Claude-SearchBot (Anthropic)

Anthropic uses it to index content for search quality. Its help page warns that disabling it may reduce your visibility and accuracy in Claude's search results.

4. Google-Extended (Google)

This one is odd: it has no user agent of its own. Google-Extended is a robots.txt token that controls whether Google may use your content to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search.

5. GPTBot (OpenAI)

GPTBot crawls content that may be used to train OpenAI's foundation models. Disallowing it signals that your content should not be used for training. OpenAI treats it as independent of OAI-SearchBot, so you can block one and allow the other.

6. ClaudeBot (Anthropic)

ClaudeBot collects web content that could contribute to training Anthropic's models. Blocking it signals that future material from your site should be left out. Anthropic also honors the non-standard Crawl-delay line if you only want it to slow down.

The user-triggered fetchers

These do not crawl on their own. They load a page because someone asked a question.

Bot Vendor Follows robots.txt?
ChatGPT-User OpenAI OpenAI says robots.txt rules may not apply
Perplexity-User Perplexity Perplexity says it generally ignores robots.txt
Claude-User Anthropic Anthropic says disabling it stops these fetches

The whole list at a glance

Bot Main job Cost of blocking
OAI-SearchBot ChatGPT search High
PerplexityBot Perplexity search High
Claude-SearchBot Claude search Medium to high
Google-Extended Gemini training and grounding Medium
GPTBot OpenAI training Low for reach
ClaudeBot Anthropic training Low for reach

Check before you edit

A bot obeys only the most specific group that names it, so one stray User-agent: GPTBot block can override everything under User-agent: *. seogeoaeo.ai reports access for GPTBot, ClaudeBot, PerplexityBot, Google-Extended and others in one run for 10 credits, with the exact line to change. Its AI Bot Log Analyzer shows which of these bots really visit, from a pasted access log.

When is another route better? If you run your site behind a CDN with its own bot controls, those settings can block bots before robots.txt ever matters, so check them too.