Guide
AI crawlers: which ones you allow, and what blocking each costs
Most sites have a policy on this that nobody chose. It arrived in a template, or in a plugin default, or in a line copied from a blog post two years ago, and it is now quietly deciding whether one of the fastest growing ways people find things can see your site. This page is about reading what your file actually says, agent by agent, before arguing about what it should say.
Two jobs, wearing similar names
The single most useful distinction, and the one that makes the rest of this readable:
- Training crawlers collect text that may be used to build future models. Blocking one is a decision about your content being learned from. It has no effect on whether you appear in answers today.
- Answer-time fetchers retrieve pages when somebody asks a question now, so the product can answer with current information and cite where it came from. Blocking one of these is what removes you from being cited.
They are separate user-agents. Blocking the first does not block the second, and a blanket rule blocks both plus, quite often, the search engines you did want.
The agents worth knowing by name
- GPTBot is OpenAI’s training crawler. This is the one people mean by "blocking AI", and blocking it does not remove you from ChatGPT’s answers.
- OAI-SearchBot is the OpenAI agent associated with fetching pages for search-style answers. Blocking this one does affect whether you can be surfaced.
- ClaudeBot is Anthropic’s, and PerplexityBot is Perplexity’s. Separate companies, separate names, and each obeys only rules written for it.
- CCBot belongs to Common Crawl, a public dataset many others build on. Blocking it is upstream of a lot of things you will never see a name for.
- Google-Extended is not a crawler. It is a control over whether content Googlebot already fetched may be used in certain Google AI products.
The Google-Extended trap
Because it appears in robots.txt beside real crawlers, people read it as one and assume disallowing it costs them search traffic. It does not: crawling, indexing and ranking are untouched. It is the clearest case where the file says something its owner did not intend, in either direction.
The rule that catches everyone
A crawler obeys the single most specific group that matches its name, and then ignores every other group, including the wildcard. So a file like this does not do what it looks like it does:
User-agent: *
Disallow: /private/
User-agent: GPTBot
Disallow: /drafts/
GPTBot is not kept out of /private/. It found a group with its own name, so the wildcard group stopped applying to it entirely, and /drafts/ is now the only thing it will not fetch. Whatever you want a named agent to obey has to be repeated inside its own group.
Our checker applies exactly this rule and reports what your robots.txt currently says to each AI crawler separately, which is usually not what its owner believes. Read your file agent by agent, no account needed.
The trade, stated honestly
We are not going to tell you to block or allow, because the right answer genuinely differs and anybody presenting it as settled has an interest in your answer.
- A reason to block. Your writing is the product, or your business is publishing, and being trained on gives you nothing back. That is a coherent position and plenty of serious publishers hold it.
- A reason to allow. You are small and being found at all is the problem you actually have. Answer engines are becoming a real discovery path, and a product nobody can cite does not get cited.
- The part that is not a judgement. Whatever you choose, choose it. A default inherited from a template is the one outcome with nothing to be said for it.
If your aim is being surfaced rather than withheld, the other half of this is here: what actually decides whether an answer engine can quote you, once it is allowed to fetch you at all.
What robots.txt cannot do
Worth being clear before anybody relies on it. It is a request that well-behaved crawlers honour and nothing else: it does not stop a scraper that ignores it, it cannot claw back anything already collected, and it does not keep a page out of search results, which is a job for a noindex.
It is also read far less often than people assume, so a change made today applies whenever each crawler next looks, not immediately. If something genuinely must not be fetched, the answer is authentication, not a text file asking politely.
Questions people actually ask
What are AI crawlers?
Bots run by AI companies that fetch pages, for two quite different jobs. Some collect text used to train models; others fetch pages at the moment somebody asks a question, so the product can answer with current information and cite sources. They obey robots.txt by name, and each name is separate, which is why the answer to "have I blocked AI crawlers" is usually more complicated than the person asking expects.
What is GPTBot?
OpenAI’s crawler for collecting training data. Disallowing it is a decision about whether your content is used to train models, and it is the one most articles mean when they say "block AI". It is not the agent that fetches your page when ChatGPT answers a live question, so blocking it does not remove you from answers, and blocking everything does.
Should I block AI crawlers?
It is a judgement rather than a best practice, and anybody presenting it as settled is selling something. Blocking training crawlers keeps your writing out of model training, which some publishers care about a great deal. It can also reduce the chance of being surfaced by those products, which for a small site trying to be found is a real cost. The right answer differs by site; the wrong answer is having one you never chose.
How do I block a specific AI crawler?
Give it its own group in robots.txt and disallow what you want withheld. The trap is that a crawler obeys the most specific group matching its name and then ignores the wildcard group entirely, so rules you put under * do not also apply to a named agent. Whatever you want an agent to obey has to be repeated inside its own group.
Does blocking AI crawlers hurt my SEO?
Not directly: Googlebot and Bingbot are separate agents, and blocking an AI crawler does not touch them. The confusion comes from Google-Extended, which is not a crawler at all, and from blanket rules that disallow everything and take out the search crawlers along with the rest. Read what your file actually says per agent rather than assuming.
Is Google-Extended a crawler?
No, and its name is the single most misleading thing in this subject. It is a control over whether content Googlebot has already fetched may be used for certain Google AI products. Disallowing it changes nothing about crawling, indexing or Search ranking, which is the opposite of what most people reading their own robots.txt assume it does.
Do AI crawlers respect robots.txt?
The major, named ones publish their user-agents and state that they honour it, and independent checks generally bear that out. Robots.txt is a request rather than a fence, though: it does not stop a determined scraper, it does not require identification, and it cannot remove anything already collected. If content must not be fetched at all, that is an authentication problem, not a robots.txt one.
A robots.txt you have not read is a policy you did not choose.
Directories carry your backlinks on pages governed by files exactly like this one, and a disallow there means the link on that page is never read at all. ShipDR checks crawlability alongside the rel and the noindex on every listing you track, and tells you the day one of them changes.
- Every listing rechecked on a schedule, not once
- Live, dofollow and indexable, read from the page itself
- Told the day a backlink drops or flips
Adding a website and charting its Domain Rating is free, no card. The backlink engine is $10 once, per website, with no subscription.