Free Robots.txt Checker
Paste any URL. It reads that site’s robots.txt, parses it the way Google does, and answers the only question that matters: may each crawler fetch this path. Search engines and AI crawlers both.
Rules are matched the way Google documents: most specific user-agent group wins, longest path wins, Allow beats Disallow on a tie. One live fetch, nothing stored.
What the file does, and what it does not
robots.txt controls crawling. It is a list of paths that cooperating crawlers agree not to fetch, served from the root of an origin, and read before almost anything else. That is the entire scope, and nearly every problem with the file comes from expecting more of it.
It does not remove pages from search results. It does not hide anything: the file is public, so a list of paths you would rather nobody visited is a published list of exactly those paths. And it is honoured by agreement, not enforcement, so it keeps out the crawlers that choose to be kept out and nobody else. For anything that must not be seen, the answer is authentication, not a disallow.
The disallow that quietly costs you backlinks
A disallowed page can still end up in the index. Google may list a URL it has never fetched on the strength of links pointing at it, showing it bare, with no description, because it was never permitted to read the page. So the file failed at the one job it was given and succeeded at looking like it worked.
The part that matters here is what it does to links. A page a crawler may not fetch is a page whose outbound links it never reads. If a directory disallows its listing pages, the backlink you earned there is on a page nothing can crawl, which puts it in the same position as a noindex page: perfectly normal to look at, worth nothing to you. It is the same silent failure, one layer further out, and it is why this check belongs next to the others rather than in a general SEO audit.
The related trap: combining a disallow with a noindex cancels the noindex. Google has to fetch the page to see the directive, and you have just forbidden that, so the URL can sit in the index indefinitely. To remove a page, allow the crawl and serve a noindex.
Three rules almost every checker gets wrong
Google’s parsing differs from the original 1994 convention in ways that change real answers. This tool follows Google:
Longest match wins, not first match
Disallow: /admin followed by Allow: /admin/public leaves the public path fetchable, because the Allow is longer. Reordering the lines changes nothing at all, which is the opposite of what a top-to-bottom reading suggests.
One group applies, and it is the most specific
A crawler obeys the single closest match to its name and ignores every other group, including *. Give Googlebot its own group and it stops obeying the wildcard rules entirely, so anything you still want it to respect has to be repeated inside its group. This is the single most common way a file means something other than what its author intended.
Allow beats Disallow on a tie
When two rules match with the same length, permission wins.
Two smaller ones worth knowing: * matches any run of characters and $ anchors the end of a path, so Disallow: /*.pdf$ is a real and useful rule. And an empty Disallow: means nothing is disallowed, which is the conventional way to write a file that permits everything.
The AI crawlers, listed on purpose
This is the question people have about robots.txt now, and most checkers still answer only for Googlebot. Each of these is a separate name, and blocking one blocks nothing else:
GPTBot
OpenAI’s training crawler. Separate from OAI-SearchBot, so blocking it does not remove you from ChatGPT’s search results.
OAI-SearchBot
What fetches pages to answer ChatGPT searches. Blocking this one is what removes you from those answers.
ClaudeBot and PerplexityBot
Anthropic’s and Perplexity’s crawlers. Both feed products that cite sources back to readers.
Google-Extended
Not a crawler. A control token governing whether already-fetched content may be used for Gemini training and grounding, which means blocking it has no effect on Search at all. The one AI opt-out with no search cost.
CCBot
Common Crawl. Not an AI company, but its archive is a training corpus for many models, so it is often the one people miss.
Whether to block any of them is a judgement about your content, not a best practice, and we are not going to pretend there is a correct answer. The trade is real in both directions: keeping out of training also risks keeping out of answers people increasingly search with. What is worth insisting on is that the policy be chosen rather than inherited from a template, which is why the tool reports every one of them.
Failures worth catching
- Disallow: / left on after launch. The staging default, shipped to production. It removes the entire site from search and nothing about the site looks different.
- A Noindex: line. Unsupported since 2019 and silently ignored, so every page relying on it has been indexable for years.
- Rules before the first User-agent line. Rules must sit inside a group. Orphaned ones apply to nobody.
- An HTML page served at /robots.txt. Common on frameworks with a catch-all route. Crawlers find no parseable rules and allow everything, so it looks like it works.
- A typo in a field name. Crawlers skip what they do not understand, so
Dissalowis not an error message, it is a rule that silently does nothing. - No Sitemap: line. Not a failure, a missed opportunity: this is the first file crawlers fetch, so it is how a sitemap gets found without being submitted. The sitemap checker reports the same thing from the other side.
Honest limits
This reports what the file permits, which is not the same as what happens. A crawler may be allowed to fetch a page and still never visit it, or visit and decide not to index it. Directives are also honoured by convention, so a rule keeps out the crawlers that cooperate and no others. We report on the crawlers we list; a file can name dozens more, and the raw text is shown so you can read the rest yourself.
One genuine ambiguity: crawler names change, and companies occasionally split one token into several or introduce a new one without announcement. A verdict here is only as current as the name it tested. And robots.txt applies per origin, so rules on example.com say nothing about a subdomain, which needs its own file and its own check.
Your robots.txt is yours. Theirs decides what your backlink is worth.
A directory that disallows its listing pages has put your backlink somewhere no crawler may read it, and the listing looks completely normal from the outside. ShipDR verifies every listing you track, reads whether the page is live, dofollow and indexable, and tells you the day any of that changes.
- Every listing rechecked on a schedule, not once
- Live, dofollow and indexable, read from the page itself
- Told the day a backlink drops or flips
Adding a website and charting its Domain Rating is free, no card. The backlink engine is $10 once, per website, with no subscription.
Fair questions
What does robots.txt actually control?
Crawling, not indexing. It tells cooperating crawlers which paths they may fetch. It cannot remove a page from search results, it cannot hide anything from a determined visitor, and it is a request rather than an enforcement: well-behaved crawlers honour it and nothing stops the rest. If you need a page kept out of the index, that is a noindex, and if you need it kept private, that is authentication.
Does a disallowed page still get indexed?
It can, and this is the trap the file is most often used wrongly for. Google may index a URL it has never fetched, on the strength of links pointing at it, and shows it with no description because it was never allowed to read the page. Worse, the disallow blocks the fix: a page Google may not fetch is a page whose noindex Google can never see. To remove a URL, allow the crawl and serve a noindex.
Which rule wins when two match?
The longest one, not the first. Google matches by path length rather than document order, and on an equal-length tie the Allow wins. So Disallow: /admin followed by Allow: /admin/public leaves the public path fetchable, and reordering the lines changes nothing. This tool applies exactly that rule, which is why it sometimes disagrees with checkers that read top to bottom.
If I give Googlebot its own group, does it still follow the * rules?
No, and this surprises nearly everyone. A crawler obeys the single most specific group that matches its name and ignores every other group, including the wildcard. So a file with Disallow: / under * and a Googlebot group listing one path leaves Googlebot free to crawl everything except that path. Whatever you want a named crawler to obey has to be repeated inside its own group.
Can I use noindex in robots.txt?
No. Google supported an unofficial Noindex directive for years and stopped in September 2019, so any page relying on it has been eligible for indexing ever since without anything appearing to change. It is still pasted into files today. This checker flags it, because the failure is completely silent otherwise.
Should I block AI crawlers?
That is a judgement rather than a best practice, and the honest framing is that it is a trade. Blocking training crawlers keeps your content out of model training. It can also keep you out of the answers those products give, which is a growing way people find things, and the two are not always separable by one token. What matters more than the answer is that it be deliberate: each crawler is a separate name, blocking one blocks nothing else, and plenty of sites have a policy they never chose. This tool lists the main ones so you can see what your file currently says to each.
What is the difference between Google-Extended and Googlebot?
Googlebot crawls for Search. Google-Extended is not a crawler at all: it is a control token that governs whether content already fetched may be used for Gemini training and grounding. Blocking Google-Extended therefore does not affect your search rankings or crawling in any way, which makes it the one AI opt-out with no search cost attached. Blocking Googlebot, by contrast, removes you from Search.