Robots.txt tester
Enter your site and a page, pick a crawler, and this tool reads your live robots.txt and tells you whether that crawler may fetch the page. It shows the group it followed and the single rule that made the decision, so you can see why, not just whether.
It’s for makers who added a Disallow line and want to be sure it blocks what they meant, and nothing else. It reads the file the way Google describes: the crawler follows the group that names it, or the * group, and the longest matching rule wins.
Free with a ShipScroll account.
How to use the Robots.txt Tester
- Enter your website address. The tool reads robots.txt from the root of that site.
- Enter the page to test, as a full URL or a path such as /blog/post. Leave it blank to test the address you entered.
- Choose the crawler, or type its user-agent token, such as Googlebot or GPTBot.
- Press Test. If the result surprises you, look at the rule it quotes and the group it came from.
How a crawler picks its rules
- It looks for a group whose User-agent line names it. Only if there’s none does it use the User-agent: * group.
- Inside that group, the most specific rule wins: the one whose path matches the most characters.
- If an Allow and a Disallow rule are equally long, Google uses the less restrictive one, so Allow wins.
- * matches any characters, and $ marks the end of the address, so Disallow: /*.pdf$ blocks every PDF.
- A page no rule matches is allowed.
Blocking crawling isn’t the same as hiding a page
Disallow stops a crawler reading the page. It doesn’t stop the page appearing in search: if other sites link to it, Google can still list the address, usually without a description. To keep a page out of results, let it be crawled and add a noindex meta tag or X-Robots-Tag header instead. A crawler has to be able to read the page to see that noindex.
Example: the rule that blocked more than planned
A site adds Disallow: /app to keep its dashboard out of search. It also blocks /apple-pie-recipe, because rules match the start of the path. Disallow: /app/ (with the slash) blocks only the dashboard folder. The tester shows which rule matched, so this kind of slip is easy to spot.
Questions
Does this test my live file or a draft?
Your live file, read from /robots.txt on the site you enter. To try a change, publish it first, or test it on a staging site with a public address.
Why does my crawler follow the * rules?
Because robots.txt has no group naming it. A crawler only uses User-agent: * when no group names it, and then it ignores every other group, even ones that seem related.
Do all crawlers obey robots.txt?
Major search engines and the main AI crawlers say they do, but it’s a request, not a lock. Anything private should sit behind a login, not just a Disallow line.
Can robots.txt be too big?
Google reads the first 500 KiB of the file and ignores the rest. Most files are tiny, so this only matters if yours is generated with thousands of lines.
Related tools
- AI Visibility CheckerSee which AI crawlers your robots.txt allows, plus llms.txt and schema readiness.
- Sitemap CheckerFind a site’s sitemap, count its URLs, and catch errors against the protocol.
- Redirect CheckerFollow every hop from an address to the page it ends on, with each status code.