AI visibility checker
Enter a page and this tool checks whether AI companies’ crawlers are allowed to read it under your robots.txt. It also checks that your content is in the HTML without JavaScript, whether you have structured data, and whether you’ve published an llms.txt file.
It’s for makers who want their product to come up in AI answers from tools like ChatGPT, Claude and Perplexity, or who want to control how their content is used. It shows access only: being readable doesn’t mean an AI will cite you.
Free with a ShipScroll account.
How to use the AI Visibility Checker
- Enter your page address and press Check page.
- Look at the Used for column: blocking search crawlers can keep you out of AI answers, while blocking training crawlers is a separate choice.
- Fix any blocked search crawler you didn’t mean to block, by editing robots.txt.
- If little text shows without JavaScript, render your key pages on the server.
Search crawlers and training crawlers are different
This is the distinction most people miss. Companies publish separate crawlers for different jobs, and blocking one doesn’t block the others. You can opt out of model training and still appear in AI search.
| Crawler | Company | What it does |
|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search |
| GPTBot | OpenAI | Collects content that may train models |
| ChatGPT-User | OpenAI | Visits a page when a person asks |
| Claude-SearchBot | Anthropic | Indexes content for search |
| ClaudeBot | Anthropic | Collects content that may train models |
| Claude-User | Anthropic | Fetches a page when a person asks |
| PerplexityBot | Perplexity | Surfaces and links sites in its results |
| Perplexity-User | Perplexity | Fetches a page when a person asks |
| Google-Extended | A control for AI training; doesn’t affect Google Search |
Is llms.txt worth adding?
llms.txt is a proposal for a Markdown file at your site’s root that gives AI agents a short overview with links to your key content. It’s not a formal standard, and only its title is required. It’s cheap to add, so it’s reasonable if you have docs or a lot of content, but don’t expect it to change anything by itself. That’s why this tool shows a missing file as something to consider, not an error.
What robots.txt does and doesn’t do
robots.txt asks crawlers not to fetch pages. It’s a request that well-behaved crawlers follow, and it isn’t access control. Fetches a person asks for may not follow it either: OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. To protect private content, put it behind a login.
Example: the rule that blocks more than you meant
A site adds “User-agent: * / Disallow: /” to staging and it reaches production, or copies a list of AI bots to block training and includes the search crawlers too. The site keeps working for people, but this tool shows those crawlers as blocked. Check this after any change to robots.txt.
Questions
If everything shows as allowed, will AI tools cite me?
Not necessarily. Access is only the first step. Being cited depends on whether your content answers the question and how those systems choose sources, which they don’t fully publish. This tool checks what you can directly control.
Should I block AI training crawlers?
That’s your call, and it depends on how you feel about your content being used for training. Blocking GPTBot or ClaudeBot doesn’t block those companies’ search crawlers, so you can opt out of training and stay visible in their search products.
Does blocking Google-Extended remove me from Google?
No. Google says Google-Extended doesn’t affect inclusion in Google Search and isn’t a ranking signal. It’s a control over use of content for Gemini models, and it has no separate crawler of its own.
Why check just one page instead of the whole site?
robots.txt rules can differ by path, so a page deep in your site may be treated differently from the home page. Check the pages you most want found, such as product pages and articles.
Is this the same as GEO or AEO?
Those terms, generative and answer engine optimisation, describe the wider effort to appear in AI answers. This tool covers the technical part of that: whether AI systems can reach and read your pages.
Related tools
- On-Page SEO AuditOne report on a page’s title, headings, images, links, schema and robots.txt.
- Schema Markup CheckerExtract a page’s JSON-LD, list its schema.org types, and catch syntax errors.
- Sitemap CheckerFind a site’s sitemap, count its URLs, and catch errors against the protocol.