Why an invisible block costs visibility
More and more people find businesses through AI assistants that read the web and cite sources. To appear in
those answers, your site has to let in the bots that collect that content. The problem is that blocking them is
almost never a conscious decision: it comes from a template that shipped with those directives, a general rule
someone set thinking of other bots, or a security service that blocked them by default. The owner does not know
it, and meanwhile their site is invisible precisely in the fastest-growing channel.
This tool exists so that decision stops being an accident. In seconds you know which AI bots you have allowed
and which blocked, and you distinguish whether the block is specific or comes from the general rule, because the
fix changes by case. If you find you are closing doors unintentionally, the next step is to generate a correct
robots.txt, and for that we have the
llms.txt and robots.txt generator.
The layer robots.txt does not show
There is an honesty this tool does not skip: robots.txt is only one of the two layers where AI can be blocked.
The other is the network layer, and it is the one almost nobody checks. Services like Cloudflare offer options
to block AI bots that act before they reach your site, returning an error, and that block does not appear in any
robots.txt no matter how carefully you read it. It is perfectly possible to have a flawless robots.txt and still
be invisible to AI because Cloudflare is stopping it at the door.
So if this tool gives you green but you use Cloudflare or another security service, you should still check its
AI bot rules, which sometimes turn on by themselves. That network-level check cannot be done from the browser
—it requires testing access with each bot\'s identity— and it is part of what we review in our AEO work. If you
want to be sure nothing is blocking you, on either layer, let's talk.
Frequently asked questions
What exactly does this tool check?
It reads your robots.txt and tells you, AI bot by AI bot, whether you are allowing or blocking it. It reviews the crawlers that matter most in 2026 —OpenAI's GPTBot and OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, and others— and for each one it indicates whether it is blocked by a specific rule, blocked by the general rule that applies to all, or allowed. So instead of reading a file full of directives by hand, you see at a glance which doors you have open and which closed to AI. It is the fast way to detect whether you are losing visibility in AI search engines without knowing it.
Why can it sometimes not read my robots.txt automatically?
Because of a browser restriction called CORS, which prevents a site from directly reading files from another domain. We try first directly and, if that fails, through a public proxy; but some sites block both paths, which is perfectly normal. When that happens, the tool tells you honestly and gives you a link to open your own robots.txt: you just copy its content and paste it in the "Paste robots.txt" tab, which analyzes exactly the same. The result is the same, only how the file arrives changes.
If my robots.txt comes out clean, am I sure I am not blocking AI?
Not necessarily, and this is the tool's most important warning. The robots.txt is the declarative layer —what you ask the bots to do— but there is another layer, the network one, where services like Cloudflare can block AI bots before they reach your site, returning an error. That block does not appear in robots.txt, so a flawless robots.txt can coexist with a network block that leaves you just as invisible. If you use Cloudflare or another security service, you should also check its AI bot rules, which sometimes activate by default without the owner realizing. That network-level check is something we can do.
What does "blocked by the general rule" mean?
It means that bot has no rule of its own in your robots.txt, but the rule that applies to all bots —the one written as User-agent colon asterisk— applies to it, and that rule blocks access. It is a common and treacherous situation: someone set a general block thinking of something else, and unintentionally left out the AI bots too. The tool distinguishes that case from a specific block —where someone named the bot on purpose— because the fix is different: if it is the general rule, you have to explicitly allow the AI bots you do want to let in.
Should I unblock all AI bots?
For almost all businesses that want visibility, yes: being available for ChatGPT, Perplexity or Gemini to read and cite you is a growing discovery channel. If you find you are blocking these bots unintentionally, you most likely want to open up. The reasonable exception is that you want to prevent the use of your content to train models but still allow citations: in that case you block the training bots and allow the search ones. If you want to generate the correct robots.txt according to the policy you choose, we have a tool for that: the <a href="/en/tools/llms-txt-generator/">llms.txt and robots.txt generator</a>.
Does the tool store the domains I check?
No. The robots.txt analysis happens in your browser and we do not store the domains you query or associate anything with you. When the public proxy is needed to read a robots.txt, that request passes through a free external service over whose logs we have no control, but on our side we keep no history and do not use the data for anything beyond showing you the result. If you prefer not to use the proxy, you can copy your robots.txt and paste it directly, and then nothing leaves your browser.