Why these two files matter in 2026
The way people search changed: more and more, answers come from AI assistants that read the web and cite
sources. For your site to take part in that conversation, AI first has to be able to access and understand it,
and that is where these two files come in. The robots.txt is the permission layer —it decides which bots get
in— and the llms.txt is the guidance layer —it gives them a curated map of what matters. Together, they remove
technical barriers and make it easier for AI to do its job well with your content.
The detail almost nobody checks is that many sites unintentionally block AI bots, sometimes because a template
shipped with those directives, sometimes because a security service added them by default. The result is that
they lose visibility in AI search engines without realizing. Generating your robots.txt deliberately, knowing
exactly whom you allow and whom you do not, is the way not to leave that decision to chance. And if you want to
check whether your current site is already blocking these bots, we have a separate tool for that: the
AI bot block checker.
Allow, block training, or block all
The tool gives you three policies, and each answers a different intention. Allowing all is what suits almost
any business that wants visibility: you let AI index and cite you, just as you let Google do it. Blocking
training but allowing search is the sophisticated middle ground: you stop your content from feeding model
training, but you allow them to cite you in answers with a link back. Blocking all is the option to close the
door completely, which only makes sense with a concrete reason. The tool translates the policy you choose into
the exact directives for each bot, without you having to memorize names or syntax.
Frequently asked questions
What is the llms.txt file and what is it for?
llms.txt is a text file you place at the root of your site to give AI models a clean, readable map of your most important content. The idea is similar to sitemap.xml for Google, but designed for assistants like ChatGPT, Claude or Perplexity: instead of making them guess through your HTML, you offer a curated list of your key pages with their title and link, in a format they understand effortlessly. It is not mandatory or an official standard yet, but it is a growing practice because it costs little and helps AI understand and cite your site more precisely. This tool generates it ready to upload.
What does the robots.txt block it generates do?
It generates the directives that tell each AI bot whether or not it can access your site. There are dozens of these bots —OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot, Google-Extended, and many more— and each is controlled separately in robots.txt. The tool assembles that block according to the policy you choose, without you having to remember the exact name of each bot or the syntax. You paste it into your existing robots.txt, at the root of the site, and it is applied. It is the standard, declarative way to indicate your preferences to these bots.
Should I allow or block AI bots?
For the vast majority of businesses, allowing is the right call, because appearing cited in ChatGPT, Perplexity or Gemini is new visibility you do not want to miss. Blocking AI bots is blocking a discovery channel that is growing fast. The reasonable exception is the middle option the tool offers: block the use of your content to train models, but allow them to cite you in answers with a link. That protects your content from feeding training without your permission, without giving up visibility. Blocking them all only makes sense if you have a concrete, deliberate reason not to be in AI search engines.
Is this difference between "training" and "search" real?
Yes, and it is a distinction worth understanding. Some bots collect content to train the underlying models —there is no link back or citation there, your material simply feeds the model. Others collect content to index and be able to cite you in real time when someone asks, with a link to your site. The tool's middle option separates those two uses: it blocks training-oriented bots and allows search-and-citation-oriented ones. It is not an exact science because each bot's classification can change, but it reflects the current best understanding of what each one uses your content for.
Does the tool store what I type?
No. Everything happens in your browser: the files are generated locally with what you type and nothing is sent to any server. We do not store your site, your pages or your configuration, and there is no usage record associated with you. You can copy or download the generated files and use them wherever you want. Privacy is by technical design: there simply is no backend that receives your data.
Does uploading these files guarantee AI will cite me?
No, and it is honest to say so: these files are a helping condition, not a guarantee. The llms.txt and a well-configured robots.txt remove technical barriers and make it easier for AI to understand and access your site, but being cited also depends on the quality and structure of your content, your authority as an entity and many more signals. Think of this as opening the door and putting up a clear sign: necessary for them to enter, but not enough for them to choose you. If you want to work on the rest, that is where AEO and brand entity come in, which are separate work.