Get in touch
Have a project in mind? Tell us a bit about it.
Do I need llms.txt? For most business websites, no. Google says its search results and AI features ignore the file, and the crawler pages from OpenAI, Anthropic and Perplexity name robots.txt, not llms.txt, as the thing that decides whether their bots can read your site. Adding one does no harm and takes a few minutes in Rank Math or Yoast, so set one up if you like, but don’t pay anyone who sells it as the way into ChatGPT or Google AI Overviews.
The question comes up a lot on Reddit. One r/SEO thread with more than 70 answers starts with an employee whose boss wants an llms.txt file and who isn’t sure it’s worth doing.
What is an llms.txt file?
It’s a short Markdown file at yourdomain.com/llms.txt that describes your site and links to its most useful pages, written for AI tools rather than people. Jeremy Howard published the llms.txt proposal in September 2024. The format is simple: a top heading with the site’s name (the only required part), a one-paragraph summary in a blockquote, then sections of links with a short note after each.
The proposal says the file is meant mainly for use “on demand”, when an AI agent needs information to help a user, and less for training. Its own site says the files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials.
It has nothing to do with access. llms.txt can’t allow or block a crawler. That is still the job of robots.txt, which we cover below.
Does llms.txt actually work?
Not for Google, and none of the main AI companies lists it as a control in its crawler documentation. Google’s guide to optimizing for generative AI features on Google Search (last updated July 10, 2026) lists llms.txt under things you don’t need to do. It says creating the files is fine “for other services or systems that use these files”, and that “Doing so will neither harm nor help your site’s visibility or rankings in Google Search, as Google Search ignores them.” Google’s separate page on AI features and your website also says you don’t need to create AI text files or special markup to appear in AI Overviews or AI Mode.
Here is what each company’s own documentation says controls its crawlers, checked this week:
| Company | What its official page says controls access | Does it name llms.txt as a control? |
|---|---|---|
| Normal Search eligibility: the page must be indexed and allowed a snippet, and the site must be included in Search generative AI features in Search Console | No. Google Search ignores llms.txt | |
| OpenAI (ChatGPT) | OAI-SearchBot in robots.txt. Sites that opt out of OAI-SearchBot are not shown in ChatGPT search answers (OpenAI crawlers) | No. The page only links to OpenAI’s own llms.txt as an index of its documentation |
| Anthropic (Claude) | robots.txt directives for ClaudeBot, Claude-SearchBot and Claude-User (Anthropic crawler help page) | No mention |
| Perplexity | PerplexityBot in robots.txt. Perplexity-User fetches on a user’s request and generally ignores robots.txt (Perplexity crawlers) | No. The page links to Perplexity’s own llms.txt for its developer docs |
OpenAI and Perplexity both publish llms.txt files for their developer documentation, which is the use the proposal was written for. None of them says it reads a business’s llms.txt when deciding what to put in an answer.
You may also see llms.txt in Chrome’s Lighthouse tool. Its agentic browsing category includes an audit that checks for a machine-readable summary at the domain root. That’s a Chrome developer tool for sites that want AI agents to browse them. It isn’t part of how Google Search ranks pages, and the same Google Search guide above still applies.
Can llms.txt stop AI from training on my content?
No. llms.txt has no allow or block rules, so it can’t keep any bot out. If you want ChatGPT search to show your pages but don’t want them used for model training, that’s done in robots.txt, for example by allowing OAI-SearchBot and disallowing GPTBot. Our guide to whether you should block AI crawlers goes through each company’s bots and what blocking each one costs you.
Should my business add an llms.txt file?
Add one if it’s free and quick on your setup, skip it if it costs real time or money. Use this to decide:
| Your situation | What we’d do | Why |
|---|---|---|
| WordPress with Rank Math or Yoast | Switch the plugin’s llms.txt option on and move on | It takes a few minutes and Google says it neither helps nor harms |
| Wix, Squarespace or another builder with no llms.txt option | Skip it | No AI company has said it uses the file to cite businesses, so a workaround isn’t worth your time |
| Software company with developer docs or an API | Add one for the docs | This is the use the proposal was designed for, and the use OpenAI and Perplexity follow on their own docs |
| An agency or freelancer is quoting to “set up llms.txt” | Don’t pay for it as a separate job | The free plugin options do the same thing. Ask what they’ll do about crawler access, content and mentions instead |
| You’ve just found AI crawlers blocked in robots.txt | Fix robots.txt first | A blocked crawler can’t read your pages, whatever llms.txt says |
We keep a hand-written llms.txt on ppcfitter.com. It lists what we offer, who we work with and links to our pricing and service pages. We wrote it because it took one sitting and makes those facts easy for any tool that does read it. We don’t count it as part of anyone’s AI visibility plan, and we have no evidence it has changed what any assistant says about us.
How do I create an llms.txt file?
On WordPress, let your SEO plugin build it. Elsewhere, you can upload a plain text file to the root of your site.
Rank Math. In wp-admin, go to Rank Math SEO, then Dashboard, find the LLMS Txt module and switch it on. It’s listed as a free module on our own install. Then open Rank Math SEO, then General Settings, then Edit llms.txt. Rank Math’s llms.txt guide says you pick the post types and taxonomies to include, set a limit (100 by default), add any extra links or text, and save. Pages set to noindex are left out.
Yoast SEO. Yoast’s help page gives the path as Yoast SEO, then Settings, then Site features. Under AI tools, open LLMS.txt, switch it on and click Save changes. You can choose the pages manually under Customize llms.txt file. Yoast says the feature isn’t available on multisite installs.
By hand. Write the file in a text editor, save it as llms.txt and upload it to the top folder of your site through your host’s file manager, so it opens at yourdomain.com/llms.txt. Here’s an example for a made-up plumbing company, following the format in the proposal:
# Smith Plumbing
> Smith Plumbing is a licensed plumbing company serving Leeds and Bradford. We handle emergency call-outs, boiler repairs and bathroom installs for homes and small businesses.
## Services
- [Emergency plumber](https://example.com/emergency-plumber/): 24-hour call-outs across Leeds and Bradford
- [Boiler repair](https://example.com/boiler-repair/): repairs and servicing for gas boilers
- [Bathroom installation](https://example.com/bathrooms/): full bathroom refits
## About
- [Areas we cover](https://example.com/areas/): towns and postcodes served
- [Contact](https://example.com/contact/): phone, email and opening hoursKeep it to pages you’d want quoted, and check every link opens. A file full of old or redirected URLs is worse than no file.
What’s the difference between llms.txt and llms-full.txt?
llms.txt is the short index of links. llms-full.txt is a name some documentation sites use for a single file holding the full text of their pages. It isn’t part of the format on the proposal’s own site, which instead suggests offering a clean Markdown copy of each page at the same address with .md added. For a typical business website, neither is worth building by hand.
How can I tell if AI bots read my llms.txt?
Check your server’s raw access logs, which your host can tell you how to download. Search them for requests to /llms.txt and look at the user agent on each line. For example, on a downloaded log file you could run:
grep "llms.txt" access.logEach matching line shows the date, the address that asked for the file and the bot’s name. Compare it with requests for robots.txt and your service pages from the same bots. A request for the file shows that a bot fetched it. It doesn’t show that the file changed what any assistant said, so don’t read more into it than that. Google Analytics won’t help here, because its known bot exclusion filters out known bots automatically and can’t be switched off.
What should I do instead to show up in ChatGPT and AI Overviews?
Start with the things the official pages do name. Google’s guide says SEO best practice still applies because its AI features are rooted in its core ranking and quality systems, and that unique, useful content is what’s likely to influence how you appear. In order:
- Check robots.txt and any security or CDN setting so OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot can reach your pages.
- In Search Console, check that your site is included in Search generative AI features, which Google’s guide lists as a condition for appearing in them.
- Write pages that answer, in plain words, the questions customers ask before they buy, with the answer near the top.
- Get your business named on the other sites assistants read: directories, review platforms and articles comparing businesses like yours.
- Track it. Our guide to tracking AI traffic in Google Analytics shows where ChatGPT and Perplexity visits turn up.
For the detail on each assistant, see how to get your business to show up in ChatGPT and how to get into Google AI Overviews. If you’d rather hand it over, our AI SEO service covers crawler access, answer-first pages, mentions on other sites and monthly tracking of where you’re named in AI answers, alongside your normal Google rankings. Nobody can promise an assistant will cite you, and we don’t. The first two months come with no setup, audit or management fees, and after that you only keep paying if you think it’s worth it.
Common llms.txt mistakes
- Paying for it as an AI SEO fix. The fix is free in the main WordPress plugins and Google says it does nothing for Search. Spend the budget on content and mentions.
- Using it to block AI training. It can’t block anything. Put the rules in robots.txt.
- Adding llms.txt while robots.txt still blocks AI crawlers. Check robots.txt, and any bot blocking in your security plugin or CDN, first.
- Letting it go stale. A plugin-generated file updates itself. A hand-written one needs checking whenever you add, rename or remove pages.
- Listing every page. Keep it to the pages that explain who you are, what you do and where.
FAQ
Is llms.txt a standard?
No. It’s a proposal published at llmstxt.org, and no search engine or AI company has adopted it as a rule for crawling or citing sites. Google says Google Search ignores it.
Will llms.txt hurt my SEO?
No. Google’s guide says creating an llms.txt file will neither harm nor help your visibility or rankings in Google Search. The only risk is wasting time or money on it.
Do I need llms.txt for ChatGPT?
No. OpenAI’s crawler page says ChatGPT search inclusion depends on OAI-SearchBot being allowed in robots.txt, and it doesn’t list llms.txt as a requirement. OpenAI also says changes to robots.txt can take about 24 hours to take effect for search.
Is there a free llms.txt generator?
Yes. Rank Math’s LLMS Txt module is in its free version, and Yoast SEO has an llms.txt switch under Site features. On other platforms, a hand-written file like the example above takes about as long as writing a short email.
Sources: 1. Optimizing your website for generative AI features on Google Search (Google Search Central). 2. AI features and your website (Google Search Central). 3. The /llms.txt file (llmstxt.org). 4. Overview of OpenAI crawlers (OpenAI). 5. Does Anthropic crawl data from the web, and how can site owners block the crawler? (Anthropic). 6. Perplexity crawlers (Perplexity). 7. Agentic browsing scoring (Chrome for Developers). 8. llms.txt (Rank Math Knowledge Base). 9. How to enable llms.txt with Yoast SEO (Yoast). 10. Known bot traffic exclusion (Google Analytics Help).
