Get in touch
Have a project in mind? Tell us a bit about it.
Should I block AI crawlers? For most businesses that want to be named in ChatGPT, Google AI Overviews, Perplexity or Claude, no. Keep the search bots and the bots that fetch pages for a user’s question allowed. If you also want your content kept out of model training, block only the training bots (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended) and you stay eligible for AI search answers.
The trouble is that plenty of sites block AI bots without the owner ever deciding to, through a security plugin, a host setting or a Cloudflare default. So the useful question is usually “what is my site blocking right now?” The sections below cover which bot does what, how to check your own setup and what to put in robots.txt. Checking crawler access is part of our AI SEO service, and these are the same checks we run on client sites.
What’s the difference between AI training bots and AI search bots?
Training bots collect pages that may be used to train AI models, while search bots index pages so an assistant can quote and link them in its answers. Most AI companies now run them under separate names, so you can block one and allow the other. A third type fetches a page only when a person asks the assistant about it.
| Company | Training bot (block to opt out of training) | Search bot (block it and you drop out of its AI answers) | Fetches pages when a user asks |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot (ChatGPT search) | ChatGPT-User; OpenAI says robots.txt rules may not apply because a user started the fetch |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None listed; Perplexity says PerplexityBot isn’t used to crawl content for AI foundation models | PerplexityBot | Perplexity-User, which Perplexity says generally ignores robots.txt |
| Google-Extended (controls use for training future Gemini models) | No separate bot: AI Overviews and AI Mode use the normal Googlebot index. Use the Search Console setting below to opt out | Not applicable | |
| Microsoft Bing | No separate bot: use the NOCACHE or NOARCHIVE robots meta tags | Bingbot | Not documented |
| Apple | Applebot-Extended (it doesn’t crawl; it only controls how Applebot’s data is used for training) | Applebot (Spotlight, Siri and Safari search) | Not documented |
One detail matters for OpenAI. Its bot documentation treats each robots.txt setting separately, and says that if you allow both GPTBot and OAI-SearchBot it may use the results of one crawl for both purposes. Blocking GPTBot leaves you in ChatGPT search. Blocking OAI-SearchBot takes you out: OpenAI says opted-out sites “will not be shown in ChatGPT search answers”, apart from plain navigational links. For Perplexity, see how to rank on Perplexity AI.
Will blocking AI crawlers hurt my Google rankings?
Blocking Google-Extended won’t. Google states that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal in Google Search” (Google’s crawlers page). Blocking Googlebot itself would hurt, because Googlebot is the crawler for Search and for AI Overviews.
Google’s AI features guide says a page only needs to be indexed and eligible to show in Search with a snippet to appear in AI Overviews or AI Mode, and that “There are no additional technical requirements.” So if Googlebot can reach a page and you haven’t added nosnippet or noindex, Google’s AI features can use it. Our guide to getting your business into Google AI Overviews covers what decides whether you’re picked.
Can I stay out of AI Overviews but keep my normal Google results?
Yes. Search Console has a site-wide switch for this, which Google says reached all websites by 31 August 2026, and it leaves your normal blue-link results alone. In Search Console, open Settings and find Search generative AI under AI controls. The options are Include (the default), Exclude and, for a child property, inherit the parent’s setting. When we opened ppcfitter.com’s own Search Console on 26 October 2026, it showed Include.
Google’s help page for the control says it covers AI Overviews, AI Mode and generative AI features in Discover. It isn’t used as a ranking or inclusion signal for the rest of Search, and it doesn’t affect AI training, which is Google-Extended’s job. Google says content is excluded within one to two days, though caching can make some pages take longer. The cost is plain: with Exclude, “You won’t receive any traffic or impressions from these features.” For most service businesses that is traffic worth keeping, and you can see how much it is worth to you with the steps in how to track AI traffic in Google Analytics.
For control over single pages rather than the whole site, Google’s AI features guide points to nosnippet, data-nosnippet, max-snippet and noindex.
How do I check whether my site is blocking AI crawlers right now?
Open yourdomain.com/robots.txt in a browser and read it, then check your firewall or CDN settings, because a block can sit in either place. It takes about ten minutes:
- Open
https://yourdomain.com/robots.txt. Look for aUser-agent:line naming any bot from the table above followed byDisallow: /. Also check there is noDisallow: /underUser-agent: *, which blocks every bot that doesn’t have its own group. - If the file has blocks you didn’t write, look at your SEO or security plugin and your host’s control panel. Some add AI bot rules for you.
- If your site uses Cloudflare, go to Security Settings and open Configure AI bot policies. Cloudflare’s documentation lists three categories, Search, Agent and Training, each set to Block, Block on pages with ads, or Allow. The older Block AI bots toggle is being retired from 15 September 2026.
- Still in Cloudflare, check the robots.txt that visitors actually get. Cloudflare’s managed robots.txt turns your AI bot policies into Disallow rules for bots such as GPTBot, ClaudeBot, Google-Extended and Applebot-Extended, and places them before the contents of your own file. The file served to bots can differ from the one on your server.
- In Google Search Console, open Settings. Check Search generative AI under AI controls (it should say Include if you want AI Overviews) and the robots.txt report under Crawling, which shows whether Google could fetch and read your file.
- View the source of an important page and search for
nosnippet,noarchiveandnocache. On Bing, NOARCHIVE keeps a page out of Bing’s chat answers entirely, and NOCACHE limits them to the URL, title and snippet.
Cloudflare deserves extra care. Its documentation says that from 15 September 2026, new domains block Training and Agent bots on pages that display ads, with Search left allowed. Cloudflare’s July 2026 blog post adds that “Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training”, because those crawlers do both jobs. If you set Training to Block in Cloudflare, check your Google and Bing crawling straight away.
What should my robots.txt say?
If you want to be found everywhere, you don’t need any AI rules at all: a file with no blocks lets every bot in. You only add groups when you want to opt out of something.
Here is a version for a business that wants to appear in AI search answers but keep its pages out of model training. It blocks the training bots by name and leaves everything else open:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yourdomain.com/sitemap_index.xmlThe rule people miss is that a crawler follows only the most specific group that names it and ignores the rest. Google’s robots.txt specification puts it as “Only one group is valid for a particular crawler.” So if you give a bot its own group, any line you want it to obey, like the /wp-admin/ block, has to be repeated inside that group.
Our own file on ppcfitter.com allows every AI bot, because we want to be cited. It names Googlebot, Google-Extended, GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot, Applebot-Extended and Bingbot, and repeats the /wp-admin/ lines in each group for exactly that reason. It also still lists two older Anthropic names, Claude-Web and anthropic-ai, which Anthropic’s current help page doesn’t mention, and it has no group for Claude-SearchBot or Claude-User. That does no harm here, since bots without their own group fall under User-agent: *, which allows everything except the admin area. It shows how robots.txt files drift as companies rename their bots, so check yours against each company’s own documentation once or twice a year.
Which setup fits my business?
Most businesses that sell a service should allow all AI bots and make sure nothing blocks them by accident. Use this table to decide:
| Your situation | What to do | Trade-off |
|---|---|---|
| You want enquiries from people who ask ChatGPT, Perplexity or Google for a recommendation | Allow everything. Check robots.txt, plugins and Cloudflare for blocks you didn’t choose | Your content may be used for training |
| You don’t want your writing used to train models but still want AI search visibility | Block GPTBot, ClaudeBot, Google-Extended and Applebot-Extended; keep OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot allowed | Applebot’s own data may still feed Apple’s AI answers; only Applebot-Extended controls training use |
| Your income is the content itself (publisher, course or paid research) and AI Overviews are answering for you | Consider Exclude in Search Console’s Search generative AI setting, or nosnippet on the pages concerned | No traffic or impressions from AI Overviews, AI Mode or Discover’s AI features |
| You’re on Cloudflare and show ads on your pages | Review Configure AI bot policies now, especially the Training setting | A Training block also stops Googlebot, Applebot and Bingbot, according to Cloudflare |
| You have private pages (client portal, staff area) | Use a login or noindex, not robots.txt | Google says robots.txt “is not a mechanism for keeping a web page out of Google” |
What mistakes do people make when blocking AI bots?
The costliest is blocking a search bot while meaning to block training. These are the ones we see most, with the fix for each:
- Blocking OAI-SearchBot or PerplexityBot “to stop AI”. Those two decide whether you can appear in ChatGPT search and Perplexity answers. Unblock them, and block GPTBot instead if training is your concern.
- Expecting robots.txt to stop every AI visit. Perplexity says Perplexity-User generally ignores robots.txt, and OpenAI says robots.txt may not apply to ChatGPT-User, because a person asked for the page. A hard stop takes a firewall rule.
- Turning on a CDN or plugin “block AI” switch without reading what it covers. Cloudflare’s Training block also catches crawlers with more than one job, such as Googlebot and Bingbot.
- Giving a bot its own group and forgetting to repeat your other Disallow lines in it, so that bot can now crawl your admin or checkout pages.
- Using old bot names only. Check the current list on each company’s own page; Anthropic, for example, now documents ClaudeBot, Claude-User and Claude-SearchBot.
- Blocking Google-Extended and expecting to leave AI Overviews. AI Overviews are controlled through Search Console or snippet controls.
Allowing the bots only makes you reachable. Whether an assistant then names your business depends on what your pages say and what other sites say about you, which we cover in how to get your business to show up in ChatGPT.
FAQ
Does blocking GPTBot remove my site from ChatGPT?
No. ChatGPT search inclusion is controlled by OAI-SearchBot, and OpenAI treats the two settings separately. Blocking GPTBot tells OpenAI not to use your content for training its models.
How long does a robots.txt change take to work?
OpenAI says its search systems take about 24 hours to adjust after a robots.txt update. Google generally caches robots.txt for up to 24 hours. Google says the Search Console Search generative AI setting excludes content within one to two days, sometimes longer.
Do I need an llms.txt file before AI crawlers can read my site?
Not for Google. Its AI features guide says you don’t need to create new machine readable files, AI text files or markup to appear in AI Overviews or AI Mode. Crawler access is decided by robots.txt and your firewall settings.
Can AI tools still read my pages if I block their crawlers?
Sometimes. The fetchers that act on a user’s request, such as Perplexity-User and ChatGPT-User, may not follow robots.txt, according to Perplexity and OpenAI. To stop them completely you would need to block them at your firewall or CDN.
If you want your robots.txt, Cloudflare and Search Console settings checked, and then the work that gets you cited, see our AI SEO service. The first two months are free, with no setup, audit or management fees, and after that you only pay if you think it’s worth it.
Sources: 1. Overview of OpenAI crawlers (OpenAI). 2. Anthropic’s crawlers (Claude Help Center). 3. Perplexity crawlers (Perplexity docs). 4. Google’s common crawlers (Google Search Central). 5. AI features and your website (Google Search Central). 6. Search generative AI control (Search Console Help). 7. Introduction to robots.txt (Google Search Central). 8. How Google interprets the robots.txt specification (Google Search Central). 9. New options to control usage of content in Bing Chat (Bing Webmaster Blog). 10. About Applebot (Apple Support). 11. AI bot policies (Cloudflare docs). 12. Managed robots.txt (Cloudflare docs). 13. Cloudflare blog, July 1, 2026.
