Growth Marketing
November 10, 2026

Should You <blue>Block AI Crawlers?</blue> What Robots.txt Choices Cost in Visibility

Troy Diffenderfer
Troy Diffenderfer
Title card: Should You Block AI Crawlers? What Robots.txt Choices Cost in Visibility
Table of contents

Scale Smarter with

Connect your sources, clean your data, and watch performance marketing soar with a flat fee and zero drama. Get started with a discovery call

Blocking GPTBot on its own costs very little AI search visibility, because OpenAI uses separate crawlers for model training and for ChatGPT search. What does cost visibility is blocking the search and retrieval bots, such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot, or blocking every bot at once. So the useful question is not "block AI or not." It is which crawler does which job, and what each robots.txt rule takes away.

Key takeaways

  • AI companies now run separate bots for training, search indexing, and user-requested fetches. Each one can be allowed or blocked independently.
  • Blocking training bots (GPTBot, ClaudeBot, Google-Extended) keeps your future content out of model training without removing you from AI search answers.
  • Blocking search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) removes you from those assistants' cited answers.
  • Robots.txt is a request, not a lock. User-triggered fetchers may not follow it, and CDN or firewall settings can block bots your robots.txt allows.

Why "AI crawler" is no longer one thing

A few years ago, most AI crawling was about collecting training data. Today the major companies split crawling into three jobs, and the visibility cost of blocking depends entirely on which job you block. For brands that want to be recommended by ChatGPT, Claude, Perplexity, or Gemini, this is the core of technical AI search visibility services: making sure the bots that feed answers can reach the pages you want cited.

  • Training crawlers collect content that may be used to train future models.
  • Search crawlers index pages so an assistant can find and cite them when answering questions.
  • User-triggered fetchers load a specific page because a person asked the assistant to look at it.

Which bot does what, as of November 2026

Matrix of AI crawler tokens by company and job: training, search, and user-triggered bots for OpenAI, Anthropic, Google, and Perplexity
Training, search, and user-triggered bots for each AI company.

OpenAI

OpenAI's crawler documentation says each setting "is independent of the others." GPTBot is the training crawler, and disallowing it "indicates a site's content should not be used in training generative AI foundation models." OAI-SearchBot powers ChatGPT search. Sites that block it "will not be shown in ChatGPT search answers, though can still appear as navigational links." ChatGPT-User handles user-initiated visits, and OpenAI notes that "robots.txt rules may not apply" to it.

Anthropic

Anthropic lists three bots. ClaudeBot collects content that could contribute to training. Claude-SearchBot indexes content for search, and Anthropic says blocking it "may reduce your site's visibility and accuracy in user search results." Claude-User retrieves pages in response to a user's request.

Google

Google-Extended is a robots.txt token, not a separate crawler. It controls whether content is used to train and ground Gemini Apps and Vertex AI models. Google states that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Googlebot is the token that governs Google Search, so blocking Googlebot to avoid AI features would also remove your pages from regular search results.

Perplexity

PerplexityBot surfaces sites in Perplexity's search results, and Perplexity says it "is not used to crawl content for AI foundation models." Perplexity-User fetches pages that users ask about and, per Perplexity's own docs, "generally ignores robots.txt rules."

What each robots.txt choice costs

  • Block training bots only. Low visibility cost. Your pages stay eligible for AI search citations, but future models are trained without your newest content.
  • Block search bots. High visibility cost on that assistant. You lose the cited answers that send referral traffic, and the assistant has to describe you from older or third-party sources.
  • Block user-triggered fetchers. Mixed. Some may ignore the rule, and those that follow it can no longer read your page when a buyer pastes your URL and asks a question.
  • Block everything with a wildcard rule. The most expensive choice. A blanket disallow also blocks Googlebot and Bingbot, which removes you from traditional search too.

When blocking training bots makes sense

Blocking training crawlers is a reasonable choice for businesses whose content is the product: publishers, research firms, paid course creators, and companies with proprietary data. For them, keeping content out of future training data protects something they sell.

For most ecommerce, B2B, and hospitality brands, the calculation is different. Their content exists to be found. If a model learns accurate facts about your products and positioning during training, that can help when it answers without searching the web. There is no published evidence that allowing training crawlers improves rankings in AI answers, so treat this as a judgment call about how you want models to know your brand, not a visibility lever.

A balanced robots.txt setup

Hypothetical example: A DTC brand wants to appear in AI shopping answers but keep its paid buying guides out of model training. It disallows GPTBot, ClaudeBot, and Google-Extended on the /guides/ folder only, and leaves OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot fully allowed. Product, category, and policy pages stay open to every search crawler.

OpenAI and Perplexity both note that changes can take about 24 hours to take effect, so test after a day rather than immediately.

Hypothetical robots.txt that blocks training crawlers on a guides folder while allowing AI search crawlers everywhere
A hypothetical setup that keeps search crawlers open.

What robots.txt cannot do

  • It cannot undo past training. Blocking applies to future crawling.
  • It does not bind every fetcher. User-triggered agents from OpenAI and Perplexity may not follow it.
  • It is not your only gate. In July 2025, Cloudflare announced it would block AI crawlers by default unless they pay creators. If your site runs behind a CDN or bot-management tool, its settings can quietly block search bots your robots.txt allows.
  • New signals are arriving. Anthropic describes a new "no AI training" preference signal as of October 30, 2026, and Google says it intends to support the AIPREF training signal in early 2027. Revisit your setup as these roll out.

How to check what is actually happening

  1. Read your robots.txt file and list every rule that names an AI user agent, plus any wildcard rules.
  2. Check your CDN and firewall for AI bot blocking that overrides robots.txt.
  3. Search server logs for each user agent name and confirm the bots you allow are getting successful responses. OpenAI and Perplexity publish IP ranges you can use to verify real traffic.
  4. Test AI answers for prompts your buyers use, and note whether your pages are cited.
  5. Track referral traffic from AI assistants, which the guide to measuring LLM traffic covers in more depth.

Crawler access is one item in a broader technical review. The AEO/GEO checklist covers it alongside schema, answer-first structure, and citation tracking.

How Onward handles AI crawler access

Onward's AEO work includes technical setup such as schema, entity consistency, and AI crawler access, followed by prompt-level tracking from visibility to traffic to customers. Access matters because cited answers can drive real revenue: Villa Alberti went from near-zero AI visibility to $100K+ in bookings from AI answers.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot is OpenAI's training crawler. ChatGPT search relies on OAI-SearchBot, which you can allow separately. Blocking OAI-SearchBot is what removes your pages from ChatGPT search answers, apart from navigational links.

Does Google-Extended affect AI Overviews?

Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls use of your content for Gemini Apps and Vertex AI training and grounding. Google Search features are governed by Googlebot.

Should I block all AI bots to protect my content?

Only if keeping content out of AI products matters more than being cited by them. A blanket block removes you from AI search answers and, if it uses a wildcard, from traditional search as well.

The bottom line

Decide bot by bot. For most brands that want AI assistants to recommend them, the sensible default is to keep search crawlers open, make a deliberate choice about training crawlers, and check that your CDN is not overriding both.

If you are not sure which AI bots can reach your site today, talk to Onward's team.