
Β© 2025 Onward. All rights reserved.
Designed by BX Studio

Blocking GPTBot on its own costs very little AI search visibility, because OpenAI uses separate crawlers for model training and for ChatGPT search. What does cost visibility is blocking the search and retrieval bots, such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot, or blocking every bot at once. So the useful question is not "block AI or not." It is which crawler does which job, and what each robots.txt rule takes away.
A few years ago, most AI crawling was about collecting training data. Today the major companies split crawling into three jobs, and the visibility cost of blocking depends entirely on which job you block. For brands that want to be recommended by ChatGPT, Claude, Perplexity, or Gemini, this is the core of technical AI search visibility services: making sure the bots that feed answers can reach the pages you want cited.

OpenAI's crawler documentation says each setting "is independent of the others." GPTBot is the training crawler, and disallowing it "indicates a site's content should not be used in training generative AI foundation models." OAI-SearchBot powers ChatGPT search. Sites that block it "will not be shown in ChatGPT search answers, though can still appear as navigational links." ChatGPT-User handles user-initiated visits, and OpenAI notes that "robots.txt rules may not apply" to it.
Anthropic lists three bots. ClaudeBot collects content that could contribute to training. Claude-SearchBot indexes content for search, and Anthropic says blocking it "may reduce your site's visibility and accuracy in user search results." Claude-User retrieves pages in response to a user's request.
Google-Extended is a robots.txt token, not a separate crawler. It controls whether content is used to train and ground Gemini Apps and Vertex AI models. Google states that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." Googlebot is the token that governs Google Search, so blocking Googlebot to avoid AI features would also remove your pages from regular search results.
PerplexityBot surfaces sites in Perplexity's search results, and Perplexity says it "is not used to crawl content for AI foundation models." Perplexity-User fetches pages that users ask about and, per Perplexity's own docs, "generally ignores robots.txt rules."
Blocking training crawlers is a reasonable choice for businesses whose content is the product: publishers, research firms, paid course creators, and companies with proprietary data. For them, keeping content out of future training data protects something they sell.
For most ecommerce, B2B, and hospitality brands, the calculation is different. Their content exists to be found. If a model learns accurate facts about your products and positioning during training, that can help when it answers without searching the web. There is no published evidence that allowing training crawlers improves rankings in AI answers, so treat this as a judgment call about how you want models to know your brand, not a visibility lever.
Hypothetical example: A DTC brand wants to appear in AI shopping answers but keep its paid buying guides out of model training. It disallows GPTBot, ClaudeBot, and Google-Extended on the /guides/ folder only, and leaves OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, and Bingbot fully allowed. Product, category, and policy pages stay open to every search crawler.
OpenAI and Perplexity both note that changes can take about 24 hours to take effect, so test after a day rather than immediately.

Crawler access is one item in a broader technical review. The AEO/GEO checklist covers it alongside schema, answer-first structure, and citation tracking.
Onward's AEO work includes technical setup such as schema, entity consistency, and AI crawler access, followed by prompt-level tracking from visibility to traffic to customers. Access matters because cited answers can drive real revenue: Villa Alberti went from near-zero AI visibility to $100K+ in bookings from AI answers.
No. GPTBot is OpenAI's training crawler. ChatGPT search relies on OAI-SearchBot, which you can allow separately. Blocking OAI-SearchBot is what removes your pages from ChatGPT search answers, apart from navigational links.
Google says Google-Extended does not affect inclusion or ranking in Google Search. It controls use of your content for Gemini Apps and Vertex AI training and grounding. Google Search features are governed by Googlebot.
Only if keeping content out of AI products matters more than being cited by them. A blanket block removes you from AI search answers and, if it uses a wildcard, from traditional search as well.
Decide bot by bot. For most brands that want AI assistants to recommend them, the sensible default is to keep search crawlers open, make a deliberate choice about training crawlers, and check that your CDN is not overriding both.
If you are not sure which AI bots can reach your site today, talk to Onward's team.