If you want AI engines to recommend your business, let their crawlers read your site. Blocking GPTBot, Google-Extended, ClaudeBot, and PerplexityBot pulls your pages out of the pool those systems retrieve from when they build an answer — including answers that would have recommended a business like yours. The case for blocking is almost entirely about training data, and training is handled by different bots than the ones that fetch pages to cite. You can treat the two separately.
Related reading: the schema markup guide and how to write content AI engines cite.
Two reasons AI companies crawl
Training. Building the model. The crawler collects text that may enter the next version’s training set. This is the crawling people object to on copyright and compensation grounds. Opting out here does not change what already-deployed models know.
Retrieval and grounding. Answering a live question. When ChatGPT search, Perplexity, or Gemini respond to a prompt, they fetch and cite current pages. Block these crawlers and you are simply absent from that engine’s answers.
Most operators want out of the first and firmly into the second. Helpfully, the major providers split the two across separate user-agents.
The crawlers worth knowing
| User-agent | Operator | Purpose | Blocking it means |
|---|---|---|---|
GPTBot | OpenAI | Model training | You opt out of OpenAI training collection |
OAI-SearchBot | OpenAI | Index for ChatGPT search | You lose ChatGPT search citations |
ChatGPT-User | OpenAI | Live fetch when a user shares a link | User-initiated lookups of your pages break |
Google-Extended | Gemini training and grounding | You opt out of Gemini; no effect on Google Search or AI Overviews | |
ClaudeBot | Anthropic | Model training | You opt out of Anthropic training collection |
Claude-User | Anthropic | Live fetch for Claude answers | Claude cannot retrieve your pages to cite |
PerplexityBot | Perplexity | Index for Perplexity answers | You lose Perplexity citations |
CCBot | Common Crawl | Open dataset used by many model builders | You reduce third-party training use of your content |
Applebot-Extended | Apple | Apple Intelligence training | You opt out of Apple training (Applebot itself still powers Siri and Spotlight) |
Bytespider | ByteDance | Training | You block a crawler with little retrieval upside for most sites |
Two things to watch. Google-Extended is the one sites get wrong most often: it is a robots token, not a bot, and it governs only Gemini — blocking it does nothing to your standard Googlebot crawl or your AI Overviews eligibility. And Perplexity has been reported to fetch pages through undeclared agents when PerplexityBot is disallowed; robots.txt is a request, and not every operator honors it equally.
What blocking actually costs
For the retrieval crawlers, blocking means:
- OpenAI: no citations in ChatGPT search or in answers where it browses. Given how many discovery questions now start in ChatGPT, this is the most expensive block for a business that sells anything.
- Perplexity: no citations in Perplexity answers, which skew toward the comparison-shopping prompts that come right before a purchase.
- Google (via
Google-Extended): no use in Gemini grounding. Your Search presence and AI Overviews eligibility are untouched. - Anthropic: no use when Claude retrieves live pages to answer a question.
Blocking a training crawler costs nothing in visibility today. It is a position on how your content is used and a bet on future licensing.
What llms.txt is, and what it is not
llms.txt is a proposed convention: a markdown file at your domain root that hands an AI system a curated map of your most important pages, sometimes with clean-text versions alongside. The goal is to help models find and quote the right content instead of inferring it from your navigation.
What it is not: enforced, universal, or a ranking factor. Adoption across the major engines is still partial and informal. Publishing one is cheap and low-risk if your site is large or your key content is buried, and it does no harm. Just do not confuse it with robots.txt. llms.txt invites; robots.txt permits or denies. Only the second one controls access.
A setup for a site that wants to be cited
The sensible default for most businesses: allow every retrieval and search crawler, allow or block training crawlers based on your position, and block scrapers that offer no citation upside.
# Retrieval and search crawlers — these earn you citations
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-User
Allow: /
# Training crawlers — allow to be quoted from broadly, or disallow to opt out
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: CCBot
Disallow: /
# No upside
User-agent: Bytespider
Disallow: /
Adjust the training block to taste. If you sell access to content — courses, research, a members’ library — disallow GPTBot, ClaudeBot, CCBot, and Google-Extended, and keep the search agents allowed so your marketing pages still earn answers while your paid content stays out of training sets.
When blocking is the right call
- Paywalled or licensed content you intend to monetize directly or through a licensing deal.
- Proprietary data or tools where being quoted undercuts the product itself.
- Membership communities where the content belongs to the members, not to you to feed into a model.
For everyone else — every business whose real problem is that not enough people know it exists — these crawlers are how the newest discovery channel finds you. Let them in, then give them pages worth quoting.
Frequently Asked Questions
Should I block GPTBot?
Block it only if you object to OpenAI using your content for model training and are willing to forgo that. GPTBot is the training crawler; it is separate from OAI-SearchBot and ChatGPT-User, which fetch pages so ChatGPT can cite them. If you want to appear in ChatGPT's answers, leave the search and user agents allowed regardless of what you decide about GPTBot.
What is llms.txt and do I need one?
It is a proposed convention: a markdown file at your domain root that gives an AI system a curated map of your most important pages. It is not enforced, not universal, and not a ranking factor. Publishing one is low-cost and harmless if your site is large or your key content is buried, but it does not control access the way robots.txt does.
Does blocking AI crawlers stop my content being used for training?
Partly, and only going forward. Blocking training crawlers like GPTBot, ClaudeBot, and CCBot signals that you opt out of future training collection by those operators. It does nothing about models already trained on earlier crawls, and it does not bind operators that ignore robots.txt or crawl through undeclared agents.
Will allowing AI crawlers slow down my site?
For almost all sites the added load is negligible — these crawlers fetch at a modest rate and respect crawl-delay where set. If a specific bot is hitting you hard, rate-limit that user-agent rather than blocking the whole category and losing the citations.
What is the difference between GPTBot, OAI-SearchBot, and ChatGPT-User?
GPTBot collects text for model training. OAI-SearchBot builds the index ChatGPT search draws citations from. ChatGPT-User is the live fetch triggered when a person asks ChatGPT to open or summarize a specific link. They are separate user-agents so you can allow the ones that earn you visibility and decide about training on its own.