Two kinds of bots, two completely different consequences
An AI provider does not send a single bot, it sends several, each with a different job. Some collect material to train a model. Others fetch content at the exact moment a user asks a question and help determine who appears as a source in the answer. That second kind is your line into the AI response.
Blocking a training bot keeps your material out of future models. You may well want that, and the impact on your current visibility is small. Blocking a citation bot, on the other hand, removes you from the game entirely. OpenAI states this plainly in its own documentation:
“Pages that disallow OAI-SearchBot will not appear in ChatGPT search responses.”
OpenAI, developer documentation
Anthropic says the same for its citation bot, Claude-SearchBot: blocking it can reduce your visibility and accuracy in Claude responses. The problem is that most site owners never make this distinction at all. They block everything and have no idea which side of the line they are standing on.
Who is who? The main AI crawlers at a glance
| Bot | Provider | Purpose | Blocking means |
|---|---|---|---|
| GPTBot | OpenAI | Training | no visibility loss |
| OAI-SearchBot | OpenAI | Citations in ChatGPT Search | gone from ChatGPT answers |
| ClaudeBot | Anthropic | Training | no visibility loss |
| Claude-SearchBot | Anthropic | Citations in Claude | less visible in Claude |
| Google-Extended | Training for Gemini | no effect on Google Search | |
| PerplexityBot | Perplexity | Citations in Perplexity | no mentions in Perplexity |
| CCBot | Common Crawl | Open training dataset | no direct visibility loss |
| Bytespider | ByteDance | Training | no loss for Western providers |
The logic is straightforward. The training bots in this list can be blocked if you do not want your material feeding into third-party models. The citation bots, OAI-SearchBot, Claude-SearchBot and PerplexityBot, should be allowed through if you want to appear in AI responses at all. Google-Extended is a special case: it only governs whether your content is used for Gemini training and has no bearing on your normal Google ranking.
The trap you may already be in
Since July 2025 the situation has quietly tightened. A major infrastructure provider that handles a substantial share of web traffic now blocks AI crawlers by default for new customers. Anyone who does not configure otherwise ends up blocking both categories, including the citation bots. The decision against your own AI visibility gets made without anyone consciously making it.
There is a second uncomfortable detail. The classic robots.txt file, the standard way to tell bots to stay out, is no longer a reliable lock. Research shows that a growing share of AI requests simply ignores those rules. Anyone who genuinely wants to block, say, a publisher protecting its content, needs network-level controls. Anyone who wants to be visible needs to actively ensure the citation bots can get through.
Why the most credible sites are locking themselves out
There is an observation worth sitting with. Two out of three of the most respected news sites now block AI bots, while the majority of dubious sites leave all doors open. A research paper from autumn 2025 drew the obvious conclusion: when reliable sources close their doors and unreliable ones stay open, AI services inevitably learn from and cite the weaker half of the web.
For you as a business, this means: if you are not a publisher with licensing interests and have no particular protection needs, every citation bot you block works against you. You are handing your spot in the answer to someone who has less to offer but left their doors open.
Check your blocks before you think about visibility
Before investing time in AI visibility, it is worth checking whether your site actually lets the citation bots in. Surprisingly often, that is where the quiet problem sits. I look at which bots your site currently reaches and which ones are being turned away at the door. Let’s talk.
Frequently asked questions
Should I block AI crawlers on my website?
It depends on the bot. Training bots such as GPTBot or ClaudeBot can be blocked if you do not want your material feeding into third-party models, without short-term visibility consequences. Citation bots such as OAI-SearchBot or PerplexityBot should be allowed through; blocking them removes you from AI-generated answers.
What is the difference between training bots and citation bots?
Training bots collect material to train AI models. Citation bots fetch content at the moment a user asks a question and help determine who gets named as a source in the answer. Blocking a training bot affects future models; blocking a citation bot affects your visibility today.
Does blocking Google-Extended hurt my Google ranking?
No. Google-Extended only controls whether your content is used for Gemini training. It is separate from ordinary Google Search. You can block it without affecting your ranking or indexing in the classic search results.
Is robots.txt enough to block AI bots?
Not reliably. Research shows that a growing share of AI requests ignores the rules in robots.txt. Anyone who genuinely wants to keep bots out needs network-level controls. robots.txt works well for selectively letting bots through; it is not a dependable lock.
How do I know if I have accidentally blocked myself?
It often comes down to default settings. Since mid-2025, some security services automatically block AI bots for new sites, including the citation bots. A look at your robots.txt and your security service’s configuration shows which bots are currently being turned away.
I am a small business, not a publisher. What should I do?
As a rule, let the citation bots through. Blanket blocking offers no advantage without licensing interests, and it costs you visibility in AI-generated answers. You can block the training bots if you want to. That is a question of stance. It hardly changes visibility.