Search, training and retrieval crawlers
“Bot traffic” is not one thing. A crawler that collects your text to train a model, a crawler building an index so an assistant can cite you, and a crawler opening your page right now because a person asked a question are three different events with three different consequences — and they often come from the same company.
Telling them apart is the difference between a hit count and a finding.
The four things a bot can want
Section titled “The four things a bot can want”| Category | Wants | A hit means |
|---|---|---|
| Search | To index you for a search engine | The classic crawl. Ranking depends on it |
| AI training | To collect content in bulk for training | Your content was ingested. Nobody read it |
| AI assistant | To answer a question, now or later | Either you were indexed for citation, or someone just asked |
| Social | To render a link preview | Somebody shared your URL |
Everything else Log Hero groups as Other — SEO tools, uptime monitors, scanners, libraries, scripts. A large and mostly legitimate group.
Search crawlers
Section titled “Search crawlers”Googlebot, bingbot, Applebot, YandexBot, Baiduspider, PetalBot, DuckDuckBot and the rest. Their behaviour is decades old and well documented: they discover URLs, fetch them, decide whether to index them, and come back at a rate they choose based on how useful and how fast your site is.
What is worth watching here is the shape rather than the total: which sections they favour, how often they re-crawl, what proportion of their requests end in an error, and whether any of that changed after your last release.
Note that one search engine is several crawlers. Google runs separate agents for desktop, smartphone, images, video and ads, and they appear as separate names in your log.
AI training crawlers
Section titled “AI training crawlers”GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent,
VelenPublicWebCrawler and others.
These collect at scale, for building or refining a model. The important thing to understand about the numbers: volume here measures how thoroughly your content is being ingested, and says nothing at all about whether anyone will ever read it. A page fetched a thousand times by training crawlers has not been seen by a single person.
This is the category where the robots.txt question actually bites, because what you are trading away — your content as training data — is not the same as what you are trading for.
AI assistant crawlers
Section titled “AI assistant crawlers”This is the category worth splitting, and Log Hero splits it in two.
AI search index
Section titled “AI search index”OAI-SearchBot, PerplexityBot, YouBot, DuckAssistBot, xAI-SearchBot,
meta-webindexer, Google-CloudVertexBot, LinkupBot.
These build the index an assistant reaches into when it needs a source. Being crawled by them is the AI equivalent of being in a search index: it is the precondition for ever being cited, and it happens ahead of time rather than in response to anything.
AI user fetcher
Section titled “AI user fetcher”ChatGPT-User, Perplexity-User, Claude-User, Gemini-Deep-Research,
OpenAI Bot, GPT User Bot.
A hit from one of these means a person asked a question and an assistant opened your page to answer it. There was a human in the loop, seconds ago.
This is the closest thing to AI-driven demand that exists in a server log, and it is the row to trend over time. It is also completely absent from every JavaScript-based analytics tool you have — see Why analytics cannot see bots.
AI Visibility separates the roles. User-triggered fetches is the one with a person behind it.
Why the distinction changes what you do
Section titled “Why the distinction changes what you do”The categories are not trivia. They lead to different decisions, and conflating them produces expensive mistakes.
Blocking training does not block citation, and blocking citation does not
block training. These are separate crawlers with separate names, often from
the same operator. Disallowing GPTBot in robots.txt stops OpenAI collecting
your content for training. It does not stop ChatGPT-User from opening your
page when a user asks about you, and it does not remove you from what ChatGPT
can cite — that is OAI-SearchBot, a different agent entirely.
So the two decisions are genuinely independent:
| You want | Then |
|---|---|
| Not to be training data, but still be citable | Disallow the training crawlers, allow the search-index and user-fetch ones |
| To be cited as widely as possible | Allow everything and make sure they get 200s, not errors |
| To be left alone by AI entirely | Disallow all of them — and accept that you become invisible to assistants |
Errors cost more in the AI categories than in search. Googlebot has been
crawling you for years and will retry a 404 patiently. A user fetcher gets one
attempt while a person waits: if it hits a 404 or a slow response, the
assistant answers from something else, and there is no second chance. An AI 404
rate materially higher than your Googlebot 404 rate is worth fixing first.
A social spike is not a crawl. facebookexternalhit, Twitterbot,
Slackbot, WhatsApp Crawler and friends fetch a page once so a preview card
can be drawn. A sudden burst almost always means a link of yours is being
shared, not that anything is crawling you.
Seeing this in Log Hero
Section titled “Seeing this in Log Hero”AI Visibility is built on exactly this split: AI share of bot hits, user-triggered fetches, the AI 404 rate, and how much of your sitemap AI bots have actually reached. Bot Activity gives you the same population by name and by day.
One honest limit: classification depends on the crawler identifying itself. A bot that sends no recognisable User-Agent, or one Log Hero has not learned yet, appears under Other or as Unknown bot — with its hits counted, but without a category. The Bot catalogue lists every name currently recognised and which category it lands in.