Bot catalogue
Log Hero sorts the bots it recognises into four categories. Everything it does not recognise is Other — which is a large and legitimate group, not a failure: SEO tools, uptime monitors, scanners, libraries and one-off scripts all land there.
| Category | What these bots want |
|---|---|
| Search | To index your pages for a search engine. |
| AI training | To collect content for training a model. |
| AI assistant | To answer a question, now or from an index built for it. |
| Social | To render a link preview when someone shares your URL. |
| Other | Everything else. |
Search
Section titled “Search”Googlebot, Googlebot-Image, Googlebot-Video, bingbot, Amazonbot, Baiduspider, Baiduspider-render, Baiduspider-image, YandexBot, YandexImages, YandexAccessibilityBot, Sogou, Sogou web spider, DuckDuckBot, Applebot, coccocbot-web, YisouSpider, PetalBot, IbouBot, Yeti, Exabot, Yahoo! Slurp.
AI training
Section titled “AI training”ClaudeBot, GPTBot, Bytespider, meta-externalagent, CCBot, Grammarly, VelenPublicWebCrawler, TimpiBot.
These collect in bulk. Volume here says how thoroughly your content is being ingested, and nothing about whether anyone reads it.
AI assistant
Section titled “AI assistant”ChatGPT-User, OAI-SearchBot, PerplexityBot, Perplexity-User, YouBot, DuckAssistBot, meta-webindexer, xAI-SearchBot, Google-CloudVertexBot, Claude-Web, Claude-User, Gemini-Deep-Research, LinkupBot, OpenAI Bot, GPT User Bot.
Social
Section titled “Social”facebookexternalhit, Facebook External Hit, WhatsApp Crawler, Slackbot-LinkExpanding, VKShare, TelegramBot, Discordbot, Twitterbot, PinterestBot, Yahoo Link Preview.
A spike here is almost always someone sharing a link, not a crawl.
AI roles
Section titled “AI roles”Inside the two AI categories, AI Visibility
splits bots again by what they are doing. This is the more useful cut, because
the three roles mean genuinely different things for your site —
Search, training and retrieval
crawlers is the long version,
including what each role means for your robots.txt.
| Role | Bots | What a hit means |
|---|---|---|
| AI training | ClaudeBot, GPTBot, Bytespider, meta-externalagent, CCBot, Grammarly, VelenPublicWebCrawler, TimpiBot | Your page was collected for training. |
| AI search index | OAI-SearchBot, PerplexityBot, YouBot, DuckAssistBot, meta-webindexer, xAI-SearchBot, Google-CloudVertexBot, LinkupBot | Your page was indexed so an assistant can cite it later. |
| AI user fetcher | ChatGPT-User, Perplexity-User, Claude-User, Gemini-Deep-Research, OpenAI Bot, GPT User Bot | Someone asked a question and an assistant opened your page to answer it. |
Claude-Web counts as an AI bot but has no role assigned, so it appears as
AI (other).
The user-fetcher row is the one to trend. It is the closest thing in a server log to AI-driven demand — though a fetch is not a citation, and your logs cannot tell you whether the assistant used what it found.
Not a rejects bin. Common residents:
- SEO tools — AhrefsBot, SemrushBot and the rest.
- Scanners and probes — CensysInspect, vulnerability scanners, anything
hunting for
/wp-admin. - Generic user agents —
Mozillaon its own, scraping libraries such as Scrapy, and scripts that never set a User-Agent. - Unknown bot — the address was recognised as a crawler but the User-Agent is not in the name table. See How bot verification works.
When a bot you know is missing
Section titled “When a bot you know is missing”The classifier learns names over time, and an unmapped name is not an error — it shows up under Other with its name intact and its hits counted. If a crawler that matters to you sits in Other, tell us at support@log-hero.com and it gets a category.