How bot verification works
A User-Agent header is a claim, not an identity. Any client can send
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)and nothing about the request makes that harder than sending any other string. Scrapers do it constantly, because sites treat Googlebot gently: no rate limit, no CAPTCHA, no paywall.
So Log Hero records the claim, and separately checks it.
Two independent things
Section titled “Two independent things”Every recorded request carries two facts that come from different places:
- The name comes from the User-Agent. It is what the client said it was.
- The verification comes from the IP address. It is whether the request came from a range that crawler actually uses.
These can disagree, and the disagreement is the interesting part. The Verified vs unverified bots panel on Bot Activity shows the split per bot.
| What you see | What it means |
|---|---|
| Verified | The User-Agent claimed a crawler, and the address is in that crawler’s known ranges. |
| Unverified | The User-Agent claimed a crawler, and the address is not. |
The spread is the point: bingbot and Amazonbot verify completely, AhrefsBot manages 1% and YouBot none at all.
Unverified does not mean fake
Section titled “Unverified does not mean fake”This is the distinction worth internalising, and it is why the panel says unverified rather than fake.
An unverified hit means our address lists did not recognise the IP. That is evidence of a spoofed User-Agent, not proof of one. The other explanation is mundane: a legitimate crawler using a range we do not track yet. Operators add ranges, and smaller crawlers publish them inconsistently or not at all.
What the two readings look like in practice:
- A bot that is 100% unverified, with a familiar name. Suspicious. Real Googlebot traffic is verified in the high nineties.
- A bot that is mostly verified with a small unverified tail. Normal. A new range, or a request that arrived through something that rewrote the address.
- A bot that is 100% verified. Confirmed, and nothing to do.
Why the client IP has to survive the trip
Section titled “Why the client IP has to survive the trip”Verification compares the address your server saw. If something in front of your origin replaces that address with its own, every crawler looks unverified, because every request now comes from your proxy.
Behind a reverse proxy or a load balancer, make sure the real client address is
restored before the request is logged — mod_remoteip on Apache, the
equivalent trusted-proxy setting in your framework. Edge integrations do not
have this problem: the CDN logs the address it received directly.
The bot with no name
Section titled “The bot with no name”Roughly one bot hit in eleven arrives with no name at all, shown as Unknown bot.
That sounds like junk data and is the opposite. Log Hero derives the name from the User-Agent and the identity from the IP address, and these are separate lookups. An empty name means the address was recognised as a crawler but the User-Agent is not in the name table — a real bot we cannot label yet, rather than a row to discard.
What verification is not
Section titled “What verification is not”- Not a block. Log Hero records; it does not intervene. Deciding what to do about a spoofed crawler is a job for your WAF.
- Not retroactive. Verification is evaluated when the request is recorded, so it only exists for the period since it was switched on for your account. Older ranges show no split.
- Not a judgement of intent. A scraper wearing Googlebot’s name and a monitoring tool with a lazy default look identical here.