Anatomy of a log line

A server log is not a complicated format. It is one line per request, with the fields in a fixed order, separated by spaces. Once you can read one line you can read all of them.

66.249.66.1 - - [30/Sep/2026:16:09:05 -0400] "GET /pricing/?ref=nav HTTP/1.1" 200 12847
"https://example.com/" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

That is the Combined Log Format, the default on both Apache and nginx and by far the most common thing you will meet. Read left to right.

Field by field

#	Field	In the example	What it is
1	Client IP	`66.249.66.1`	The address the request came from — and where the answer was sent
2	Identity	`-`	An ancient identity-protocol field. Always `-`. Ignore it
3	User	`-`	The username, if the resource needed HTTP authentication. Usually `-`
4	Timestamp	`[30/Sep/2026:16:09:05 -0400]`	When the request was handled, with the offset from UTC
5	Method	`GET`	What kind of request. `GET` is nearly all crawler traffic
6	Path	`/pricing/?ref=nav`	What was asked for, query string included
7	Protocol	`HTTP/1.1`	The HTTP version spoken
8	Status	`200`	What your server answered
9	Bytes	`12847`	Size of the response body. `-` when nothing was sent
10	Referrer	`https://example.com/`	The page that linked here, as claimed by the client
11	User-Agent	`Mozilla/5.0 (compatible; Googlebot/2.1; …)`	What the client says it is

A - anywhere means this field has no value, not zero. It is the format’s way of keeping the columns aligned.

You will also meet the Common Log Format, which is the same line without the last two fields. It is a downgrade: without the User-Agent you cannot tell one crawler from another, or a crawler from a person. If you get to choose, choose Combined.

Which fields you can trust

This is the part that decides what a log analysis can honestly claim. The fields fall into two groups: things your server observed, and things the client told it.

Field	Set by	Trust
Timestamp, Status, Bytes, Method, Path, Protocol	Your server	Reliable. These are facts about what happened
Client IP	The network	Hard to forge. The response has to go back there, so a faked address gets no reply. But see proxies below
Referrer	The client	Unreliable. A free-text header. Anyone can put anything in it
User-Agent	The client	Unreliable. Also free text. This is the important one

The User-Agent deserves the emphasis. It is the field everyone uses to identify crawlers, and it is a string the client types about itself. Writing Googlebot into it takes one line of code, and plenty of scrapers do exactly that to get past rules that treat search engines kindly.

Which is why the useful question is never what does it claim to be but does the address back up the claim — a reverse and forward DNS check against the operator’s own records. How bot verification works covers the mechanism, and Detect fake Googlebots what to do when the numbers look wrong.

What Log Hero stores per request

Log Hero records the same fields, normalised so they mean the same thing across every integration, plus a few your raw log does not have:

Stored	Where it comes from
Hostname	Which of your domains answered — a raw log often omits this entirely
Protocol, method, path with query string, status code	The request, as above
Client IP, User-Agent, referrer	The request, as above
Timestamp	The request, as above
Page load time	How long your server took, in milliseconds
Bot name	Derived: the crawler the User-Agent and address resolve to
Is bot	Derived: whether this was machine traffic at all
Official bot	Derived: whether the claim survived verification
Spam bot type	Derived: set when the claim did not survive

The last four are the difference between a log file and a log analysis. Your raw log has the evidence; it does not have the conclusion.

Metrics defines what each figure built on these fields means, and the GA4 property lists them as they appear in Google Analytics.

The hostname problem

A single server usually answers for several names — example.com, www.example.com, a staging host, maybe an old domain still redirecting. In many default configurations all of them write into the same log file, and the line does not say which host was asked.

If your analysis shows traffic you cannot place, this is a good first suspect. Log Hero stores the hostname explicitly for exactly this reason, and shows a hostname selector once it has seen more than one. See Find crawl on the wrong hostname.

Behind a proxy, the client IP is not the client

If a CDN, load balancer or reverse proxy sits in front of your server, then from your server’s point of view every request arrives from the proxy. Field 1 stops being the visitor and becomes your own infrastructure.

The real address is passed along in a header — usually X-Forwarded-For, or CF-Connecting-IP on Cloudflare — and your server has to be configured to prefer it (mod_remoteip on Apache, real_ip on nginx).

This is not a cosmetic problem. Bot verification is a check on the client address, so if every request appears to come from your load balancer, nothing can be verified and every crawler shows up as unverified. A verification rate that suddenly collapses to zero is almost always this, not an attack.

What is not in a log line, ever

Just as important as what a log records is what it cannot:

Fragments. The #section part of a URL is used by the browser after the page loads and is never transmitted. It appears in no log file anywhere.
POST bodies. The line records that a POST happened, not what was in it.
Whether JavaScript ran, or whether anything was rendered. The log ends when the response is sent.
Whether a crawled page was indexed. A fetch is not an index entry. Pair your log with Search Console for that half.
Whether an assistant cited you. A fetch is not a citation either.
Soft 404s. A page that returns 200 with “sorry, not found” content is a 200 here. Your log records your server’s verdict, not the page’s honesty.
Anything finer than a request. Time on page, scroll depth, clicks — all of that is browser-side, and belongs to analytics.