Sitemap fetch failed

Fetch now or Detect did not give you an inventory. Find the message you got below. The sections are ordered by how often each cause is the real one.

Log Hero prints fetch errors in three places:

On Project settings ▸ Sitemap, as a red or amber notice with the failing file listed under it.
On the Coverage report, in the Sitemap health, from the logs panel, as Last fetch OK, Last fetch partial — some files failed, Last fetch failed, Last fetch blocked by the site or Last fetch looked suspicious — previous inventory kept.
In the panel header on every report page, as Last fetch: {status} — {lastError}.

Upload is the escape hatch for almost every case on this page. Upload is exempt from the rate limit and never touches your site.

Your CDN or WAF blocked the fetch

{host} blocked KeywordHeroBot ({vendor}, HTTP {code}). Allow the user agent KeywordHeroBot or our IP in your CDN/WAF, or upload the sitemap file instead.

Cloudflare, Akamai, Imperva and PerimeterX all produce this. Log Hero names the vendor when it recognises the challenge page. If it does not, the message reads (access denied, HTTP {code}) instead. On Detect the same block reads The site blocked our crawler ({vendor}) — HTTP {code}. Upload the sitemap file instead.

What it means. Your edge answered 401, 403, or a challenge page, so Log Hero never got the file.

What to do. Allow this user agent in your WAF or CDN:

Mozilla/5.0 (compatible; KeywordHeroBot/1.0; +https://keyword-hero.com/bot)

Every request also carries the header From: integrations@keyword-hero.com.

Log Hero does not publish a source IP address for the fetch, so you cannot allowlist one. The Learn more link in the notice points at https://keyword-hero.com/bot, and that page is not live yet. Allow the user agent, or upload the file.

Your robots.txt disallows the fetch

{host} blocked KeywordHeroBot — your robots.txt disallows our user agent for this path.

On Detect the same cause reads robots.txt blocks KeywordHeroBot — enter the sitemap URL manually or upload the file.

What it means. Log Hero reads your robots.txt before it fetches, and it obeys a Disallow for its own user agent.

What to do. Allow KeywordHeroBot for the sitemap path in robots.txt, or upload the file.

The host is not one of this project’s hosts

{host} is not one of this project's hosts

What it means. Log Hero fetches only from hostnames it has seen in your log data. Today that list is empty on existing projects, because it is built from a field the app no longer fills. The result is that Fetch now refuses every address you type.

What to do. Upload the file. Upload does not use the host list.

No sitemap found at the usual locations

No sitemap found at the usual locations — enter the URL manually or upload the file

What it means. Detect read your robots.txt for Sitemap: lines and probed the usual paths, and found nothing usable. It probes at most three hosts and gives up after 20 seconds.

Today this message also appears when the host list behind the probe is empty. That is the same defect as the section above.

What to do. Upload the file.

A related message appears when the project has no traffic at all: No traffic recorded for this project yet — upload the sitemap file instead.

The fetch timed out

timeout fetching {host} · timeout reading body from {host} · stopped at cap: wallClock

What it means. One fetch has 40 seconds for everything: the robots.txt reads and every sitemap file. Each file has at most 10 seconds to connect and 20 seconds to deliver its body. A file that fails on the network is retried twice. An HTTP error is never retried.

When the 40 seconds run out, the run stops and the status becomes partial. Log Hero keeps the URLs it already collected.

What to do. Fetch a smaller set of files, one index branch at a time. If your sitemap is large or your server is slow, upload the file instead.

The response is not a sitemap

response is neither a sitemap XML nor a URL list

What it means. Log Hero got a body it does not recognise. An HTML error page or a login page at the sitemap address produces this.

Related messages on the same file:

Message	Cause
`HTTP {status}`	The server answered with a non-2xx status other than a block.
`compressed content could not be decoded`	The gzip or deflate stream is corrupt.
`unsupported encoding "{label}"`	The XML declares a character encoding Log Hero cannot read.

Broken XML is not rejected. Log Hero keeps every <loc> and <lastmod> it read before the fault and stores them. There is no “invalid XML” message. A sitemap with a truncated tail therefore produces a short inventory and a successful status. If the URL count is far below what you advertise, open the file and look for a syntax error.

What to do. Open the sitemap address in a browser. Make sure that it returns XML or a plain URL list, and not a page.

You uploaded a sitemap index

this is a sitemap index — upload the individual sitemap files, or enter the index URL instead

What it means. An index file lists other sitemaps. It advertises no URLs of its own, and Log Hero does not follow the children of an uploaded file.

What to do. Upload a child sitemap file. An upload holds one file, so a site split over several child sitemaps cannot upload them all today.

A cap stopped the fetch

Message	Cap
`urls must be a list of 1..50 sitemap URLs`	50 addresses in the text box.
`at most 5 distinct hosts per sync`	5 hostnames per fetch.
`invalid URL: {s}` · `only http(s) URLs are allowed: {s}`	The address is malformed or not `http`/`https`.
`response larger than 20971520 bytes` · `content larger than …`	20 MB per file, after decompression.
`stopped at cap: maxUrls`	500 000 URLs per snapshot.
`host cap exceeded`	A child sitemap sits on a sixth hostname.
`File is larger than 50 MB`	The upload limit.

Two more caps have no message of their own. A fetch reads at most 50 sitemap files over the whole index tree, and follows an index at most 10 levels deep. Beyond either, the run stops and the status becomes partial.

What to do. Split the fetch into smaller runs, or upload the file.

URLs disappeared without a message

What it means. Log Hero drops an advertised URL whose hostname is not one of your project’s hosts, and it drops an entry that is not an http or https address. It prints no warning for either. The only sign is the number in the Dropped column of the files table on Project settings ▸ Sitemap.

What to do. Read the Dropped column after every fetch. If the count is high, make sure that the sitemap advertises your own hostnames.

Please wait before fetching again

Please wait {n} more minute(s) before fetching again

What it means. Log Hero permits one fetch per 10 minutes for each project.

What to do. Wait, or upload the file. Upload is exempt from the rate limit.

The fetch looked suspicious

Latest fetch found {N} URLs (previous {M}). Kept the previous inventory.

What it means. The new snapshot has less than half the URLs of the last one. Log Hero stores it but does not promote it, so your reports keep using the old inventory.

What to do. If your site really lost that many pages, choose Use the new one anyway. If it did not, find out why the fetch was short before you promote it.

The inventory is older than 180 days

What it means. Log Hero stores a snapshot for 180 days. Nothing re-fetches on its own, so an inventory that is never refreshed passes that age.

What to do. Fetch or upload the sitemap again. Do the same after every large change to the site.

If none of these match

Send an email to integrations@keyword-hero.com with the subject “My sitemap fetch failed”. Include:

The domain as registered in Log Hero.
The sitemap address you entered, or the name of the file you uploaded.
The message you got, copied exactly.
The time of the attempt.