Home › Free tools › Server log analyser
Your access log is the only record of what search engines and AI crawlers actually did on your site, rather than what you hope they did. It is also a list of your visitors' IP addresses, which is why this reads it in your browser and uploads nothing.
Runs in your browser. Nothing is sent anywhereApache or nginx, combined or common format, plain or gzipped. The file is read in this tab and never leaves it. Logs hold your visitors' IP addresses, which is exactly why this one does not upload anything.
access.log, access_log or
yourdomain.com-ssl_log, often gzipped. Large files are fine: they are read in
chunks, so nothing has to fit in memory at once.
Grouped by what each request said it was. Search engines and AI crawlers are separated because you may well want to allow one and refuse the other.
| Crawler | Kind | Requests | URLs | Errors | Last seen |
|---|
The most useful thing in any log. A search engine that keeps being handed a 404 or a 500 spends your crawl budget on nothing, and a page it cannot fetch is a page it cannot rank.
| URL | Status | Hits | Crawlers |
|---|
Crawl budget is finite. This is how yours was spent.
| Most crawled URL | Requests | Crawlers |
|---|
A sudden stop usually means a block, a robots.txt change or an outage. A sudden spike is usually something crawling you that is not a search engine.
Any request can put Googlebot in its user agent, and plenty do. A log on its own cannot prove otherwise, so this flags what looks off and then shows you how to settle it properly.
host 66.249.66.1googlebot.com or
google.com, Bingbot to search.msn.com. Google, Bing and the
larger AI crawler operators also publish the address ranges their crawlers use, which you
can check an address against directly.
Optional. Paste your sitemap.xml, or just a list of URLs,
and find the pages you have asked search engines to index that they have never once
fetched.
No. The file is opened with the browser's own File API and read in chunks inside the page. Nothing is transmitted, there is no server to transmit it to, and you can confirm it by opening your browser's network tab: the page makes no requests while it works. This matters more here than on the other tools, because an access log contains the IP addresses of everyone who visited you, which in most jurisdictions is personal data.
The combined and common formats, which is what Apache and nginx write by default and what almost every shared host produces. It copes with the usual variations: a virtual host name in front of the address, a missing referer or user agent, IPv6 addresses, and quotes escaped inside a user agent string. Gzipped logs are unzipped in the browser.
No. The file is streamed and only counters are kept, never the individual lines, so memory use stays flat whether the log is five megabytes or five hundred. Very large files simply take longer to read.
Because a log cannot answer that, and a tool that claims otherwise is guessing. Any request can put Googlebot in its user agent. Confirming identity means a reverse DNS lookup on the address followed by a forward lookup on the name that comes back, and a browser cannot make DNS queries at all. So the tool flags the claims that contradict themselves, such as one address that is both Googlebot and a person browsing in Safari, and gives you the command to settle the rest.
Three things, in order. Errors crawlers hit, because a search engine handed a 404 or a 500 has spent your crawl budget on nothing. Then where the crawling went, because a site whose crawl budget is mostly parameter URLs and static files has a structural problem. Then crawling by day, where a sudden stop usually means a robots.txt change or an outage nobody noticed.
Yes, and they are listed separately from search engines for that reason. GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider and others appear with their own request counts. Whether you want to allow them is a decision, not a setting: some fetch to train models, some fetch to answer a question a person just asked, and the same robots.txt rule treats both the same.