Free tool

Free log file analyzer for SEO and AI crawlers

Drop your access log and see how Googlebot and the AI crawlers use your site, which pages they read, the errors they hit and who arrives from ChatGPT. It runs in your browser.

What this log file analyzer shows

  • Every bot, by what it is for: search engines, AI training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider), AI search indexes (OAI-SearchBot, PerplexityBot, Claude-SearchBot) and on-demand visits (ChatGPT-User, Claude-User, Perplexity-User), with requests, pages, errors and last visit.
  • Requests per day: search engines against AI crawlers, to spot spikes and the day a crawler stopped coming.
  • The pages AI crawlers read most, and the ones where they get 4xx or 5xx errors.
  • Visits from AI assistants: people who clicked a link in ChatGPT, Perplexity, Gemini, Copilot or Claude, and the pages they land on.

How to get your access logs

  • Nginx: usually /var/log/nginx/access.log and its rotated .gz files.
  • Apache: /var/log/apache2/access.log or /var/log/httpd/access_log.
  • Hosting panels: cPanel and Plesk have a “Raw access logs” or “Logs” section to download them.
  • Cloudflare, Vercel and other CDNs: export them as JSON lines with Logpush or a log drain; the request URI, status, user agent and referer fields are enough.

What to do with what you find

If an AI crawler you want is missing, check whether robots.txt or your firewall blocks it with the AI crawler checker, and test your rules with the robots.txt tester. If crawlers spend their visits on redirects, 404s or parameter URLs, fix or block those so they reach the pages that matter. If the pages they read most are not the ones you want AI assistants to quote, give them a map with an llms.txt .

Crawling is the start of the chain, not the end. Whether ChatGPT, Perplexity and Google’s AI Overviews then recommend your brand is what Mencoro tracks , every day, next to your competitors, from €15 a month.

FAQ

Log file analyzer: frequently asked questions

It is reading your server’s access logs to see what crawlers actually do on your site: which pages Googlebot and other bots request, how often, and what your server answers. Unlike crawl simulations, logs show real visits, so they reveal wasted crawl budget, errors served to bots and pages crawlers never reach.
No. The analyzer runs in your browser: the file is read on your computer, a piece at a time, and nothing is sent to our servers. That is why it can handle logs with IP addresses and other data you would not want to upload.
The combined format that Nginx and Apache write by default, including variants with a virtual host in front or extra fields at the end; W3C extended logs from IIS and some CDNs; and JSON lines such as Cloudflare Logpush, Vercel or most log shippers. Gzipped files (.gz) are read as they are, and you can add several files at once.
Look for its user agents in your logs: GPTBot collects training data, OAI-SearchBot builds the index ChatGPT searches, and ChatGPT-User fetches a page when a user asks about it. The analyzer counts each one, with the pages they request and the errors they get. Visits from people who clicked a link in ChatGPT show up as referrals from chatgpt.com or with utm_source=chatgpt.com.
Yes. Anyone can send a request that claims to be Googlebot or GPTBot. For a strict audit, check the IP addresses against the ranges Google, OpenAI, Anthropic, Perplexity and Microsoft publish for their crawlers.

More free tools

AI visibility checker

Ask ChatGPT, Gemini and Perplexity what your customers ask, and see whether they name you or your competitors.

Check your brand →

AI Overview checker

See which of your keywords show a Google AI Overview or AI Mode answer, whether it cites you and which sites it cites instead.

Check your keywords →

ChatGPT rank checker

Run your own prompts on ChatGPT and see where it ranks your brand in its answer, next to your competitors.

Check your rank →

Perplexity rank checker

Run your own prompts on Perplexity and see where it ranks your brand and which sources it cites.

Check your rank →

Share of voice calculator

Work out your share of voice against your competitors from mentions, impressions or clicks, or from AI answers weighted by how they mention you.

Calculate yours →

AI crawler checker

See whether GPTBot, ClaudeBot, PerplexityBot, Googlebot and more can read your site, from robots.txt to your server.

Check your site →

robots.txt tester

Test whether Googlebot, Bingbot and the AI crawlers may crawl a URL, which rule decides it, and what a change to the file would do.

Test a URL →

Schema validator

Check a page's JSON-LD and Microdata against schema.org and Google's rich result requirements, with fixes for every error.

Validate a page →

llms.txt generator

Build a draft llms.txt from your sitemap, with your key pages grouped by section, ready to copy or download.

Generate yours →

See all free tools →

Start tracking your brand in AI search today

Monitor how AI engines cite your brand, track keyword positions, and benchmark against competitors, all in one platform.