Free tool

Free robots.txt tester: can Google and AI bots crawl this URL?

Test any URL against your live robots.txt or an edited version, for Googlebot, Bingbot and the AI crawlers, and see the exact rule that allows or blocks it.

User agents

Leave it empty to test the live file. Paste or edit a version to see what a change would do before you publish it.

What this robots.txt tester does

This robots.txt tester reads the robots.txt of the site you enter and applies it the way crawlers do, following RFC 9309, the standard Google documents:

  • Allowed or blocked: for each user agent you pick, whether it may crawl the URL, including its query string.
  • The deciding rule: the Allow or Disallow line that decides, its line number and the group it belongs to, so you know exactly what to change.
  • Validation: lines crawlers skip or misread, such as rules before any User-agent, Noindex or Crawl-delay directives Google ignores, missing colons and files over 500 KiB.
  • Test before you publish: after a live test the file loads in the text box. Edit it and test again to see the effect of a change without touching your site.

Common robots.txt mistakes

  • A group for one bot that forgets the general rules. A crawler with its own group ignores the * group entirely, so rules you want it to follow must be repeated there.
  • Blocking CSS and JavaScript. Google renders pages; if it cannot load their resources it may not see the content.
  • Using Disallow to deindex. A blocked page can stay in the index. Use noindex on a crawlable page instead.
  • A server error on robots.txt. If /robots.txt answers 5xx, Google stops crawling the whole site until it recovers.
  • Blocking AI search crawlers by accident. A Disallow for every bot, or a firewall rule, can take you out of ChatGPT and Perplexity answers.

robots.txt and AI crawlers

AI companies run several crawlers with different jobs. GPTBot, ClaudeBot and CCBot collect training data. OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes their assistants search. ChatGPT-User and Perplexity-User fetch a page when a user asks about it. Google-Extended and Applebot-Extended are switches with no crawler of their own: they tell Google and Apple whether they may use your content for their AI models. Block training if you want to, but keep the search crawlers allowed if you want to appear in AI answers. Your server logs show which of them actually come. Our AI crawler checker tests all of them at once, including how your server answers each one, and the llms.txt generator gives the ones you allow a map of your key pages.

Being crawlable is the first step. Whether AI engines then recommend you is what Mencoro tracks , every day across ChatGPT, Perplexity and Google’s AI Overviews and AI Mode.

FAQ

robots.txt tester: frequently asked questions

Enter the URL you want to check and pick the user agents. The tester reads your live robots.txt, applies the same rules Google documents and shows, for each crawler, whether the URL is allowed or blocked and which line of the file decides it. Paste an edited version in the text box to see what a change would do before you publish it.
Google retired the robots.txt Tester in Search Console at the end of 2023 and replaced it with a robots.txt report that shows the file Google fetched and its errors, but does not test a URL against it. This tester fills that gap and adds the AI crawlers.
It follows the group for its own user agent, or the * group when it has none; it does not combine both. Within the group, the longest matching rule wins, and when an Allow and a Disallow of the same length match, Allow wins. * matches any sequence of characters and $ marks the end of the URL.
No. robots.txt controls crawling, not indexing: a blocked page can still be indexed without its content if other sites link to it. To keep a page out of search results, let Google crawl it and add a noindex meta robots tag or X-Robots-Tag header. Noindex lines inside robots.txt are ignored.
It depends on what you want. Blocking training crawlers such as GPTBot or CCBot keeps your content out of future training sets. Blocking search and user crawlers such as OAI-SearchBot, PerplexityBot or ChatGPT-User also keeps you out of the answers those assistants give, which usually costs visibility. Many sites block training and allow search.

More free tools

AI visibility checker

Ask ChatGPT, Gemini and Perplexity what your customers ask, and see whether they name you or your competitors.

Check your brand →

AI Overview checker

See which of your keywords show a Google AI Overview or AI Mode answer, whether it cites you and which sites it cites instead.

Check your keywords →

ChatGPT rank checker

Run your own prompts on ChatGPT and see where it ranks your brand in its answer, next to your competitors.

Check your rank →

Perplexity rank checker

Run your own prompts on Perplexity and see where it ranks your brand and which sources it cites.

Check your rank →

Share of voice calculator

Work out your share of voice against your competitors from mentions, impressions or clicks, or from AI answers weighted by how they mention you.

Calculate yours →

AI crawler checker

See whether GPTBot, ClaudeBot, PerplexityBot, Googlebot and more can read your site, from robots.txt to your server.

Check your site →

Schema validator

Check a page's JSON-LD and Microdata against schema.org and Google's rich result requirements, with fixes for every error.

Validate a page →

Log file analyzer

See in your server logs how often GPTBot, ClaudeBot, PerplexityBot and Googlebot crawl you, which pages they read, and who comes from ChatGPT.

Analyze your logs →

llms.txt generator

Build a draft llms.txt from your sitemap, with your key pages grouped by section, ready to copy or download.

Generate yours →

See all free tools →

Start tracking your brand in AI search today

Monitor how AI engines cite your brand, track keyword positions, and benchmark against competitors, all in one platform.