robots.txt Tester
Test URLs against a robots.txt for Googlebot, Bingbot or AI crawlers.
About this tool
Paste your robots.txt, pick a crawler (or type any user-agent token) and list the URLs or paths to test. Each one shows Allowed or Blocked and the line that decided it. Parsing problems and Sitemap lines are reported too.
How crawlers read robots.txt
robots.txt rules are grouped by User-agent. A crawler follows only the most specific group that names it, so Googlebot-Image uses its own group and ignores the * group entirely, which surprises many site owners. Within the chosen group, the longest matching path wins. Disallow: /admin/ is beaten by Allow: /admin/public/ for /admin/public/logo.png, and when an Allow and a Disallow match with equal length, Allow wins. * matches any sequence of characters, and $ anchors the end of the URL. Disallow: /*.pdf$ blocks /report.pdf but not /report.pdf?v=2, and paths are prefix matches, so Disallow: /search also blocks /searching. These are the rules Google documents and that RFC 9309 standardised. The tester shows the deciding rule and its line number, so you can debug unexpected results before you deploy. It also lets you check AI crawlers such as GPTBot, ClaudeBot and CCBot. Honest limits: robots.txt controls crawling, not indexing. Blocked URLs can still be indexed from links, so use a noindex tag on a crawlable page to remove pages from results. Some bots ignore robots.txt entirely, and Bing and other engines differ slightly in edge cases. The robots.txt generator, the sitemap validator and the meta tag generator are related.
Frequently asked questions
How do I test if a URL is blocked by robots.txt?
Paste the robots.txt here, choose the crawler and enter the URL. The result shows the deciding rule.
Does Allow override Disallow?
The longest matching rule wins. If an Allow and a Disallow are equally long, Allow wins.
Does blocking a page in robots.txt remove it from Google?
No. It stops crawling, not indexing. Use noindex on a crawlable page to remove it.
Is anything uploaded or fetched?
No. What you paste is processed in your browser, and nothing is fetched, stored or sent.