Check a URL against your robots.txt rules

One line in robots.txt can stop Google from crawling a whole site. This tester checks a URL against your rules the way Google documents it: it finds the group for the crawler, then the rule that applies, and tells you whether the URL is allowed. The test runs in your browser, so your rules and URLs never reach our servers.

The tester needs JavaScript to run in your browser.

The tester follows the rules Google documents for robots.txt. Other crawlers can read the file differently.

Runs in your browser. Nothing you paste here is sent anywhere.

How to use it

  1. Paste your rules

    Copy the contents of your robots.txt, or the rules you plan to add.

  2. Enter a URL and a crawler

    Use a path on the same host, for example /blog/my-article, and a user agent such as Googlebot.

  3. Read the verdict

    See whether the URL is allowed or blocked and which rule decided it.

How Google reads the rules

  • The group

    Google's crawlers follow the group with the most specific user agent that matches them and ignore the other groups.

  • The rule

    Within that group, the most specific rule wins, meaning the one with the longest path. When an allow rule and a disallow rule conflict, Google uses the least restrictive one.

  • Wildcards and case

    An asterisk matches any sequence of characters and a dollar sign marks the end of the URL. Paths are case-sensitive.

  • Other crawlers

    This tester follows Google's documentation. Google notes that other crawlers may interpret the same rules differently.

Questions

Does blocking a page in robots.txt remove it from Google?

No. robots.txt controls crawling. A blocked URL can still be indexed, without a description, if other sites link to it. To keep a page out of results, use noindex and leave the page crawlable.

Does example.com/robots.txt apply to blog.example.com?

No. The rules apply only to the host, protocol and port where the file is. Each subdomain needs its own robots.txt.

How fast does Google notice a change?

Google generally caches robots.txt for up to 24 hours. In urgent cases you can request a recrawl of the file in Search Console's robots.txt report.

What happens if robots.txt returns an error?

If the file returns a 4xx error other than 429, Google acts as if there were no robots.txt and crawls without restrictions. If it returns a server error, Google stops crawling the site for the first 12 hours and keeps trying to fetch the file.

All free tools

Publishing that keeps the basics in order

Ranketto publishes articles on your website and keeps your sitemap and language versions up to date. It publishes only on domains whose ownership you have confirmed.

Sign-ups aren't open yet. Until then, you can follow ranketto.com's own Search Console numbers.