Check a URL against your robots.txt rules
One line in robots.txt can stop Google from crawling a whole site. This tester checks a URL against your rules the way Google documents it: it finds the group for the crawler, then the rule that applies, and tells you whether the URL is allowed. The test runs in your browser, so your rules and URLs never reach our servers.
Runs in your browser. Nothing you paste here is sent anywhere.
How to use it
Paste your rules
Copy the contents of your robots.txt, or the rules you plan to add.
Enter a URL and a crawler
Use a path on the same host, for example /blog/my-article, and a user agent such as Googlebot.
Read the verdict
See whether the URL is allowed or blocked and which rule decided it.
How Google reads the rules
The group
Google's crawlers follow the group with the most specific user agent that matches them and ignore the other groups.
The rule
Within that group, the most specific rule wins, meaning the one with the longest path. When an allow rule and a disallow rule conflict, Google uses the least restrictive one.
Wildcards and case
An asterisk matches any sequence of characters and a dollar sign marks the end of the URL. Paths are case-sensitive.
Other crawlers
This tester follows Google's documentation. Google notes that other crawlers may interpret the same rules differently.
Questions
Does blocking a page in robots.txt remove it from Google?
No. robots.txt controls crawling. A blocked URL can still be indexed, without a description, if other sites link to it. To keep a page out of results, use noindex and leave the page crawlable.
Does example.com/robots.txt apply to blog.example.com?
No. The rules apply only to the host, protocol and port where the file is. Each subdomain needs its own robots.txt.
How fast does Google notice a change?
Google generally caches robots.txt for up to 24 hours. In urgent cases you can request a recrawl of the file in Search Console's robots.txt report.
What happens if robots.txt returns an error?
If the file returns a 4xx error other than 429, Google acts as if there were no robots.txt and crawls without restrictions. If it returns a server error, Google stops crawling the site for the first 12 hours and keeps trying to fetch the file.
All free tools
Find the phrases that make a text sound machine-written
Paste a text and see which typical AI writing patterns it contains, from "not just X, but Y" to one-line closers. Free, based on the open humanizer list.
Tell Google which page is for which language
Enter the language versions of a page and get hreflang link tags that list every version, including the page itself, with valid language and region codes.
Check your title and description against Google's advice
Check a page title and meta description against Google's published advice: descriptive, specific, no keyword stuffing and no boilerplate. Free tool.
Structured data without writing JSON by hand
Fill in a few fields and get JSON-LD structured data to paste into your page, in the format Google recommends, plus what it can and cannot do for you.
Check a title and description before you publish them
Type a title and a meta description and see roughly how your page could look in Google results, including where long text may be cut off. Free tool.
Publishing that keeps the basics in order
Ranketto publishes articles on your website and keeps your sitemap and language versions up to date. It publishes only on domains whose ownership you have confirmed.
Sign-ups aren't open yet. Until then, you can follow ranketto.com's own Search Console numbers.