Robots.txt Tester

How the rules are read

The tester follows the same rules as Google: crawlers use the group that names them most specifically, the longest matching rule wins, and when an Allow and a Disallow rule are the same length, Allow wins. Wildcards * and end markers $ are supported.

When to use it

• Before uploading a new robots.txt.
• When a page is not showing in Google and you suspect it is blocked.
• To check whether AI crawlers can read your content.

Frequently asked questions

Can I test a robots.txt before it is live?

Yes. Switch to “Paste robots.txt” and test your draft rules.

Why is my page blocked when I did not block it?

A broader rule such as Disallow: /blog may match it. The tester highlights the exact rule that applies.

Is the result the same for every crawler?

No. Each crawler follows its own group of rules. Pick the crawler from the list to test it.