Does this fetch my live robots.txt file?
No. Paste the content you want to test; all evaluation stays in your browser.
Developer
robots.txt Tester
Check a crawler and URL path against pasted robots.txt rules without sending site configuration anywhere.
Uses specific user-agent groups and longest matching Allow or Disallow rule precedence.
✓ Tool inputs are processed in your browser✓ No account
Test crawler access against robots.txt rules locally.
Test whether robots.txt blocks a URL
Paste a robots.txt file, crawler name and URL path to check whether the rules allow crawling before you publish a change or investigate a blocked page.
- Test whether Googlebot can crawl a URL path.
- Check an Allow exception inside a blocked folder.
- Review wildcard, user-agent and sitemap directives locally.
Related tools
Related tools
FAQ
Questions
Questions
How are conflicting rules resolved?
The most specific matching path wins, with Allow winning when matching rules have equal length.
Does this guarantee search engine indexing?
No. robots.txt controls crawling, while indexing also depends on page content, status, canonicals and crawler decisions.