What this tool does
Robots.txt Tester checks whether a path appears allowed or blocked by pasted robots.txt rules.
← All Tools
🤖 Robots.txt Tester
Paste robots.txt rules and test whether a URL path is allowed or blocked for a crawler user-agent.
Sensitive path note
- Robots.txt paths can reveal private admin routes, staging paths, or internal naming conventions.
- Do not paste rules that expose sensitive paths unless you are comfortable seeing them in this browser session.
Why it is useful
It helps debug crawler access before changing production robots.txt files or submitting pages for indexing.
How it works
The tool evaluates pasted robots.txt directives against a selected user agent and URL path using practical matching rules.
Best input
Paste the exact robots.txt content and test specific paths such as /tool/json-formatter/.
Crawler tip
Test important bots separately because user-agent groups can have different rules.
Privacy note
Pasted robots.txt rules are tested locally in your browser.
Important limitation
Robots.txt controls crawling, not indexing guarantees or access control. Do not use it to protect private content.
Quick answers
Can robots.txt hide private pages?
No. Use authentication for private content.
Why do crawlers disagree?
Different crawlers may interpret edge cases differently, and cached robots.txt can persist temporarily.
Related Tools
Common Use Cases
Test if a URL is blocked by robots.txt
Quickly check whether search engines are allowed to crawl specific pages on your site.
Debug indexing issues caused by robots.txt
Verify if missing pages in search results are due to robots.txt rules.
Validate robots.txt changes before deploying
Test new rules to ensure they don't accidentally block important pages.