Paste the draft rules
Use robots.txt text with User-agent groups and Allow / Disallow directives.
Publishing & SEO / Made for the task at hand
Before the crawl.
Follow the rules.
Test draft robots.txt Allow and Disallow rules against several paths. Inspect the winning line locally before checking crawler-specific behavior on your live site.
Your pasted robots.txt rule tester workspace
02 / SOURCE
03 / RESULT
Ready when you arePath decision report
A little less work starts here.
Add your source, or try the example, then choose “Test these paths”.
Results are temporary. Download anything you want to keep.
Your original data is never changed.
◇ Local processing↧ Real, downloadable results⌁ No account required
A practical little workflow
From “I need to…” to done.
Choose a crawler and paths
Enter a crawler product token and several paths. Matching groups are combined; longer rules win and Allow wins ties.
Inspect the winning line
Export decisions with matching rule lines. Confirm the real crawler’s behavior and your live HTTP response separately.
Before you get started
Useful answers.
Clear boundaries.
Built for a focused job.
Review important results before use.
Is this a complete robots standard validator?
No. It simulates common path patterns, * wildcards and terminal $. Percent-encoding equivalence, unusual user-agent behavior, HTTP handling and crawler caches are not modeled.
Does Allowed mean Google will index a page?
No. Crawl permission and indexing are different. noindex, canonical tags, authentication, HTTP errors and other signals are outside this tool.
Can robots.txt protect private data?
No. A robots file is public guidance to cooperating crawlers, not access control. Use authentication and server permissions for private content.
What happens to my data?
Working text and local files are processed in this browser tab and are not sent to a server or an AI service. Reloading clears this workspace. Optional analytics and feedback have separate privacy controls; see Privacy below.