Built and fact-checked by the DocNectar team — see our editorial standards
Key Features
Instant Calculation
Get accurate results in real time with our optimized algorithm.
Mobile Friendly
Fully responsive design. Works on all devices & screen sizes.
Privacy Focused
Your data stays on your device. We don't store any inputs.
100% Free
No hidden costs. This tool is completely free forever.
A robots.txt file can have overlapping Allow and Disallow rules for the same path, and figuring out which one actually wins by eye is easy to get wrong. This tester checks a specific URL path against a robots.txt file's rules for a given crawler, applying the same longest-match precedence rule real search engines use.
How it works
Paste your robots.txt content, enter the path you want to test, and specify a user-agent (or use "*" for the default group). The tester finds the matching user-agent group, then applies the longest-matching-rule-wins precedence rule (ties favor Allow) to determine the verdict.
- Enter robots.txt content.
- Enter path to test.
- Enter user-agent.
- Click Calculate to see your results.
Examples
An allowed exception inside a disallowed folder
With "Disallow: /admin/" and "Allow: /admin/public/", testing "/admin/public/page" is Allowed — the longer, more specific Allow rule wins over the shorter Disallow rule.
Who should use it
- Verifying that an important page isn't accidentally blocked by an overly broad Disallow rule.
- Testing robots.txt changes before deploying them to a live site.
Industry applications
- SEO and technical website auditing
- Web development and site launches
Advantages
- Applies the real longest-match-wins precedence rule, not just a naive first-match check.
- Falls back to the wildcard "*" group automatically when no exact user-agent match exists.
Limitations
- Tests the rules as pasted — it does not fetch a live robots.txt file from a URL.
Common mistakes to avoid
- Assuming the first matching rule in the file wins — actual precedence is based on the LONGEST matching path, not the order rules appear in.
- Forgetting that robots.txt rules are case-sensitive and path-prefix based, not full pattern matches, unless wildcards are used.
Best practices
- Test every important path exception (like an allowed subfolder inside a disallowed folder) directly rather than assuming how the rules interact.
Tips
- After generating or editing a robots.txt file with the Robots.txt Generator, test your most important pages here before publishing the file live.