Last updated October 13, 2026 · Report an error
robots.txt is the first file most well-behaved crawlers check before indexing a site — it tells them which parts of the site they're allowed to crawl and, often, where to find the sitemap. Getting its syntax exactly right matters, since a small mistake can accidentally block an entire site from search engines or fail to block what you actually intended to keep out of the index.
This tool builds a syntactically correct robots.txt from a simple form: which paths to disallow or explicitly allow, an optional crawl-delay, and a sitemap URL — ready to save as robots.txt in your site's root directory.
How it works
Choose which user-agent the rule applies to (leave as * to target all crawlers), list any paths you want to disallow and, optionally, any sub-paths you want to explicitly allow within a disallowed area. Add a crawl-delay if you need to slow down aggressive crawlers, and a sitemap URL so crawlers can find it directly. The tool builds the correctly formatted robots.txt content.
- Add paths to disallow (and optionally allow within them).
- Optionally set a crawl-delay and sitemap URL.
- Generate and copy the resulting robots.txt content.
Examples
Blocking an admin area while allowing everything else
Disallowing "/admin" for the * user-agent, with a sitemap URL added, produces a robots.txt that keeps crawlers out of the admin area while letting them index the rest of the site and find the sitemap.
Allowing one page within a blocked folder
Disallowing "/private" but explicitly allowing "/private/public-page" lets crawlers index that one specific page while the rest of the folder stays blocked.
Who should use it
- Blocking admin, private or staging areas from being crawled.
- Pointing crawlers to a sitemap for more efficient indexing.
- Setting a crawl-delay for a resource-constrained server.
Industry applications
- SEO and technical SEO
- Web development
- Site launches and migrations
Advantages
- Generates syntactically correct robots.txt content from a simple form.
- Supports allow rules within disallowed paths, not just blanket blocking.
- Includes sitemap and crawl-delay support in the same generator.
Limitations
- Single user-agent rule group per generation — multiple different rule sets for different crawlers need separate generations combined manually.
- Doesn't validate that your disallowed paths actually exist on your site.
Common mistakes to avoid
- Accidentally disallowing the entire site (Disallow: /) when only a specific folder was meant to be blocked.
- Assuming robots.txt guarantees a page is excluded from search results, rather than just from crawling.
- Forgetting to include the sitemap reference, making it harder for crawlers to discover it.
- Placing the file anywhere other than the exact root of the domain, where it won't be found.
Best practices
- Test your robots.txt in Google Search Console's robots.txt tester (or equivalent) after publishing it, to confirm it behaves as intended.
- Use a noindex meta tag, not robots.txt, for pages that must never appear in search results at all.
- Keep the file as simple as possible — overly complex rule sets are more likely to contain mistakes.
Tips
- After publishing, verify the file is reachable at yoursite.com/robots.txt and returns a 200 status, not a 404.
- If you need different rules for different crawlers (like Googlebot vs. Bingbot), generate each rule block separately and combine them in one file.