Robots.txt Generator

About this tool

Build a simple robots.txt file for one user-agent. Enter the crawler name (use * for all), paths to disallow and allow separated by commas, an optional sitemap URL and an optional crawl delay. The text updates as you type and is generated in your browser. The tool writes a single rule group, so for different rules per crawler you will need to add more groups by hand. Upload the finished file to the root of your site, for example example.com/robots.txt.

How to use it

1. Set the User-agent (the default * means all crawlers).
2. List the paths to disallow and allow, separated by commas.
3. Add your sitemap URL and, if wanted, a crawl delay.
4. Copy the output and save it as robots.txt in your site root.

Example

User-agent *, Disallow /admin, /private, Sitemap https://example.com/sitemap.xml

User-agent: *
Disallow: /admin
Disallow: /private

Sitemap: https://example.com/sitemap.xml

Common problems

The file blocks nothing

With no disallow or allow paths the output is "Disallow:" with no value, which allows everything. Add the paths you want blocked.

Pages still appear in search results

robots.txt asks crawlers not to fetch pages, but it does not remove already indexed URLs or guarantee compliance. Use a noindex directive or authentication for that.

Crawl-delay has no effect

Not every crawler supports Crawl-delay; Google ignores it.