robots.txt Generator

A robots.txt generator is a tool that creates a valid robots.txt file from a visual interface, eliminating the need to write the file manually. The robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages or directories they are allowed or not allowed to access. Our robots.txt generator supports all standard directives — User-agent, Allow, Disallow, Crawl-delay, Sitemap, and Host — and includes presets for 14 common bots including Googlebot, Bingbot, and AI training crawlers like GPTBot and Claude-Web. All generation runs locally in your browser with no data sent to any server.

1. 1. Configure User-Agent Groups: Select a bot from the preset list (Googlebot, Bingbot, GPTBot, etc.) or enter a custom user-agent. Add multiple groups to set different rules for different crawlers.
2. 2. Add Allow and Disallow Rules: Add Allow and Disallow rules for each user-agent group. Enter the path to allow or block (e.g. /admin/). Add a Crawl-delay value to throttle aggressive crawlers.
3. 3. Generate, Copy, and Deploy: Click Generate robots.txt to produce the formatted file. Copy it to your clipboard or download it directly. Upload the file to the root of your website at yourdomain.com/robots.txt.

• User-agent: Specifies which crawler the following rules apply to. Use * (wildcard) to target all bots, or specify a named bot like Googlebot. Each user-agent block applies until the next User-agent directive.
• Disallow: Tells the specified crawler not to access the given path. An empty Disallow value means the crawler can access all pages. Use /admin/ to block a directory, or /page.html to block a specific page.
• Allow: Explicitly permits access to a path that would otherwise be blocked by a Disallow rule. Allow takes precedence over Disallow when both match the same path (longest-match rule applies).
• Crawl-delay: Requests that the crawler wait the specified number of seconds between requests. Not supported by Googlebot (use Google Search Console instead), but respected by Bingbot, Yandex, and others.
• Sitemap: Specifies the full URL of your XML sitemap. This helps search engines discover all your pages. You can include multiple Sitemap directives. The Sitemap directive is not user-agent specific.
• Host: Specifies the preferred domain for your website. Only supported by Yandex — not recognized by Google or Bing. Use it if you have multiple domain variants and want Yandex to prefer one.

robots.txt is a request, not a technical barrier — well-behaved crawlers respect it, but malicious bots may ignore it. Disallowing a page in robots.txt does not remove it from search results if other pages link to it; use a noindex meta tag for that. Google does not support Crawl-delay — use Google Search Console to set crawl rate instead. Always validate your robots.txt after generating it using our Robots.txt Validator tool to catch syntax errors before deploying.