Robots.txt Generator
Build a valid robots.txt with disallowed paths, a sitemap reference, an optional crawl delay and optional rules blocking known AI training crawlers.
Runs on your device
Result
Results appear here after you select “Generate”.
What it does
robots.txt tells well behaved crawlers what not to fetch. It is advisory, not security, never rely on it to hide private pages.
How to use it
- List paths to block, one per line (for example /admin/).
- Add your sitemap URL.
- Download robots.txt and upload it to your site root.
Example
The example fills in realistic values and runs the tool. Select Try an example above to see it.
When it helps
- Keeping search engines out of admin and checkout pages.
- Pointing crawlers to your sitemap.
- Opting out of AI training crawlers.
Questions
Does Disallow remove a page from Google?
Not reliably. A blocked page can still be indexed from links. Use a noindex meta tag on a crawlable page to remove it.
Does Google respect Crawl-delay?
No, Googlebot ignores it. Bing and some other crawlers honour it.
More seo tools
See all 10- Google SERP Snippet Preview Preview how a title and meta description may be truncated in Google desktop results, using an approximate pixel width model rather than plain character counts.
- Keyword Density Checker Count how often words and two- or three word phrases appear in your copy and check the density of a target keyword, to catch stuffing or missing topics.
- FAQ Schema Generator Turn questions and answers into valid FAQPage JSON-LD structured data, safely escaped for embedding in a script tag.
- Hreflang Tag Generator Generate hreflang link tags for multilingual or multiregional pages, with validation of language region codes and an optional x-default.
- Heading Structure Checker Paste a page’s HTML source to see its H1 to H6 outline, skipped heading levels, title, meta description, canonical and images missing alt text.
- .htaccess Redirect Generator Generate Apache .htaccess rules for 301 or 302 page redirects, forcing HTTPS and adding or removing www, for cPanel and Hostinger hosting.
People also use
Browse all 200 tools- RC and RL Filter Cutoff Calculator Calculate the −3 dB cutoff frequency and time constant τ of first order RC and RL filters, plus the 5τ settling time. Electrical & engineering
- Roman Numeral Converter Convert numbers to Roman numerals and back, from 1 to 3999, with validation that catches invalid forms like IIII or VX. Quick utilities
- Net Worth Calculator Add up your assets and debts to calculate your net worth, with a breakdown you can update every few months to track progress. Personal finance
- Find and Replace Find and replace text in bulk with options for whole words, case insensitive matching and regular expressions with $1 capture references. Text & writing