Robots.txt Generator Build a Valid File in Seconds
Tell search engines exactly what to crawl. Pick a preset, block the paths you don't want indexed, add your sitemap and crawl-delay, then copy or download a ready-to-upload robots.txt file.
Configure your rules
Your robots.txt
Upload the file to the root of your domain: https://yourdomain.com/robots.txt
Not sure what to block?
A single wrong line in robots.txt can hide your entire site from Google. Our free technical SEO audit checks your robots.txt, crawlability, indexation and Core Web Vitals — and tells you exactly what to fix.
Getting robots.txt right
The robots.txt file is powerful but unforgiving. A stray Disallow: / left over from a staging site is one of the most common — and most damaging — technical SEO mistakes, silently blocking every page from being crawled.
Use it to keep crawlers out of low-value areas like internal search results, cart and checkout pages, or admin folders, and to point bots to your XML sitemap. Remember that robots.txt controls crawling, not indexing — to remove a page from Google use a noindex tag instead.
After uploading, always test your file in Google Search Console's robots.txt report to confirm your important pages are still crawlable.