Robots.txt Generator
Create robots.txt files for your website
Select Bots
Crawl Rules
Generated robots.txt
Frequently Asked Questions
What is robots.txt?
Tells search crawlers which pages to index and which to skip. Placed at site root.
Does it block from Google?
Prevents crawling but not indexing. Use noindex meta tag to truly block indexing.
What to block?
Admin pages, duplicate content, private directories, API endpoints, and search result pages.
Can robots.txt prevent my pages from appearing in Google search?
Robots.txt prevents crawling but not indexing. If other sites link to your blocked pages, Google may still index the URLs without content. Use the noindex meta tag to fully prevent indexing.
Where should I place the robots.txt file on my website?
Robots.txt must be placed in the root directory of your domain (e.g., example.com/robots.txt). Search engines only check this exact location. Subdirectory placement is ignored.
About robots.txt
The robots.txt file tells search engine crawlers which URLs they can access on your site. It's placed at the root of your website (e.g., example.com/robots.txt).
User-agent: * applies rules to all bots.
Disallow: /path/ blocks access to a path.
Allow: /path/ explicitly allows access (useful after a broader Disallow).
Sitemap: tells bots where to find your XML sitemap.
Common Patterns
Disallow: / - Block all pages
Disallow: (empty) - Allow all pages
Disallow: /*.pdf$ - Block all PDF files
Disallow: /private/ - Block specific folder