Tells search crawlers which pages to index and which to skip. Placed at site root.

Does it block from Google?

Prevents crawling but not indexing. Use noindex meta tag to truly block indexing.

Admin pages, duplicate content, private directories, API endpoints, and search result pages.

Can robots.txt prevent my pages from appearing in Google search?

Robots.txt prevents crawling but not indexing. If other sites link to your blocked pages, Google may still index the URLs without content. Use the noindex meta tag to fully prevent indexing.

Where should I place the robots.txt file on my website?

Robots.txt must be placed in the root directory of your domain (e.g., example.com/robots.txt). Search engines only check this exact location. Subdirectory placement is ignored.

Robots.txt Generator

Create robots.txt files for your website

Select Bots

All Bots (*) Googlebot Bingbot Yandex DuckDuckBot Baiduspider GPTBot (OpenAI) ClaudeBot (Anthropic)

Crawl Rules

Sitemap URL (optional)

Crawl Delay (optional, seconds)

Generated robots.txt

About robots.txt

The robots.txt file tells search engine crawlers which URLs they can access on your site. It's placed at the root of your website (e.g., example.com/robots.txt).

User-agent: * applies rules to all bots.

Disallow: /path/ blocks access to a path.

Allow: /path/ explicitly allows access (useful after a broader Disallow).

Sitemap: tells bots where to find your XML sitemap.

Common Patterns

Disallow: / - Block all pages

Disallow: (empty) - Allow all pages

Disallow: /*.pdf$ - Block all PDF files

Disallow: /private/ - Block specific folder