SEO Essential

Robots.txt Generator

Tell Google which pages to crawl and which to ignore. Protect your admin areas and optimize your crawl budget.

Default Access (All Robots)

Allow Everything

Default for most sites.

Disallow Everything

Private development sites.

Crawl Delay (Seconds)

Use only if your server is crashing from bot traffic.

Sitemap

Sitemap URL

WordPress & Security

Disallow /wp-admin/ (Recommended) Disallow /wp-includes/ Disallow Search Pages (/?s=)

robots.txt

User-agent: *
Allow: /

What is a robots.txt file?

A robots.txt file is a text file that resides in the root directory of your website. It gives instructions to web robots (also known as crawlers or spiders) about which pages they are allowed to crawl and index.

Common Directives

User-agent: Specifies which bot the rule applies to (e.g. * for all, Googlebot for Google).
Disallow: Tells the bot NOT to visit a specific URL path.
Allow: Explicitly allows a path (often used to override a parent Disallow).
Sitemap: Points the bot to your XML sitemap for easier discovery.