ToolsRobots.txt Generator

Robots.txt Generator

Generate a valid robots.txt file with common rules, custom paths, and sitemap URL in seconds.

Crawl Rules

Allow all robots

Default — all bots can crawl everything.

Block all robots

Prevent all bots from crawling any page.

Block Googlebot

Block only Googlebot from crawling.

Allow only Googlebot

Block all bots except Googlebot.

Block common admin paths

Block /admin/, /wp-admin/, /private/.

Path

Action

User-agent

User-agent: *
Allow: /

Select the crawl rules that apply to your site.
Add any custom path-level allow/disallow rules.
Optionally enter your XML sitemap URL.
Copy the generated file and save it as robots.txt in your website's root directory.
Test it using Google Search Console's robots.txt tester.

Place robots.txt in the root of your domain.

robots.txt is a suggestion — not a block. Use noindex for sensitive pages.

Never block CSS/JS needed to render your pages.

Always include your sitemap URL.

Test changes with Google Search Console before deploying.