Robots.txt Generator

Control how search engine bots crawl and index your website's pages.

1 Default Access Settings

Default (All Robots) Allowed (Recommended) Refused

Crawl-Delay (Seconds) Default (No Delay) 5 Seconds 10 Seconds 20 Seconds 60 Seconds

* Crawl-delay tells bots how long to wait between requests to prevent server overload.

2 Sitemap URL (Optional but recommended)

3 Specific Search Robots

Advanced

Googlebot Same as Default Allow Refuse

Googlebot-Image Same as Default Allow Refuse

Bingbot Same as Default Allow Refuse

Slurp (Yahoo) Same as Default Allow Refuse

Baiduspider Same as Default Allow Refuse

YandexBot Same as Default Allow Refuse

4 Restricted Directories

Paths you want to block robots from indexing (e.g. /admin/, /private/).

Generated robots.txt

Copied!

What is a robots.txt file?

A robots.txt file is a text file webmasters create to instruct web robots (typically search engine bots) how to crawl pages on their website. It tells them which areas of the website should not be processed or scanned.

How to install it:

1. Generate your code using the tool above.
2. Click "Download File" to save it as robots.txt, OR copy the text.
3. Upload the file to the root directory of your website. (e.g., www.yourdomain.com/robots.txt)
4. For Blogger: Go to Settings > Crawlers and indexing > Enable custom robots.txt > Paste the copied code.

Important Warning:

• Be very careful with what you disallow. If you accidentally select "Refused" for Default Access, you will block Google from indexing your entire website!
• Robots.txt is not a security mechanism. Do not use it to hide sensitive data (passwords, private emails), as bad bots can ignore this file.