Free robots.txt Generator

A robots.txt file is a text file placed at the root of your website (e.g., https://example.com/robots.txt) that tells search engine crawlers which pages or sections of your site they are allowed or not allowed to crawl. It follows the Robots Exclusion Protocol.

Upload the robots.txt file to the root directory of your website so it is accessible at https://yourdomain.com/robots.txt. It must be at the root — crawlers will not look for it in subdirectories.

No. robots.txt controls crawling, not indexing. A page blocked by robots.txt can still appear in search results if other pages link to it. To prevent indexing, use a noindex meta tag or X-Robots-Tag HTTP header instead.