Robots.txt Generator

A robots.txt file is a plain text file that lives at the root of your domain and tells search engine bots which pages or folders they can or cannot crawl. It helps you protect private areas, save crawl budget on unimportant pages, and guide bots toward your most valuable content.

No. Disallow only stops search engines from crawling the page content, but if other sites link to that page, Google can still index the URL without a description. To fully remove a page from search results, use a noindex meta tag or HTTP header instead of, or alongside, robots.txt.

Upload the file to the root directory of your domain so it is reachable at yourdomain.com/robots.txt. If you're on WordPress, most SEO plugins like Yoast or Rank Math let you edit it directly from your dashboard without needing FTP access.

This is a personal choice. Blocking AI crawlers stops your content from being used to train some AI models, but it may also reduce your visibility inside AI-powered search summaries. Our tool includes a one-click preset to block the most common AI bots if you decide that's the right approach for your site.

Googlebot ignores the crawl-delay directive entirely and instead manages its own crawl rate through Google Search Console settings. However, Bing, Yandex, and several other smaller crawlers do respect it, so it's still useful to include if you want to reduce server load from those bots.

Yes, you can add as many Sitemap: lines as needed, which is common for large websites that split their sitemap into separate files for posts, pages, images, and products. Each sitemap line should contain the full absolute URL, not a relative path.