My Toolyard

robots.txt Generator

Tell search engines what to crawl.

robots.txt
User-agent: *
Disallow: /admin
Disallow: /api
About this tool

Builds a valid robots.txt file telling search engine crawlers what to index.

How to use
  1. 1Pick user agents and disallow paths.
  2. 2Add a sitemap URL.
  3. 3Copy the generated robots.txt.
Good to know

Verify rules with Google Search Console's robots.txt tester before deploying.