Free Tool

robots.txt Generator & Tester

Build crawl rules, check the file you already have, and test whether any URL is actually blocked — before Google finds out the hard way.

Start from a preset

Sitemap URL

The single most useful line in the file. Don't skip it.

Rules

Your robots.txt

Goes at the root of your domain

Domain to check

We fetch /robots.txt from that domain. 25 checks per hour.

robots.txt is one of several ways a page quietly disappears from search. We find the rest.