Free Tool

robots.txt Generator & Tester

Build crawl rules, check the file you already have, and test whether any URL is actually blocked — before Google finds out the hard way.

The single most useful line in the file. Don't skip it.

Your robots.txt

Goes at the root of your domain 


            

              
              
            


            

          

        

      

    


    
    

    
    
  






  

    
Pages missing from Google?

    

      robots.txt is one of several ways a page quietly disappears from search. We find the rest.
    

    Get a free strategy session