Monday, February 05, 2007

Robots.txt Generator.

A robots.txt file is a short text file that resides in your home directory. Before search engines spider your site, they look into this file to see which files/file types and/or directories they are not allowed to see.

This tool will make your robots.txt file. As a little 'extra' it also includes a block to many unwanted spiders that only crawl your site in order to collect the email addresses stored on your pages.(SPAM-bots).

[ Robots.txt Generator ]