robots.txt

L

Logan

Perch
I have been looking at my site IIS logs and noticed 500 errors often when looking for a robots.txt file. I did some surfing to understand what this is.

Does anyone else have one of these? Has anyone had any problems with their site with spiders in general?
 
The robots.txt is the file many search engine spiders search for, before they start indexing your site. This robots.txt contains information on folders or files which are not to be indexed by these bots... And if the file doesn't exist, an error is returned to the spider software, which - in most cases - means that the robots.txt-file doesn't exist, and your complete site can be indexed.

Regards,
Niels
 
You must log in or register to reply here.
Back
Top