Where is the robots.txt file?

jag5311 · October 21, 2003, 4:18pm

I am just curious, because I wanted to see if googlebot is hitting my site.

Thanks
Bryan

Yash · October 21, 2003, 4:23pm

You are suppose to create that file. It can be used to set rules for the bots I believe

jag5311 · October 21, 2003, 4:47pm

What folder do I put it in if I create it?

Where the database is? in the webalizer folder? where all my files are?

Thanks

Yash · October 21, 2003, 5:53pm

It should be put in your domain directory.

Which database are you referring to? Databases should not be in the domain folder or any sub folders within it. Create a separate DB folder in the root

StPatrick · October 22, 2003, 8:33am

Hmm, actually you should enable either raw logs, or use Webalizer to see if search engine hits your site.
I’m not quite sure, but as I remember right, robots.txt is used to tell to search bot what to index and what not. At least this is what I’m using it for