I have a customer on linux server at Jodo.
He studies his logs religiously and is seeing all kinds of bots crawling his site and wants to stop them.
Here are a couple of snippets from log file he sent:
Here is what he says is his current .htaccess file:
He is saying it is not working and bots are still coming through. He would like to know how to stop them. Is he missing something? Not sure why he wants to stop Bing as well, but he does.
Thanks,
Greg
He studies his logs religiously and is seeing all kinds of bots crawling his site and wants to stop them.
Here are a couple of snippets from log file he sent:
HTML:
124.115.6.14 - - [17/Mar/2013:02:44:56 -0500] "GET /robots.txt HTTP/1.1" 200 115 "-" "Mozilla/5.0(compatible; Sosospider/2.0;
http://help.soso.com/webspider.htm)"199.21.99.80 - - [17/Mar/2013:02:45:37 -0500] "GET /images_iob/reserve_it1.gif HTTP/1.1"
304 - "-" "Mozilla/5.0 (compatible; YandexImages/3.0; http://yandex.com/bots)"108.59.8.80 - - [17/Mar/2013:03:16:57 -0500] "GET /robots.txt HTTP/1.0" 200 115 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php? )"208.115.111.71 - - [17/Mar/2013:04:12:59 -0500] "GET /robots.txt HTTP/1.1" 200 115 "-"
"Mozilla/5.0 (compatible; Ezooms/1.0; [email protected])"2.Here is what he says is his current .htaccess file:
HTML:
order deny,allow
deny from all
SetEnvIfNoCase User-Agent "^bingbot/2.0" bad_bot
SetEnvIfNoCase User-Agent "^ezooms/1.0" bad_bot
SetEnvIfNoCase User-Agent "^Ezooms/1.0" bad_bot
SetEnvIfNoCase User-Agent "^Mail.RU_Bot/2.0" bad_bot
SetEnvIfNoCase User-Agent "^MJ12bot" bad_bot
SetEnvIfNoCase User-Agent "^MJ12bot/v1.4.3" bad_bot
SetEnvIfNoCase User-Agent "^Sosospider" bad_bot
SetEnvIfNoCase User-Agent "^Sosospider/2.0" bad_bot
SetEnvIfNoCase User-Agent "^YandexBot" bad_bot
SetEnvIfNoCase User-Agent "^Yandex/1\.01\.001" bad_bot
SetEnvIfNoCase User-Agent "^YandexBot/3\.0" bad_bot
SetEnvIfNoCase User-Agent "^YandexBot/3.0" bad_bot
order allow,deny
allow from all
deny from env=bad_bot
deny from 93.63.195.11
deny from 91.232.96.40He is saying it is not working and bots are still coming through. He would like to know how to stop them. Is he missing something? Not sure why he wants to stop Bing as well, but he does.
Thanks,
Greg
