Drupal Traffic

microweb · September 17, 2012, 3:06pm

I have a website that we recently converted from a CMS system that was written by their previous web company to Drupal. Since we transitioned their site to Drupal, their web traffic has gone crazy. The reports show that they have had over 7GB of http traffic since 9/1/2012. This does not match the traffic that the site should be getting. It is a small restaurant chain and wasn’t getting anythings near that before switching to Drupal.

I have noticed that some of our other drupal sites have seen a big increase in traffic also. Is this something that is normal. Or do you think that the site has been hacked and there are scriptse running that are causing this.

I am not very familiar with Drupal and don’t even know where to start looking for what could be causing this.

Bunchadogs · September 17, 2012, 4:25pm

After we converted a site to Joomla we were getting a HUGE amount of traffic (3-4 GB) from the search engines (way more than normal visitors) - apparently re-indexing. It went on for a couple of months before it finally fell back in line with normal traffic. I finally discovered this in the stats logs.
I’m not sure if there is a better/easier way to do this, but here’s what I did:
Under AWStats - Robots/Spiders

Here’s an example:

38 different robots* Hits Bandwidth Last visit Python-urllib 4867+3 24.71 MB 16 Sep 2012 - 23:53 BaiDuSpider 4195+67 178.64 MB 17 Sep 2012 - 01:38 Googlebot 3456+42 112.56 MB 17 Sep 2012 - 01:55

I was also worried about some type of hack, and this helped alleviate my fears since the robot traffic matched my increase pretty closely.

Stephen · September 17, 2012, 4:41pm

wow 100+MB a night is pretty intense for just bots traffic.

microweb · September 17, 2012, 4:47pm

I just tried to access awstats and webalizer and got a 403 error. Stephen, can you make sure that the site is set correctly for monitoring stats? www.thetinfish.net

microweb · September 17, 2012, 4:54pm

never mind… I got access to the reports.

microweb · September 17, 2012, 5:05pm

It does seem to be a lot of bot traffic: When can I expect it to slow down. It’s been about 3 months of over 1GB of monthly traffic.

[FONT=verdana][CENTER][SIZE=13px]Robots/Spiders visitors (Top 10) - [SIZE=11px]Full list[/SIZE] - [SIZE=11px]Last visit[/SIZE][/SIZE][/CENTER] [CENTER][SIZE=13px] [/SIZE][/CENTER]
[CENTER][SIZE=11px]
[CENTER][SIZE=11px]37 different robots*[/SIZE][/CENTER][CENTER][SIZE=11px]Hits[/SIZE][/CENTER][CENTER][SIZE=11px]Bandwidth[/SIZE][/CENTER][CENTER][SIZE=11px]Last visit[/SIZE][/CENTER]
[SIZE=11px]Yahoo Slurp[/SIZE] [CENTER][SIZE=11px]6682+3[/SIZE][/CENTER] [CENTER][SIZE=11px]122.60 MB[/SIZE][/CENTER] [CENTER][SIZE=11px]17 Sep 2012 - 00:57[/SIZE][/CENTER]
[SIZE=11px]Unknown robot (identified by ‘bot*’)[/SIZE] [CENTER][SIZE=11px]340+322[/SIZE][/CENTER] [CENTER][SIZE=11px]7.92 MB[/SIZE][/CENTER] [CENTER][SIZE=11px]17 Sep 2012 - 01:46[/SIZE][/CENTER]
[SIZE=11px]Unknown robot (identified by ‘crawl’)[/SIZE] [CENTER][SIZE=11px]337+7[/SIZE][/CENTER] [CENTER][SIZE=11px]5.40 MB[/SIZE][/CENTER] [CENTER][SIZE=11px]16 Sep 2012 - 21:44[/SIZE][/CENTER]
[SIZE=11px]Unknown robot (identified by empty user agent string)[/SIZE] [CENTER][SIZE=11px]243+39[/SIZE][/CENTER] [CENTER][SIZE=11px]8.47 MB[/SIZE][/CENTER] [CENTER][SIZE=11px]17 Sep 2012 - 01:10[/SIZE][/CENTER]
[SIZE=11px]Googlebot[/SIZE] [CENTER][SIZE=11px]171+34[/SIZE][/CENTER] [CENTER][SIZE=11px]4.60 MB[/SIZE][/CENTER] [CENTER][SIZE=11px]17 Sep 2012 - 01:21[/SIZE][/CENTER]
[SIZE=11px]BaiDuSpider[/SIZE] [CENTER][SIZE=11px]112+60[/SIZE][/CENTER] [CENTER][SIZE=11px]1.95 MB[/SIZE][/CENTER] [CENTER][SIZE=11px]17 Sep 2012 - 00:54[/SIZE][/CENTER][/SIZE][/CENTER][/FONT]

Bunchadogs · September 17, 2012, 6:11pm

This one shows 151MB, which does seem to be pretty excessive - is this all from Tin Fish?
My experience was with a site that had several hundred pages and several thousand images in dozens of galleries and lots of image descriptions. I wasn’t too surprised by the search engines spending a lot of time…
However - and I’m not an expert - I wouldn’t think they could spend that much time downloading Tin Fish unless there was a lot of content I didn’t see!

To feed the paranoid side of this - I had quite a few Joomla sites get hacked with all kinds of stuff uploaded, and hundreds of MBs of data stored in the database tables.
You might check PHPMyAdmin and see if any of your tables have suddenly grown - anything more than 1 MB would be suspect. In my Joomla hack I found several tables with 100’s of MB of data (most tables were only 100’s of KB in size, so they were easy to spot).