Stephen, need some help/advice/direction...

Phurion said:
When you say "when this domain is processing it is really slowing down all else..." you're referring to the offending/abusive domain/site and not mine, correct?
Click to expand...
CORRECT, the abusing one, which your site is not in that pool, there are about 6 domains there and at this point we are sorting them a bit to find the offending one, at the moment it has stopped abusing out of the blue...and we are in a holding pattern waiting.

Edit, should have said it sooner, as soon as I posted it came back..so back to work again.
 
Phurion said:
Yep, still not working right. Getting PHP/MySQL time-outs now. Site won't even load.
Click to expand...
We are still working on it, my net at the office has been down for 2.5 hours and I had to give up on it coming back and come to a bookstore with wifi...where I am now checking it myself. However the site is coming up well at this time.
 
Looks like the RAID lost sync this morning, and it is in a rebuild mode, that is one of the key issues behind being slow this time but we are still trying to check into it, and there is in fact one domain that is using far more resources than others, just more noticeable now than normally.
 
Think you will find it much better now, still working on it AND monitoring it, and the rebuild is still going, but it is performing much better.
 
Back in the bit bucket again this morning.

Do I need to request to move this domain to a different server/cluster? This has been an ongoing issue since day one with this particular domain/website. If you guys can't get the abuse under control, I'll need to review other options.
 
Phurion said:
Back in the bit bucket again this morning.

Do I need to request to move this domain to a different server/cluster? This has been an ongoing issue since day one with this particular domain/website. If you guys can't get the abuse under control, I'll need to review other options.
Click to expand...
Sure you can request a move(but not to another cluster), but it is opening fine here, I tried from aletra as well and the longest was 6 seconds and that was in Oz...
 
I've tried from two different locations today, different networks on different backbones, and the site is taking forever to load.

Right now I have it loading in another tab and it's been churning for 2+ minutes already.

What's going on?
 
Phurion said:
I've tried from two different locations today, different networks on different backbones, and the site is taking forever to load.

Right now I have it loading in another tab and it's been churning for 2+ minutes already.

What's going on?
Click to expand...
DDOS filtering has heavy load on the server and PHP is not working well with that. With tweaking of the filters we got asp and asp.net to load well, but PHP is pretty slow, we are aware.

I was using your site as a heavy code php page to test and had it between 8-24 seconds regular, right now it just loaded in 16, but it is very sensitive to the amount of attack still incoming.

The domain being attacked is redirected to another IP, and down on the server, but many of the bots still attacking the old shared IP. If it were a dedicated IP resolution would be as simple as null routing or simply dismounting the IP address on the server, but with shared IP that is not an option except in extreme bandwidth usage cases where we don't have much choice. (i.e a multi gigabit attack that impacts the entire network not just the server/shared ip even)
 
Any update on this?

Nearly 24 hours later and the site is still performing like ass.

This needs to get fixed. This particular domain is about to become more important than it has been in the past. It will be a bands official public facing site, and it needs to perform better than it has been lately.

There seems to be a lot of excuses lately as to why it performs poorly, but never any real solutions or answers.
 
Phurion said:
Any update on this?

There seems to be a lot of excuses lately as to why it performs poorly, but never any real solutions or answers.
Click to expand...

I understand you but his isn't excuses, when you renewed this thread the raid had broken and was an issue, we resolved it and it was good after raid rebuilt.
come Sunday I thought it was this again and prepped a server to migrate ( see status threads for this).
I thought I'd gotten it working with raid rebuilding only to get a mail saying now sites are down randomly. I checked it and there were many thousands of open connections due to ddos.
I did some tweaking and found the needed logs, being generated at over 50MB/min at one point(this is just in errors not even successful hits, there were couple 100MB of those too), this was causing a big overload. I've got the logs down to less than 3MB/min now so it isn't an io overload now, but to do that I made a lot of blocking rules (over 500) and the server overhead is pretty high. I accept that I should have looked at ddos sooner on sunday, but it wasn't generating large packets or traffic like normal that send us alerts, so I hadn't even thought more than some server issue making this current problem.

we have an open offer on forums and in ticket replies to move people to another server setup for moves. That will be the best bet now as the shared ip is still being attacked so I can't lift filter set.

ETA: I was finally taking a lunch break and replied this from phone want to add a few things that were a bit tedious on phone. Main thing is that this is hitting the shared IP, if the site being attacked was on a dedicated IP I would have already dropped it and the server would not even need to filter further.
I have maintained a lot of communication, and little rest for myself in this, right now we have to do this with a software solution. We have have seen an increase in these, and are working on a way we can filter these with some dedicated hardware in the future on an as needed basis, this will help by not taking server resources to filter requests legit/illegit.
 
You must log in or register to reply here.
Back
Top