I know your hands are busy - but the last updates was 2.5 hrs ago - i need to be able to tell something to the client.
thanks
it is up as you may know by now. the only left is pgsql and it is a bit of a manual process and still being worked. We had a hiccup on the web17 restore that slowed it down over the others in that the backup server went offline, it needed a bit of maintenance and fsck that made it take longer, but overall for such an incident it went pretty well.
yup
thanks for all your efforts
Web 10,11, and 17 (while it lasted - currently 17 is down) all seemed much slower after the repair.
Is there still a lot going on “behind the scenes” that is causing things to seem slower?
The sites I tested were all DB driven - several WordPress, Joomla, but also several sites that are relatively simple PHP /MySQL (not a large CMS). I didn’t have any straight HTML sites to test, so this might only be DB related.
I didn’t check to see if they were all on the the same MySQL servers either - just thought I’d ask if we need to wait for a while longer for things to truly return to normal.
There are SYN floods that seem to be causing some slowness for some CMS sites as it turns out. We are going to be tweaking servers for helping with this overnight tonight. We noticed an uptick in SYN floods last week and have been blocking them with little impact but they have spread out more and we are moving to some more automated ways and tweaks to help them.
Initially on the node once rebuilt there was a sync process going on the RAID arrays that made them slower but it was really in sync before the servers even came live. The hardware it is on is the same so there’s no ‘downgrade’ in parts etc.
Web17 had a complex issue in that people that made changes on the Hsphere control panel after the restore were going down on their sites, as it tried to reconfigure it was taking sites off, of course several tickets came in about this matter. Then we tried several ways to fix it and mid process all sites went off due to the same type of problem. It took some time but the matter was resolved it ended up being a complex permissions issue after the restoration that was the core issue. Made worse because logs weren’t being made fully of what the error was due to the same permissions problem.