Discussion in 'Network Outages and Updates' started by Stephen, Jan 18, 2017.
We're checking as there is a lot of packetloss coming right now.
We're seeing a large attack coming in on one IP, and checking this now.
The Win29 cluster1 shared IP is being hit, we're working on blocking it, it's taking a bit longer than expected, working to get done ASAP.
Win29 is the only thing impacted right now, we got it blocked from the rest.
Win29 is back up now for sites NOT on the shared IP address, we're still working to block the main attack.
ok nevermind on that, going to have to take it back down because it's killing the network again.
attacks are even worse now, I'm headed to the datacenter to unplug the node with win29 and work on getting it going because null routes are working to block it properly.
We're having the win29 IP blocked at upstream networks now due to the huge nature of them and it causing others to lose so many packets.
And down again, even with the win29 serve entirely down. We're still working on making sure the upstream networks have null routes in place.
I'm very sorry this is taking so long, I've made multiple phone calls and they are not understanding the words 'URGENT' and taking forever to put blocks in place.
ok, now again all is up but Win29, we are still awaiting confirmation that this is due to a null route in place, and then we will post further updates. I don't want any more false hope.
We're continuing to monitor the situation, at this time Win29 based sites will not be live, we are going to attempt to bring up the dedicated IP sites ASAP but taking in small steps here so that it does not return with more attacks.
We had the null route removed as it had not been hit since 9am, but now it is back after the null route, we're having it blocked back again.
We have lifted the null route and will be enabling the shared IP very soon.
The null route was put back in place and in fact extended all the way to further upstream peers because it is very huge in amount of data being pumped in, even after being down for over 72 hours. We will be moving accounts to other servers and dedicated IP addresses to work around this. As this was not an HTTP based attack, we're really not sure its target domain, so we are on guard for any return of attacks.
We've been actively making solutions for clients on Win29 shared IP to move their sites over to either dedicated IPs, or a single dediated IP that we manually move their DNS over to for their domains so that this doesn't keep their site down any longer.
Separate names with a comma.