Discussion in 'Network Outages and Updates' started by Stephen, Dec 20, 2007.
We are seeing quite a bit of packetloss and checking it out now.
Traffic coming in via Cogent appears to be fine, however there is certainly an issue with traffic coming in over Internap.
We are experiencing about 50-80% packetloss on internap uplink and working to get this resolved ASAP.
We have an inbound DDoS with over 800,000 packet per second and 500 megabit per second steady that is causing this.
doing a bit better still there are some slowdowns however.
the DoS died for about 4 min and then came back even worse!
Over a gigabit per second at this time
this attack is so major we are having to work with upstreams to block well before our network.
Win14 shared IP is the destination of this DDOS attack and is being null routed now.
Carriers are actually taking steps to do this because the attack is very heavy. There is still major packetloss now over 70% if you can get a page to load it probably will not load in full.
There is still a major problem in this, and the null route is taking longer than expected the traffic levels have been sustained at well over 1 million packets per second and above 1.5 at times.
What a night the null route is in effect but it seems the network has gone down on the fiber uplink for us. We are working to resolve ASAP.
I have been on the phone with the fiber uplink providers for the last 15 minutes and they said there is a switch having a problem that is causing this issue, we are waiting IMPATIENTLY to get this resolved at the earliest.
Fiber team is trying to locate the issue with their Fiber switch. This may well be an aftereffect of the DDOS issue, however we won't know unless they are done.
It is going to be at least another 20 minutes as they say there is no link, but we show a link and the operations manager is coming in to check at the MDF where the cross connects are all linked.
I am very sorry about this situation, we run on a redundant link system and something happened during the DDoS that has gone haywire.
We are up now. Details will be furnished shortly
The issue with the fiber link has been isolated and corrected. This whole incident is going to be analyzed in detail.
We will be evaluating our options for more redundancy to avoid getting into link problems like this.
As for the original DDOS issue, the target server was Win14. Shared IP for this server stays null routed at this moment. Further updates about Win14 will be posted on this thread:
A mass-mail will soon be sent to all our customers as well, with the incident details as posted here.
Our team sincerely apologize for the issue and we would like to thank all our customers for their cooperation.
Separate names with a comma.