Packetloss problem

Discussion in 'Network Outages and Updates' started by Stephen, Dec 20, 2007.

  1. Stephen

    Stephen US Operations Staff Member

    We are seeing quite a bit of packetloss and checking it out now.
  2. tanmaya

    tanmaya APAC Operations Staff Member

    Traffic coming in via Cogent appears to be fine, however there is certainly an issue with traffic coming in over Internap.
  3. tanmaya

    tanmaya APAC Operations Staff Member

    We are experiencing about 50-80% packetloss on internap uplink and working to get this resolved ASAP.
  4. Stephen

    Stephen US Operations Staff Member

    We have an inbound DDoS with over 800,000 packet per second and 500 megabit per second steady that is causing this.
  5. Stephen

    Stephen US Operations Staff Member

    doing a bit better still there are some slowdowns however.
  6. Stephen

    Stephen US Operations Staff Member

    the DoS died for about 4 min and then came back even worse!
  7. tanmaya

    tanmaya APAC Operations Staff Member

    Over a gigabit per second at this time
  8. Stephen

    Stephen US Operations Staff Member

    this attack is so major we are having to work with upstreams to block well before our network.
  9. tanmaya

    tanmaya APAC Operations Staff Member

    Win14 shared IP is the destination of this DDOS attack and is being null routed now.
  10. Stephen

    Stephen US Operations Staff Member

    Carriers are actually taking steps to do this because the attack is very heavy. There is still major packetloss now over 70% if you can get a page to load it probably will not load in full.
  11. Stephen

    Stephen US Operations Staff Member

    There is still a major problem in this, and the null route is taking longer than expected the traffic levels have been sustained at well over 1 million packets per second and above 1.5 at times.
  12. Stephen

    Stephen US Operations Staff Member

    What a night :( the null route is in effect but it seems the network has gone down on the fiber uplink for us. We are working to resolve ASAP.
  13. Stephen

    Stephen US Operations Staff Member

    I have been on the phone with the fiber uplink providers for the last 15 minutes and they said there is a switch having a problem that is causing this issue, we are waiting IMPATIENTLY to get this resolved at the earliest.
  14. tanmaya

    tanmaya APAC Operations Staff Member

    Fiber team is trying to locate the issue with their Fiber switch. This may well be an aftereffect of the DDOS issue, however we won't know unless they are done.
  15. Stephen

    Stephen US Operations Staff Member

    It is going to be at least another 20 minutes as they say there is no link, but we show a link and the operations manager is coming in to check at the MDF where the cross connects are all linked.

    I am very sorry about this situation, we run on a redundant link system and something happened during the DDoS that has gone haywire.
  16. tanmaya

    tanmaya APAC Operations Staff Member

    We are up now. Details will be furnished shortly
  17. tanmaya

    tanmaya APAC Operations Staff Member

    The issue with the fiber link has been isolated and corrected. This whole incident is going to be analyzed in detail.
    We will be evaluating our options for more redundancy to avoid getting into link problems like this.
    As for the original DDOS issue, the target server was Win14. Shared IP for this server stays null routed at this moment. Further updates about Win14 will be posted on this thread:
    http://support.jodohost.com/showthread.php?goto=newpost&t=11455

    A mass-mail will soon be sent to all our customers as well, with the incident details as posted here.
    Our team sincerely apologize for the issue and we would like to thank all our customers for their cooperation.

Share This Page

JodoHost - 26,000 hosting end-users in 100 countries
Plesk Web Hosting
VPS Hosting
H-Sphere Web Hosting
Other Services