Network Maintaince - 12 AM Tonight

Y

Yash

Bass
Dated: December 08, 2006

Maintenance Window - 12:01 AM until 12:31 AM EST, Saturday Dec 09

Hardware: Core Router

Reason: We will be making some routing changes that are purely software. There will be a momentary loss of connectivity as the routes reload after the configuration change.

There should be no more than 5 minutes of outage in this change, but we are allocating a 30 minute window if there is any problem in the process.

Last week, we attempted the same Router maintenance and it resulted in issues with broadcasting with our upstream provider. We have replanned this maintenance from scratch, and have co-ordinated with all parties involved. We expect a flawless maintenance procedure. This is necessary to maintain our network and prevent possible issues with our router in the future
 
This maintenance will be happening shortly.

We will keep www.jodopulse.com updated with the latest news if there is any change in the 30 minute window or any other news to note.
 
Change has not been made to this point. The are reviewing all the scripts again before running them.
 
They have said they are not quite ready to run it now, it will be a little longer than anticipated before the start of the maintenance.
 
Still there has been no maintenance, they are making some last minute phone calls.
 
Thee are still some issues with subnets and we have been asking for updates the last 30 minutes with few to any. We are not happy at all about this situation.
 
Our routing is at a total mess from InterNAP. InterNAP engineers have been putting us on hold for the last hour. We are doing everything at this point to get them to work on our problem
 
we got a hold of an internap engineer. We are working on this.

All our mail servers are up. However, a large number of web servers are down
 
There is a massive looping issue of traffic between our router and internap causing major packet loss and loads of issues.
Their engineers are finally working on it :(
 
The issue has been pretty much resolved. We are back to full-speed. I am speechless and exhausted. This has been a nightmare with InterNap and our providers. We will not attempt this again.

We are going to be following through this very aggressively tomorrow, and laying blame where it deserves.

Never in JodoHost history (4 years) have we had a network outage this long. We deeply apologise for it.
 
this email was sent out to clients:

During ordinary router maintenance that was scheduled to be performed, we were to co-ordinate with our upstream provider InterNAP for routing and broadcasting changes. The maintenance was to be routine/simple and flawless. However, after performing the maintenance, our network went into a state where packets were looping back and forth and there was major packet loss between InterNAP and our router. When we began contacting InterNAP for support, and resolution to this issue, they put us on hold telling us their top engineers were busy or not available. We in fact, remained in this position for almost an hour till threats of suing them got them to page their techs and finally get someone to work on our case a short while later.

As a result, we faced one of the worst network outages in our 4-year history. Our network was totally down for about 20 minutes, and partially down for about 2 hours. During this 2 hour period, our email servers functioned normally. We?ve learnt a lot from this disaster. We do deeply apologies for this issue. We plan to take this issue up with InterNAP?s top management to prevent further cases like this in the future. InterNAP is considered a premium tier-1 provider with some of the highest rates for bandwidth compared to our providers. For the most of the 2 years we have been using them, we?ve had no such issues and they have delivered excellent uptime to us.

We once again apologise for this issue
Click to expand...
 
You must log in or register to reply here.
Back
Top