Network Card Issue

[LegalAlien]

Win5 went down at 2:11am according to my monitoring service. I waited 10 minutes and submitted a ticket at 2.22am.

1. Why did it take a whole hour to acknowledge there was a problem?
2. Why did it take over 2 and a half hours to fix it?

Thanks for fixing it, but this is another major outage in my book and I'm not very happy. I guess I should be thankful it happened in the dead on night.

 

[Yash]

The network card failed at 2:15AM
Issue identified by 2:35AM
We tried to use IP recreator of HSphere IPs. Our efforts were abandoned at 2:55AM
Shared IP assigned by 3:00AM. 90% of sites came back after this.
Took another 5 to 10 to reassign dedicated IPs and the remaining 10% sites were up

Your ticket:
Created: Dec 11, 2004 2:28:25 AM
Last Mod: Dec 11, 2004 3:16:40 AM
Subject: Win5 Down

Your ticket was updated at 3:16AM with the issue stated as "resolved". You created your ticket less than 1 hour before that time.

Win5 was definately not down for 2 hours and 30 minutes. We got everything up as fast as humanly possible. This was a hardware failure and its the first time we have ever seen a network card failure

 

[LegalAlien]

I assume it was because it took you longer to reassign the dedicated IP's as per Atuls thread. Correction, my website was down from 2.11 - 4.15am

 

[Yash]

Actually, I am sorry. Had wrong info with me.
There were about 100 to 120 dedicated IPs to be reassigned. That took an additional 45 to 60 minutes.

It's very unfortunate that this happened very close to a major Win5 outage. But we did everything as fast as humanly possible

We use 1gbps Intel network cards. These are the best quality network cards you'd find with any host

 

[SubSpace]

Seems to me you need to complain loudly to PSoft to get their IP recreator to actually work. That should shave off an hour of recovery time should it ever occur again..

 

[Yash]

no other control panel actually has an IP resource recreator

But yes, we will be figuring out why this didn't work with PSOFT on a test server so we can avoid something like this in the future

 

[Logan]

no other control panel actually has an IP resource recreator

But yes, we will be figuring out why this didn't work with PSOFT on a test server so we can avoid something like this in the future

Yet another problem with HSphere and PSoft that makes me worry about your dependance on them.

 

[Yash]

Yet another problem with HSphere and PSoft that makes me worry about your dependance on them.

The issue was not caused by HSphere/PSOFT. A tool that could reduce the outage time was not functioning. However such a tool is not available with other win cps such as Helm.

The issue was caused by an unexpected network card failure. I believe your sites were on a shared IP so they were not affected by the extended downtime dedicated IP customers faced.