I know you are all eager to know more, as am I. I am powerless here, I can’t access their cage to help(different companies, I don’t know their telco gear, etc), and I only delay by talking to them there. On the phone they are going to keeping me on hold more than giving me updates.
Multiple onsite carriers are going on the same fiber sets, so it is affecting multiple companies here, they have 6 on the lines right now to the NOC where I am on hold.
On Jan 25th they said they sent a notice of 15 minutes maintenance(they told me this tonight at start of issue), but I have checked and none came then, I am really unhappy. As I am sure all of you are as well.
Just trying to keep you updated as much as possible!
Appreciate your dilligence and updates Stephen.
Its early saturday evening here in Australia and only a few of my customers have contacted me. I’m at least able to give some detailed information to them due to your updates
I have had a very full day emotionally, physically, and now mentally.
Seems some days you just get hit hard.
I wont get into it all but my sister was in a bad wreck and walked away from it safe and sound.
(there is a reason I post this only to customer only as well, I don’t want it public, as it is not scapegoat or complaint by any means, just a fact)
I am off the phone for the moment as basically they have told me nothing can happen in the next 15 minutes while they go get all the needed hardware, then they start work again.
Oh manager just told me why the memory card was fouled up now. It seems it had a bad sector/whatever it is called on flash memory, it was corrupting, and the old backup of the OS and new OS image used all the space, with the bad area, it did not have enough space and corrupted the boot.
they have the config files and all, but the OS image is not loading.
lol, so this is the “state of the art” datacenter we moved to some time back? whoever it is can’t even send an email to its customers.
I’ve never tried to understand “redundant servers” and all the other fantastic dot-points that are highlited about these datacentres by hosting companies…i just guessed it meant we were protected to a large extent anyway from this kind of thing.
cca, redundancy seems to always have an unseen point of fail somewhere in the stream.
We did not go in and fully design every element of it, and this one is sadly out of our hands.
in fact this is the first such in this facility. The last issue we had similar was at the NAP of the Americas side, and we have a solution to go around that if such happened again, but not this crops up and killed that.
They did email so they say, just somehow we got skipped. My mail address where it goes for notices has gotten numerous notices and updates from them in the past, and has no spam filtering at all. it only marks as spam and leave to me, so I am highly confident there was an error in sending. For such a major maintenance I am not sure why they did not mail on Jan 25th, and then later as well however. just to be clear even had we posted a maintenance for them, it would have far exceeded that, but at last the impact may not have been quite as strong due to it having been some level of notice given.
Datacenters are a funny thing, NAP of the americas is considered a premier datacenter in Miami, Miami Exchange was linked directly to it, however if you dig deeper, most of NAP is interconnected to two other datacenters in Miami and not exactly the ‘network access point’ for a good amount of the traffic. I’ve learned a lot over the years here, and actually putting it to use in some future plans.
you are right that redundancy in servers alone, is not much use when there are other points of failure, even the catastrophic levels that can take out most any best plans. there are ways around this with geolocation redundancy however, which is something i have a personal interest in such can be done on a very independent basis if designed right.
I saw Manish’s note that all is back and running. Thank you to all of you who had a sleepless night over this. Give 'em hell for not notifying you. Seems a little ironic that the entire data center relies on one little flash memory card. I hope they at least bought a brand name rather than a Walmart special!
She is doing quite well, and parents are raving about how well Progressive handled the claim and all, they overnighted a check for the car and she already has another (used) car.
BTW MSSQL2 is a situation I have power to do something, keep fully informed, and much more in control, something I can handle!
such can be done on a very independent basis if designed right.