We are having to do a full recovery on Win2, the RAID rebuild process hung at 32%, and actually corrupted the good HD, and the target HD.
We are recovering Win2 to a new server, it is a Dual Xeon with 2.8GHz processors and 4GB or DDR2 ECC RAM. We are doing this all as quickly as possible and will keep you updated here.
We have the hsphere tools still running now. Will update when we know a bit more later, ETA is approx 5-6 hours.
Physical resource creation 65% done. Website content restoration will follow once this process is completed.
Physical creation of resources done. User content restoration starting now. IIS will be stopped during this time.
Also, for everyone’s information, our backup is only 20 mins old.
Estimated Completiton time: 3 hours from this point
Hopefully much sooner. I understand this has been a long outage but there is a huge amount of data being restored.
We have also decided to put Win2 on much better hardware. It is now on a Dual XEON server. The old server had a total motherboard and hard disk crash.
I am going to have to revise the recovery time to 3 to 5 hours. it very much depends on how fast this data gets written on the disks now. 9AM EST is what we should have things up by
We have just over 10GB restored as of now, we are having to do this in the old file restore way as we had a series of failures, the good thing is that we do plan for the failures and have good backups, it is just that they take a bit longer to bring back to a live state.
Just wanted to send in an update, 15GB restored andover half a million files.
These small files are harder to restore that a couple thousand larger ones.
We are going to have to revise our estimate here once again. We see the entire restoration process being complete by 5 PM EST (GMT - 5). The restoration process is slow and there is absolutely nothing we as administrators can do to speed it up. Since there are so many files, the system can really never restore at the I/O speeds it supports.
We will offer SLA credit to those customers who have been down as a result of this.
While I understand customers are disappointed, I’d like to stress that if any host loses a disk array, they must always do a conventional backup from a compressed source either over network or over tape. Such conventional restores do not yield speeds any faster than what we are getting.
Losing both disks in RAID however is rare and this sort of even is a once in 1 or 1.5 year thing
Letter A restored. Letter L to Z restored. You’ll notice these sites are UP. However FTP is not working and only read permissions are currently granted..
Other letters continue to be restored… Once all letters are restored, a script would be run to give full write permissions
A & H to Z are now restored. Sites on these domains show be up. If they aren’t, there is no need to worry as permissions have still not be set correctly and it would be done after all directories are restored as per the announced time-frame
It has been a while since an update, I am up in the D’s now, (some E’s and one huuuuuuggge account starting with T are still going however)
We are almost complete. You’d notice most sites are running and we should be running the permissions script in 60min
We fixed the issues with CFMX a 5 minutes ago.
Permissions script is now running. The permission script will restore write permissions on your folders. Right now your sites are working in read-only so it couldnt create new files in its directory. Estimated completion is 2 hours.
If your site is not working at all, or you need full asp.net permissions, open a ticket please
All permissions set about 30 minutes ago. Everything from our side in this restore is complete. Please open a ticket if your site isnt working