Mail1 is online now - recovery completed

Stephen

US Operations
Staff member
I am checking mail1 rigght now, I am here at the datacenter and it just shows both HDs in the RAID1 as offline, I am going to restart the server right now and check further.
 
Re: Mail1 is offline

We have recovered the server for the most part, SMTP service is still down at the moment and we are checking further, ther raid IS degraded and we will have to rebuild this tonight.
 
Re: Mail1 is offline

the drive just offilned again, it is going to be a long night here in the datacenter. I am on it right now.
 
Re: Mail1 is offline

the raid volume at this point is offlining under any load, I am working my hardest to bring this back online but at the same time I am preparing a new server for disaster recovery if required.
 
Re: Mail1 is offline

I have paused the alert for mail1, We know it is down, it is my singular task right now.
 
Re: Mail1 is offline

We are preparing a new server and going for a recovery of files from our backups.
 
Re: Mail1 is offline

We have started copy from old data backup. Once done, we will update this with newer backups. This is being done specifically to get the as latest copy of data as possible and without crashing the source of data itself.

ETA here is 5 hours.

We will be trying out best to get it done early.
 
Re: Mail1 is offline

We have the data syncing, Tanmaya is supervising this operation and I am going to relax a moment being done on the hardware/OS side.

The new server is, quad core cloverton Intel Xeon, 4GB FDBIMM RAM, 3ware RAID10 with 250GB SATA2 WD Raid Edition2 drives, 128MB RAID cache memory.
 
Re: Mail1 is offline

I would like to mention, no mails are being lost here.
our anti-spam gateway is still collecting them.
Those who opted not to use these new gateway shouldn't worry too, as mails are just being queued on sender mail-servers and will get delivered soon after the new mail1 is up.
 
Re: Mail1 is offline

We have recovered from the backups, but are attempting to get the most current data from either of the two RAID1 volume members to update that with the most current data available. We have not started mail services at this point. it will likely be 45 minutes to 1.5 hours more.
 
Re: Mail1 is offline

We were done with restore uptil 13th April an hour back, but any attempt to restore the newer data seems not be working now. The drives storing this updated data are failing every minute.
Last resort will be to leave mail server with mails upto 13'th april, but we are trying hard to get updated data.
We apologize for all problems being faced here and would like to assure there is we are still trying with all efforts we can put in.
 
Re: Mail1 is offline

Mail1 is online with data upto April 13th. Any data between April 14th and April 19th is not available and we deeply apologize for the inconvenience caused.
We will still be trying to get data upto April 15th and if any success, it will be posted here.
 
Re: Mail1 is offline

As conlusion of our final efforts in this restore, we have managed to restore data up to April16, 6:27 AM. Rest of the mail data is still unrecoverable.
 
Re: Mail1 is offline

We have managed to recover latest backup from our archives.
Now we have only data that is just 3 hours behind the crash and this is when the drives failed. We were still able to restore 100% pre-crash state for approx 300 domains and here drives died completely. We even tried a lot to recover for the rest of the domains as well, using various techniques and softwares, etc, but it didn't work. Customer may see old mails being downloaded. This step was essential for quick recovery of the service.

Mail1 restore process is considered successful at this point.
In case of any problem, kindly report via a support ticket.

We thank our customers for their patience and cooperation throughout the issue.
 
Back
Top