I am checking mail1 rigght now, I am here at the datacenter and it just shows both HDs in the RAID1 as offline, I am going to restart the server right now and check further.
We have recovered the server for the most part, SMTP service is still down at the moment and we are checking further, ther raid IS degraded and we will have to rebuild this tonight.
the raid volume at this point is offlining under any load, I am working my hardest to bring this back online but at the same time I am preparing a new server for disaster recovery if required.
We have started copy from old data backup. Once done, we will update this with newer backups. This is being done specifically to get the as latest copy of data as possible and without crashing the source of data itself.
I would like to mention, no mails are being lost here.
our anti-spam gateway is still collecting them.
Those who opted not to use these new gateway shouldn't worry too, as mails are just being queued on sender mail-servers and will get delivered soon after the new mail1 is up.
We have recovered from the backups, but are attempting to get the most current data from either of the two RAID1 volume members to update that with the most current data available. We have not started mail services at this point. it will likely be 45 minutes to 1.5 hours more.
We were done with restore uptil 13th April an hour back, but any attempt to restore the newer data seems not be working now. The drives storing this updated data are failing every minute.
Last resort will be to leave mail server with mails upto 13'th april, but we are trying hard to get updated data.
We apologize for all problems being faced here and would like to assure there is we are still trying with all efforts we can put in.
Mail1 is online with data upto April 13th. Any data between April 14th and April 19th is not available and we deeply apologize for the inconvenience caused.
We will still be trying to get data upto April 15th and if any success, it will be posted here.
As conlusion of our final efforts in this restore, we have managed to restore data up to April16, 6:27 AM. Rest of the mail data is still unrecoverable.
We have managed to recover latest backup from our archives.
Now we have only data that is just 3 hours behind the crash and this is when the drives failed. We were still able to restore 100% pre-crash state for approx 300 domains and here drives died completely. We even tried a lot to recover for the rest of the domains as well, using various techniques and softwares, etc, but it didn't work. Customer may see old mails being downloaded. This step was essential for quick recovery of the service.
Mail1 restore process is considered successful at this point.
In case of any problem, kindly report via a support ticket.
We thank our customers for their patience and cooperation throughout the issue.