Tonight we will be doing maintenance on the Mail1, Mail3, and Mail7 servers.
This maintenance will include a reboot, and all of them will require a fsck on restart because they have a long uptime. The maintenance will be staged with Mail1 first, then Mail3, and finally Mail7, we hope not to have any major overlap in the maintenance. The window of maintenance is 10PM CDT tonight (GMT-5) Until 1AM Saturday morning. We expect each server will be down between 15 and 45 minutes (due to the fsck that is required).
Maintenance on Mail1 will be starting shortly.
Mail1 is in fsck now and expected to be for some time, but it is moving along.
Mail1 is up mail3 will be starting in next 15 minutes or so.
mail3 is down now for maintenance.
Mail3 is having to go into extended maintenance, on reboot the server did not come up and we checked there was a blown capacitor near the PCI-express slots that was leaking some gunk. This is obviously the problem and we are doing a motherboard swap now.
During the swap I found another capacitor under the pci-e slot also busted and leaking ‘acid’, so its good we caught this in a maintenance time instead of a random reboot that was needed due to performance we’d have been in a lot worse situation than the planned maintenance when I was here anyway.
The RAID card and drives are all swapped to new hardware and FSCK is running now.
Mail3 is now up and running well.
MAil1 and MAIL3 are DONE. Since the maintenance window is now gone, we will reschedule the MAIL7 maintenance for 11PM CDT Saturday Aug 30th. (tonight)
