Scheduled Maintenance, Cluster 1 Mail1, Mail3, Mail7 Between 10PM and 1AM, MAIL7 rescheduled - Done

Stephen

Stephen

US Operations
Staff member
Tonight we will be doing maintenance on the Mail1, Mail3, and Mail7 servers.

This maintenance will include a reboot, and all of them will require a fsck on restart because they have a long uptime. The maintenance will be staged with Mail1 first, then Mail3, and finally Mail7, we hope not to have any major overlap in the maintenance. The window of maintenance is 10PM CDT tonight (GMT-5) Until 1AM Saturday morning. We expect each server will be down between 15 and 45 minutes (due to the fsck that is required).
 
Mail3 is having to go into extended maintenance, on reboot the server did not come up and we checked there was a blown capacitor near the PCI-express slots that was leaking some gunk. This is obviously the problem and we are doing a motherboard swap now.
 
During the swap I found another capacitor under the pci-e slot also busted and leaking 'acid', so its good we caught this in a maintenance time instead of a random reboot that was needed due to performance we'd have been in a lot worse situation than the planned maintenance when I was here anyway.

The RAID card and drives are all swapped to new hardware and FSCK is running now.

14892874169_abf0726050.jpg
 
Last edited:
Mail3 is now up and running well.

MAil1 and MAIL3 are DONE. Since the maintenance window is now gone, we will reschedule the MAIL7 maintenance for 11PM CDT Saturday Aug 30th. (tonight)
 
You must log in or register to reply here.
Back
Top