We will be turning off winvps33 for a quick hardware maintenance between 1am and 2am EDT tonight, maintenance will last about 15 minutes.
We need to do some maintenance on the RAID Array on this machine and add a new hard drive as well.
Tonight will be hardware maintenance on several machines, so the window is a bit longer as it may need a little flex on exactly when it starts.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
This will be starting about 5-10 min till 2am.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
Starting maintenance now.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
just started up the server after maintenance now.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
Last VPS is starting, checking on other status.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
Well the VPSes are up and going which is a good thing, but the new HD is not seen.
We will now have to have another maintenance tomorrow at 9:30PM EDT. We suspect the power feed is bad on this hard drive as there is no way two have failed the new one being brand new.
I have edited the original topic to quickly reflect the new status, but keeping the old as listed for continuity. As before it will be about 15 minutes down.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
Bumping this tread to the top after other maintenance is done.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
We will be starting this maintenance in about 5-8 minutes time.
There was heavier than expected traffic on the way in tonight.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
server is up after maintenance work done, checking on status now and VPSes launching.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
ok with the change out of both the power and the data cable, the drive is working properly and being utilized now to rebuild the RAID array, we are keeping a watch on the rebuild at this time. WinVPS26 and 34 yesterday took about 2 hours to rebuild fully, we expect the same here.
IO may be a bit less during the rebuild, but not a significant impact.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
the raid rebuild was progressing along well, but seems to have hung, and the server is hanging now as well. I am rebooting it.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
The server is back up and raid rebuild resumed from where it was, however the error was caused by an error reading the source port, so we are watching it very closely.
We have backups of the VPSes if required.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
it seems possible that there are so many bad sectors on the source disk, that it will not be possible to rebuild the RAID.
I am going to let it hang about 5 minutes and see if it can force through a copy of these bad sectors. Otherwise we will prep to move with backups, or remove the drive that is syncing to and do a live migration to a new node.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
We have it up on one HD now, and will be migrating to WinVPS43 node for all clients on it. This one HD mode is not perfect and shown some issues in the last days, but it is better than hanging in the RAID rebuild so often.
Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe
The server is running more stable and the first VPS is migrating, we will update later once more done.
Moves mostly going well, last one to go, one failed and will work on it. Once last done.
All VPs are migrated successfully.
The servers stayed up from the time we removed the 2nd hard drive with the rebuild making it hang, so we did not have to do any additional reboots and the only additional downtime during migrations were the few pings down for new hardware addresses to be seen by switching.