WINVPS33 Maintenance start 9:30PM EDT 9/25 - 15 Minutes timeframe - continued

Discussion in 'Virtual Private Server' started by Stephen, Sep 24, 2010.

  1. Stephen

    Stephen US Operations Staff Member

    We will be turning off winvps33 for a quick hardware maintenance between 1am and 2am EDT tonight, maintenance will last about 15 minutes.

    We need to do some maintenance on the RAID Array on this machine and add a new hard drive as well.

    Tonight will be hardware maintenance on several machines, so the window is a bit longer as it may need a little flex on exactly when it starts.
  2. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    This will be starting about 5-10 min till 2am.
  3. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    Starting maintenance now.
  4. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    just started up the server after maintenance now.
  5. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    Last VPS is starting, checking on other status.
  6. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    Well the VPSes are up and going which is a good thing, but the new HD is not seen.

    We will now have to have another maintenance tomorrow at 9:30PM EDT. We suspect the power feed is bad on this hard drive as there is no way two have failed the new one being brand new.

    I have edited the original topic to quickly reflect the new status, but keeping the old as listed for continuity. As before it will be about 15 minutes down.
  7. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    Bumping this tread to the top after other maintenance is done.
  8. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    We will be starting this maintenance in about 5-8 minutes time.

    There was heavier than expected traffic on the way in tonight.
  9. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    server is up after maintenance work done, checking on status now and VPSes launching.
  10. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    ok with the change out of both the power and the data cable, the drive is working properly and being utilized now to rebuild the RAID array, we are keeping a watch on the rebuild at this time. WinVPS26 and 34 yesterday took about 2 hours to rebuild fully, we expect the same here.

    IO may be a bit less during the rebuild, but not a significant impact.
  11. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    the raid rebuild was progressing along well, but seems to have hung, and the server is hanging now as well. I am rebooting it.
  12. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    The server is back up and raid rebuild resumed from where it was, however the error was caused by an error reading the source port, so we are watching it very closely.

    We have backups of the VPSes if required.
  13. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    it seems possible that there are so many bad sectors on the source disk, that it will not be possible to rebuild the RAID.

    I am going to let it hang about 5 minutes and see if it can force through a copy of these bad sectors. Otherwise we will prep to move with backups, or remove the drive that is syncing to and do a live migration to a new node.
  14. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    We have it up on one HD now, and will be migrating to WinVPS43 node for all clients on it. This one HD mode is not perfect and shown some issues in the last days, but it is better than hanging in the RAID rebuild so often.
  15. Stephen

    Stephen US Operations Staff Member

    Re: WINVPS33 Maintenance starting 1am EDT 9/25 - 15 Minutes timeframe

    The server is running more stable and the first VPS is migrating, we will update later once more done.
  16. akshay

    akshay Administrator Staff Member

    Moves mostly going well, last one to go, one failed and will work on it. Once last done.
  17. akshay

    akshay Administrator Staff Member

    All VPs are migrated successfully.
  18. Stephen

    Stephen US Operations Staff Member

    The servers stayed up from the time we removed the 2nd hard drive with the rebuild making it hang, so we did not have to do any additional reboots and the only additional downtime during migrations were the few pings down for new hardware addresses to be seen by switching.

Share This Page

JodoHost - 26,000 hosting end-users in 100 countries
Plesk Web Hosting
VPS Hosting
H-Sphere Web Hosting
Other Services