WinVPS3 - resolution process

Stephen

Stephen

US Operations
Staff member
Well it seems problems come all at the same time. WinVPS3 rebooted during scheduled backups without warning, and it is mostly back up now. Looks like it will be fully up within the next 5 minutes.

I guess that means there won't be a reboot this weekend on Winvps 1/3, since they happened early.
 
Re: WinVPS3 - problem

All VPSes have been up for the last 10 minutes, this has been a rough night.
 
Re: WinVPS3 - problem

WinVPS3 had a hard disk failure that caused the inital problem and I am restarting it again now to try and find a resolution.
 
Re: WinVPS3 - problem

WE have a tech enroute to the datacenter to check on the server now, and make a hardware change.
 
  • Deleted by N/A
Show…
Re: WinVPS3 - problem

We are working on the issue. 2 of the 4 drives on the system appear to have failed. We may have lost the array
 
Re: WinVPS3 - problem

We are going to put 4 fresh drives in this server, reinstall the system and do a restore.
Our off-site restores are unfortunately 10-14 days old for winvps3 due to an issue with its remote backup mechanism. Our other VPS servers rotate off-site backups every few days. We deeply apologise for issue with WinVPS3's backup system, and we do recognise this is a bad mark on our otherwise spotless record of maintaining near-daily backups of all our systems. We do plan to investigate and ensure that in future, our WinVPS systems do not have any break down of the remote-backup mechanism.

Any customers that experience significant data loss can claim compensation under our SLA policy once we have the server restored.

This restore requires significant coordination with SWSOFT due to the propritary nature of their software. At best, this restore shouldn't take more than 10 hours (or 3PM EST). We are hoping to however get the server up much quicker. Updates will be posted here
 
Re: WinVPS3 - problem

ok, we have replaced ALL hard drives that were in winvps3, 2 were dead 1 would not come on and one would come on for less than 2 seconds and then quit.
We have the server prepped but are having to wait on the raid to complete building right now.
 
Re: WinVPS3 - problem

We are making steady progress here now. There will sadly be a few VPSes that we must recreate, and we will notify you of this with the new information and also provide you with a credit for the time and data.

The newly configured and changed hardware WinVPS3 is already configured like winvps1/2 to do all backups remote now as well. So any backup, whether from the power panel, or the backups that we force to be taken on a weekly basis will be off-server backups.
 
Re: WinVPS3 - problem

We have VPSes coming back up and running now. This is by no means all of them, but some are back up, some with older data that others. We are restoring to the most current working backup that we had off-server.

Again this will by default make EVERY backup off server from now on.
 
Re: WinVPS3 - problem

All VPSes that we are able to restore have been restored. We are working to re-create a handfull of VPSes that can not be restored due to either errors in the restore that a force restore is not resolving, or because they are newer than the last off-server backup that was made.
 
Re: WinVPS3 - problem

I now need to issue one last reboot to WinVPS before we can recreate those accounts, this should be the last reboot that we require today.
 
Re: WinVPS3 - problem

We are re-creating VPSes with the same ID's and IPs manually and making the base config the same for users that we were unable to restore. We wll be emailing each of them directly as well as soon as we have completely finished the process. The process of manually creating the ID numbers, IPs, and settings it quite time consuming compared to normal VPS setup times, where we utilize some automation features to assist.
 
We have now created VPSes with the same ID's and IPs manually and making the base config the same for users that we were unable to restore. All are now up and running.

We will be corresponding with clients via email as well.
 
You must log in or register to reply here.
Back
Top