Scheduled Power Maintenance Window, April 28th 11pm-5am CDT

Discussion in 'Network Outages and Updates' started by Stephen, Apr 20, 2018.

  1. Stephen

    Stephen US Operations Staff Member

    Scheduled maintenance announcement

    On the evening of Saturday, April 28th, 2018, the datacenter facilities will be performing some major preventative maintenance within our critical power infrastructure at the Dallas data center that houses our VPS, reseller and shared hosting infrastructure, and we may be impacted by this maintenance. The maintenance will be performed on several core components of the power distribution system and switching equipment. This work will directly impact our equipment and some servers and network gear within the facility. As a result, several areas within the data center will be without power for up to 30 minutes, which will mean we need to plan for some servers to be taken down properly (preventing file system errors) and may have an impact up to an hour for some services.

    The maintenance is scheduled to start at approximately 11PM CDT and has a window of 6 hours in length. We will have technical staff on site for immediate updates and hands on access to the servers should there be any unexpected problems or extended errors on restarting servers.

    Regards,
    JodoHost Customer Service
  2. Stephen

    Stephen US Operations Staff Member

    Just a bump reminder, this is happening tonight!
  3. Stephen

    Stephen US Operations Staff Member

    I've got some very bad news, and not a happy customers right now.

    It turns out our A/B circuits aren't A/B circuits, both both the same, and both going down. This is very bad because we intentionally ordered the 2nd circuits after power problems caused some reboots last year with the UPS system transfer switch, to prevent this from happening, and it turns out had there been more problems it would have still gone down.

    On the other hand, we'll be able to do proper shutdowns and prevent major file system rebuilds and checks by doing so, but it will mean a bit longer downtime and for everyone instead of just possibly a few customers. We will keep updates going on the JodoHost facebook page and here (but DNS may be an issue here)

    Thanks for your understanding, and it will all still be completed within the maintenance window as announced, just with a bit earlier start time due to having to issue a shutdown to the nodes before 11PM start time.
  4. Stephen

    Stephen US Operations Staff Member

    We're going to start shutting down nodes in about 7 minutes now. Trying to hold off as long as possible while still being safe!
  5. Stephen

    Stephen US Operations Staff Member

    Whew this did not go as planned at all. Router went a bit haywire after the power was up, keeping the network down as it wasn't establishing the BGP sessions properly. We had to upgrade the firmware there, then the VPS nodes for client VMs dind't want to start, worked past that. And, the nameservers while booted, weren't talking to the outside and answering queries, so we had to restart them and then all came live as it should be.

Share This Page

JodoHost - 26,000 hosting end-users in 100 countries
Plesk Web Hosting
VPS Hosting
H-Sphere Web Hosting
Other Services