ISSUE ON MULTIPLE NODE
- Thread starter Shubham
- Start date
Dear Customers,
On Tuesday July 7th, 2020 4:10am CST one of our VPS customers contacted us on LiveChat complaining that their disk was out of space and they were not able RDP.
Upon our investigation we found that the node on which this VPS resides has been hacked with a message saying files are encrypted and to contact the hacker. We found out that files were encrypted with ".waiting ransomware". This type of ransomware software is typically used to extort money from its victims, as only the hackers can decrypt the files.
We immediately powered down the server upon detecting the ransomware
After powering down the server, at 6:56am CST we opened a ticket with our Dallas Datacenter informing them about the hacking. We asked them to check if data on the attached drive was encrypted or not, and if they could decrypt it.
The Datacenter disconnected the node from our network and gave us IPKVM access. They booted the server on a new OS drive with anti-malware software installed.
From this point onward our team worked all night to recover the node from this malware. At 2:03am CT Wednesday we discovered another node affected by the same ransomware. Upon discovery of this, we shut down all our windows nodes and asked our DC to disconnect our net and network cabling to NAS and SAN servers.
Currently we are checking each node to determine what data is useful and recoverable. We hope to come to a determination of how much data has been lost and recoverable in the next 24 hours. At that time we will provide again a detailed report.
Our DNS, Mail, Web and all other services on Linux nodes continue to work as expected.
Please know that we are doing everything we can to recover your data. Our team has slept only a few hours since this hacking. We are a small company trying to survive in the current Covid situation. However our commitment to customers is the same as when we started years ago.
On Tuesday July 7th, 2020 4:10am CST one of our VPS customers contacted us on LiveChat complaining that their disk was out of space and they were not able RDP.
Upon our investigation we found that the node on which this VPS resides has been hacked with a message saying files are encrypted and to contact the hacker. We found out that files were encrypted with ".waiting ransomware". This type of ransomware software is typically used to extort money from its victims, as only the hackers can decrypt the files.
We immediately powered down the server upon detecting the ransomware
After powering down the server, at 6:56am CST we opened a ticket with our Dallas Datacenter informing them about the hacking. We asked them to check if data on the attached drive was encrypted or not, and if they could decrypt it.
The Datacenter disconnected the node from our network and gave us IPKVM access. They booted the server on a new OS drive with anti-malware software installed.
From this point onward our team worked all night to recover the node from this malware. At 2:03am CT Wednesday we discovered another node affected by the same ransomware. Upon discovery of this, we shut down all our windows nodes and asked our DC to disconnect our net and network cabling to NAS and SAN servers.
Currently we are checking each node to determine what data is useful and recoverable. We hope to come to a determination of how much data has been lost and recoverable in the next 24 hours. At that time we will provide again a detailed report.
Our DNS, Mail, Web and all other services on Linux nodes continue to work as expected.
Please know that we are doing everything we can to recover your data. Our team has slept only a few hours since this hacking. We are a small company trying to survive in the current Covid situation. However our commitment to customers is the same as when we started years ago.
JodoHost “.waiting ransomware” Incident Report (July 10th, 2020)
Dear Customers,
As reported in our incident report dated June 10th, some of our Windows nodes were hacked on Tuesday July 7th, 2020 at 4:10am CST with ransomware
As part of our recovery plan, we’ve created a secure backup space where we have copied over files from the impacted Windows nodes. In this backup space, we have been sifting through the files to determine what’s recoverable. We plan to begin recovering VMs from the backup starting today, at around 12PM CT. We expect to be recovering VMs through the weekend, and are still unsure how many are recoverable or bootable.
We will provide another update in 12 to 24 hours
Dear Customers,
As reported in our incident report dated June 10th, some of our Windows nodes were hacked on Tuesday July 7th, 2020 at 4:10am CST with ransomware
As part of our recovery plan, we’ve created a secure backup space where we have copied over files from the impacted Windows nodes. In this backup space, we have been sifting through the files to determine what’s recoverable. We plan to begin recovering VMs from the backup starting today, at around 12PM CT. We expect to be recovering VMs through the weekend, and are still unsure how many are recoverable or bootable.
We will provide another update in 12 to 24 hours
We are still working to restore VMs from backups, it’s a slow process and the entire team is running on very few hours of sleep, but we are working as hard as we can.
We also have hired a Cyber Security firm to try to help us recover encrypted files. They are advising us and assisting our team
We also have hired a Cyber Security firm to try to help us recover encrypted files. They are advising us and assisting our team
Dear Customers,
Since last report, our Linux based services i.e. DNS, Web and Mail are running unimpacted.
We continue to work on restoring service for our Windows machines. As of Tuesday morning, 20% of our Windows fleet has been restored. If your server is online, it has been recovered.
We are expecting to recover 40% of our Windows VMs in the next 24-48 hours, and another 20% by this weekend. Since we are turning over all our Windows servers, the availability of hardware has become a bottleneck. We have rushed-shipped newer, updated servers to our DC to aid in the recovery.
With the new hardware in place, we expect to reach an 80% recovery by this weekend. We are continuing to work with our security firm to recover the remaining 20% of data, and should have an update on the progress of that by the end of the week.
The ransomware attack was sophisticated, and the hacker attacked not only our VMs, but our remote backups as well. To prevent a future attack, we are rebuilding our entire Windows fleet on new VMs, and have purchased additional hardware to help. All new VMs have new security software installed to prevent such a future attack.
We value your business, and want to assure our customers we are doing everything possible to restore remaining services as fast as we can. In our 15+ year history, we’ve never had a hacker hold us and our customers hostage, but we are fighting back and we will not give up.
We will update you more once we have some additional information to share with you.
Since last report, our Linux based services i.e. DNS, Web and Mail are running unimpacted.
We continue to work on restoring service for our Windows machines. As of Tuesday morning, 20% of our Windows fleet has been restored. If your server is online, it has been recovered.
We are expecting to recover 40% of our Windows VMs in the next 24-48 hours, and another 20% by this weekend. Since we are turning over all our Windows servers, the availability of hardware has become a bottleneck. We have rushed-shipped newer, updated servers to our DC to aid in the recovery.
With the new hardware in place, we expect to reach an 80% recovery by this weekend. We are continuing to work with our security firm to recover the remaining 20% of data, and should have an update on the progress of that by the end of the week.
The ransomware attack was sophisticated, and the hacker attacked not only our VMs, but our remote backups as well. To prevent a future attack, we are rebuilding our entire Windows fleet on new VMs, and have purchased additional hardware to help. All new VMs have new security software installed to prevent such a future attack.
We value your business, and want to assure our customers we are doing everything possible to restore remaining services as fast as we can. In our 15+ year history, we’ve never had a hacker hold us and our customers hostage, but we are fighting back and we will not give up.
We will update you more once we have some additional information to share with you.