First, please allow me this opportunity to thank all of your for your incredible patience and understanding as we’ve worked through this very challenging scenario. We are absolutely dedicated to ensuring the happiness and recovery of every single impacted customer. Now that we’ve begun to get ahead of the support load I’m able to share more specific details on our current status. At this time all customer services should be online and available. A small subset of customers may still be offline as their containers will require further investigation and attention. A few nodes have been condensed/transferred/migrated so please do not be alerted if your node ID has changed. Direct access to SolusVM remains inactive as we wait for their internal and external security audits to be completed (as discussed here: lowendtalk/discussion/11327/solusvm-audit-update#latest). During the interim you are able to control your containers through our billing/support system and may request OS reloads via ticket. Initially the mitigation strategy called for all impacted nodes to be inspected for logs, reloaded, and then images from our backup database installed. This process proved far too timely and dramatically slowed down the recovery effort which is also partially to blame for our poor communication during the first 48 hours post-compromise. We altered the plan two days ago, deciding to instead install all impacted customer service from scratch; this revised strategy has allowed us to rapidly return to full online status and reduce pressure as we work diligently to recover data for all customers who require it. ChicagoVPS has two separate backup facilities, a free public facing system called Central Backup and a secondary backup, which automatically ran each week. Unfortunately a small group of nodes were not yet setup for the automatic/secondary system or the backups were corrupted. For customers on those nodes, if you ran a Central Backup your data is absolutely safe and you may request a restoration via ticket. Customers who were not on the nodes with corrupted secondary backups can request a restoration regardless of whether they utilized the free Central Backup feature. The restoration process currently has a 6-12 hour lead time once you’ve requested it. ChicagoVPS will be analyzing this event closely so we may implement refined plans to both protect against repeat issues and to ensure our communication and reaction strategies are improved. While SolusVM has released multiple updates in the past few days to fix vulnerabilities which allowed this event, and others, to occur we take full responsibility for our outage. We simply should have reacted more quickly, more effectively and provided better communication. We’ve already learned from this past week, and we will continue to learn. As we push towards final resolution on all fronts we will continue to update our customers via email, Facebook (https://facebook/chicagovps) and the “Low End Talk” message board (lowendtalk/discussion/11304/chicagovps-update#latest). Thank you again for your patience, loyalty and understand. We appreciate you. Regards Chris Fabozzi Director of Operations ChicagoVPS
Posted on: Sun, 23 Jun 2013 05:08:20 +0000