This post will be updated as information is made available to us from the data center technicians. Please check back often for updates. Newest updates will be towards the top, with earlier updates at the bottom of the post.
Friday 3 May 2019
After no update from the data center yesterday, we proceeded to submit a new request for information this morning. Unfortunately, after 6 hours we have still not received a reply. We have been in communication with other users of the same data center, some of which have reported that their servers have been restored and are now online, but it appears that the process is slow going. We will continue to monitor the situation, and as soon as we have an update from the data center, we will post the new information here. For the mean time all clients are active on the failover server, although database services remain offline.
Thursday 2 May 2019
A more revealing update has been provided by the data center company, unfortunately, it is not very promising at this time. The company has admitted to us that a ransomware virus had infiltrated their network, and that the reasoning behind pulling all Windows servers offline on the 23rd of April was in efforts to stop the spread of this ransomware virus. In response to the ransomware infection, the data center company has brought in an outside cyber security firm, which is now in the process of combing through the data on all the servers, identifying what data is infected and what data can be restored. Additionally, the data center company has contacted and enlisted the support of law enforcement to further investigate the cause, and source, of this infection.
To our surprise, the data center company has not been forthcoming with specific details about the IdealMSP server and database server. Our recent inquires to their support staff have resulted in vague answers, or simply not responded to. A request for our backup files for both web and email as well as database server files was denied, saying that they could not provide our backups at this time. This response makes us believe that the ransomware infection might have directly affected the server housing all of our backup files as well, however at this time we do not even know if the ransomware infected any of our servers. The data center company has not confirmed, nor denied infection of IdealMSP servers, despite our continued requests for answers.
We will be closely monitoring the situation, and in the event we receive additional information from our inquires today we will post another update. At this time all clients are setup on our failover servers, however without access to our system backups and database backups, email history and database services are unavailable. If you are experiencing any issues with your service on the failover server, please contact us directly.
Wednesday 1 May 2019
We are seeing some signs of life from our main data center server, after responding to port pings the last couple of days the server is now offline, which we are assuming is an indication that the data center technicians are completing the restoration process on our server, however, at this time they have still not provided us with any update as to when we can expect full service restoration. We will continue updating this post as new information is made available.
7:00 PM EST – no new communication from the server since it last responded to port pings. Maybe the restoration process is still underway. We will monitor the situation and report if any new updates are available.
Tuesday 30 April 2019
Clients that were experiencing issues with downtime on the failover server have been verified as their name server issues resolved and now all clients are verified online with the failover servers. We are continuing to monitor the main data center server, however at this time there has been no update from the data center technicians as to an estimate of when the restoration process will be completed and our main servers brought online.
Monday 29 April 2019
The weekend has unfortunately produced no results for IdealMSP servers. Some clients have reported that their websites were still showing as being offline after the swap to the failover servers – the only clients that reported continued downtime were those which do not have their domain names registered with IdealMSP and have been provided with instructions for updating their name server records to our dedicated DNS provider.
Our server monitoring service has indicated that our main data center web and email server is online, we anticipate that this means our server is in the rotation for service restoration and will be monitoring it diligently in anticipation for service restoration.
Friday 26 April 2019
As of this posting, we have still not received a confirmed estimate of when we can expect our servers to be restored to full service. According to the updates provided from the data center there have been many servers restored to date and we are hopeful that the IdealMSP servers will be coming up in the rotation soon.
In the meantime, we have confirmed that all web hosting clients are active on the failover servers we have established, and more importantly, email services are working as intended, allowing for our web hosting clients to access their webmail and send and receive emails from their domain email accounts.
As previously stated, email history and contacts are not available on the failover servers at this time, however once our main data center servers are restored, we will be migrating all email history to the failover server. Additional information about our failover/redundancy plans will be made available in the coming weeks. In the meantime, please continue to access your webmail services at http://webmail.434tech.com – once main servers are restored you will be able to access webmail at your regular domain address.