Charlottezweb

Charlottezweb Hosting => Server Updates & Outages => Topic started by: Jason on September 07, 2012, 06:46:40 PM

Title: Sept 7, 2012 :: Emergency reboots by datacenter (multiple servers)
Post by: Jason on September 07, 2012, 06:46:40 PM
I've become aware of some emergency reboots due to maintenance at one of our datacenters. 

The maintenance was apparently urgent as they normally provide ample advanced notice of situations like this.

They are doing the reboots by zones at the datacenter.

Two of our servers were down but are now back online whereas the others one there are now in the process of rebooting.

Impacted servers:

Heatwave
Deluge
Avalanche
Wildfire
Sandstorm
Thundersnow

I will post back once all of them are back online.

Thank you,
Jason

Title: Re: Sept 7, 2012 :: Emergency reboots by datacenter (multiple servers)
Post by: Jason on September 07, 2012, 06:54:39 PM
It looks like all servers are back online.

If you are facing any issues, please open a ticket or email me directly.

Thanks!

Regards,
Jason
Title: Re: Sept 7, 2012 :: Emergency reboots by datacenter (multiple servers)
Post by: Jason on September 08, 2012, 03:09:23 PM
Further information from the datacenter on the emergency maintenance yesterday:

Quote
In our ongoing effort to keep all of our customers informed of
potential issues, we are reaching out to you now to further explain why
the reboot we performed was required.

A vulnerability was detected in servers running XEN hypervisor,
worldwide.

For customers who would like further information about the specifics
about the vulnerability in XEN, please see
http://www.securityfocus.com/bid/55413/info

In order to ensure the stability and security of your server, should it
have been among those affected, we proactively patched your server.
While this measure will prevent your instance from suffering the
potential issues that could result from this vulnerability, it did
require a server reboot. It is our policy to provide as much advance
notice as possible in the event of any service affecting events. This
circumstance, unfortunately, required immediate action in order to
prevent possible serious repercussions.

Our focus lies in delivering functionality and security at the highest
level possible for our customers while also providing a sterling level
of communication and customer service. As part of that commitment,
while we were unable to provide significant advance notice, we do want
to ensure that all affected customers are provided a full explanation
of why this occurred.