Charlottezweb

March 26/27 (overnight) :: Tsunami emergency maintenance

Discussion started on Server Updates & Outages

This thread serves as a placeholder for a write-up with further details on some emergency maintenance we performed overnight and into this morning on server Tsunami.

I will update this thread with more information on what occurred and why later on today (Thursday).

Thank you,
Jason
#1 - March 27, 2014, 04:22:03 AM

A quick update on this topic -- I may have more information later.  I'm waiting on confirmation of a few things.

At the most basic level, we ran a MySQL update last night that was required for an urgent cPanel update that was released.  The MySQL update had some issues with the version of PHP we were running which didn't become apparent until after we ran the initial recompiles.  This required a few attempts to resolve -- hence the extended outage required.

Normally a MySQL upgrade and Apache/PHP recompile should take nowhere near the time this took last night.  Unfortunately the process appears to have been repeated multiple times throughout our troubleshooting as we attempted to find the root cause of why PHP wasn't functioning properly. 

Thank you again for those of you who reached out to me and for the patience of everyone involved.  Whereas other services on the server were up throughout (email, html-based sites, etc), php is obviously a core requirement for the majority of our customers and I can certainly understand how big of an impact this was.  It was certainly frustrating for me throughout.

I will likely have a separate post a bit later in our Clients's-only section of the forum but I wanted to provide what I can at this time for those interested.

Thank you again,
Jason
#2 - March 27, 2014, 09:04:29 PM

Members:

0 Members and 1 Guest are viewing this topic.