August 17, 2007: Mysql update issue on two servers

Jason · August 17, 2007, 04:33:06 AM

Mysql updated overnight with a new stable release of cpanel.

At this point, mysql is not working properly on Blizzard and Cyclone.

We're looking into this now.

If you have a site (like a forum) that runs entirely on mysql, you will likely have a database connection error right now until this is resolved.

Jason · August 17, 2007, 05:42:17 AM

We are reverting to cpanel 10 on these impacted boxes right now.

Powerbob · August 17, 2007, 07:19:56 AM

Hi Jason,

How much longer will it be in your estimation until everything is back to normal. I have had 2 sites down for several hours now?

Bob

Jason · August 17, 2007, 07:45:56 AM

I am trying to get an update now. We're reverting cpanel to an earlier version on these two servers so it's not a quick fix.

All services are up except mysql so if you're site is a forum only, you can add a normal html page to advise of the outage in the meantime if you'd like.

As soon as I get a timeframe, I will post it here immediately.

Jason · August 17, 2007, 08:21:48 AM

Cpanel has been reverted and mysql is up again, however, a few sites I've tested are generating blank pages.

We're looking into that presently.

Jason · August 17, 2007, 08:44:35 AM

Please check your sites and post here if you're having mysql-related issues still so I can have a list of examples to work with. Thanks

akheir · August 17, 2007, 08:47:58 AM

Client www.celebratingtheword.com is still having issues.

sam · August 17, 2007, 09:22:15 AM

on edit:
working again - thanks

akheir · August 17, 2007, 09:45:45 AM

Up again on www.celebratingtheword.com.

Quote from: akheir on August 17, 2007, 08:47:58 AM
Client www.celebratingtheword.com is still having issues.

Jason · August 17, 2007, 09:53:24 AM

Ok, please try again now.

akheir · August 17, 2007, 09:55:49 AM

Still up.

Quote from: Jason on August 17, 2007, 09:53:24 AM
Ok, please try again now.

Jason · August 17, 2007, 10:18:00 AM

Thank you.

I will provide you a full status on this as soon as I can today in this thread.

Jason · August 17, 2007, 11:46:03 AM

Here's an explanation on what happened with mysql on Cyclone and Blizzard today.

Please note that this only impacted mysql functionality (which is certainly very important!) but did not impact any other services including http.

We determined the root cause and it's due to a version upgrade (from version 10.x to 11.x) of cPanel that happened overnight. Cpanel has updates that happen all the time but a version change is significant and this new release has a LOT of new features and configurations.

On several servers we have mysql located in spots that are non-standard. In other words, we have purposely moved mysql to store databases in a location different than what the stock operating system install would define. Because of this, the cPanel update overnight wasn't able to handle that and this caused the internal linking to our database locations to break and we had to manually fix it all.

Now that we're on version 11 and back up and running, I don't anticipate any further issues related to this problem but it's certainly something we'll keep in mind for any next major releases. cPanel 11 has been in the making for years so you can rest assured that these sorts of upgrades don't happen often. I think cPanel 10 has been out for years prior to this but I'd have to check to be sure.

On a side note, cPanel 11 has been out for months now but hasn't been released as "stable" by their standards. The way our servers are configured, they update ONLY on stable versions (or for security releases of course). The fact that 11 went from "release" to "stable" this week triggered automated updates to run. At this point, I think all but a few of our servers are on 11 now. I'm going to verify that tonight and will likely push those still on version 10 to version 11 manually asap just to be confident those migrations go smoothly.

I know of at least 3 of our other servers on cPanel 11 that upgraded without issue so we unfortunately hit a snag with these two.

Thank you for your patience and understanding. Unfortunately this was an upgrade incompatibility with the software that happens from time to time.

Regards,
Jason

Jason · August 17, 2007, 06:06:06 PM

There were 3 servers (Cyclone, Maelstom and Thunderstorm) still on cPanel 10.

They have "standard" partitions which should make for a normal upgrade.

To be sure, we're manually upgrading these to cPanel 11 right now.

I will update this thread when that's complete and confirmed to be successful.

Thanks

Mark · August 18, 2007, 12:17:19 AM

Are there any key features in this release or is it mainly non important stuff?