October 23-24, 2007 :: All server updates

[Jason]

We will be performing scheduled updates (as needed) across all servers starting tomorrow evening.

Based upon what software we're presently running, the upgrades will bring all servers up to speed with the latest cpanel stable and php.  This will also include kernel updates in some cases.

"Activating" these updates may require a reboot (definitely in the case of kernel upgrades) which may result in a few minutes of service outage per server.

I will update this thread as we process these tomorrow.

Please feel to post any questions, comments or concerns here. 

Thank you!

[Mark]

Thanks for the heads up! But, do you have an estimated time frame of the updates? I know you said evening, but what about an actual time?

[Jason]

The "loose" expectation is around 9pm EST.  Beyond that, I don't have a definite timeframe or the order we'll be doing it in.

I'm going to post all updates (and advance notices) I get as soon as I have them in this thread.

[Mark]

Ok I was jusk asking because the company uses e-mail from 7AM to 6PM MST so I was wondering if I needed to let everyone know of a possible outage.

[Jason]

If my math is correct, 6pm mst should be 8 est if I'm not mistaken so that should be outside that window. 

Given the way email services are used, the chances of that being a visible problem are pretty slim but there's no harm in sharing the possibility.  The more notice the better. 

[Mark]

Oh yeah, I already knew it was outside that time window, I was just saying why I was asking.

[Jason]

Roger roger. 

[Jason]

This process is being started now.  I will update this thread with what I can.

[Jason]

Update as of 7am EST.

The server software upgrades were completed around 5am EST this morning.  That seemed to go smoothly. That covers the latest stables of php, mysql, perl and cpanel.

Following that, the kernels were upgraded.  To activate the new kernels, all servers were rebooted between 5am and now.  For most servers, that meant about a 5 minute outage with the exception of:

Thunderstorm experienced a kernel panic on the new kernel and had to be rebooted back into the old one.  We're looking into this presently.  (That's why you may have noticed my site being down for approx 30 mins)

Blizzard came up but appears to be having a configuration issue causing sites not to load.  We're looking into this as well presently.

The other servers look good from the sites I've tested with.  Please check your sites, email, mailing lists, ssl certs, etc. and make sure everything looks good for you.  (If you're on Blizzard, please give me awhile to get the issue above resolved -- I will keep this thread updated as often as possible).

If you're seeing problems, please post them here or open a ticket through your Client Area so they can be prioritized.

Thanks!

[Jason]

Blizzard came up but appears to be having a configuration issue causing sites not to load.  We're looking into this as well presently.

Our focus is on Bind.  I hope to have this resolved quickly.

[Jason]

Bind has been rebuilt on Blizzard and all sites are up again.

I will update this thread regarding Thunderstorm.  It will likely be tomorrow before we repair/reattempt the kernel upgrade on that box.  It's fine in the meantime on the previous kernel version.

Thanks

[Mark]

All looks well on my end. Thanks for keeping us in the loop!

[Jason]

I do my best! 

[Finner]

Yup thanks Jason

[akheir]

Hey Jason,

Were any servers downgraded to PHP 4.4.7?