December 7, 2004 :: outtage due to cpanel update

[Jason]

December 7, 2004 :: outtage due to cpanel update
Here's a copy of the email sent to all clients today:

"I wanted to actively inform you of an outage that caused our websites to not display this afternoon that lasted approximately 40 minutes (around 11:20 AM to noon EST) according to our monitoring services.

Here's a shortened explanation of what occurred:

An automated security update ran last night on all servers that was necessary to patch some security exploits that have been discovered lately.  Not running these updates would not have been an acceptable solution.  The updates were scheduled for late last night so as to interfere with as few users as possible based on normal server loads.  Unfortunately, the security updates had some incompatibilities with cPanel which caused it to render Apache unable to restart once stopped.  Everything was fine until around 11:20am when a normal Apache restart occurred and was unable to restart. It was at this point that our sites were no longer reachable via Http.

Our techs already had a team allocated and working on reconfiguring last night's updates at this point.  Rather then just removing the updates which would've left us still vulnerable, they recoded the fixes to restore compatibility with cPanel. They were therefore already working the issue when I first noticed it a few minutes after it started. 

Overall, it was a good drill to demonstrate that our security and technical bases are well covered.  I apologize for the outtage taking 40 minutes, but that's certainly better than leaving unpatched security holes which could've done far more damage.  As always, I'll alert you of any issues with as much advanced notice as I can.  In this case, it wasn't until this morning that apache restarts started failing that the issue was discovered.

Please let me know if you have any questions or concerns or feel free to respond on the forum.

Thank you for your continued business and I hope you have a relaxing holiday season.

Regards,
Jason

[Jason]

Here's the more technical explanation:

"The cPanel Apache configuration would not accept the security module (installed last night) that was needed to run the server safely -- without it the server would be vunerable to PHP injection exploits that would enable hackers to break in, deface websites etc. This was an automated install that upgraded mod_secure (http://www.modsecurity.org/) to prevent exploits including the phpBB exploit that has allowed many people to hack servers recently. As far as we can tell cPanel machines on the 'stable' release were affected by this problem.

Apache had to be re-configured to accept that module and keep the server secure.  The unfortunate side effect was that Apache was required to be offline while this was done. The main part of the module was upgraded on servers last night.  Ensim, Plain and Plesk servers accepted it and worked fine however cPanel servers had a issue with that had to be resolved before Apache could restart without failing."

Regards,
Jason

[Patrick]

  Thanks again for keeping us updated ! Your hosting is truly the best. 

[alancurry]

Thanks for the info Jason!