February 27 and March 4, 2017 :: Lightning Outages

Started by Jason, March 06, 2017, 05:00:49 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Jason

On February 27 and March 4, we faced service outages on server Lightning.  The server was up during both outages but http and mysql services were impacted.

Each outage was approximately 10-15 minutes in length from the time Pingdom alerted me to when http and mysql were restarted.  In some cases, some customer databases were corrupted due to the service stop and required a repair to be run.

In reviewing both instances, we're fairly confident that we've found a service running on the server that may have been to blame.  Unfortunately I'm not 100% positive (yet) but that's what the data is showing us.  I have disabled this service so that at worst case, we'll be able to rule that out if this happens again.

Service has been normal since this weekend after disabling that so I'm hopeful that's our solution.  If not, we'll review this again/further.

As posted on our Status Page, if you're seeing any errors on your site (that uses a MySQL database) on this server, running a database repair in cPanel will typically resolve it.  If not or if you'd like assistance, please email me or open a ticket.  I've created an article here on how to run a database repair in cPanel.

Thank you,
Jason