For those of you who noticed, the server just underwent a series of reboots. Our techs are researching the issue now and I hope to post the results of their findings as soon as they analyze the logs.
Around 10:55AM EST, the server was rebooted and the restart didn't initialize various services on the server. Causes for this are being looked into.
There was a miscommunication that prevented the techs from authorizing an additional restart. Rest assured this has been corrected and WILL NOT occur again. Because of that, I was able to call in the restart (since the server was unresponsive) when I noticed the outtage about 15 minutes later. Typically (and definitely from now on) we will have a 5 minute turnaround in the event this occurs again now that authorization has been cleared.
After services didn't restart, we rebooted a third time which returned all services to normal. The server is now being monitored closely until we determine the cause.
Total outtage was right at 30-35minutes from the first reboot until it was rebooted a third time by the datacenter.
Feel free to post any questions/comments. Otherwise, I'll update this thread as information becomes available.
Regards,
Jason
update
We're focusing on some httpd and mysql issues that occurred earlier this morning around 7:05 AM EST. It appears that mysql was automatically shutdown at that time because of high server loads. It restarted a few minutes later and was fine. It's possible that the latter problems around 11am arose from similar issues and we're looking into that.
Server loads have returned to normal and everything has been fine for the past 1.5 hrs.
I'll update this thread if new information is determined.
Regards,
Jason