November 29, 2007 :: Hurricane Outage

MaryBeth · November 29, 2007, 04:46:08 PM

Well I'm not good. My site is up but not my message board. Is there a PHP problem?

Jason · November 29, 2007, 05:11:15 PM

Quote from: MaryBeth on November 29, 2007, 04:46:08 PM
Well I'm not good. My site is up but not my message board. Is there a PHP problem?

There is not a php problem. My techs have "disabled" a few of the key sites they identified that were causing the high loads.

I have been travelling the last half hour but am back online and will be looking into this closely with them now.

Your site was one of those sites. We're going to be re-enablling them and watching things now.

The problem today thus far appears to be extremely high mysql loads -- possibly due to some hangs. I'm trying to find out if any configurations changed or what caused this to happen this afternoon. This server generally runs very efficiently so I want to try to find out what the root cause is if possible.

MatthewSchenker · November 29, 2007, 05:29:53 PM

Jason,
I'm glad to know I'm not the only one experiencing this problem! Thank you for your efforts in
working on this, and I am confident you'll figure it out!

I know that my forum has been getting an unusual number of hits from spiders recently, sometimes as many as 300 at a time. Could that be part of the problem?

Matthew

MaryBeth · November 29, 2007, 05:31:17 PM

OK thanks. Let me know if there is something I need to do on my end to help. I haven't noticed an increase in activity lately but who knows - it could be something I am unaware of.

MaryBeth · November 29, 2007, 05:36:13 PM

Jason do we have a possible time frame? I just want to try and pass on the info to my members. They are having withdrawal.

Jason · November 29, 2007, 05:40:26 PM

Quote from: MatthewSchenker on November 29, 2007, 05:29:53 PM
I know that my forum has been getting an unusual number of hits from spiders recently, sometimes as many as 300 at a time. Could that be part of the problem?

I certainly wouldn't count that out -- I'll pass that on, that could certainly be a big problem if they're all using a mysql connection.

Jason · November 29, 2007, 05:56:23 PM

We're going to run some repairs on the databases for the sites causing the high loads right now.

Note: we "unlocked" the accounts we suspended so server performance has slowed down again as the loads have climbed again. We are aware of this but that's going to allow us to troubleshoot more effectively.

It's possible the reboot we did when the server first stopped responding this afternoon has had some adverse effects on some databases. If so, the repairs should resolve that.

I'll keep this thread current.

To answer the earlier question, no, I can't provide any solid ETA at this time though it's certainly my top priority to restore service before I get up from my desk again -- nature willing.

akheir · November 29, 2007, 06:04:03 PM

Hey Jason,

I notice that some of my client's sites are unreachable again.

HKmod · November 29, 2007, 06:08:01 PM

We were up but we're gone again.

Jason · November 29, 2007, 06:11:08 PM

Yes, all sites are presently unreachable. We're running some load intensive repairs. When I think things are "back to normal" I'll say but for now, things may be shaky for awhile.

HKmod · November 29, 2007, 06:21:06 PM

Oh, sorry about that Jason. I misunderstood and thought they were up and slow.

I get it!

Kabres · November 29, 2007, 07:00:18 PM

Was my forums one of the ones that was shut down as well? sanguiphoria.com

texasflats · November 29, 2007, 07:34:02 PM

mine is down as well, I submitted a ticket before I found this thread.

Jason · November 29, 2007, 07:56:16 PM

Quote from: Kabres on November 29, 2007, 07:00:18 PM
Was my forums one of the ones that was shut down as well? sanguiphoria.com

No, your's wasn't purposely disabled like some of the others earlier.

At this point, none are purposely disabled presently -- all sites are going up and down as httpd and mysql respond to the loads which are fluctuating dramatically.

The database repairs didn't provide the results we were hoping so this has been passed to a senior admin now.

I'll continue updating as we learn more.

Jason · November 29, 2007, 08:47:10 PM

Ok, we're making some progress now. No specifics to share just yet (before we know for sure) though we're thinking we found the source and the loads are returning close to normal now. It's still going to be a bit sluggish for a little while due to the increased load upon restoration but things are quickly getting back to fast again.

Thank you all for your patience. I hope to have a more full update shortly.