I'm sure you already know but Hurricane has been down for about 10-15 minutes now and while I usually would just sit and wait for it to come back up, I have an employee who was in the middle of sending out a very important time sensitive e-mail. So I was just wondering what is going on.
Though I know as soon as I submit this post it will be back up. ;)
Yes, we're aware. We're awaiting a reboot.
Thanks
Service has been restored.
Thanks
edited: works fine thanks.
Quote from: Mishenka on November 29, 2007, 02:09:40 PM
I'm still down.
Your site works fine for me. What are you seeing?
I notice that service is sketchy right now (up, down, slow). I think the server loads are peaking (I saw 156 at one point!), but can you check on it? :)
Thanks
ok, it's down again. the page times out
They're looking into the cause and situation.
I'll keep this posted as I get more info.
Yes please keep us updated, it appears to be down still for me at least.
Oh and sorry for posting in the wrong forum I don't know what I was thinking. :P
Thanks for looking into it.
Not to induce a pile-on, but I'm still down as well. Just wanted mostly to tag this thread for future updates...
Current update is that they're still actively looking into it. This has top priority so we'll get to the bottom of it asap.
OK thank you for the update!
Seems to be up and running now for about 5 minutes but loads seem to be fluctuating quite a bit.
Looks like we are good to go? Loads seem to have leveled out at a very low number.
Well I'm not good. My site is up but not my message board. Is there a PHP problem?
Quote from: MaryBeth on November 29, 2007, 04:46:08 PM
Well I'm not good. My site is up but not my message board. Is there a PHP problem?
There is not a php problem. My techs have "disabled" a few of the key sites they identified that were causing the high loads.
I have been travelling the last half hour but am back online and will be looking into this closely with them now.
Your site was one of those sites. We're going to be re-enablling them and watching things now.
The problem today thus far appears to be extremely high mysql loads -- possibly due to some hangs. I'm trying to find out if any configurations changed or what caused this to happen this afternoon. This server generally runs very efficiently so I want to try to find out what the root cause is if possible.
Jason,
I'm glad to know I'm not the only one experiencing this problem! Thank you for your efforts in
working on this, and I am confident you'll figure it out!
I know that my forum has been getting an unusual number of hits from spiders recently, sometimes as many as 300 at a time. Could that be part of the problem?
Matthew
OK thanks. Let me know if there is something I need to do on my end to help. I haven't noticed an increase in activity lately but who knows - it could be something I am unaware of.
Jason do we have a possible time frame? I just want to try and pass on the info to my members. They are having withdrawal. ;D
Quote from: MatthewSchenker on November 29, 2007, 05:29:53 PM
I know that my forum has been getting an unusual number of hits from spiders recently, sometimes as many as 300 at a time. Could that be part of the problem?
I certainly wouldn't count that out -- I'll pass that on, that could certainly be a big problem if they're all using a mysql connection.
We're going to run some repairs on the databases for the sites causing the high loads right now.
Note: we "unlocked" the accounts we suspended so server performance has slowed down again as the loads have climbed again. We are aware of this but that's going to allow us to troubleshoot more effectively.
It's possible the reboot we did when the server first stopped responding this afternoon has had some adverse effects on some databases. If so, the repairs should resolve that.
I'll keep this thread current.
To answer the earlier question, no, I can't provide any solid ETA at this time though it's certainly my top priority to restore service before I get up from my desk again -- nature willing. :)
Hey Jason,
I notice that some of my client's sites are unreachable again.
We were up but we're gone again.
Yes, all sites are presently unreachable. We're running some load intensive repairs. When I think things are "back to normal" I'll say but for now, things may be shaky for awhile.
Oh, sorry about that Jason. I misunderstood and thought they were up and slow.
I get it!
Was my forums one of the ones that was shut down as well? sanguiphoria.com
mine is down as well, I submitted a ticket before I found this thread.
Quote from: Kabres on November 29, 2007, 07:00:18 PM
Was my forums one of the ones that was shut down as well? sanguiphoria.com
No, your's wasn't purposely disabled like some of the others earlier.
At this point, none are purposely disabled presently -- all sites are going up and down as httpd and mysql respond to the loads which are fluctuating dramatically.
The database repairs didn't provide the results we were hoping so this has been passed to a senior admin now.
I'll continue updating as we learn more.
Ok, we're making some progress now. No specifics to share just yet (before we know for sure) though we're thinking we found the source and the loads are returning close to normal now. It's still going to be a bit sluggish for a little while due to the increased load upon restoration but things are quickly getting back to fast again.
Thank you all for your patience. I hope to have a more full update shortly.
:-[
Everything is looking great again. We have pinpointed the problem with one account and will be cleaning it shortly. In the meantime, the loads are back to optimal levels again.
Thank you all for your patience and understanding throughout this afternoon/evening. Unfortunately, not all problems can be isolated as quickly as we'd like but luckily some of our configurations do help tremendously with this. In the end, finding a problem that can be easily reversed is a blessing.
Regards,
Jason
I am very impressed with the speed in which you fixed it AND in your communication.
I've been with a couple of servers before now and I have never been so happy!
Quote from: HKmod on November 29, 2007, 10:21:10 PM
I am very impressed with the speed in which you fixed it AND in your communication.
I've been with a couple of servers before now and I have never been so happy!
Glad to hear it. :)
I'm not very happy with the speed to be honest but sometimes something weird like this comes up that takes awhile to locate. I like to keep everyone posted with as much as I know because I remember how painful it can be just sitting there wondering what's going on.
Well, I won't deny that I've been a bit upset and frustrated today. But since this doesn't happen a lot (well, hopefully not :P), life goes on and service here is still 2 thumbs up.