Charlottezweb

Charlottezweb Hosting => Server Updates & Outages => Topic started by: Jason on September 11, 2006, 03:44:34 PM

Title: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Jason on September 11, 2006, 03:44:34 PM
There is presently a router issue at the datacenter causing extreme slowness for Cyclone.

Unfortunately, when it rains it pours around here it seems.

They have been working on it and expect to have it resolved very quickly.

Note:  The server is running perfectly, it's a router in the datacenter that appears to be bottlenecking traffic.

I will keep this thread updated once resolved.

Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Knu on September 11, 2006, 03:47:43 PM
I was about to ask! :)

Jason, you must sleep with your servers! :D
Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Greentoes on September 11, 2006, 03:56:08 PM
Why is it that I can get into your site but not mine?
Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Jason on September 11, 2006, 03:58:28 PM
Quote from: Knu on September 11, 2006, 03:47:43 PM
Jason, you must sleep with your servers! :D

Seems like it the past week or so unfortunately.  :)

Quote from: Greentoes on September 11, 2006, 03:56:08 PM
Why is it that I can get into your site but not mine?

Two of our servers (including the one Charlottezweb is hosted on) are located in a datacenter in Texas.  Our two latest servers are located in Atlanta, Georgia.  It's the Georgia datacenter having the router problem.
Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Greentoes on September 11, 2006, 04:04:07 PM
Quote from: Jason on September 11, 2006, 03:58:28 PM

Two of our servers (including the one Charlottezweb is hosted on) are located in a datacenter in Texas.  Our two latest servers are located in Atlanta, Georgia.  It's the Georgia datacenter having the router problem.

Thanks, I will be so glad when all of these problems are resolved.
Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Jason on September 11, 2006, 04:26:01 PM
I don't have any new information at this time other than the fact that the datacenter's website is now also loading just as slowly. 

I'm on my way home and should be online again in about 45 mins.  In the meantime, I'm going to give them a call in the next few minutes.  I know from an earlier post on their forums that their CEO is aware of the problem so I know it's getting their full attention.



Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Jason on September 11, 2006, 05:32:31 PM
I spoke with the datacenter.  It appears they are experiencing a Denial of Service attack on their datacenter.  This is what's overloading the routers.  Their head network technician is filtering the traffic and they hope to have this cleaned shortly.

Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Jason on September 11, 2006, 06:05:11 PM
Well, if anyone wants to follow along, there are threads on WHT now for our Gnax (Cyclone's datacenter):

http://www.webhostingtalk.com/showthread.php?t=546488

Not only that, but Ev1 (our other provider) seems to be having a major outage today.  The biggest some people say they've seen in years.  (That's the datacenter where our servers Thunder and Lightning are).

Hopefully this is all just a coincidence.

http://www.webhostingtalk.com/showthread.php?t=546509

Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Jason on September 11, 2006, 06:59:07 PM
Looks like speeds are returning to normal.  They're still much slower than they should be but sites are loading.  I expect the latency to drop as the initial spike drops.

I'll update this once the dc posts their description of what occurred.
Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Ken on September 11, 2006, 07:03:50 PM
Thanks for keeping us informed Jason!
Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Greentoes on September 11, 2006, 09:08:28 PM
My site is doing much better too. Thanks
Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Jason on September 11, 2006, 09:10:47 PM
Glad to hear it :)
Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Mark on September 11, 2006, 10:13:06 PM
Ouch! Both datacenters in one day. :-\

Just thought I'd let you know that as of the time of this reply, my website (on Cyclone as you know) seems to be a little slow still. It's loading, but slow.
Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Jason on September 11, 2006, 10:21:57 PM
Quote from: Killer Possum on September 11, 2006, 10:13:06 PM
Just thought I'd let you know that as of the time of this reply, my website (on Cyclone as you know) seems to be a little slow still. It's loading, but slow.

That doesn't surprise me.  I think something like this will take overnight to get back to normal.  I'm still awaiting their official response to see how resolved this truly is...
Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Mark on September 11, 2006, 11:15:05 PM
I see, thanks for the information... as always. :D
Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: CountryLady on September 13, 2006, 01:34:18 AM
Thanks for giving us all the details. Really helps. :)
Title: Re: Mon, Sept 11, 2006 :: Cyclone, network slowness at DC
Post by: Jason on September 13, 2006, 09:50:03 AM
No problem.

The cause was a bad ten gig interface on one of the core routers that was throwing errors and causing bgp instability.