Wed, Jan 10, 2007 :: Cyclone outage

Jason · January 10, 2007, 07:08:57 PM

I'm looking into an outage with Cyclone that started a few mins ago.

I will update this thread asap.

itsjustme · January 10, 2007, 07:18:31 PM

Thanks! I was just coming to see what was going on. Dropped off about 25 minutes ago for me

Terri

Jason · January 10, 2007, 07:19:18 PM

Update: It's not just us (which is actually a good sign) so I expect this should be getting major attention.

I'll update this as soon as I have more info...

Jason · January 10, 2007, 07:20:13 PM

Quote from: itsjustme on January 10, 2007, 07:18:31 PM
Thanks! I was just coming to see what was going on. Dropped off about 25 minutes ago for me

Terri

Yes, that's probably about right. I got a page approx 20 mins ago...

Seems to be network related so I hope to have something to report quickly.

Jason · January 10, 2007, 07:36:05 PM

Update from the datacenter:

"We currently have one of our core routers down, our 2 most senior techs are on site working on the issue. We have spare parts for all routers so the issue will be resolved shortly."

itsjustme · January 10, 2007, 07:46:23 PM

Thanks!

Jason · January 10, 2007, 08:04:32 PM

Ok, connectivity is restored. I'm going to confirm some things and will post the explanation I get from the dc later this evening. The server itself didn't have any problems but they lost network connectivity for approx the last hour.

itsjustme · January 10, 2007, 08:17:59 PM

Jason · January 11, 2007, 01:54:28 PM

For anyone interested in the full story from the dc:

Quote
Switch and Data - which is in telx at 56 marietta had a pdu failure last night that took our router down that is colocated there.

The reason the outage was so irritating is that we are in an in between phase on our move.

The rest of the network is fully redundant - that is why people at dc3 only saw minor issues as bgp providers reconverged our network across the ones that did not go down.

We are also adding further redundancy soon - which I will talk about if the deal goes through - which will expand our network coverage to another building in atlanta so that all our outbound is not in telx. This will give us outbound in 2 locations in telx - another building in downtown with another carrier and of course cogent is lit here at the nap so we have outbound through them already.

(The move he's referring to is the combining of GNAX's 3 datacenters into one huge new one. This is the move that I've discussed here in a thread about Cyclone being physically moved to a new datacenter. That was supposed to happen back in November but has been temporarily delayed. I will update that thread accordingly as I get more information. I don't expect it to be anytime sooner than a few months.)

Regards,
Jason

weekend camper · January 11, 2007, 10:55:37 PM

nvm.