July 21, 2005 :: Lightning being rebooted

[Jason]

Lightning has stopped responding and is awaiting a reboot by the datacenter.

I will update this thread with more information as it arrives.

[Jason]

The reboot failed.  The datacenter is looking into the box now.

[marcobarrera]

It's up now.

[bitwiz44]

The reboot failed.  The datacenter is looking into the box now.

Thanks. 

[Jason]

I think it may be back down at the second though.

Here's the "good" news from the datacenter...

At this time we are experiencing problems with the switch on your subnet. Our networking tech is now replacing the switch and server will be up shortly. Thank you for your patience..

So at least it's nothing with our server which is a nice relief.  They should have the switch replaced shortly if not already to resolve this.

[Jason]

Service has been up since the last post.  The datacenter just verified the same which should signify that the switch was replaced.  In the event it's presently on the faulty one awaiting replacement (doubtful since ev1 closed my ticket), it's possible we may see another short outage when they replace the switch later tonight.  As it stands, we believe this to be resolved.

Total outage time by my alert system was reported at approx 40 mins (3:03am EST to 3:42am EST).

[Jason]

UPDATE:  As described above, a switch needs replacing and we're are going to do that hopefully within the next hour. This is an email I'm sending to everyone on Lightning right now:

-----------------
Lightning is going to be powered down for hopefully less than 5 minutes within the next hour or so.  As part of last night's outage, the Master Reboot Switch failed and is running on temporary power. 

They need to swap us over to a new one.  This should be a quick swap, but I wanted to pass this on with as much advance notice as possible.  Since it's 4pm EST, I decided to authorize them to do this right away when traffic is lighter.

Regards,
Jason