06/13/2011 to 06/19/2011 (Scheduled server reboots and Network issue)

Started by Jason, June 20, 2011, 07:02:04 pm

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Jason

We had a number of scheduled (and non-scheduled) issues this week:

1.  Scheduled:  June 12-16 -- we rebooted two servers per night to activate recent kernel upgrades. 
Click for the server list and schedule.

2:  Non-Scheduled:  June 19 -- we experienced a DDoS at the datacenter that caused Alertra to detect outages on Jetstream and Typhoon.  In actuality, multiple other servers also experienced periods of intermittent connectivity although they weren't detected as being fully down.  For example, Whiteout and Tsunami had reports of very slow response times. 
Click for details.


All times are in EST.

Device:    Whiteout
Protocol:    HTTPb
URL:    http://207.210.108.178
Frequency:    Every 5 Minutes
Status:    UP Since 2010-06-17 20:15:44
Date    Outages    Downtime    Uptime%
2011-06-13 Mon    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-14 Tue    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-15 Wed    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-16 Thu    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-17 Fri    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-18 Sat    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-19 Sun    0    0 hrs, 0 mins, 0 secs    100.000%
Summary:    0    0 hrs, 0 mins, 0 secs    100.000%

Device:    Blizzard
Protocol:    HTTPb
URL:    http://72.9.246.186
Frequency:    Every 5 Minutes
Status:    UP Since 2011-06-20 06:22:45
Date    Outages    Downtime    Uptime%
2011-06-13 Mon    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-14 Tue    1    0 hrs, 3 mins, 3 secs    99.788%
2011-06-15 Wed    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-16 Thu    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-17 Fri    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-18 Sat    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-19 Sun    0    0 hrs, 0 mins, 0 secs    100.000%
Summary:    1    0 hrs, 3 mins, 3 secs    99.970%

Device:    Firestorm
Protocol:    HTTPb
URL:    http://63.247.66.42
Frequency:    Every 5 Minutes
Status:    UP Since 2011-06-13 22:03:37
Date    Outages    Downtime    Uptime%
2011-06-13 Mon    1    0 hrs, 4 mins, 39 secs    99.677%
2011-06-14 Tue    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-15 Wed    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-16 Thu    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-17 Fri    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-18 Sat    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-19 Sun    0    0 hrs, 0 mins, 0 secs    100.000%
Summary:    1    0 hrs, 4 mins, 39 secs    99.954%

Device:    Tempest
Protocol:    HTTPb
URL:    http://207.210.107.194
Frequency:    Every 5 Minutes
Status:    UP Since 2011-06-14 22:01:59
Date    Outages    Downtime    Uptime%
2011-06-13 Mon    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-14 Tue    1    0 hrs, 2 mins, 48 secs    99.806%
2011-06-15 Wed    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-16 Thu    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-17 Fri    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-18 Sat    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-19 Sun    0    0 hrs, 0 mins, 0 secs    100.000%
Summary:    1    0 hrs, 2 mins, 48 secs    99.972%

Device:    Monsoon
Protocol:    HTTPb
URL:    http://63.247.66.90
Frequency:    Every 5 Minutes
Status:    UP Since 2011-06-12 22:26:29
Date    Outages    Downtime    Uptime%
2011-06-13 Mon    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-14 Tue    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-15 Wed    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-16 Thu    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-17 Fri    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-18 Sat    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-19 Sun    0    0 hrs, 0 mins, 0 secs    100.000%
Summary:    0    0 hrs, 0 mins, 0 secs    100.000%

Device:    Tsunami
Protocol:    HTTPb
URL:    http://207.210.103.250
Frequency:    Every 5 Minutes
Status:    UP Since 2011-06-12 22:26:29
Date    Outages    Downtime    Uptime%
2011-06-13 Mon    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-14 Tue    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-15 Wed    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-16 Thu    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-17 Fri    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-18 Sat    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-19 Sun    0    0 hrs, 0 mins, 0 secs    100.000%
Summary:    0    0 hrs, 0 mins, 0 secs    100.000%

Device:    Tornado
Protocol:    HTTPb
URL:    http://72.9.230.170
Frequency:    Every 5 Minutes
Status:    UP Since 2011-01-14 12:36:50
Date    Outages    Downtime    Uptime%
2011-06-13 Mon    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-14 Tue    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-15 Wed    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-16 Thu    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-17 Fri    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-18 Sat    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-19 Sun    0    0 hrs, 0 mins, 0 secs    100.000%
Summary:    0    0 hrs, 0 mins, 0 secs    100.000%

Device:    Supercell
Protocol:    HTTPb
URL:    http://72.9.251.146
Frequency:    Every 5 Minutes
Status:    UP Since 2011-05-29 23:22:37
Date    Outages    Downtime    Uptime%
2011-06-13 Mon    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-14 Tue    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-15 Wed    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-16 Thu    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-17 Fri    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-18 Sat    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-19 Sun    0    0 hrs, 0 mins, 0 secs    100.000%
Summary:    0    0 hrs, 0 mins, 0 secs    100.000%

Device:    Typhoon
Protocol:    HTTPb
URL:    http://72.9.243.194
Frequency:    Every 5 Minutes
Status:    UP Since 2011-06-19 20:59:13
Date    Outages    Downtime    Uptime%
2011-06-13 Mon    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-14 Tue    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-15 Wed    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-16 Thu    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-17 Fri    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-18 Sat    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-19 Sun    1    0 hrs, 9 mins, 47 secs    99.321%
Summary:    1    0 hrs, 9 mins, 47 secs    99.903%

Device:    Jetstream
Protocol:    HTTPb
URL:    http://209.51.153.58
Frequency:    Every 5 Minutes
Status:    UP Since 2011-06-19 21:22:44
Date    Outages    Downtime    Uptime%
2011-06-13 Mon    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-14 Tue    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-15 Wed    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-16 Thu    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-17 Fri    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-18 Sat    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-19 Sun    2    0 hrs, 7 mins, 58 secs    99.447%
Summary:    2    0 hrs, 7 mins, 58 secs    99.921%

Device:    Sirocco
Protocol:    HTTPb
URL:    http://207.210.68.50
Frequency:    Every 5 Minutes
Status:    UP Since 2011-06-15 22:00:02
Date    Outages    Downtime    Uptime%
2011-06-13 Mon    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-14 Tue    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-15 Wed    1    0 hrs, 0 mins, 47 secs    99.946%
2011-06-16 Thu    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-17 Fri    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-18 Sat    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-19 Sun    0    0 hrs, 0 mins, 0 secs    100.000%
Summary:    1    0 hrs, 0 mins, 47 secs    99.992%

Device:    Avalanche
Protocol:    HTTPb
URL:    http://67.225.235.5
Frequency:    Every 5 Minutes
Status:    UP Since 2011-06-13 22:04:13
Date    Outages    Downtime    Uptime%
2011-06-13 Mon    1    0 hrs, 4 mins, 39 secs    99.677%
2011-06-14 Tue    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-15 Wed    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-16 Thu    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-17 Fri    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-18 Sat    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-19 Sun    0    0 hrs, 0 mins, 0 secs    100.000%
Summary:    1    0 hrs, 4 mins, 39 secs    99.954%

Device:    Stormsurge
Protocol:    HTTPb
URL:    http://50.28.10.47
Frequency:    Every 5 Minutes
Status:    UP Since 2011-02-04 19:33:40
Date    Outages    Downtime    Uptime%
2011-06-13 Mon    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-14 Tue    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-15 Wed    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-16 Thu    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-17 Fri    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-18 Sat    0    0 hrs, 0 mins, 0 secs    100.000%
2011-06-19 Sun    0    0 hrs, 0 mins, 0 secs    100.000%
Summary:    0    0 hrs, 0 mins, 0 secs    100.000%