May 6, 2013 :: Emergency maintenance on Monsoon

Discussion started on Server Updates & Outages

Monsoon experienced two outages early this morning.  As a result, we need to do some maintenance (run a file system check) which will require downtime.  Typically the range is roughly 30 minutes to maybe an hour.

Due to the impact, we will likely be starting this shortly vs. scheduling it for later in the day/evening.

#1 - May 06, 2013, 08:32:10 AM

The maintenance was completed earlier but the filesystem has locked up again.

We're evaluating the primary harddrive and may need to do an emergency replacement.

I will keep this thread updated as I get further information.
#2 - May 06, 2013, 12:06:37 PM

The second file system check is complete and sites should be functioning normally again for now.

The datacenter is investigating the primary drive to determine whether we start a replacement process.
#3 - May 06, 2013, 01:12:14 PM

Unfortunately the tests are showing the primary hard drive is failing.  We will begin the replacement process shortly.  This will involve about 2-4 hours to replace the harddrive and get the server reconfigured.  From there we'll start restoring accounts until complete. 

I will keep this thread updated as this process unfolds.

Thank you,
#4 - May 06, 2013, 02:54:02 PM

Any progress?
#5 - May 06, 2013, 08:07:59 PM

The new drive is in place and configured.  The first account was just restored and looks good so it's a matter of us running through all the restores now. 

Thank you for your patience -- I know it's been a rough start to the week.
#6 - May 06, 2013, 08:11:26 PM

No problem.  It's not snowing in Minneapolis, so it's a good day.  It's been like 3 days now !
#7 - May 06, 2013, 08:31:29 PM

We're flying through restores now so with any luck all of them should be complete within the hour.
#8 - May 06, 2013, 08:35:42 PM

All accounts were restored as of about 50 minutes ago.  I've gone through all of them and from what I can see they appear to be good.

If you're having an issue with your site or anything related, please feel free to email me or open a ticket.

Thank you!

#9 - May 06, 2013, 10:00:14 PM

Thank you for keeping us informed, Jason.   :)
#10 - May 06, 2013, 10:20:06 PM

Thank you for keeping us informed, Jason.   :)

Absolutely!  :)
#11 - May 06, 2013, 10:32:58 PM


0 Members and 1 Guest are viewing this topic.