May 6, 2013 :: Emergency maintenance on Monsoon

Started by Jason, May 06, 2013, 08:32:10 AM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Jason

Monsoon experienced two outages early this morning.  As a result, we need to do some maintenance (run a file system check) which will require downtime.  Typically the range is roughly 30 minutes to maybe an hour.

Due to the impact, we will likely be starting this shortly vs. scheduling it for later in the day/evening.

Regards,
Jason

Jason

The maintenance was completed earlier but the filesystem has locked up again.

We're evaluating the primary harddrive and may need to do an emergency replacement.

I will keep this thread updated as I get further information.

Jason

The second file system check is complete and sites should be functioning normally again for now.

The datacenter is investigating the primary drive to determine whether we start a replacement process.

Jason

Unfortunately the tests are showing the primary hard drive is failing.  We will begin the replacement process shortly.  This will involve about 2-4 hours to replace the harddrive and get the server reconfigured.  From there we'll start restoring accounts until complete. 

I will keep this thread updated as this process unfolds.

Thank you,
Jason


Jason

The new drive is in place and configured.  The first account was just restored and looks good so it's a matter of us running through all the restores now. 

Thank you for your patience -- I know it's been a rough start to the week.

itsjustme

No problem.  It's not snowing in Minneapolis, so it's a good day.  It's been like 3 days now !

Jason

We're flying through restores now so with any luck all of them should be complete within the hour.

Jason

All accounts were restored as of about 50 minutes ago.  I've gone through all of them and from what I can see they appear to be good.

If you're having an issue with your site or anything related, please feel free to email me or open a ticket.

Thank you!

Regards,
Jason

Ann

Thank you for keeping us informed, Jason.   :)

Jason