Server Status

Update #2: June 24 @ 12:15am Eastern:  Lightning is online again as of approx midnight.  We'll be monitoring the maintenance window to close in a few hours but the impact to Lightning should be complete.

Update #1: June 23 @ 10:35pm Eastern:  This maintenance is in progress.  Lightning is currently unavailable. 

---------

Network Engineers will be performing maintenance within our datacenter that will likely result in a brief period of downtime for our server "Lightning."

The scheduled maintenance window (4 hours) is this Friday, June 23 from 10pm until 2am Eastern (New York) on Saturday, June 24.

Lightning will likely become unavailable at some point during this timeframe although the experienced downtime should be limited to 5-10 minutes.

Customers with hosting accounts on Lightning will be emailed a link to this notice on June 20th to share awareness.

This post will be removed once the maintenance is complete or updated as necessary.

 

UPDATE: 4/01/2023 @ 11pm:  Maintenance has completed with no experienced outage.

---------

Network Engineers will be performing hardware maintenance on a subset of network switches within our datacenter that may result in downtime for our server "Vortex."

The scheduled maintenance window is this Saturday, April 1 between 10pm - 11pm Eastern(New York) time.

Vortex will likely become unavailable at some point during this timeframe although the experienced downtime generally is much less than the scheduled window.

This post will be removed once the maintenance is complete.

Update: These reboots have successfully completed.  Pingdom monitored 1 minute of downtime for each server.

---------

We will be rebooting the following four servers tomorrow (Saturday, March 18, 2023) around 6:30am Eastern/New York time.

• Lightning
• Seabreeze
• Tornado
• Vortex

Expected downtime should be under 10 minutes for each.  Normally this action completes in 5 minutes or less.

Sites/services will be unavailable during the reboot activity.

This post will be updated if needed and closed upon completion.

Thank you

Since Tuesday around 4pm Eastern (New York) time, server Seabreeze has been experiencing occasional high loads resulting in some sporadic website outages due to automatic service restarts.  The server restarts the services automatically when they fail which is usually about a 10 minute process.

We're narrowing down the cause and hope to have it addressed today.

This post will be updated (and ultimately closed) once resolved.

Update March 10 @ 9:15am Eastern:
Multiple adjustments were made yesterday morning that have improved the load issue.  There haven't been any additional service outages since then.  This appears to have resolved most of the issue but we're still monitoring a few things before we declare this fully resolved.

There is scheduled maintenance for server "Downpour" starting Wednesday, November 30 at approximately 9pm Eastern (New York) time. 

The window will run for 6 hours until 3am, December 1. 

Visible downtime to Downpour should be limited to 30 minutes or less.

This posting will be updated if further information needs to be shared.

Please be aware that our datacenter has scheduled maintenance that will require a reboot of server "Lightning."

The window is scheduled to begin at approximately:  November 2nd, 2022, at 09:00 PM (EDT). 

The maintenance window is schedule for several hours but the reboot itself should cause 5-10 minutes of outage (potentially less) at some point close to the first hour.

This post will be updated if any new updates are needed.

FINAL UPDATE: The issue for customers opening their Acronis backups via cPanel is now restored.

 

UPDATE, 7:10am Eastern:  The reboot was completed as scheduled with only a few minutes (if that) of impact.  Pingdom detected no downtime.   The Acronis plugin is now launching properly but we're investigating an error message when you try to browse into a backup.  I'll update this post once we learn more.

-----------

We've scheduled a reboot of server "Tornado" to happen at approximately 6am Eastern (5am Central) US time on Sunday, September 25.

Typically, the downtime associated with this type of activity should just be a matter of minutes unless the system starts a file system check upon restart.

This reboot is to activate a fix to the integration between cPanel and Acronis (our cloud backup storage system).  Acronis is running backups as expected however the integration that allows customers to view their own backups to run restores, etc., is currently not working from within cPanel.

I will update/remove this post upon completion.

We will be rebooting the two servers above at approximately 3am Eastern (New York) time on Tuesday, May 24, 2022.

This is to enable some recent updates we've made as part of a new backup system we're deploying.

Typically a reboot only takes a few minutes to complete but these servers haven't been rebooted in some time (over 600+ days) so they may run some file system checks that could take longer to complete.  It is expected the outage should be less than 30 minutes and possibly as little as several minutes.

This posting will be removed once the reboots are complete.

Our datacenter will be performing maintenance that has the potential to cause service disruption.  Usually these types of exercises don't cause downtime but it's possible:

"Starting on March 4th, 2022, at 10:30PM (EST), our Network Team will be performing maintenance on the network and switches that service your [servers]. This maintenance does have the possibility to cause traffic latency or brief packet loss, but any service impacts should be minimal. Even so, we are sending out this notification out of an abundance of caution and transparency. This maintenance window will remain open for one and a half hours, until March 5th, 2022 at 12:00 AM (EST). This will not require any action on your part.

Although this maintenance does have the possibility of causing minimal service impacts, this maintenance is necessary to improve network functionality in order to provide a more stable platform going forward."

Update:  Nov 10, 6:30pm Eastern/New York:  We believe the issue has been resolved as of approximately 1.25 hours back but we'll be monitoring it overnight to confirm.

---------

Since yesterday afternoon, server Seabreeze has been experiencing issues causing it to stop and restart the httpd (web) service.  When this happens, websites are unavailable for approx 10-15 minutes from the time the server stops it to when it restarts it again.

We're actively investigating the cause and hope to have it resolved as soon as possible.

Thank you

We will be upgrading server "Vortex" on the morning of Tuesday, March 2nd at approximately 3am Eastern (New York) time.

We anticipate a downtime of Charlottezweb.com (including the website and Client Area) for possibly up to 15 minutes.

Customers who do not have hosting accounts on server Vortex will not be impacted.

Server Seabreeze is currently experiencing a service outage.  Datacenter technicians are currently working to resolve.

I will post more information as I receive it.

Update 4:15am Eastern:  The server is responsive again although loads are high.  Technicians will continue working to isolate the issue.

Update: Dec 1, 2020:  The issue has been isolated to a security software we subscribe to for our servers.  It was unfortunately causing incorrect blocks against certain php-fpm functions.  The developers of that software have a fix that I've applied to our servers.  If you are still facing issues, please open a ticket.  Thank you for your patience!

--------------------------

Update: Nov 30, 2020:  The issue appears to be related to php-fpm.  I have turned off php-fpm for the customers that have reported the problem so far.  If you have issues and haven't reached out, please do so. 

What is php-fpm?  It stands for "PHP - FastCGI Process Manager" -- Over-simplified, it's a software process that allows PHP (the software language used by most website software these days) to run faster and more efficiently.

Does disabling php-fpm hurt my site?  Not necessarily.  If you have a very busy site, it will help with efficiency but is not required if the server has capacity to accommodate.

--------------------------

On server Lightning, some customers are experiencing error pages sporadically that look like:

--------------------------

Service Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

Additionally, a 503 Service Unavailable error was encountered while trying to use an ErrorDocument to handle the request.

--------------------------

This is not a server capacity issue as it states but appears to be related to some security rules that are causing php-fpm to fault.  These rules appear to have been pushed to servers on Monday/Tuesday of this week.

I will explain further once we have more specifics but the good news is that we appear to have narrowed down the cause as of 5pm CT Nov 28.

For now, refreshing is resolving but I can certainly appreciate that this is quite a nuisance and is service-impacting.

We're working to resolve this completely as soon as possible.

Update:  This upgrade is complete. 

We will be performing software upgrades on our website and Client Area for Charlottezweb.com on Wednesday, November 25, 2020.

All clients have been informed via email.

Client websites will not be impacted but Charlottezweb.com will be unavailable for a short period during the maintenance window.  This should be less than an hour at some point between 7am-11am Eastern (New York).

Update: This upgrade was compleated at approximately 9am Eastern.

We will be upgrading MySQL on server Lightning.charlottezweb.com at this time:

Friday, October 23 at 7am Eastern (New York)

Customers have been emailed a notification of this maintenance window.

If all goes as expected, the MySQL service may be down for as long as 30 minutes but generally it takes 15 minutes or less.

This posting will be updated in the event of anything unexpected.

Regards,
Jason

Update: This work was completed at approx 8:10am Eastern. 

We will be upgrading MySQL on server Vortex.charlottezweb.com at this time:

Thursday, October 22 at 7am Eastern (New York)

Customers have been emailed a notification of this maintenance window.

If all goes as expected, the MySQL service may be down for as long as 30 minutes but generally it takes 15 minutes or less.

This posting will be updated in the event of anything unexpected.

Regards,
Jason

We have a maintenance window that will run for approximately four hours overnight on Monday (Eastern timezone).

It will begin on Monday, June 22, 2020 at approximately 11:00 PM (EDT) and should end at approximately 3:00 AM (EDT) on Tuesday, June 23, 2020.

During this timeframe, there is expected to be a short outage for two of our servers:

Downpour
Vortex

Although the maintenance window will remain open for four hours, these servers should only be down for a short time during this window. The expected down time should be 30 minutes or less.

Vortex serves several clients (including Charlottezweb.com, itself).  This means that our website will be offline as part of the outage.

A notice of this outage will be emailed to all customers for awareness.

We are currently investigating slowness issues on server Tornado.  The server is up and most pages do eventually load but incredibly slow with some timeouts.

This posting will be updated as we learn more.

We're currently working on the httpd service on server Tornado.

The server is up but web sites services are repeatedly restarting.

We should have info to share shortly.

 

Charlottezweb is launching a new website design during the next 36 hours. 

Customers have been emailed to advise in advance.

Details can be found here:  https://www.charlottezweb.com/forums/index.php?topic=2153

Update: The migration is complete and customers have been informed. Please open a ticket if you are facing any issues.  Thank you!

We will be patching and rebooting our servers over the next three nights to address a few known security vulnerabilities.

No action is required on your end though you may see several minutes of downtime when the reboots occur.

The maintenance windows are from 11pm - 7am Eastern (New York) time over the next three nights. 

Each server will only be rebooted one time over the next three nights, not each night. 

If you face an issue after or have any questions, please open a ticket or email me and I'll respond as quickly as possible.

Update Aug 18, 11am Eastern:  All reboots have been completed as of this morning.  The longest downtime experienced was 6 minutes.

This is a placeholder posting for our migration of server Landslide to new server Tornado on June 25.

Customers on this server have been emailed with details on the move. 

If we encounter any issues, this posting will be updated with the latest information.

Update:  9pm Eastern - We're starting the final data sync now.

Update: 9:30pm Eastern -- The move is complete.  We're finalizing dns changes. 

This is a placeholder posting for our migration of server Thunderbolt to new server Seabreeze on June 23.

Customers on this server have been emailed with details on the move. 

If we encounter any issues, this posting will be updated with the latest information.

Update 7:25pm Eastern :  The final file synchronization is in progress.

Update 7:40pm Eastern:   The move is complete. DNS may take 10-15 minutes to resolve. Please email or open a ticket if you are facing any issues.

Maintenance has been scheduled for server Thunderbolt for next Friday, June 14 at approximately 2am Eastern (New York) time.

There is an expected outage to all services on Thunderbolt in the range of 30 to 45minutes as it requires powering down the box to replace one of the hard drives.

I will email a link to this notice to all customers with accounts on Thunderbolt.

Update:  Maintenance was completed in just shy of 14 minutes.

We're currently addressing an issue with server Thunderbolt that has been causing very high loads since last evening.  There may be some web service downtime associated with this work.  I'll update this posting if we're not able to correct it quickly.

Regards,
Jason

For server "Thunderbolt" - We will be conducting maintenance at 6am Eastern (New York) on Wednesday, November 21, 2018.

One of the hard drives in our RAID storage array has failed and needs to be replaced.

The downtime may take as long as an hour, although hopefully it will be in the range of 15-30 minutes.

Update: This maintenance was completed around 6:20am Eastern today.  Pingdom monitored the downtime at approx 19 minutes.

To further protect against the Spectre and Meltdown vulnerabilities, we will be rebooting all of our servers to activate new security patches.

The maintenance timeframes when this will happen is during the overnight North American hours starting Thursday through Sunday morning this weekend.  (Oct 25-27).

Typically a reboot completes within just a few minutes (sometimes less than one) and is often unnoticeable.

We will update this status posting as reboots are completed.

As of Saturday morning, October 27, all of our servers except for one have completed their reboots.

If you have any questions, please email or submit a support ticket.  Thank you! 

We're going to quickly reboot servers Landslide and Thunderbolt to activate kernel updates.

Outage time for this is typically a couple of minutes.

I will update this post if we encounter any issues.

Thank you,
Jason

MySQL and PHP on server "Landslide" will be upgraded on Sunday, June 10, 2018 at 10am US Eastern (9am US Central).

This post will be kept current if any issues arise during this process. 

UPDATE 11:50am Eastern:  This upgrade is complete.  MultiPHP is now available in cPanel.  Please contact me (or open a ticket) if you notice any issues with your site(s).

Regards,
Jason

The following servers are planned to be rebooted tonight as a follow-up to the recent Meltdown/Spectre security vulnerabilities.

Landslide, Lightning, Thunderbolt and Vortex

The maintenance window is scheduled from:  Thursday, February 1st, from 10:00 PM EST to Friday, February 2nd 08:00 AM EST

Update:  January 10, 2018:
Servers Thunderbolt and Landslide were rebooted back on January 6th per the update below.  We still have several remaining servers and I received an update from the datacenter today that there will be an additional maintenance window for Wednesday, January 10th, from 10:00 PM EST to Thursday, January 11th at 09:00 AM EST.  Our other servers should be rebooted during that time if all goes to plan.

Update:  January 11, 2018:
Servers Vortex and Lightning were completed today.  We have one more remaining.

---
Original Post:
All of our servers will be rebooted at some point during the next 72 hours. Due to the number of servers the datacenter has in scope, we are unfortunately unable to provide the exact timing.

These reboots are required to patch against the Spectre/Meltdown vulnerabilities you may have read about online:

http://money.cnn.com/2018/01/03/technology/computer-chip-flaw-security/index.html

https://meltdownattack.com


Update:  4:50am Eastern, 1/6/2018
So far servers Thunderbolt and Landslide have been rebooted.  I will update this as more are completed.

Update:  8:00am Eastern, 1/8/2018
We are still awaiting a few more reboots.  I will update this once completed.

Some customers on server Thunderbolt have received failure notices the last two nights as it relates to their cPanel-provisioned SSL certificates.

We're currently investigating this.

This issue is not impacting certificates that are already active.  What we're seeing is failures to validate the free cPanel certificates that are set to expire in late December.

There's no immediate impact in other words, however we're working to resolve this asap.

Further updates will be posted here.

Update: 3:15pm Eastern, 11/16/2017:  We're finalizing next steps and should have more to share shortly.
Update: 9:30pm Eastern, 11/17/2017:  We're currently upgrading some software on the server.  There may be short disruptions of service. 
Update: 10:45pm Eastern, 11/17/2017:  We've upgraded Apache on the server and will be monitoring throughout the weekend to see if additional adjustments are needed.
Update: 9:10pm Eastern, 11/20/2017: It appears that the issues have been resolved per our previous update however a reboot is needed to fully activate everything we've done.  We will do this in about an hour.  Several minutes of downtime may be experienced during the reboot.  Once that's completed I will update/close this posting.
Update: 10:05pm Eastern, 11/20/2017:  The reboot is complete - Downtime was recorded at 2 minutes per Pingdom.  I'll leave this post open while I check with customers to see if all is well.

Thank you,
Jason

We will be rebooting server Lightning at approx 9:30pm Eastern (New York) time tonight (Saturday) to enable some cPanel updates.

This should hopefully only involve a few minutes of outage -- perhaps less.

We've scheduled a reboot for server Landslide to happen at approximately 7:30am Eastern time on Saturday, October 21. 

This reboot is to enable some recent updates.

Typically this should result in a brief service disruption -- typically 5 minutes or less.

We're currently investigating an issue on server Lightning that's causing some sites to not load. 

I should have an update shortly.

As part of recent cPanel upgrades, we need to reboot all our servers.

At 7am Eastern (approximately) on Friday, September 1 (timezone converter), we will be rebooting the following servers:

Downpour
Landslide
Lightning
Thunderbolt

The expected downtime is typically under 5 minutes.

This status posting will be updated throughout that process and/or if any issues arise.

-----
Update: 7:15am Eastern

All servers have been rebooted.

We're troubleshooting an issue with Lightning that's causing sites not to load yet.  I hope to have an update on that in the next few minutes.

-----
Update: 7:20am Eastern

The issue with Lightning has been addressed.  Thank you

As part of recent cPanel upgrades, we will be rebooting all our servers in the coming days.

We will start with Vortex at 7:30am Eastern time on Thursday, July 31.

Vortex hosts Charlottezweb.com so this means this site will be offline briefly at that time.

The datacenter is currently investigating high loads that started on server Lightning late this morning (Eastern time). 

I will update this posting once more is known if we don't have it sorted shortly.

We're currently investigating some service issues on server Thunderbolt.

So far I've received notification of at least one MySQL site not loading properly (whereas others I've tested are not impacted).

I've also received reports of email disruptions.

This post will be updated as we learn more.

Update: 12:30pm Eastern:  I wanted to advise that we are still actively investigating this issue with the datacenter.  It's top priority and I'll update as details progress.

Update: 2:30pm Eastern:  From initial reports, it looks like the email issues are now resolved.  If you are still experiencing email-related issues on Thunderbolt, please reach out.  We do still have one site I'm aware of that's having a MySQL issue.  There may also be SSH issues.  We're continuing to investigate those items.

Update:  3:45pm Eastern:  While I was monitoring, server load spiked dramatically and has taken down most services.  We're looking at this now and may need to perform a reboot.

Update:  3:55pm Eastern:  We're initiating a reboot as the server is unresponsive.

Update:  4:50pm Eastern:  Apologies for the delayed response in posting this.  The reboot from approximately an hour ago took less than a minute to complete and so far it appears all services are back to normal operation.  We're continuing our investigation into the cause and anything further we need to do.  If you still are facing any issues, please contact us.

Thank you for your patience!

Update:  8:00pm Eastern:  We're not aware of any issues at this time.  If you have any issues, please contact us.  Thank you!

We're going to be performing a PHP upgrade on server Lightning.  There may be a short outage to the web service as we complete the process.

Regards,

Jason

Lightning experienced a failure of its http (web) service that lasted approx 45minutes.  The server was online the whole time but sites weren't loading. 

We're currently investigating the cause however we did have to suspend one account temporarily to allow the load to drop to where we could repair a few mysql databases that were compounding the problem.  It looks like we're stable as of that activity however I hope to get more information on the root cause.

We're investigating an issue with http / mysql on server Lightning.  I will update this post as soon as it's resolved or we have more information.
---
Update: This was resolved at approximately 7:04pm Eastern. This seems to be very similar (if not the same) issue that happened earlier this week on Monday.  We're looking for root cause now.  I will post to our forums once I have more information.

Update March 6, 2017:  A forum posting with additional details has been posted here:  https://www.charlottezweb.com/forums/index.php?topic=2113.0

As with earlier this week, if you're seeing any errors on your site on this server, running a database repair in cPanel will typically resolve it.  If not or if you'd like assistance, please email me or open a ticket.  I've created an article here on how to run a database repair in cPanel.

Thank you,
Jason

We're working on a mysql (database) issue on server Lightning currently.  This started after an http (web service) outage.

I will post an update once we know more.

Update: This was resolved at approximately 2:30pm Eastern however we're looking into the root cause.  If you're seeing any errors on your site on this server, running a database repair in cPanel will typically resolve it.  If not or if you'd like assistance, please email me or open a ticket.  I've created an article here on how to run a database repair in cPanel.

Thank you!

Regards,

Jason

We will be upgrading PHP on server Landslide on Friday around 6am Eastern time.  This may result in a brief outage of php service to activate once complete.

The retirement of server "Blackice" is complete as of approximately 8:10pm Eastern, November 2.

Please visit this post for all details including the latest updates:

http://www.charlottezweb.com/forums/index.php?topic=2089

Please see the following post regarding a scheduled upgrade to server Lightning.

http://www.charlottezweb.com/forums/index.php?topic=2093.0

Please see the following post regarding a scheduled upgrade to server Vortex.

http://www.charlottezweb.com/forums/index.php?topic=2092.0

Note that Charlottezweb.com is on this server and will experience two outages on Saturday as a result of this planned maintenance.

Update: This was completed per schedule.

Reminder that server Wildfire will be shut down on Thursday, September 15, 2016 at approx 10pm Eastern.

This is related to the server migrations that are discussed here:

http://www.charlottezweb.com/forums/index.php?topic=2089

If you have an account on this server and manage your own domain name(s), your action is required as discussed in the post above.

CIRA.ca  (the agency that controls .ca domains) has been stepping up compliance audits lately. 

If you use a .ca domain for your site and you find that your site isn't loading, please open a ticket immediately so we can investigate.

UPDATE:  Charlottezweb's maintenance should now be complete.  If you spot any errors or issues, please contact us.
-------------

We are replacing server "Thundersnow" over the weekend of May 6-8, 2016 via a migration to a new server:  "Vortex."

Charlottezweb.com is hosted on this server and will be impacted by this migration.

I'm targeting a window of 4pm-6pm Eastern (New York) time on Saturday, May 7 to do the migration of Charlottezweb.com. 

What does this mean?
During this time, you may see our forums and the Client Area in a "locked down" maintenance mode.  This is to prevent the loss of data during transfer.

Please see this post on our forum regarding two upcoming datacenter maintenance windows.

http://www.charlottezweb.com/forums/index.php?topic=2074

Many of our servers will be rebooted tonight starting around 10pm Eastern.  This is a semi-scheduled event however I have purposely not posted advanced notice.

This is related to a largescale security issue that has been exposed in the past few days.  I will share more details on our forum once our reboots are complete and our servers are secure.

Thank you for your understanding of your security.

Regards,

Jason

Our datacenter is currently working on server "Blackice" due to an unplanned emergency maintenance issue that arose.

They expect the maintenance to be brief however I don't I have specific timeframes yet.

I will update this posting as soon as more information is available.

Regards,

Jason

Server "Wildfire" will be upgraded tonight to a more powerful configuration.

The upgrade will involve two periods of service disruption.  The first is complete.

Please review the following thread on our forums for full details:

http://www.charlottezweb.com/forums/index.php?topic=2072.0

Update 12pm Eastern, Feb 6:  The upgrade is still in progress.  I will update the forum post above once complete.
Update 2:48pm Eastern, Feb 6:  The server appears to be in the final stages of the resize and is currently offline while this completes.
Update 3:49pm Eastern, Feb 6:  The server is in the final stage of resizing its partioning.  Unfortunately there's not a way of knowning how much time is remaining at this phase of the process.  I will keep monitoring it for completion. 
Update 4:30pm Eastern, Feb 6:  The resize has completed as of 4:21pm Eastern.

Thank you,
Jason

The datacenter is working on a some scheduled network maintenance that has been causing intermittant network connectivity issues over the last few hours for two of our servers:

Landslide
Thunderbolt

These servers have been unreachable at times starting when the maintenance began approximately 4 hours ago.

The latest update is that they expect the maintenance to be complete within the next hour at which point full redundancy should be in place again.

Thank you for your patience

Landslide required emergency datacenter maintenance last night which caused approximately one hour and 45 minutes of complete downtime.  When the server was brough back online, MySQL did not properly recover and required additional maintenance as well as the repair of a number of databases that corrupted when the server went down.  Unfortunately the server did not send any MySQL alerts which led to an extended period of outage to sites that utilize MySQL.  This was restored in the past 45 minutes and we'll be investigating today why the notices were not sent and what we can do to better prepare for a scenario like this going forward.

Update:  This work has been completed.  As PHP/MySQL have been upgraded, please check your sites and scripts and contact Support if you have any issues.  Thank you!

We have scheduled upgrades to MySQL and PHP on server Thunderbolt to occur on Friday (October 30) between 6-8am Eastern.  During this time, there may be periods of disruption to MySQL and PHP.

I will update this posting once complete.

On Sunday, October 25, we upgraded Apache, MySQL and PHP on server Wildfire. 

On Wednesday, October 28, we will be doing another upgrade of both MySQL and PHP. 
 
The reason for two updates in the same week is that we're incrementally stepping up (version-wise) so that we can test in between vs. making a significant jump between versions.

I expect any interruption experienced to be similar to Sunday's which was a few minutes to restart services once we have the upgrades complete.

I will update this thread as we proceed.  Thank you!

Update:  This work has been completed.  As PHP has been upgraded, please check your sites and scripts and contact support if you have any issues.  Thank you!

We have scheduled upgrades to MySQL and PHP on server Wildfire to occur on Sunday between 5:30-7:30am Eastern.  During this time, there may be periods of disruption to MySQL and PHP.

I will update this posting as we proceed.

As an update -- All servers are currently functioning normally but we have experienced issues with 7 of them over the past two days that have caused anywhere from 30 minutes to a few hours of outage at a time.  One server (Wildfire) has seen significant downtime today related to this.

We believe the issue has been isolated to a known bug with the most recent release of cPanel.  We're implementing a workaround to address the issue until they officially release a fix.

These are the servers that were impacted to varying degrees either yesterday, today or both:

Avalanche, Blackice, Deluge, Heatwave, Sandstorm, Thundersnow, Wildfire

Thank you for your patience.

Our datacenter is currently working on some issues causing very high loads to several of our servers.  I've been told it may take them upwards of an hour to have it addressed.  I will post more information as it's available but this is impacting servers Avalanche, Blackice and Wildfire as I type this.

Update: mitigation is in place and services have been restored. Wildfire is still experiencing higher loads than normal but is stabilizing.

We're currently investigating high loads impacting servers Avalanche and Wildfire.  This is causing sites to be unresponsive at this time.

We will be rebooting all servers over the next hour for a security patch implementation.  Details will be posted following completion.

Update: All servers are complete except for Wildfire which we're working on now.

Regards,

Jason

Two DDoS attacks against servers Blackice and Landslide resumed this morning.  We'll post more information as it becomes available.

Almost all of our servers were rebooted this morning (around 7:30am Eastern) to enable a security patch critical for securing our environment. 

The reboots all completed successfully and quickly.  Pingdom reported each server outage to be 1-2 minutes.

Please visit this posting on our forum for full details.

Thank you

We're starting some maintenance to the web service (Apache) on Landslide.  This will cause loads to be potentially elevated for the next 30 minutes (approximately) followed by a restart of the service.  This may result in under 5 minutes of outage.

We're doing this to address what appears to be a bug that's caused some service interruptions over the past week.

This maintenance was successfully completed this morning. 

Please see the following post regarding scheduled outages that will impact the following servers on Tuesday, March 3:

http://www.charlottezweb.com/forums/index.php?topic=2033.0

(This was originally scheduled for Monday, March 2 but the datacenter postponed it).

Avalanche
Blackice
Deluge
Heatwave
Sandstorm
Thundersnow
Wildfire

We will be rebooting all servers over the course of the next 24 hours. Details will be shared after the reboots are complete.

We are currently investigating high loads impacting server Landslide. Customers may experience page timeouts and errors while this occurs.

Sandstorm is currently experiencing high loads due to an account restore. We're presently monitoring this and expect loads to drop once complete. The loads are causing slow site load times and some interruption.

Please visit this thread for information on two outages required for server Wildfire that will occur on Friday, December 5, 2014.

http://www.charlottezweb.com/forums/index.php?topic=2024.0

Reminder that servers Jetstream and Monsoon will be retired (wiped and shutdown) later today.

Please visit the forum for details:

http://www.charlottezweb.com/forums/index.php?topic=2002.0

Update 1 - we've been advised these reboots will occur early morning October 1 between 1am - 8am Eastern
Update 2 - All maintenance is now complete.

We've received notice of some software maintenance occurring at one of our datacenters that will require reboots of many of our servers.

Impacted servers will include:

• Avalanche - COMPLETE
• Blackice - COMPLETE
• Deluge - COMPLETE
• Heatwave - COMPLETE
• Sandstorm- COMPLETE
• Thundersnow - COMPLETE
• Wildfire - COMPLETE

Please see the following post on our forum for details around scheduled downtime for server Thundersnow this coming Saturday (August 30) at 1am Eastern.

This maintenance will mean Charlottezweb.com's website will be down.

http://www.charlottezweb.com/forums/index.php?topic=1995

We've scheduled reboots to occur at approximately 5am Eastern on Saturday, August 23 for the following servers:

Jetstream
Monsoon
Sirocco

Details will be posted here:

http://www.charlottezweb.com/forums/index.php?topic=1992.0

Thank you

Loads are very high on server Jetstream currently causing services to restart in response.

This is causing an interruption in web pages at this time.

We're currently investigating.

If you are looking for information on service disruptions experienced today, please see the following post on our forum:

http://www.charlottezweb.com/forums/index.php?topic=1992.0

The datacenter is working on a segment of their network where server "Thunderbolt" is located.   The server is up and running but is appearing inaccessible at times due to this issue.  I will post an update as soon as I hear further from them.

Please see this post regarding scheduled datacenter network maintenace that will potentially cause service interruptions for server "Thunderbolt" early morning Sunday, August 3, 2014.

http://www.charlottezweb.com/forums/index.php?topic=1990.0

Thank you

We will be rebooting the following servers this coming Saturday, July 19th approximately between the hour of 4 to 5am Eastern (New York time) to implement a security update:

Avalanche
Blackice
Deluge
Heatwave
Sandstorm
Thunderbolt
Thundersnow
Wildfire 

Please visit this thread on our forum regarding a scheduled maintenance window this coming Sunday morning for server Sandstorm.  There will be an outage to all of Sandstorm's services for approximately 30 minutes at that time:

http://www.charlottezweb.com/forums/index.php?topic=1984.0

Please refer to this forum thread regarding scheduled MySQL maintenance for server "Avalanche."

http://www.charlottezweb.com/forums/index.php?topic=1983.0

Please refer to this forum post regarding emergency datacenter (upstream) network maintenance that may cause latency for up to an hour overnight:

http://www.charlottezweb.com/forums/index.php?topic=1982.0

Potential impact is limited to servers:  Jetstream, Monsoon and Sirocco.

Our server Wildfire will be upgraded tomorrow starting at 5am Eastern.

Update 6/20 at 8:30am Eastern:  The resize attempt failed this morning.  We'll reschedule once the root cause is corrected.

Please visit this forum post for updates as they arrive:

http://www.charlottezweb.com/forums/index.php?topic=1977

Thank you.

Please see this thread regarding a very brief service disruption scheduled for 5am Eastern Friday, June 20.

http://www.charlottezweb.com/forums/index.php?topic=1978.0

Thank you

There were several disruptions of service to server "Thunderbolt" today due to emergency network maintenance done by the datacenter.

I've just received an update that it's possible there could be further outages overnight as they continue work on this.

"Additional complications have prevented us from completing the network maintenance .... that was originally planned to be completed in the early morning of June 18th. We are moving forward with a solution that will enable us to restore services to affected customers while vendor issues are resolved. Users losing service earlier today will see this improve although, until additional resources are restored, [service] will continue to run at a diminished capacity. This means customers with services in that Zone may continue to experience intermittent latency and connection issues. Fixing this is of critical importance to us and we are working diligently to ensure the service of each and every one of these customers is personally addressed. This will be updated with additional information as we continue to approach a comprehensive solution. Customers affected by these networking issues ... will also continue to receive emails with additional details as they develop."

We are seeing sporadic network instability today impacting server Thunderbolt.  This is due to a network issue our datacenter is working.  Here is their latest update that should end in the next 30 minutes:

We attempted to complete an update to significantly improve our network
performance late last night. Unfortunately, an unexpected issue arose in
Lansing Zone B due to additional complications with our most recent
maintenance. We are moving forward with an enhanced solution that will
return users in this section to peak performance, however this will
incur another brief round of latency and temporary loss of connectivity
for those involved. It is our aim to have this completed within the next
90 minutes by proceeding with this fix immediately.

Thunderbolt is has had some intermittant outages (two in the past hour and a half, approximately) related to some emergency network maintenance the datacenter is completing.   Their maintenance window ends at 6am so hopefully the interruptions have completed.

In the event we experience further issues to Thunderbolt within the hour, this could be related.

The current time is 5:24am Eastern.

Thank you for your patience.

An image (backup) process is running on all our servers right now however Wildfire is experiencing some issues with it causing high loads and unresponsiveness.

Update - a crash during the backup corrupted a database for one site that was causing it to consume server memory extremely quickly.   This should now be sorted.

We'll continue monitoring for the morning.

Thank you

For those impacted by the outage to server "Wildfire" earlier this morning, an event write-up has been posted here:

http://www.charlottezweb.com/forums/index.php?topic=1975.0

Wildfire is currently rebooting after hanging.

Update 1:50pm Eastern:  The datacenter is working on a network-related issue that seems to be tied into this.  The issue may not be with Wildfire directly but with the segment of the network it's on.  I will update this posting as details are made available.

Thank you,
Jason

Wildfire became unresponsive and is currently being rebooted.

Wildfire is being rebooted now.

Note:  This is now complete.  I'll leave this posting up for the rest of today.

There was an extended outage to server Avalanche on Friday, April 25 caused by multiple hardware failures on the parent server.  Whereas this has been addressed, we will be migrating Avalanche to a new parent server starting at 4am Eastern, Sunday, April 27, 2014.

For details of this planned maintenance, please visit:

http://www.charlottezweb.com/forums/index.php?topic=1967.0

For details around the extended outage on Friday, April 26, please visit:

http://www.charlottezweb.com/forums/index.php?topic=1968.0

A notification of the information above has been emailed to all customers with active accounts on Avalanche.

Initial Issue Description - This server initially became unresponsive and required a reboot just before 8am Eastern today.  The reboot completed but the startup required a file system check which lengthened the time to get it back online. 

Update 9:50am Eastern - the dc confirmed it is running a file system check after the reboot.  These can take some time to complete.  Given how long it's been already (approx 50 minutes since the reboot), it should complete fairly soon.

Update 10:17am Eastern - the file system check appears to have completed and services are starting to come up now.  Loads are very high due to the downtime and everyone trying to reconnect repeatedly so it will take some time for it to stabilize.

Update 10:35am Eastern - I see that the datacenter has initiated another reboot just now.  I'm awaiting an update from them on why that was initiated.

Update 10:45am Eastern - I've spoken to the datacenter.  The loads after reboot spiked so high that it became unresponsive again. After the current reboot we're going to temporarily block web traffic so they can get in and diagnose what's driving the loads.

Update 11am Eastern - The server is up however we're blocking all HTTP traffic while they investigate the load issue.  This means that your sites are still down but the server itself is up and running again.  I hope to have their update shortly.

Update 11:50am Eastern - The firewall has been updated to re-allow HTTP traffic again.  Loads are very high (but are dropping) due to the influx of inbound traffic.  Hopefully we will see stabilization soon.  I know that many sites are not loading due to the resources required to generate them.  This should correct itself as the server stabilizes.

Update 1:20pm Eastern - The datacenter is still actively working on this server.  Whereas it's technically up, it's essentially unable to serve sites due to high load.
 
Update 2:50pm Eastern - Unfortunately the server required another reboot due to load spike (unresponsiveness) and it's running an automated file system check again.  As mentioned before, a file system check can take anywhere from 15 minutes or so to much longer. Earlier today it took just over an hour before access was restored.  The tech onsite I'm working with is connected to the server so he'll know the moment that completes so he can login and attempt to isolate the load cause before it spikes again.  

Update 3:20pm Eastern - The datacenter is now aware of an issue with the parent server that Avalanche runs from.  They are performing emergency maintenance right now on it.  Hopefully that is the cause of what we've been experiencing today (vs. an issue with Avalanche itself).   We'll know if that's the case once that maintenance is complete.
 
Update 3:35pm Eastern - The datacenter is quite confident the issue we're seeing is due to the parent issue and not with Avalanche.  Once they complete their emergency maintenance (currently in progress) we'll know for sure.  They will also provide a root cause for us. 

Update 4:50pm Eastern - The datacenter maintenance on the parent server is still in progress.  I will update this once they've advised of completion. 
 
Update 6:20pm Eastern - I've been advised that they're currently rebuilding (cloning) the parent server.  If all goes as planned they're hoping to have that restored in the next 1-2 hours.
    
Update 9:25pm Eastern - Here is the present status as of right now.  The parent server suffered from a hard drive hardware failure that could not wait for a normal maintenance window or process to address.  That led to the need to do a complete rebuild/clone of the parent as I mentioned above.  Given the massive amounts of data involved in something like this, it's taking time to restore everything.  The parent server has a configuration that uses four hard drives.  The first drive has been restored.  At present, they are rebuilding the second drive and that should take approximately another hour.  When that drive is complete, there's a chance Avalanche will be part of the first and second drive space to where we could be back online while drives 3 and 4 rebuild in the background.  If we're part of the 3rd/4th drive space, however, we would remain down for the timeframe needed to rebuild drives 3 and 4.  That could be several more hours.  
 
To summarize - the parent server suffered a hardware problem related to storage and a rebuild of that is ongoing.
 
Update 9:45pm Eastern - Avalanche is back online.  I'm able to load a few sites but they're a little slow which is to be expected given traffic and load after being down all day.  I don't know if this means the maintenance that impacts us is complete or not.  I will attempt to confirm.
     
Update 8:10am Eastern, April 26 - Avalanche has remained up overnight while the datacenter continued their rebuild.  It was fortunate that we were setup to where we fell into the first half of the restore.  Whereas the parent is up and not under an immediate threat of disruption, they would like to migrate us to a new parent that's more stable.  We've had to do this in the past on a few of our other instances.  It's a normal proces they do when they need to upgrade or perform (non-emergency) maintenance on a parent.  This involves an initial brief outage (usually around 15 minutes) and then the server is back up and online while data transfers in the background to the new parent.  The transfer can take anywhere from a few hours to 10+.  During this time there's typically no performance issue experienced and sites will be online.  When the transfer is complete, a second outage occurs to sync up and finalize the move.   They would like to schedule this for 12am Monday morning however I'm looking into possibly doing it sometime late tonight so it completes over the weekend.  I don't want to impact another business day (especially a Monday) if avoidable.  I will confirm here once I have full details.  I will also be moving all of these updates into a thread on our forum and will send a summary email to everyone with an account on Avalanche.
      
Thank you for your ongoing patience.  Hardware-related issues like this are never pleasant.
 
Regards,
Jason

We will be upgrading PHP and MySQL on Thundersnow.

Please refer to this forum thread:

http://www.charlottezweb.com/forums/index.php?topic=1960.0

We will be upgrading PHP and MySQL on Avalanche.

Please refer to this forum thread:

http://www.charlottezweb.com/forums/index.php?topic=1959.0

We will be upgrading PHP and MySQL on Avalanche.

Please refer to this forum thread:

http://www.charlottezweb.com/forums/index.php?topic=1958.0

We will be performing a security update and rebooting server "Thunderbolt."

The reboot will likely occur within the hour however I made the decision to proceed now vs. scheduling this for off-hours due to the security issue involved.

Please visit this thread for details:

http://www.charlottezweb.com/forums/index.php?topic=1955.0

This maintenance is complete.   I will be posting a full write-up with details on what occurred later today in this post on our forum:

http://www.charlottezweb.com/forums/index.php?topic=1953.0

-------------

We're currently working on Tsunami.  PHP-driven sites are currently not loading related to an emergency software upgrade that was needed. 

Update 12:15am Eastern -- this is still being actively worked.  We hope to have resolution shortly.

Thank you for your patience.

Please see this forum posting about a planned outage (approx 30 minutes) this week for server "Tempest."

http://www.charlottezweb.com/forums/index.php?topic=1946.0

Sandstorm has crashed.  Before I reboot, I'm waiting on the datacenter to login and gather logs as this has happened a few times already this week.

Sandstorm is currently being rebooted after becoming unresponsive.

An upgrade of MySQL is scheduled for Blackice for this coming Sunday morning.  Please visit this thread on our forum for further details:

http://www.charlottezweb.com/forums/index.php?topic=1944.0

Servers Heatwave and Deluge became inaccessible around 10 minutes back.  I'm looking into a possible network/datacenter issue now.

Will update once details are known.

Thanks

Tsunami quit responding and is in the process of being rebooted.

Wildfire is currently running at very high loads due to a backup process.  It should stablilize shortly.

Update:  The server became unresponsive a few moments ago and is currently rebooting.

Sandstorm was experiencing this as well but has now stabilized.

Scheduled maintenance this coming Thursday (11:30pm Eastern) potentially impacting multiple servers.

Please visit this thread for details:

http://www.charlottezweb.com/forums/index.php?topic=1934.0

Deluge will be undergoing a server move starting at 4am, Sept 25.  There will be a reboot followed by normal operation (while data transfers) and then a second reboot later in the day to complete.   This is ONLY for our server "Deluge."

We're working on controlling high email-related loads on Tempest right now.  You may be seeing slow server performance as a result of the mail processing.

We've received notification of an emergency datacenter maintenance window that may impact any of the following servers for a period on Friday, September 6, 2013 starting at 10pm Eastern.

    Jetstream
    Monsoon
    Sirocco
    Supercell
    Tempest
    Tsunami

Please visit this thread on our forum for additional details:

http://www.charlottezweb.com/forums/index.php?topic=1915.0

Server "Tempest" is in the process of being rebooted.  

We're working on isolating high loads on Monsoon.  Mysql is currently not responding due to this.  Sites not using mysql are online but running slowly at this time.

Server: Sandstorm
Date:  Monday, June 10, 2013 -- starting at 11pm EST

There will be two planned outages to server "Sandstorm" as part of a scheduled maintenance this next Monday, June 10 starting at 11pm EST. 

Please see this forum post for full details:

http://www.charlottezweb.com/forums/index.php?topic=1897.0

Thanks,
Jason

Monsoon has been experiencing symptoms of primary hard drive failure today.  We're going to be starting emergency drive replacement shortly.

Please visit this thread for full details and updates:

http://www.charlottezweb.com/forums/index.php?topic=1894.0

We're running a quota fix on Jetstream which is driving high loads.  It should stabilize shortly.

On Tuesday, April 23 at 5am EST, we will be doing required MySQL maintenance on server "Supercell." 

This maintenance will require MySQL services to be stopped for approx 30-40 minutes.  Worst-case, we may be looking up to an hour though I don't expect it to take that long.

Any sites or scripts on Supercell that use MySQL databases for functionality will experience an outage during this time.

If you have any questions, please visit this thread:

http://www.charlottezweb.com/forums/index.php?topic=1890.0

Thank you,
Jason

One of our datacenters will be performing some network maintenance on Sunday, April 21th from approx 2am - 3am EST.

This may impact the following servers:

Jetstream
Monsoon
Sirocco
Supercell
Tempest
Tsunami

Window duration: 1 hours
Anticipated impact: 15 minutes

Description:
Upstream Network Maintenance
 
Impacted services:
Atlanta Gold and Silver network connections
 
Details of Maintenance:
Hardware upgrades will be carried out on equipment carrying connections for nLayer and some peering links, resulting in a short disruption of connectivity for traffic passing through these connections.  Some network instability and suboptimal routing may occur during the scheduled window as traffic is rerouted through other upstream providers.

Supercell is currently experiencing issues.  We're working with the datacenter now.

We're rebooting Avalanche right now. 

Update:  The reboot did not resolve.  We're working with the datacenter now.  Updates to follow as I get them.

Thanks,
Jason

Avalanche is currently experiencing high loads.  We're investigating this now.

Thank you,
Jason

Avalanche is being rebooted right now.

Scheduled PHP upgrade.  Please refer to this thread on our forums:

http://www.charlottezweb.com/forums/index.php?topic=1878.0

Thanks

The website we use for uptime stats provided by Pingdom is presently not loading.

Their main site is up as is my admin area but the "public-facing" stats page is not loading.

I've reported this issue to them to address.

There have been multiple reboots of server "Avalanche" in the past 24 hours.  Full detail are available on our forum here:

http://www.charlottezweb.com/forums/index.php?topic=1874.0

Regards,
Jason

Server "Avalanche" is currently in the progress of being rebooted.

I'm in the progress of migrating Charlottezweb.com (the website) to a newer server.  This does not impact any customer accounts -- this is only for Charlottezweb.com.

The Client Area (billing/support system) and Forum will be locked during the move.  If you need to contact support, please email support@charlottezweb.com.

Server "Monsoon" is currently being rebooted.  

There will be a brief PHP outage on Sunday as we upgrade the version on server Wildfire. 

Details are available here:  http://www.charlottezweb.com/forums/index.php?topic=1855.0

Planned maintenance on Sandstorm will require two brief outages.  Please view this thread on our forums for full details:

http://www.charlottezweb.com/forums/index.php?topic=1853.0

Avalanche became unresponsive -- We're rebooting it right now.

Reminder that our server Blizzard is being retired tomorrow.

Tempest has been rebooted after becoming unresponsive.  It's back up and accessible however sites aren't loading yet.  We're currently investigating.

Thanks,
Jason

I'm scheduling a 30 minute maintenance window for tomorrow evening at 10pm EST for our server Tsunami.  We're going to be altering a setting that has been causing one of our storage partitions to fill up incorrectly.  It's possible there may be a 5-15 minute service disruption to web services.

If you have any questions, there's a forum notice here:

http://www.charlottezweb.com/forums/index.php?topic=1840.0

Thank you,
Jason

One of our datacenters just started some emergency maintenance that required a reboot of our servers.  This was not planned and unfortunately I wasn't advised until I logged into their site to confirm.

They are essentially rebooting the servers after doing the work required on their end.  Two servers were down about 10 minutes (rebooting) and are up again now.  It looks like our other servers with them are now in the process of rebooting.

They are very good about scheduling maintenance and providing advance notice so I assume this was for something they felt was urgent.

Here is the forum thread where I'll post an update once complete:

http://www.charlottezweb.com/forums/index.php?topic=1838.0

Thank you,
Jason

Scheduled Apache Update.  Please visit this thread for details:

http://www.charlottezweb.com/forums/index.php?topic=1831.0

Starting just before midnight last night, Tsunami has been facing some sporadic network issues. Services may appear to be up/down as we continue our investigation.

Update 6:45pm EST:  I've posted a summary of this issue here: 
http://www.charlottezweb.com/forums/index.php?topic=1828.0

Update 11:35pm EST: 
We believe this to be resolved.  If you're still experiencing issues, please open a ticket, post in the thread above or email us.

We are working on issues with Wildfire currently related to high loads and mysql.  Details will follow as we have them.

Update:  Mysql service was restored awhile ago but loads remain high.  We've found an issue that we're working to address now.

Thank you for your patience!

Regards,
Jason

This server is currently being rebooted.

Avalanche is currently being rebooted.

This coming Friday evening (April 13, 2012) at approximately 11pm EST, our server "Jetstream" will begin planned emergency maintenance to replace its primary hard drive.

For full details or to ask questions, please visit this thread for full details.

An email notification has been sent to all customers on Jetstream directing them to the forum post above.

Wildfire is still experiencing high loads.  This is being investigated.  Please view this thread for full details or discussion:

http://www.charlottezweb.com/forums/index.php?topic=1792.0

Please see this forum post for full details:

http://www.charlottezweb.com/forums/index.php?topic=1791.0

I'm getting reports from some customers of slow or unreachable sites.  It appears there's a larger (perhaps regional) network issue presently.

All servers are up and running fine however depending on your location, you may be facing issues reaching them.  This is not a server or even a datacenter issue from what I'm seeing in tracerts.  I'm seeing slowness reaching our servers in Atlanta and if I trace to our severs in Michigan, it's also very slow until it gets a few states north of here at which point the speeds increase dramatically.  Because of that, I believe it's a network/provider issue somewhere in this general southeast region.

I will add to this post if I can find information online about anything currently happening.

Update:  Our servers Avalanche and Wildfire are responding very slowly due to a network issue at the datacenter.

Details here:

http://www.charlottezweb.com/forums/index.php?topic=1788.0

For more details (including potentially impacted servers) or to post questions, please visit this post on our forum.

One of our datacenters has advised us today of a pending upstream network maintenance window for this coming Friday night at approx 11pm EST.   Normally we don't see any impact from these but I like to post them in the event you experience slower response times during the maintenance window.

Maintenance Notification:
Facility: AtlantaNAP
Time: 23:00 EST on Friday March 2nd.
Window duration: 5 hours
Anticipated impact: Routing instability

This has been rescheduled for midnight Sunday evening (12:01am EST, Monday, March 5)

This coming Saturday morning (March 3, 2012) at approximately 4am EST, Avalanche will  experience some scheduled datacenter maintenance that will result in two periods of outage separated by some potentially slower response times in between.  Each outage should be under 30 minutes however it could be less.

For full details or to ask questions, please visit this thread for full details.

An email notification has been sent to all customers on Avalanche directing them to the forum post above.

We're working on an issue with Tsunami currently.  It looks like a process just used the full temp partition.

http://www.charlottezweb.com/forums/index.php?topic=1778.0