September 22, 2005 :: Hurricane Rita Email Alert

[Jason]

This post is being partially emailed (with a link back) to all clients on the morning of September 22, 2005 as Hurricane Rita approaches the US Gulf coast.

- - - - - - - - - - -

Greetings from Charlottezweb,

I wanted to pass on a few important notes to all clients.  This is a little lengthy, but please read it as soon as possible.

Our datacenter is in Houston, TX which is geographically next to Galveston.  This happens to be right in Hurricane Rita's present path.  All preparation that could be done is in order [see the info from the datacenter CEO below], but it would be a great idea to backup your data now just in case.

As normal, Charlottezweb does backup your data nightly to remote servers in Atlanta, Georgia, but I would urge you to make your own local backups as appropriate for peace of mind.  This should be a regular chore for everyone.  We offer the backup security as an added value to our service but cannot guarantee its effectiveness.  We've used it for client requests several time where it's proved its value, but I can't guarantee it 100%.  Please take responsibility for your data!

If you are a design-only client who relies on me to administer your site, this is already in order.

In the event that we do have a weather-related issue that takes down our servers, we can be reached via:  Charlottezweb@gmail.com.  Please do not email this address directly unless our servers are down.  This address is not checked or monitored like my normal one.  Please use this only in the event that you need to reach us if our servers are down.

The datacenter is extremely prepared [see quote below] and I'm working on some relocation plans in case things really get messy. 

This leads me to item number 2.  I'd like to highly recommend everyone to sign up on our forum so that you can be contacted in situations like these.  I can email through the server like I'm doing now, but using the forum to send out announcements is a lot more efficient and allows us to maintain a searchable source of historic information.  Not to mention, it's a great way to post questions to learn more.  At the least, please create an account so that in the future I can contact everyone that way.

On a more positive note, Charlottezweb has purchased server #3.  Per your votes, this box will be named Cyclone and it's a powerhouse located in Atlanta at the GNAX datacenter.  We've used them for over a year now for our remote server backup accounts and I've decided to give them a try for this next server after personal experience and research over the past year.  I still highly support EV1 and have no plans to move Thunder and Lightning from there but for performance and reliability, I owe it to all of us to see if GNAX is really as amazing as everyone claims.    

This server is up but I will not be installing accounts to it until Lightning reaches its ideal capacity.  I'll have a few posts soon on our forum regarding this server and my decision to try GNAX for those of you who are interested.

Thank you for your time.  Let's hope the storm lessens in strength and impact.  However, please be prepared in the event of the worst.

Regards,
Jason




From the CEO of the EV1 datacenter:
http://forum.ev1servers.net/showthread.php?t=58244

Disaster planning is currently and has been underway for some time. Anytime we have a potential storm, we reassess our current level of readiness.

Should the hurricane make a direct path for the upper Texas coast, we will operate our core services with a limited staff so that the bulk of our employees can be with their families.

Employees will be on hand to handle emergency repairs and tickets. Routine tickets and upgrades will most probably have a delayed response.

As an extra precaution, we have even sourced an additional rental generator. While this unit would not be needed for a brief outage, if we were top experience a loss of power lasting several days, we would need to perform normal maintenance on our generators, and this would give us a generator to run while that maintenance is taking place. Maintenance such as oil changes is required under our operating specifications every 100 to 150 hours, meaning that we would need to perform maintenance sometime after 4-5 days.

All total, we have in excess of 10,000 gallons of fuel on site. We have guaranteed contracts for fuel delivery and two fuel depots are located within 2 miles of our facility.

Our staff is working to ensure an uninterrupted continuity of service.

While I am current attending to business out of town, I have made arrangements to return to Houston on Wednesday and will weather the storm with our employees in one of our facilities.

We anticipate that the coming storm will have no impact on our operations. However, we are prepared to deal with any eventuality.

Reference:

http://quickfacts.census.gov/qfd/maps/texas_map.html

Map of Texas.  Notice Houston in red and Galveston right to the south-east.

[Jason]

Here's the latest news from the CEO of our datacenter:

Preparations continue 24/7. A core staff of approximately 25 will be on hand throughout the storm. This includes a facilities specialist as well as an a/c specialist.

We are monitoring the storm on a continual basis. Our buildings are designed to withstand Cat 4 conditions. It would be a highly improbable situation for us to sustain a cat 4 this far inland. It is anticipated that we will see high Cat 3 conditions .... based upon the current track and our location.

This could easily escalate to cat 4 conditions.

ALL of our providers have disaster and contingency plans and we are in contact with all of these providers ensuring that our mutual emergency contacts .. including contact outside of the pstn.

I understand everyones concern. Remember ... if you suffer a significant outage, we suffer as well.

From a financial perspective, we have mobilized on that front as well, completely drawing available lines of credit, making nearly $5 million in cash immediately available to complete any necessary repairs, or any contingency contemplated.

We have also drawn an undisclosed amount of greenbacks in the event that we have needs that require green cash.

In addition, we have another $500k available on our corporate charge account with American Express.

In short, we are prepared from a facilities and a financial perspective to weather the storm and its aftermath.

Further, we had additional millions available in business continuity funds through our business continuity insurance.

Due to evacuations and limited staff, ticket responses will likely slow starting with tomorrow's shifts. No upgrades will take place until we restore complete operations. New server purchases will be limited to that on hand and no new servers will be placed into service.

I will personally be on site during the storm.

I have complete confidence in our team. They have proven themselves in many previous times of crisis. During these times, our facilities have remained completely functional. (Remember the transformer fire?) I fully expect the same during this storm. Like the transformer fire, I'll be onsite during the duration.

I ask for your patience during the storm. We understand the importance of your business and the importance that our servers play in your business.

I know that you probably have a million questions. I've tried to address the largest points. However, PLEASE do not call our offices or open tickets with inquiries related to storm status or plans. It is unreasonable to expect that we could respond to them while we are working critical tickets and emergencies.

Also, please open only critical tickets while we are working on a short staff. They'll each be working 18 hour shifts. Please give them extra respect and understanding. They'll be working under less than ideal conditions.

and just as impressive:

We have established a secondary support center in Wichita, Kansas and will be up and running there on Friday morning.

We estimate that we will have about 15 technicians and their families living in that area temporarily by Friday morning.

This center will have internet access for both vpn access into our ticketing system as well as for voip phone calls.

More info as we implement.

Regards,
Jason

[marcobarrera]

Good news!

[Jason]

Here are some snippets from the latest news since yesterday:

I have spoken with high level reps from all of our upstream and transport providers and I have all of their emergency contact info on hand.

Maximum backup power is available at all of their points of presence (POPs) and central offices.

It is difficult to anticipate all potential impact, but they have personnel on site who are prepared with multiple contingency plans to respond to whatever they are handed.

In case of any link failure myself and David Wartell (Sr. Network Engineer) will be available on site (I am at DC1 and David at DC2) to handle communications w/ our providers and to re-route traffic (if necessary).

We all truly appreciate your support and your faith in our ability, and our willingness to care for your business interests.

Here are some photos taken by the staff there of how the place looks presently...

http://www.kodakgallery.com/Slideshow.jsp?mode=fromshare&Uc=b9df8gy.8nbp8jb6&Uy=-vjp14w&Ux=1

And an update at 6:00am today from the CEO:

Yesterday was a busy day finishing up a lot of loose ends. This included a review of our roofs and removal of ANY debris that could become airborne.

This morning, we'll remove anything (such as trash cans, etc) in our parking lots that could becoem airborne.

We took the extreme step of removing some debris from the parking lot/ truck court of an adjoining property.

With the exception of me spending the night, our other key personnel spent the night in their own homes and bed. The team is arriving as we speak to meet and discuss any other last minute preparations.

Our team in Wichita is now up and fully operationsl.

Security in our facilities will be on high alert starting this morning. As is guaranteed in the Constitution, we are "bearing arms" in the event of looting. A sedcurity K9 is also on patrol at our corporate office and will be deployed as necessary.

Hot pizza was found last night. We will be searching for one last hot meal for our crew this morning. Then, it will be on to roma noodles and frozen dinners.

More later. I'll continue posting throughout the storm and aftermath.

More to come as I get it.

Regards,
Jason

[chilly]

The gallery link's not working!

Man it sounds like they're preparing for an invasion down there...I hope everyone is safe.

[Jason]

Updated...it should work now:

http://www.kodakgallery.com/Slideshow.jsp?mode=fromshare&Uc=b9df8gy.8nbp8jb6&Uy=-vjp14w&Ux=1

Regards,
Jason

[Jason]

More updates...

Now they have a live webcam outside at our datacenter.  It's getting hit hard (traffic-wise) so you might have a hard time connecting. 

http://www.ev1.net/stats/ritacam1.html

Here's some more outside photos of the dc:  (click on the image to skip having to register)

http://www.kodakgallery.com/I.jsp?c=rvtzcb9.cxpp2tx&x=0&y=-n80fqb

Noon CEO update:

All is calm. Storm track looks like we'll now see maximum sustained winds below 100. This is excellent news.

Operations continue under emergency conditions.

Regards,
Jason

[Jason]

More info from the datacenter:

**UPDATE 3:00 A.M. CST**

As Robert mentioned we are currently on generator power in our dc1 facilities. City power is still up however it has been flickering on and off the past two hours.

Generator checks are being done 2-3 times an hour round the clock to ensure generators are operating properly (as a precaution only). I personally went outside as well and the winds are picking up the water is coming down it looks alot worse than it actually is.

As more information develops we will be informing everyone.

We truly appreciate everyones cooperation and patience with us during this time.

Here are live outdoor webcams at both Ev1 datacenters. 
http://www.ev1.net/stats/ritacams.html

Everything looks pretty good at this point -- the next danger would be flooding. But for us, the storm path moved quite north-east of the initial projections. According to the news, there is a lot of power outage in Houston but as posted above, Ev1 switched to their generators last night at 6pm just in case. 

Regards,
Jason

[Jason]

Looks like we made it through without any outage 

Wrap-up from the CEO of Ev1:

Just as an FYI, as we were preparing to go back on utility power, we found out that the utility power was out. Don;t know how long it had been out. Came back about 1 hour later, and we returned to utility power once it was safe to do so.

Diesel fueling occured right on time. I would have figured it would have been late, however it was just as requested.

Many things went right. Our employees shined. They gave more than could have been requested. However, each emergency shows you something new to look at. The next few months will have us rethinking every aspect of our plan and how it was implemented.

Thanks again to everyone: employees, competitors, and our clients. We all made it through together.