Charlottezweb

Charlottezweb Hosting => Server Updates & Outages => Topic started by: Jason on May 19, 2010, 06:54:15 PM

Title: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 06:54:15 PM
It looks like we've had some web page compromises on this server within the last 15 minutes.

At first glance, it looks similar to what we faced on Blizzard back in March.  http://www.charlottezweb.com/forums/index.php?topic=1447.0

I'll update this thread as we proceed.

As of now, we're investigating the cause so we can patch it prior to searching/replacing the code if possible.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Mark on May 19, 2010, 07:01:59 PM
I'm was going to go through and change all my passwords. Is it okay to do it, or should we wait just in case they get back in?
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 07:03:42 PM
Quote from: Mark on May 19, 2010, 07:01:59 PM
I'm was going to go through and changing all my passwords. Is it okay to do it, or should we wait just in case they get back in?

You can but if there's a thought that passwords were compromised, we may run a script to generate new ones for everyone anyway.  If you saw files being changed in realtime, then there's something running right now that we need to find and kill first.  So you might want to wait until we have more news on that.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Mark on May 19, 2010, 07:04:20 PM
Okay, I'll hold off. Thanks Jason.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Steve on May 19, 2010, 07:05:28 PM
I'm waiting.... too
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: ^ChYmAiL^GTX on May 19, 2010, 07:10:06 PM
Thanks Jason.

I was looking at this...

http://www.kisaso.com/technology/hacked-by-ghost61-my-blog-got-hacked/

I placed some folders 777 a few months ago to try some mods... but i think i placed them back to 755 again...
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Steve on May 19, 2010, 07:12:03 PM
This is what appeared to my forum


(http://img685.imageshack.us/img685/5805/47164346.gif)

(http://yfrog.com/j147164346g)
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Pam on May 19, 2010, 07:15:01 PM
Steve, I got the same "message" on my forum as well.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 07:16:15 PM
We believe the script has been isolated and killed. 

Working on next steps now.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Steve on May 19, 2010, 07:20:46 PM
I have replaced the index.php from the backup file. Do i have to do something more?
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: weekend camper on May 19, 2010, 07:21:24 PM
Just checked and both sites I have on that server were affected.

Thanks for getting on this Jason.


Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Mark on May 19, 2010, 07:22:35 PM
Quote from: Steve on May 19, 2010, 07:20:46 PM
I have replaced the index.php from the backup file. Do i have to do something more?

I only saw evidence of index files being tampered with, but I'm sure once Jason is done he'll let us know what else to do.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 07:24:54 PM
Quote from: Mark on May 19, 2010, 07:22:35 PM
Quote from: Steve on May 19, 2010, 07:20:46 PM
I have replaced the index.php from the backup file. Do i have to do something more?

I only saw evidence of index files being tampered with, but I'm sure once Jason is done he'll let us know what else to do.

Correct, I don't want to advise until I know with certainty of what's needed.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 07:48:28 PM
An email has just been set to our Tsunami customer listing alerting them to this thread.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: JPDeni on May 19, 2010, 07:56:52 PM
Be aware that if you have any subdirectories with index.html or index.php files in them, you'll likely have to replace them, too. I've just replaced a whole bunch of files.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: ShaneR on May 19, 2010, 07:58:29 PM
Thanks, Jason.

I was just about to start trouble shooting on my own when the email came through.  I'll hold off doing anything for now.

I thought it was something stupid. I did earlier as I was changing permissions on a couple folders.  Guess not (at least I hope not).

Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 08:01:09 PM
Quote from: JPDeni on May 19, 2010, 07:56:52 PM
Be aware that if you have any subdirectories with index.html or index.php files in them, you'll likely have to replace them, too. I've just replaced a whole bunch of files.

If you want to proactively replace files, feel free.  However, we're scanning all files server wide to compile a list of everything impacted.  If there's an easier (automated) restore route, we'll do it if possible.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: JPDeni on May 19, 2010, 08:06:38 PM
I'm so used to having to do things myself that I expect to have to. I still have to get used to the wonderful service we have here.  :)
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: rebelsgirl on May 19, 2010, 08:08:34 PM
This is the same person who hacked mine. Do I need to replace the index.php from a new package from SMF? I don't have any modifications on my site at all. And did they attack the server or the forum itself?
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 08:15:05 PM
Quote from: rebelsgirl on May 19, 2010, 08:08:34 PM
This is the same person who hacked mine. Do I need to replace the index.php from a new package from SMF? I don't have any modifications on my site at all. And did they attack the server or the forum itself?

We're still looking into this.  I would just wait or if you have a backup of your files, you can replace just the impacted files.  Otherwise, we'll attempt to do this for you once our investigation is complete.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: ShaneR on May 19, 2010, 08:21:50 PM
I have backups, but I'll hold of until you give the final report.  I don't want to go mucking about without a definitive cause.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 08:27:59 PM
Looks like the earlier cause was accurate and stopped at this time.  We also have a list of all impacted files complete now.

If you want to do your own file restores, feel free -- that will be fatest.  We are looking into a way to restore just those files from backups without having to do full account restores so it may take us some additional time -- especially if we end up having to do it manually.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: rebelsgirl on May 19, 2010, 08:48:45 PM
I replaced my index outside the forum and the one inside. Looks like everything is ok for now. You're going to regenerate new passwords for us?
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: ulborn on May 19, 2010, 09:01:19 PM
Everyone must be trying to restore their files, I can not connect with FTP. :(

Error:   Connection timed out
Error:   Could not connect to server
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 09:04:54 PM
Loads are high -- we're in the midst of doing automated file (not account) restores.  We've completed about 10 sites so far and it's going well.

Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 09:09:25 PM
Quote from: rebelsgirl on May 19, 2010, 08:48:45 PM
I replaced my index outside the forum and the one inside. Looks like everything is ok for now. You're going to regenerate new passwords for us?

Only if necessary.  I don't have a judgement on that just yet.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: ^ChYmAiL^GTX on May 19, 2010, 09:43:18 PM
Thanks Jason! Everything back to normal.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Pam on May 19, 2010, 09:50:15 PM
Jason, we seem to be back too.

Any updates on the need to change passwords?
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 10:08:31 PM
All file restores are done.  If you still have any issues, please post, email or open a ticket so we can look.

Not sure on the passwords yet.  Checking into that now.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 10:45:42 PM
Quote from: Jason on May 19, 2010, 10:08:31 PM
Not sure on the passwords yet.  Checking into that now.

We don't believe this to be necessary at this time, however it never hurts if you want to take that extra step.
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Pam on May 19, 2010, 10:49:02 PM
Thanks Jason for the update!

We appreciate all that you do! :)
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 10:51:08 PM
Thanks!  :)

Now on to the next task of the day -- baby delivery... 
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Pam on May 19, 2010, 10:53:47 PM
How exciting! Congrats to you and Mrs. Jason! ;)
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: akheir on May 19, 2010, 11:05:36 PM
Quote from: Jason on May 19, 2010, 10:51:08 PM
Now on to the next task of the day -- baby delivery... 

I didn't even know you were a mid-wife!

:)

All the best
Title: Re: May 19, 2010 :: Tsunami compromise
Post by: Jason on May 19, 2010, 11:06:40 PM
Thanks everyone :)