January 27, 2012 :: Avalanche load issues

[Jason]

We are currently working on some recurring load issues on Avalanche which is causing us to restart apache/php to keep the server from crashing. 

There have been multiple service disruptions during the past hour while we've been dealing with this.

I will update this thread as I learn more.

[Jason]

A bit of an update. 

We found an IP accessing a particular account suspiciously.  That's been blocked and that account is being analyzed.

We also found a somewhat large forum that is reporting database errors.  We're repairing that now as well.

Currently, the server is looking stable but we'll monitor it as we bring that forum back online.

Thank you for your patience!

[Jason]

Currently, the server is looking stable but we'll monitor it as we bring that forum back online.

The forum mentioned above is back online.  So far things are looking stable.

We will continue to monitor overnight.

[Jason]

Update --

We've done some extensive scanning and found approx 14 accounts that contained malicious files.   Whereas we don't have the initial point of entry, it appears an unauthorized person was able to gain access into the server (likely by way of a virus installed on someone's computer) at which point they were able to upload files that then were able to compromise other accounts. 

We've suspended all of these accounts and have removed the current malicious files we were able to detect.

We will now need to change all those accounts' cpanel passwords.  Those users will then need to change all their passwords.  Furthermore, I HIGHLY recommend all of those users run a current virus scan on any computer they use to connect to their account as that appears to be the initial way they were able to access the server.

I will post more info here as appropriate.

[Jason]

I am attempting to get information out to you as soon as I get it.  I'd like to clarify part of what I said above as it does not appear to be accurate but is what I was told at that time:

We've done some extensive scanning and found approx 14 accounts that contained malicious files.   Whereas we don't have the initial point of entry, it appears an unauthorized person was able to gain access into the server (likely by way of a virus installed on someone's computer) at which point they were able to upload files that then were able to compromise other accounts. 

The 14 accounts that were identified were found by a reputable malware scanner as being infected.  Sometimes these scanners have false positives. 

The way we have security setup on our servers, it does NOT appear that one account compromised others.  It may be that we're seeing false positives or that multiple accounts were independently compromised.  That can certainly happen if they are all running the same software and someone was scanning for that particular script or vulnerability.

I will being resetting accounts and reaching out to these users now.