server compromised?

Mon Mar 11 15:20:17 MST 2013

You can never be certain that your machine has been cleaned off 100% 
unless you do the clean install, however I have been in this situation 
where rebuilding was not an option.  I spent almost 3 months just 
figuring things out and finally I did what I had to in one day.  There 
are couple things that can be done, again without certainity of having 
100% clean system :

1. Re-install / replace ssh binary , ssh config and restart ssh.
http://www.amitnepal.com/server-security-verification-tips-and-tricks/
   use tcp wrappers in addition to key based authentication, iptables etc..
http://www.amitnepal.com/using-tcp-wrappers/

2. Re-install /replace iptables  and then block anything that is not 
known service.
3. Re-install /replace netstat
4. Put some kind of notification on user login, email notification is 
best way because they can remove traces from log, but will miss out 
email sent out as soon as they get a shell.
http://www.amitnepal.com/email-notification-on-root-login-on-linux-machines/

5. See what ports are listening and which ports have established 
connections, check for any mysterious IP Addresses.
6. Check for crontab for all users, cron.daily, monthly, weekly, hourly 
and all possible places for cron.
eg. I have seen, altered ssh  binary and a cron job , to check for their 
injected string in ssh binary, if the string is not present, it would 
replace the new binary with their infected one, so that they regain 
access. cron job could be in some other users cron.
7. run visudo and check for sudo access.

After few hours.. check for ssh binary and see if it has lib wrap 
support, they generally remove lib wrap support in altered binaries.
http://www.amitnepal.com/server-security-verification-tips-and-tricks/

These i think are the basic things to check, however there are many 
other verifications like rpm verification and so on.. which depend on 
how much time you would want to invest in investigation and all that..

Thanks

*Amit K Nepal
Infrastructure Engineer (RHCE)
omNovia Technologies Inc. <http://www.omnovia.com>
Amit K Nepal <http://www.amitnepal.com>
<http://www.amitnepal.com>*
On 3/11/2013 11:40 AM, Vimal Shah wrote:
> Thank you for the advice. The necessary security layer that was 
> missing has been identified and is being incorporated.
>
> Deploying a server from scratch has been tedious (running each command 
> manually). Capturing all of these commands into a python script seems 
> obvious. The python script is slow to develop due to the fact that I'm 
> trying to learn it and code it at the same time.
>
> Has anyone had any experience with Vagrant? Is it worth the time to 
> investigate?
>
> Lastly, if anyone is available for some consulting on these matters 
> (server security and deployment), please contact me.
>
>
> On Thu, Mar 7, 2013 at 4:25 PM, Paul Mooring <paul at opscode.com 
> <mailto:paul at opscode.com>> wrote:
>
>     It's likely that if he left that key in there with a valid e-mail
>     address then whoever compromised the server wasn't trying to be
>     discrete.  I would check my auth logs to see when/if someone was
>     logging in from somewhere suspect.  Next if the server was
>     compromised, it's done, you can never trust it again, no amount of
>     clean up or post-mortem investigation can ever give reasonable
>     confidence that there's no back door on it.  Move the services and
>     data and make a new server/clean install, then look very carefully
>     at what attack vector was exploited and close it (like if it was
>     brute force you should have ssh for root turned off, more
>     restrictive firewall rules and ssh guard).
>
>     Having a server compromised can be a huge headache, good luck.
>     -- 
>     Paul Mooring
>     Systems Engineer and Customer Advocate
>
>     www.opscode.com <http://www.opscode.com>
>
>     From: Vimal Shah <vimals at sokikom.com <mailto:vimals at sokikom.com>>
>     Reply-To: Main PLUG discussion list
>     <plug-discuss at lists.phxlinux.org
>     <mailto:plug-discuss at lists.phxlinux.org>>
>     Date: Thursday, March 7, 2013 4:49 PM
>     To: Main PLUG discussion list <plug-discuss at lists.phxlinux.org
>     <mailto:plug-discuss at lists.phxlinux.org>>
>     Subject: server compromised?
>
>     Hello all,
>
>     While randomly looking into the .ssh/authorized_keys file, I
>     noticed a line that shouldn't have been there. This was concluded
>     based on the last portion of the line. This portion was in the
>     form of /user at domain.com <mailto:user at domain.com>/, where the
>     domain was one of a likely competitor. Does this automatically
>     mean that this server has been compromised? The line has been removed.
>
>     Thanking everyone in advance.
>
>     -- 
>     Vimal
>
>     ---------------------------------------------------
>     PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
>     <mailto:PLUG-discuss at lists.phxlinux.org>
>     To subscribe, unsubscribe, or to change your mail settings:
>     http://lists.phxlinux.org/mailman/listinfo/plug-discuss
>
>
>
>
> -- 
> Vimal (rhymes with Kimmel) Shah
> Front-End / Infrastructure Engineer
> Sokikom
> Mobile: (480) 752-9269 <tel:%28480%29%20752-9269>
> Email:vimals at sokikom.com <mailto:vimals at sokikom.com>
> Web:www.sokikom.com <http://www.sokikom.com/>
>
> Follow us: twitter.com/sokikom <http://www.twitter.com/sokikom>
> Like us: facebook.com/sokikom <http://www.facebook.com/sokikom>
>
>
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.phxlinux.org/mailman/listinfo/plug-discuss

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.phxlinux.org/pipermail/plug-discuss/attachments/20130311/fed32a63/attachment.html>