<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">You can never be certain that your
machine has been cleaned off 100% unless you do the clean install,
however I have been in this situation where rebuilding was not an
option. I spent almost 3 months just figuring things out and
finally I did what I had to in one day. There are couple things
that can be done, again without certainity of having 100% clean
system :<br>
<br>
1. Re-install / replace ssh binary , ssh config and restart ssh.<br>
<a class="moz-txt-link-freetext" href="http://www.amitnepal.com/server-security-verification-tips-and-tricks/">http://www.amitnepal.com/server-security-verification-tips-and-tricks/</a><br>
use tcp wrappers in addition to key based authentication,
iptables etc..<br>
<a class="moz-txt-link-freetext" href="http://www.amitnepal.com/using-tcp-wrappers/">http://www.amitnepal.com/using-tcp-wrappers/</a><br>
<br>
<br>
2. Re-install /replace iptables and then block anything that is
not known service.<br>
3. Re-install /replace netstat <br>
4. Put some kind of notification on user login, email notification
is best way because they can remove traces from log, but will miss
out email sent out as soon as they get a shell.<br>
<a class="moz-txt-link-freetext" href="http://www.amitnepal.com/email-notification-on-root-login-on-linux-machines/">http://www.amitnepal.com/email-notification-on-root-login-on-linux-machines/</a><br>
<br>
5. See what ports are listening and which ports have established
connections, check for any mysterious IP Addresses.<br>
6. Check for crontab for all users, cron.daily, monthly, weekly,
hourly and all possible places for cron.<br>
eg. I have seen, altered ssh binary and a cron job , to check for
their injected string in ssh binary, if the string is not present,
it would replace the new binary with their infected one, so that
they regain access. cron job could be in some other users cron.<br>
7. run visudo and check for sudo access.<br>
<br>
After few hours.. check for ssh binary and see if it has lib wrap
support, they generally remove lib wrap support in altered
binaries.<br>
<a class="moz-txt-link-freetext" href="http://www.amitnepal.com/server-security-verification-tips-and-tricks/">http://www.amitnepal.com/server-security-verification-tips-and-tricks/</a><br>
<br>
These i think are the basic things to check, however there are
many other verifications like rpm verification and so on.. which
depend on how much time you would want to invest in investigation
and all that..<br>
<br>
Thanks<br>
<br>
<div class="moz-signature"><b>Amit K Nepal<br>
Infrastructure Engineer (RHCE)<br>
<a href="http://www.omnovia.com">omNovia Technologies Inc.</a><br>
<a href="http://www.amitnepal.com">Amit K Nepal</a><a
href="http://www.amitnepal.com"><br>
</a></b></div>
On 3/11/2013 11:40 AM, Vimal Shah wrote:<br>
</div>
<blockquote
cite="mid:CAJ6smQegTyAqTfdQbHQqcswt5+Ch16g0tYGiQVGoumYb8sKsaA@mail.gmail.com"
type="cite">Thank you for the advice. The necessary security layer
that was missing has been identified and is being incorporated.
<div><br>
</div>
<div>Deploying a server from scratch has been tedious (running
each command manually). Capturing all of these commands into a
python script seems obvious. The python script is slow to
develop due to the fact that I'm trying to learn it and code it
at the same time.</div>
<div><br>
</div>
<div>Has anyone had any experience with Vagrant? Is it worth the
time to investigate?</div>
<div><br>
</div>
<div>Lastly, if anyone is available for some consulting on these
matters (server security and deployment), please contact me.</div>
<div><br>
</div>
<div><br>
<div class="gmail_quote">On Thu, Mar 7, 2013 at 4:25 PM, Paul
Mooring <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:paul@opscode.com" target="_blank">paul@opscode.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div
style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word">
<div>
<div>
<div>It's likely that if he left that key in there
with a valid e-mail address then whoever compromised
the server wasn't trying to be discrete. I would
check my auth logs to see when/if someone was
logging in from somewhere suspect. Next if the
server was compromised, it's done, you can never
trust it again, no amount of clean up or post-mortem
investigation can ever give reasonable confidence
that there's no back door on it. Move the services
and data and make a new server/clean install, then
look very carefully at what attack vector was
exploited and close it (like if it was brute force
you should have ssh for root turned off, more
restrictive firewall rules and ssh guard).</div>
<div><br>
</div>
<div>Having a server compromised can be a huge
headache, good luck.</div>
<span><font color="#888888">
<div>
<div>
<div>-- </div>
<div>
<div
style="font-family:Consolas;font-size:medium">Paul
Mooring</div>
<div
style="font-family:Consolas;font-size:medium">Systems
Engineer and Customer Advocate</div>
<div
style="font-family:Consolas;font-size:medium"><br>
</div>
<div
style="font-family:Consolas;font-size:medium"><a
moz-do-not-send="true"
href="http://www.opscode.com"
target="_blank">www.opscode.com</a></div>
</div>
</div>
</div>
</font></span></div>
<span><font color="#888888">
</font></span></div>
<span><font color="#888888">
<div><br>
</div>
</font></span><span><span><font color="#888888">
<div style="border-right:medium
none;padding-right:0in;padding-left:0in;padding-top:3pt;text-align:left;font-size:11pt;border-bottom:medium
none;font-family:Calibri;border-top:#b5c4df 1pt
solid;padding-bottom:0in;border-left:medium none">
<span style="font-weight:bold">From: </span>Vimal
Shah <<a moz-do-not-send="true"
href="mailto:vimals@sokikom.com" target="_blank">vimals@sokikom.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>Main
PLUG discussion list <<a moz-do-not-send="true"
href="mailto:plug-discuss@lists.phxlinux.org"
target="_blank">plug-discuss@lists.phxlinux.org</a>><br>
<span style="font-weight:bold">Date: </span>Thursday,
March 7, 2013 4:49 PM<br>
<span style="font-weight:bold">To: </span>Main
PLUG discussion list <<a moz-do-not-send="true"
href="mailto:plug-discuss@lists.phxlinux.org"
target="_blank">plug-discuss@lists.phxlinux.org</a>><br>
<span style="font-weight:bold">Subject: </span>server
compromised?<br>
</div>
</font></span>
<div>
<div>
<div><br>
</div>
<div>
<div>Hello all,
<div><br>
</div>
<div>While randomly looking into the
.ssh/authorized_keys file, I noticed a line
that shouldn't have been there. This was
concluded based on the last portion of the
line. This portion was in the form of
<i><a moz-do-not-send="true"
href="mailto:user@domain.com"
target="_blank">user@domain.com</a></i>,
where the domain was one of a likely
competitor. Does this automatically mean that
this server has been compromised? The line has
been removed.</div>
<div><br>
</div>
<div>Thanking everyone in advance.</div>
<div>
<div><br>
</div>
-- <br>
<font size="3">
<div
style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font
face="arial,helvetica,sans-serif">Vimal <br>
</font></div>
</font>
</div>
</div>
</div>
</div>
</div>
</span>
</div>
<br>
---------------------------------------------------<br>
PLUG-discuss mailing list - <a moz-do-not-send="true"
href="mailto:PLUG-discuss@lists.phxlinux.org"
target="_blank">PLUG-discuss@lists.phxlinux.org</a><br>
To subscribe, unsubscribe, or to change your mail settings:<br>
<a moz-do-not-send="true"
href="http://lists.phxlinux.org/mailman/listinfo/plug-discuss"
target="_blank">http://lists.phxlinux.org/mailman/listinfo/plug-discuss</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<font size="3">
<div
style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font
face="arial, helvetica, sans-serif">Vimal (rhymes with
Kimmel) Shah</font></div>
<div
style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><span
style="font-family:arial,helvetica,sans-serif;font-size:13px">Front-End
/ Infrastructure Engineer</span></div>
<span
style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font
face="arial, helvetica, sans-serif">Sokikom<br>
Mobile: <a moz-do-not-send="true"
href="tel:%28480%29%20752-9269" value="+14807529269"
target="_blank">(480) 752-9269</a><br>
Email:<font color="#500050"> </font><a
moz-do-not-send="true" href="mailto:vimals@sokikom.com"
style="color:rgb(42,93,176)" target="_blank">vimals@sokikom.com</a></font></span>
<div
style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><font
face="arial, helvetica, sans-serif">Web:<font
color="#500050"> </font><font
style="color:rgb(42,93,176)" color="#3333FF"><a
moz-do-not-send="true" href="http://www.sokikom.com/"
style="color:rgb(42,93,176)" target="_blank">www.sokikom.com</a></font></font></div>
<div><font face="arial, helvetica, sans-serif"><br>
</font></div>
</font>
<div>
<div
style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px">Follow
us: <a moz-do-not-send="true"
href="http://www.twitter.com/sokikom"
style="color:rgb(42,93,176)" target="_blank">twitter.com/sokikom</a></div>
<div
style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px">Like
us: <a moz-do-not-send="true"
href="http://www.facebook.com/sokikom"
style="color:rgb(42,93,176)" target="_blank">facebook.com/sokikom</a></div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">---------------------------------------------------
PLUG-discuss mailing list - <a class="moz-txt-link-abbreviated" href="mailto:PLUG-discuss@lists.phxlinux.org">PLUG-discuss@lists.phxlinux.org</a>
To subscribe, unsubscribe, or to change your mail settings:
<a class="moz-txt-link-freetext" href="http://lists.phxlinux.org/mailman/listinfo/plug-discuss">http://lists.phxlinux.org/mailman/listinfo/plug-discuss</a></pre>
</blockquote>
<br>
</body>
</html>