J. Smith Fedora Project cc: Phoenix Linux User Group Security Team cc: Fedora Announce List I was present at FUDCon today when announcement of your results of the "forensics" from the recent F.A.S. web/SSH key system encroachment "hack" was made. Announce List REFERENCE: http://lists.fedoraproject.org/pipermail/announce/2011-January/002911.html Fedora Response: http://numberedhumanindustries.wordpress.com/2011/01/25/security-incident-on-fedora-infrastructure-on-23-jan-2011-dont-panic/ Not only was the FUDCon verbal discussion *incomplete* in context to standard response during a security incident, but the statements appearing in Announce listserv and websites failed to address, within industry standards, the full impact, ramifications and forensics, nor appropriate OPSEC-ish security forensics and solutions. Specifically: 0) Was the system booted into a forensics tool to look at low level hashing or unallocated space file changes (deleted files, etc.)? If it was not rebooted, did you obtain a RAM image? 1) Was a CRC check made against the systems binaries (we know the Fedora source tree was checked and found to be pristine, don't we)? How *did you determine* that no escalated privileges allowed obfuscation of actions? 2) You vaguely state that insufficient information exists, within the window, to determine who did what since "so many people changed their keys at that time". Were new users added? If, in fact *this is true*, your systems fail to meet basic logging recommendations for any server set out by CERT, LOPSA, SCALE, LPI. 3) Redesign/configure FAS to require both https access and FULLY RANDOM user passwords. 4) Remove the SSH key change feature from Web usermin-ish access, instead requiring additional information from either a database of via email exchange. A web based SUID clearly must be protected, since it can be deployed without logging from a shell account? *Since SSLStrip will provide decrypted network information, running on the same network as any attacker, as users log into FAS, the design fails to meet standard tiered security requirements, providing ease of access to the entire code base! * Either VPN (protect via OSI Layer security) or add additional code based access authentication for escalated privileges like changing keys, and/or adding to the repo (over and above a SSH key). Many security professionals, would, in fact, recommend that SSH be turned off completely outside of a VPN. 5) Clearly if the user had access to the system, they potentially have all the passwords and keys - the system MUST BE REBUILT and REDESIGNED, does it not? Perhaps Fedora has designated the entire system as a Honeypot (VLAN excluded) now in hopes of catching the return tagger? * It might be assumed that none actually WANT what Fedora systems have to offer, other than exposing Fedora for the poor security structure?* According to CERT: http://www.cert.org/tech_tips/win-UNIX-system_compromise.html The following are BASIC steps to recovery from any exploit or encroachment event: ITEMS that might be lacking here are in RED: Introduction A. Before you get started SNIP - WE ASSUME YOU HAVE COMPLETED THE BASICS! D. Contact the relevant CSIRT and other sites involved 1. Incident Reporting 2. Contact AusCERT - Australian Computer Emergency Response Team 3. Contact the CERT Coordination Center 4. Obtain contact information for other sites involved E. Recover from the intrusion 1. Install a clean version of your operating system 2. Disable unnecessary services 3. Install all vendor security patches 4. Consult AusCERT advisories and external security bulletins 5. Consult CERT advisories and vendor-initiated bulletins 6. Caution use of data from backups 7. Change passwords F. Improve the security of your system and network 1. Review security using the UNIX or NT configuration guidelines document 2. Install security tools 3. Enable maximal logging 4. Configure firewalls to defend networks G. Reconnect to the Internet H. Update your security policy 1. Document lessons learned from being compromised 2. Calculate the cost of this incident 3. Incorporate necessary changes (if any) in your security policy From JSmith's post describing the incident: The account in question was not a member of any sysadmin or Release Engineering groups. The following is a complete list of privileges on the account: * SSH to fedorapeople.org (user permissions are very limited on this machine). * Push access to packages in the Fedora SCM. * Ability to perform builds and make updates to Fedora packages. The Infrastructure Team took the following actions after being notified of the issue: 1. Lock down access to the compromised account 2. Take filesystem snapshots of all systems the account had access to (pkgs.fedoraproject.org, fedorapeople.org) 3. Audit SSH, FAS, Git, and Koji logs from the time of compromise to the present Here, we found that the attacker did: * Change the account's SSH key in FAS * Login to fedorapeople.org The attacker did not: * Push any changes to the Fedora SCM or access pkgs.fedoraproject.org in any way * Generate a koji cert or perform any builds * Push any package updates Based on the results of our investigation so far, we do not believe that any Fedora packages or other Fedora contributor accounts were affected by this compromise. While the user in question had the ability to commit to Fedora SCM, the Infrastructure Team does not believe that the compromised account was used to do this, or cause any builds or updates in the Fedora build system. The Infrastructure Team believes that Fedora users are in no way threatened by this security breach and we have found no evidence that the compromise extended beyond this single account. As always, Fedora packagers are recommended to regularly review commits to their packages and report any suspicious activity that they notice. Fedora contributors are strongly encouraged to choose a strong FAS password. Contributors should *NOT* use their FAS password on any other websites or user accounts. If you receive an email from FAS notifying you of changes to your account that you did not make, please contact the Fedora Infrastructure team immediately via admin at fedoraproject.org. Thanks for creating great community with the Fedora project and FUDCon. Please let me know if I can assist in any way to investigate or complete these important security changes. -- (503) 754-4452 (623) 688-3392 http://www.obnosis.com *Catch My MetaSploit & IP CAM Surveillence Presentations @ ABLEConf.com in April!* -- (503) 754-4452 (623) 688-3392 http://www.obnosis.com *Catch My MetaSploit & IP CAM Surveillence Presentations @ ABLEConf.com in April!*