ESX Security Omaha VMUGMay 29, 2007 m-hoesing@cox.net

• Gartner – Risks and Control considerations

• Other – Risk and Control considerations

• VMware Whitepapers

• DISA STIG – Risks and Control considerations

• VMware Security Lab at VMWorld 11/2006

• Assessment Approaches

• Assessment Demo - Ecora

• Nessus Vulnerabilities (real and otherwise)

• Questions, Resources, PCI/DSS checklist, policy, miscServer Virtualization – security contributor or detractor?

Disclaimer – I have no money, don’t sue me m-hoesing@cox.netItems shown in this presentation are for discussion only, the speaker nor conference sponsors can have knowledge of the unique attributes of each

attendees environment, therefore nothing contained herein should be implemented in your environment without complete analyses and testing, and even after that the presenter nor the conference sponsor accept any liability for the results achieved or not achieved nor any negative repercussions.

On January 19, 2007, a putative class action was filed against TJX in the United States District Court for the Districtof Alabama, Wood, et ano. v. TJX, Inc., et al., 07−cv−00147. The plaintiffs purport to represent a class of “all TJX customerswho made credit card transactions at TJX’s stores during the period that the security of [d]efendants computer systems werecompromised and the privacy or security of whose credit card, check card, or debit card account, transaction or non−publicinformation was compromised.” The complaint asserts claims for negligence per se, negligence, bailment and breach ofcontract, and also names Fifth Third Bancorp as a defendant. Plaintiffs seek compensatory damages, credit monitoring,injunctive relief, attorney’s fees and costs. On March 6, 2007, the court granted an unopposed motion to stay the actionpending disposition of the motion before the Judicial Panel for Multidistrict Litigation to transfer the action and similarfederal court actions to the District of Massachusetts for pretrial consolidation and coordination.On January 19, 2007, a putative class action was filed against TJX in the Supreme Court of British Columbia, Canada,Ryley v. TJX Companies, Inc., et al., Court File No. 07−0278. The plaintiff purports to represent a putative class of “allindividuals resident in British Columbia, or throughout Canada and elsewhere, who have communicated confidential debitand credit information to the [d]efendants in 2003, or between May 1, 2006 and December 31, 2006.” The complaint alsonames “Winners Apparel Inc.” and “HomeSense Inc.” as defendants, and asserts claims for negligence, breach of confidenceand violation of privacy. The plaintiff seeks general and pecuniary damages, punitive damages, interest, attorney’s fees andcosts.On January 19, 2007, a putative class action was filed against TJX in the Quebec Superior Court, Canada, Howick v.TJX Companies, Inc., et al., Court File No. 06−000382−073. The plaintiff purports to represent a putative class of “[a]llphysical persons in Quebec and Canada and all legal persons in Quebec and Canada who, during the twelve (12) monthperiod preceding this Motion for Authorization to Institute a Class Action, had not more than fifty (50) employees undertheir direction or control, who have communicated personal or confidential information to the [r]espondents and have

suffered damage as a result of the loss or theft of this personal or confidential information.” The complaint also names

“Winners Merchants International LP” and “HomeSense Inc.” as defendants. The plaintiff seeks general and specialdamages, punitive damages, attorney’s fees, interest and costs.On January 20, 2007, a putative class action was filed against TJX in The Court of Queen’s Bench, Alberta, Canada,Churchman, et ano. v. The TJX Companies, Inc., et al., Court File No. 0701−00964. The plaintiffs purport to represent aputative class of “individuals who communicated to the [d]efendants confidential information being their debit card numbersand credit card numbers, expiry dates, and all of the information accessible to someone in possession of those debit cards orcredit cards.” The complaint also names “Winners Apparel Inc.,” “Winners Merchants International LP” and “HomeSenseInc.” as defendants and asserts claims for negligence, breach of confidence and violation of privacy. Plaintiffs seek general

and special damages, punitive damages, attorney’s fees, interest and costs.

On January 22, 2007, a putative class action was filed against TJX in The Court of Queen’s Bench, Saskatchewan,Canada, Copithorn v. TJX Companies, Inc., et al., Court File No. 100. The plaintiff pu

TJX 10Q 12/31/2006

(we don need no stinking security)

Gartner - Risks (controls)1. Unique to Virtualization and Paravirtualization : ?????2. Unique to Virtualization Hardware i.e. VT rootkits:

https://www.blackhat.com/presentations/bh-usa-06/BH-US-06-Zovi.pdf (hypervisor on firmware [IBM])(trusted boot measurement [BIOS and other metrics])(execute disable [blocks execution in data addresses])

Memory not erased (buy more expensive hardware, no hyperthreading) [http://www.cs.nps.navy.mil/people/faculty/irvine/publications/2000/VMM-usenix00-0611.pdf

3. Exacerbated by Virtualization:a1.) Configuration Tools less effective (make sure patches, AV

signatures, configuration setting tools and assessment tools can reach the deepest vswitch levels, redeploy tools at each layer or virtualization aware tools [Configuresoft, xccdf, OVAL??],)

a2.) NATed and host only virtual networking hidden from IDS/IPS tools (security tool at every layer, proxys??)

Gartner - Risks (controls) [2]1. Exacerbated by Virtualization (continued):

b.) VM Migration (documented inventory of all live or staged or backup or “extra” guests, pre-approved destinations and access, documented change control, authorization, check for static IPs )

c.) Host Single Point of Failure (VMotion & HA, backups, cluster, load balance, dual power supplies, partitioning [/, /boot, / log])

d.) Cross Communication Between Hosts (policy on shared vnets, maintain default non-promisc vnic settings)

e.) Denial of service by over provisioning (weight the resource requests)

f.) Appliances (patch, test, and assess before deployment)g.) Remote Access (separate VLANS, SSH, SSL, appropriate

user access rights)

Gartner - Risks (controls) [3]1. Good Practice regardless if Virtual or Physical:

• Patching (inventory and a process for ALL templates, live, backups, appliances, “extras” [BlueLane ??])

• Harden & Monitor Host & Guest OS (policy and procedure, Tripwire, assessment tools like Ecora)

• SOD Access (Host admin to appropriate staff, guest admin to appropriate staff, security functions within guests by security staff, separate storage team)

• Networking (according to architecture strategy and policy)• Logging (all security events, all management events, all else if room)

2. Not Available yet ( Dr Moreau – forward thinking)• VM Meta Data (VM “mac”, creator, owner, dates…)• Automatic or Continuous Assessment (xccdf, OVAL)• NAC for VMs (maybe after VM meta data is common place)

Virtualization Risks - Other

• Confidentiality – Memory Sharing http://www.cs.nps.navy.mil/people/faculty/irvine/publications/2000/VMM-usenix00-0611.pdf (buy more secure hardware)– Guest event log, user not recorded (vendor change?)

• Integrity –Complexity of OS, VMM, Storage, Networking (train)• Availability –

– Host is a single point of failure (DR, VMotion [yes the same tools that create patching and configuration challenges also help enhance security goal of availability ])

– VC – single point of access (strong password, and limit host users )– VC configuration database (limit access, backup)– License Server – single point of multiple host DOS [30 days] (backup)– Anti-virus on host (clamav?, warranty voided??)– Remote administration (pick [default] HIGH on configuration setting)– Hosts 272 meg memory (edit GRUB menu.lst if needed)

Virtualization Risks - Other [2]• PROTECT /etc/<<everything>> on host (sudo, strong root

password, multifactor root access, sha1sum integrity monitoring)• Ports (approved by policy? Allowed thru firewall?)

– 902 – management and 80 and 443 and 22– 5988 – CIM (Common Information Model)– 5989 – WBEM (Web Based Enterprise Management)– 27000 & 27010 – license manager– 2050 , 8042 – AAM by EMC (who owns whom?)– 2049 , 3260, 8000 - NAS and iSCSI and VMotion– 2050 thru 5000 and 8042 thru 8045 DAS traffic

• Authentication– No password history (policy, configure with PAM per manual)– Cracklib present but not configured (configure PAM to check for

dictionary words)– /etc/login.defs (life, complexity = policy?) (/etc/passwd = which shells)

Virtualization Risks - Other [3]

– ESX kernel is 2.4 (agree with policy?)– Grub has no password (agree with policy?)– MOTD empty (add warning banner)– UMASK = 22 (agree with policy?)– Open SSL, Open SSH versions(agree with

policy?)– /etc/VMware-mui/ssl/mui.crt or mui.key are

credentials in clear text? – SSL certificate (see VMware security training

doc for correction)

Virtualization Risks - Other [4]

– SNMP (public only, no private : ) , however /etc/snmp/snmpd.conf (change community string from ‘public”)

– Clocks (enable NTP/UTC)– No USB on host (how is two-factor

authentication achieved? [if required])– make-3.79.1-17.1 is running – vmkload-mod –l (remove un-needed modules)– vmkmultipath –q (remove un-needed paths)– Scripts – (access control, change control)

Whitepaper ESX 2.5 • http://www.VMware.com/pdf/esx2_security.pdf

– No public interfaces – Minimal host installation (apache in default install)– Guest isolation (using files)– AV & Firewall recommended (but not supplied)– Su to root– Default non-promiscuous NIC– Code was audited (scope, auditor & methodology not

stated)– Use VLANs and place management console on

separate VLAN from production– Host OS is 100% VM ??, only drivers are open source– Management Console is from Red Hat 7.2

Whitepaper ESX 3http://www.VMware.com/pdf/vi3_security_hardening_wp.pdf

• “..attacking and individual virtual machine will result in the compromise of only the virtual machine..“ (1 hack OK?) (page 4 clarifies)

• Watch patching of dormant (turned off) virtual guests• Rotate logs to prevent DoS• Separate VLANS for management traffic• Configure the firewall (iptables provided)• Use Directory Services (NIS)for admin authentication• Protect Root (sudo)• SNMP is read only

ESX Set “Security” = HIGH

– Set “security” at HIGH 2.5 3.x

DISA STIG (draft) Virtual Computing

• Nice Architectural description (T1 [bare metal directing resources], T2 [software directing resources], Hybrid)

• ParaVirtualization – Type II (kinda) with modified OS’s handling some privileged requests

• Master Image – increases (decreases) security by standardization, MD5

• Remote Management Console - Timeout after 15 minutes

• MOM traffic will be encrypted [if technically possible]

• 3.1.3 – Passwords = length 9, at least 2 of each upper, lower, special, numeric [the ninth is your choice ] , lockout at 3 [no history specification]

• Clipboard cut n paste (disable)

• Scripting – no passing strings from the host to the VM (??)

DISA STIG (draft) Virtual Computing [2]

• Time Synchronization- sync with the host, sync the host to an authorized clock (use NTP)

• Delete VM and Files, not just VM

• 3.1.7 Reinforces the value of a complete Inventory, with accountability

• 3.1.8 Rollback to a Snapshot – check = logs, rights, patches, retired keys, – Rollback to a snapshot off the network

• 3.1.9 “..move the organization’s entire data center on any type of removable media that had sufficient space.” maybe not the whole data center but it makes a good case for controlling removable

media

DISA STIG (draft) Virtual Computing [3]

• 3.1.10 Logs – VM create, move, delete, by whom, reviewed daily, store online for 30 days (PCI compliant??)

• 3.2 Screen Savers – “… running screen savers on the host or virtual machines consumes a lot of CPU.” STIG does not specify “will” have SS’s

• 3.2.1 hosts installed in user directory????– a vmdk disk file and vmk config file can be created elsewhere

and moved it to the virtual server storage device, restrict logical access to the management console which could activate that copied set (segregate storage management & monitoring duties from server management duties)

– Up to date inventory

DISA STIG (draft) Virtual Computing [4]

• 3.2.2 “Private” guests viewable only to the creator (what about VC admin??)

• 3.2.33.2.3 Anti-Virus at non-peak, do not scan VMDKs (updating DAT files ??)– “…all off and suspended virtual machines will have

the latest up-to-date anti-virus software signatures.” (GPO can update turned-off machines??, maybe they meant to say “…before a dormant guest is brought back into production, it is updated with the latest antivirus, os patches, has its configuration settings recertified, and then is placed into production.”)

DISA STIG (draft) Virtual Computing [5]

• 3.3 Guest OS Configuration (documented planning in addition to a documented inventory, install tools??, install the OS version indicated in the build step, rotate logs, disable screen savers [PCI compliant?], )

• 3.4 Networking (remove unused vswitches, packet sniffing only by sys admin? [promisc nic?], unique MACs [how to test]

• 3.5 Hard Disk Drive management (policy, unlimited dynamic disk??, documented backup procedures, agent on host or snapshot or VCB, backup storage separate device distance, flat file backups can not be the primary strategy, snapshots before VI3 can not be primary backup strategy)

DISA STIG (draft) Virtual Computing [6]

• 4.1 ESX Configuration – usb disabled– setuid on sudo & 8 other commands– increase memory for apps running in the service console ?– shared VMFS disks only used with clustering– production - only use persistent disks– .vmx 755 .vmdk 550 = umasks 022 & 227– vmdk’s only stored in VMFS volumes [what about backups?]– use VMware-converter [not cp]– move vmdks using encryption or over a dedicated VLAN– change mgmt process for renaming, moves & other– 5 logs = a.) secure access b.) rotate to different system c.) daily

review

DISA STIG (draft) Virtual Computing [7]

• 4.1.8 Virtual Center – authorized appropriate access to this single point of _____, – warning banner– dedicated server [but license, db services are OK]– patch VC, configure/patch Windows OS + DB + Apache– define appropriate VC use standards– create a VC admin account and remove local administrators

from the default VC admin group– document & approve group construction– watch users that are members of multiple groups

DISA STIG (draft) Virtual Computing [8]

• 4.2 Networking (ports, see slide 7)– ESX 2 physical nics [are both defined?]– MAC_Address_Changes, set to reject– Forged_Transmits, set to reject– No promisc adapters [but see 3.4?]– Use VMTools to get vmxnet adapters– Use dedicated VLAN for VMotion (which is in the clear)– Disable beacon monitoring– No third party firewall on ESX (only iptables)– Check snmpd.conf for “ro” setting

VMWorld Security Lab 11/2006

• Virtual Center – do not let other users inherit parent admin permissions

• Use wheel & sudo (no remote root ssh access, check /etc/sshd.conf)

• Create a MOTD warning banner• Config NTP/UTC• Use NST appliance (Network Security Toolkit,

Nagios, Nessus, Nmap)• SNMP monitoring of guests & hosts• SSL key file permissions (other CVE’s see slide 18)

ESX 2.x Nessus Vulnerabilities • CVE-2006-2481    • Summary: VMware ESX Server 2.0.x before 2.0.2 and 2.x before 2.5.2 patch 4 stores

authentication credentials in base 64 encoded format in the VMware.mui.kid and VMware.mui.sid cookies, which allows attackers to gain privileges by obtaining the cookies using attacks such as cross-site scripting (CVE-2005-3619).

• Published: 7/31/2006 • CVSS Severity: 2.3 (Low) • CVE-2005-3620      VU#822476 • Summary: The management interface for VMware ESX Server 2.0.x before 2.0.2 patch 1,

2.1.x before 2.1.3 patch 1, and 2.x before 2.5.3 patch 2 records passwords in cleartext in URLs that are stored in world-readable web server log files, which allows local users to gain privileges.

• Published: 12/31/2005 • CVSS Severity: 1.6 (Low)

CVE-2005-3619    • Summary: Cross-site scripting (XSS) vulnerability in the management interface for

VMware ESX 2.5.x before 2.5.2 upgrade patch 2, 2.1.x before 2.1.2 upgrade patch 6, and 2.0.x before 2.0.1 upgrade patch 6 allows remote attackers to inject arbitrary web script or HTML via messages that are not sanitized when viewing syslog log files.

• Published: 12/31/2005 • CVSS Severity: 10.0 (High)

ESX 2.x Nessus Vulnerabilities (cont)

• CVE-2005-3618    • Summary: Cross-site request forgery (CSRF) vulnerability in the management

interface for VMware ESX Server 2.0.x before 2.0.2 patch 1, 2.1.x before 2.1.3 patch 1, and 2.x before 2.5.3 patch 2 allows allows remote attackers to perform unauthorized actions as the administrator via URLs, as demonstrated using the setUsr operation to change a password. NOTE: this issue can be leveraged with CVE-2005-3619 to automatically perform the attacks.

• Published: 12/31/2005 • CVSS Severity: 8.0 (High)

• Per VMware– 2006-2481SSL keys, change default ownership to root (assuming root is protected)– 2003-0386 IP restrict and enable & verify reverse mapping off, not applicable to ESX– 2003-0693 SSH 3.6 buffer overflow, not applicable to ESX– 2003-0987 Apache mod_digest replay, not applicable to ESX– 2005-2798 SSH GSSAPIDelegateCredentials, not applicable to ESX– 2006-2444 snmp trap, not applicable to ESX – 2006-3747 cross site scripting w http trace, use separate vlans

ESX 3.0.1 – No Nessus Vulnerabilities 5/11/2007

ESX 3 Assessment Tools

• Ecora Auditor Pro 4.1 tool Http://www.ecora.com/ecora/pr/06-11-2006-b.asp (automated, baseline, deltas)

• “regular” Linux assessment of ESX Host (make is installed in ESX host, not in VirtualIron nor XenEnterprise)– Nessus– CIS/Bastille --assess– LSAT– MTH script

http://members.cox.net/m-d-hoesing/MTH_Linux_Audit_V8.4.txt

Ecora Demo/Output Here

• Talk

• Demo

• Results

OTHER - Resources 133

• The Source http://www.VMware.com – Technology network http://www.VMware.com/community/index.jspa – Security topics http://www.VMware.com/vmtn/technology/security/ – Security Response http://www.VMware.com/support/policies/security_response.html

• Book by Ogelby & Herold http://www.amazon.com/VMware-ESX-Server-Advanced-Technical/dp/0971151067

• Book by Al Muller http://www.amazon.com/Virtualization-VMware-ESX-Server-Muller/dp/1597490199/ref=pd_bxgy_b_text_b/104-0393259-8012733

• Arrasjid & Mills http://download3.VMware.com/vmworld/2005/sln138.pdf

• Watch for CIS standard http://www.cisecurity.org/ • Watch for Virtualization Security Book by Wiley Publishing m-hoesing@cox.net • VM cloning of credentials http://www.thoughtpolice.co.uk/VMware/howto/VMware-

security-tips.html • Blogs http://www.virtualization.info/2003/09/virtualization-sites-blogs.html • DISA orangebook virtualization draft at

http://iase.disa.mil/stigs/draft-stigs/index.html  • Ultimate Deployment Appliance http://www.rtfm-ed.co.uk/?page_id=366 • Guest resource usage and billing http://www.v-kernel.com • “Free” iSCSI http://www.doubleh.ca/docs/Fedora%20iSCSI%20Target.pdf

OTHER 133

• Questions ??• How many Texans does it take to………….

• Gartner articles• http://www.networkworld.com/supp/2007/ndc2/031907-ciso-insight-side-virtualization.html • http://www.gartner.com/it/page.jsp?id=503192

• New Non-fud– Management tools http://www.nworks.com/VMware/– Performance/Billing tools http://www.v-kernel.com – Security tools http://www.bluelane.com/ Virtual Shield (patching)– http://ecoraccm.blogs.com/my_weblog/2007/02/esx_secuirty.html

• Big 4 –– Where are they (VMs) now? (inventory,cc, monitor)– Current Patches

http://www.VMware.com/download/vi/vi3_patches.html– High setting on connections– Appropriate user rights