http://iase.disa.mil/stigs/draft-stigs/virtual-computing-stig

DRAFT

VIRTUAL COMPUTING

SECURITY TECHNICAL IMPLEMENTATION GUIDE

Version 1, Release 0.1

13 April 2007

Developed by DISA for the DoD

UNCLASSIFIED

VIRTUAL COMPUTING V1R0.1, DRAFT DISA Field Security Operations13 April 2007 Developed by DISA for the DoD

This page is intentionally left blank.

UNCLASSIFIEDii


TABLE OF CONTENTS

1. INTRODUCTION........................................................................1

1.1 Background..........................................................................................................................11.2 Authority..............................................................................................................................11.3 Scope....................................................................................................................................11.4 Writing Conventions............................................................................................................21.5 Vulnerability Severity Code Definitions.............................................................................31.6 DISA Information Assurance Vulnerability Management (IAVM)....................................31.7 STIG Distribution................................................................................................................31.8 Document Revisions............................................................................................................3

2. SERVER VIRTUALIZATION..........................................................5

2.1 Introduction..........................................................................................................................52.2 Classical Virtualization........................................................................................................52.3 Software Virtualization........................................................................................................62.4 Virtual Machines.................................................................................................................72.5 Virtualization Vendors.........................................................................................................82.6 Virtualization Support.........................................................................................................92.7 Patch Management...............................................................................................................92.8 Product Specific Checklists...............................................................................................11

3. VIRTUALIZATION SERVER CONFIGURATION..............................13

3.1 Virtualization Server Management....................................................................................133.1.1 Master Images...........................................................................................................133.1.2 Managing Virtualization Servers and Virtual Machines..........................................153.1.3 Virtualization Server Authentication........................................................................163.1.4 Virtual Machine Tools..............................................................................................163.1.5 Time Synchronization...............................................................................................183.1.6 Deleting Virtual Machine Files................................................................................193.1.7 Virtual Machine Owners...........................................................................................193.1.8 Snapshots..................................................................................................................203.1.9 Moving Virtual Machines.........................................................................................203.1.10 Logging and Auditing...............................................................................................21

3.2 Virtualization Server Operating System Configuration....................................................213.2.1 Virtual Machine Creation.........................................................................................223.2.2 Virtual Machine Permissions....................................................................................233.2.3 Antivirus...................................................................................................................24

3.3 Guest Operating System Configuration.............................................................................253.3.1 Guest OS Selection...................................................................................................263.3.2 Guest OS Logging....................................................................................................263.3.3 Controlling Guest Resource Usage...........................................................................26

3.4 Virtualization Server Networking......................................................................................283.4.1 Networking Configurations......................................................................................293.4.2 MAC Addresses........................................................................................................31

3.5 Virtual Machine Hard Drive Disk Management...............................................................32

UNCLASSIFIEDii


3.5.1 Sharing and Moving Virtual Machine Files.............................................................323.5.2 Virtual Hard Drive Disk Types.................................................................................32

3.6 Virtualization Server Backup and Recovery.....................................................................353.6.1 Traditional Backups..................................................................................................353.6.2 Flat-File Backups......................................................................................................363.6.3 Snapshot Backups.....................................................................................................37

4. VMWARE ESX SERVER CONFIGURATION...................................39

4.1 VMware ESX Server.........................................................................................................394.1.1 Service Console........................................................................................................414.1.2 Setuid and Setgid Applications.................................................................................424.1.3 Memory Requirements.............................................................................................434.1.4 Disks and Partitions..................................................................................................434.1.5 Managing VMDK Files............................................................................................454.1.6 Renaming and Moving Virtual Machines.................................................................474.1.7 Logging and Auditing...............................................................................................484.1.8 VirtualCenter............................................................................................................49

4.2 ESX Server Networking....................................................................................................584.2.1 Physical and Virtual Network Adapters...................................................................584.2.2 Virtual Switches........................................................................................................624.2.3 VLANS.....................................................................................................................654.2.4 VLAN Modes...........................................................................................................654.2.5 Bonded Network Adapters.......................................................................................704.2.6 Bonded Modes..........................................................................................................714.6.7 ESX Server Beacon Monitoring...............................................................................724.2.7 ESX TCP/IP Ports.....................................................................................................73

4.3 SNMP................................................................................................................................74

UNCLASSIFIEDiii


APPENDIX A. RELATED PUBLICATIONS............................................75APPENDIX B. PRODUCT VMM TYPES..................................................76

UNCLASSIFIED4


LIST OF TABLES

TABLE 1.1. VULNERABILITY SEVERITY CODE DEFINITIONS.........3

TABLE OF FIGURES

FIGURE 2-1. TYPE I VMM...........................................................................6FIGURE 2-2. TYPE II VMM.........................................................................7FIGURE 3-1. MASTER INSTALLATION IMAGE....................................14FIGURE 3-2. BRIDGED NETWORK.........................................................29FIGURE 3-3. NAT NETWORK...................................................................30FIGURE 3-4. HOST-ONLY NETWORK....................................................31FIGURE 3-5. FIXED DISK..........................................................................33FIGURE 4-1. ESX SERVER ARCHITECTURE.........................................41FIGURE 4-2. VMFS WITH REDO DISK FILE..........................................45FIGURE 4-3. VIRTUALCENTER INTERFACE........................................49FIGURE 4-4. VIRTUALCENTER PORTS..................................................57FIGURE 4-5. ESX SERVER WITH TWO NETWORK ADAPTERS........58FIGURE 4-6. PROMISCUOUS MODE.......................................................60FIGURE 4-7. MULTIPLE VIRTUAL NETWORK ADAPTERS...............62FIGURE 4-8. VIRTUAL ETHERNET NETWORK....................................63FIGURE 4-9. VMOTION TECHNOLOGY.................................................65FIGURE 4-10. EST MODE..........................................................................66FIGURE 4-11. VST MODE..........................................................................68FIGURE 4-12. VGT MODE.........................................................................70FIGURE 4-13. BONDED NETWORK ADAPTERS...................................71

UNCLASSIFIEDv



UNCLASSIFIEDvi


1. INTRODUCTION

A core mission for the Defense Information Systems Agency (DISA) Field Security Operations (FSO) is to secure Department of Defense (DoD) Computing systems. The processes and procedures outlined in this Security Technical Information Guide (STIG), when applied, will decrease the risk of unauthorized disclosure of sensitive information. Security is clearly still one of the biggest concerns for our DoD customers (i.e., the war fighter).

The intent of this STIG is to include security considerations and to provide an acceptable level of risk for information being transmitted.

1.1 Background

This STIG was developed to enhance the confidentiality, integrity, and availability of sensitive DoD Automated Information Systems (AIS).

Virtual Computing infrastructures must provide secure, available, and reliable data for all customers. This document will assist sites in meeting the minimum requirements, standards, controls, and options that must be in place for secure Virtual Computing environments.

1.2 Authority

DoD Directive 8500.1 requires that “all IA and IA-enabled IT products incorporated into DoD information systems shall be configured in accordance with DoD-approved security configuration guidelines” and tasks DISA to “develop and provide security configuration guidance for IA and IA-enabled IT products in coordination with Director, NSA.” This document is provided under the authority of DoD Directive 8500.1.

The use of the principles and guidelines in this STIG will provide an environment that meets or exceeds the security requirements of DoD systems operating at the Mission Assurance Category (MAC) II Sensitive level, containing sensitive information.

It should be noted that Field Security Operations (FSO) support for the STIGs, Checklists, and Tools is only available to DoD Customers.

1.3 Scope

The requirements set forth in this document will assist Information Assurance Managers (IAM), Information Assurance Officers (IAO/SA), Network Security Officers (NSO), and System Administrators (SA) in support of protecting DoD Virtual Computing systems.

The Information Operations Condition (INFOCON) for the DoD recommends actions during periods when a heightened defensive posture is required to protect DoD computer networks from attack. The IAO will ensure compliance with the security requirements of the current INFOCON level and will modify security requirements to comply with this guidance. Password length and complexity given throughout this document must be adjusted as needed to comply with INFOCON guidance.

UNCLASSIFIED1


This document contains a set of principles and guidelines that serve as the basis for establishing Virtual Computing environments within the DoD. There are many virtual server products available as commercial off the shelf products. This STIG will focus on virtual server products with specific guidance for the ESX Server. Since many products utilize the virtual server within their names, the term virtualization server will be used to refer to any virtual server product.

The policy portions of this STIG are relevant to all virtual servers or applications connected to either the DoD Unclassified (But Sensitive) Internet Protocol Router Network (NIPRNet) or Secret Internet Protocol Router Network (SIPRNet).

1.4 Writing Conventions

Throughout this document, statements are written using words such as “will” and “should.” The following paragraphs are intended to clarify how these STIG statements are to be interpreted.

A reference that uses “will” indicates mandatory compliance. All requirements of this kind will also be documented in the italicized policy statements in bullet format, which follow the topic paragraph. This makes all “will” statements easier to locate and interpret from the context of the topic. The IAO will adhere to the instruction as written.

Each policy bullet includes the STIG Identifier (SDID) in parentheses that precedes the policy text and references the corresponding vulnerability check in the SRR Checklist and Vulnerability Management System (VMS). An example of this will be as follows: "(G111: CAT II)." If the item presently does not have an STIGID, or the STIGID is being developed, it will contain a preliminary severity code and "N/A" (i.e., "[N/A: CAT III]"). Throughout the document accountability is directed to the IAO to “ensure” a task is carried out or monitored. These tasks may be carried out by the IAO or delegated to someone else as a responsibility or duty.

A reference to “should” indicates a recommendation that further enhances the security posture of the site. These recommended actions will be documented in the text paragraphs but not in the italicized policy bullets. All reasonable attempts to meet this criterion will be made.

UNCLASSIFIED2


1.5 Vulnerability Severity Code Definitions

Category I Vulnerabilities that allow an attacker immediate access into a machine, allow superuser access, or bypass a firewall.

Virtual Computing Application: Vulnerabilities that may result in malicious attacks on virtual infrastructure resources or services. Attacks may include but are not limited to the Blue Pill, SubVirt, Trojan, DOS, and executing potentially malicious actions.

Category II Vulnerabilities that provide information that have a high potential of giving access to an intruder.

Virtual Computing Application: Vulnerabilities that may result in unauthorized users accessing and modifying virtual infrastructure resources or services.

Category III Vulnerabilities that provide information that potentially could lead to compromise.

Virtual Computing Application: Vulnerabilities that may result in unauthorized users viewing or possibly accessing virtual infrastructure resources or services.

Table 1.1. Vulnerability Severity Code Definitions

1.6 DISA Information Assurance Vulnerability Management (IAVM)

The DoD has mandated that all IAVMs are received and acted on by all commands, agencies, and organizations within the DoD. The IAVM process provides notification of these vulnerabilities and alerts require that each of these organizations take appropriate actions in accordance with the issued alert. IAVM notifications can be accessed at the Joint Task Force -Global Network Operations (JTF-GNO) web site: https://www.jtfgno.mil.

1.7 STIG Distribution

Parties within the DoD and Federal Government's computing environments can obtain the applicable STIG from the Information Assurance Support Environment (IASE) web site. This site contains the latest copies of any STIG, as well as checklists, scripts, and other related security information. The NIPRNet URL for the IASE site is http://iase.disa.mil/.

1.8 Document Revisions

Comments or proposed revisions to this document should be sent via e-mail to [email protected]. DISA FSO will coordinate all change requests with the relevant DOD organizations before inclusion in this document.

UNCLASSIFIED3

http://iase.disa.mil/



UNCLASSIFIED4


2. SERVER VIRTUALIZATION

2.1 Introduction

The term virtualization refers to the method of disconnecting the operating system from the physical server hardware and inserting a “virtualization layer” between the two. The virtualization layer presents a defined collection of virtual hardware to the virtual machine’s operating system and acts as mediator between the virtual “guest” operating system and the host computer running the virtualization software.

2.2 Classical Virtualization

The need to virtualize unmodified x86 operating systems has given rise to software techniques that go beyond the classical trap and-emulate Virtual Machine Monitor (VMM). A VMM is the piece of software that provides the abstraction of a virtual machine. The software VMMs have enabled widespread use of x86 virtual machines to offer server consolidation, fault containment, security, and resource management. A particular VMM implementation style, trap-and-emulate, was so accepted in 1974 that it was considered the only practical method for virtualization. Although other methods were available, confusion resulted over the years to equate “virtualizability” with the VMM implementation of trap-and-emulate.

In order to clarify the “virtualizability” definition, the term classically virtualizable is used to describe an architecture that is purely trap-and-emulate. As a result, the x86 is not classically virtualizable or trap-and-emulated. However, it is virtualizable by the Popek and Goldberg’s requirements using the essential requirements.

The Popek and Goldberg virtualization requirements are a set of sufficient conditions for computer architecture to efficiently support system virtualization. They were introduced by Gerald J. Popek and Robert P. Goldberg in their 1974 article "Formal Requirements for Virtualizable Third Generation Architectures". Popek and Goldberg’s 1974 paper establishes three essential characteristics for system software to be considered a VMM:

- Equivalence: a program running under the VMM should exhibit a behavior essentially identical to that demonstrated when running on the original machine directly.

- Resource control: the VMM must be in complete control of the virtualized resources.

- Efficiency: a statistically dominant fraction of machine instructions must be executed without VMM intervention.

Recent architectural changes have enabled the classical virtualization of the x86 architecture. This is accomplished through AMD’s SVM and Intel’s VT. This is referred to as hardware-assisted VMM (hardware VMM as opposed to the software VMM).

Hardware can provide explicit support for virtualization. CPU vendors are adding hardware virtualization assistance to their products. Intel's name for these extensions is Vanderpool and

UNCLASSIFIED5

http://en.wikipedia.org/wiki/Vanderpool

http://en.wikipedia.org/wiki/Intel

http://en.wikipedia.org/wiki/1974

http://en.wikipedia.org/w/index.php?title=Robert_P._Goldberg&action=edit

http://en.wikipedia.org/wiki/Gerald_J._Popek

http://en.wikipedia.org/wiki/Virtual_machine_monitor


AMD's is Pacifica. These extensions address the parts of x86 that are difficult to virtualize, providing additional support to the virtualization layer. This provides for simpler virtualization code and a higher performance.

2.3 Software Virtualization

Software virtualization is possible through the use of the hypervisor or VMM. A VMM is software for a computer system that creates representations of physical hardware to a guest virtual machine. These representations provide users with the appearance of direct access to the real machine environment. These representations are referred to as virtual machines. The VMM or hypervisor allows these virtual machines to share resources with each other. Resources that are shared may be memory, disk space, CPU, etc.

Virtualizing an operating system requires that all the instructions must be executed by either software, hardware, or a combination of both. These combinations create different types of VMMs. The amount of software and hardware execution of processor instructions determines what type of VMM it is. The three types of VMMs are Type I, Type II, and Hybrid, which is a combination of Type I and II. Appendix B lists the virtualization products and their associated VMM types.

The Type I VMM runs on directly on the machine hardware. A Type I VMM is an operating system or kernel that has mechanisms to support virtual machines. It must perform scheduling and resource allocation for all virtual machines in the system and requires drivers and hardware peripherals. Processors must comply with several virtualization requirements to support a Type I VMM. First, the execution of non-privileged instructions must be equivalent in both privileged and user mode. Second, there must be a method to protect the real system and any other virtual machines from the active virtual machine. Figure 2-1 illustrates a Type I VMM.

Figure 2-1. Type I VMM

UNCLASSIFIED6

http://en.wikipedia.org/wiki/Pacifica_(virtual_machine)

http://en.wikipedia.org/wiki/AMD


The Type II VMM runs as an application on a host operating system and relies on the host operating system for memory management, processor scheduling, resource allocation, and hardware drivers. Processors must meet all the Type I VMM requirements as well as several host operating system requirements. Once such host operating system requirement is a mechanism to allow the VMM to run the virtual machine as a sub-process. Figure 2-2 illustrates the Type II VMM.

Figure 2-2. Type II VMM

A Hybrid VMM (HVM) has all the advantages of normal VMMs and avoids the penalties of purely software based VMMs. An HVM is functionally equivalent to the real machine. The difference between an HVM and a VMM is that the HVM treats the privileged mode of hardware as a software construct, whereas the normal VMM may directly execute some privileged instructions. Also, with the HVM all non-privileged instructions are executed directly on the processor. The HVM has less strict processor requirements than the VMM in two ways. First, the HVM does not have to execute non-sensitive privileged instructions because they are all emulated in software. Because of this software emulation, the HVM does not need to map the most privileged processor mode into another processor privilege level. This increased interpretation execution usually lowers the performance of an HVM relative to a VMM.

2.4 Virtual Machines

While virtualization has been a part of the IT landscape for decades, it was only until recently that companies delivered the benefits of virtualization to industry-standard x86-based platforms. A key benefit of virtualization is the ability to run multiple operating systems on a single physical system and share the underlying hardware resources.

UNCLASSIFIED7


Virtualization allows multiple virtual machines, with heterogeneous operating systems to run in isolation, side-by-side on the same physical machine. Each virtual machine has its own set of virtual hardware (CPU, RAM, disks, network cards) upon which an operating system and applications are loaded. The virtual machine operating system sees a consistent, normalized set of hardware regardless of the actual physical hardware components. Virtual machines have many benefits such as encapsulation, isolation, and partitioning.

Virtual machines are encapsulated into files, making it possible to rapidly save, copy, and provision a virtual machine. Fully configured systems, applications, operating systems, and virtual hardware may be moved, within seconds, from one physical server to another for zero-downtime maintenance and continuous workload consolidation.

Virtual machines are completely isolated from the host machine and other virtual machines. If a virtual machine crashes, all others are unaffected. Data does not leak across virtual machines and applications can only communicate over configured network connections. Partitioning multiple applications and operating systems can be supported within a single physical system. Servers can be consolidated into virtual machines on either a scale-up or scale-out architecture. Computing resources are treated as a uniform pool which is allocated to virtual machines in a controlled manner; however, they could undermine the security architecture as well.

2.5 Virtualization Vendors

The virtualization industry is currently dominated by two main players: VMware, who has been in the virtualization market since 1998, and Microsoft, who has released VirtualPC and Virtual Server products. VMware is available in several versions including VMware Workstation, VMware Server, and ESX Server. Microsoft has the Virtual PC and Virtual Server available.

VMware and Microsoft virtualization products have different backgrounds. VMware products have been commercially available for the x86 platform since 1998. VMware is in the third and fourth generation of products for the ESX Server and Virtual Server products. Microsoft purchased their virtualization product from Connectix. Microsoft’s current version, Microsoft Virtual Server 2005 R2, was released in November 2005.

VMware ESX Server is VMware’s enterprise-class product. The most notable difference between ESX Server and VMware Server/Workstation is that ESX Server does not run on a standard “hosted” operating system. Both VMware Server and Workstation are “hosted” solutions in that they require a host operating system to be installed on the hardware prior to loading the VMware product. ESX Server is loaded directly onto the hardware and has its own microkernel operating system. VMware Server and ESX Server are licensed based on the number of processors in the host server computer.

Microsoft’s Virtual PC is designed to provide an optimal experience for a desktop user who wants to run one or more additional desktop operating systems on a single computer. The user interface is fairly simple, and there is extensive integration between the host operating system running on the physical computer and the guest operating systems running in virtual machines.

UNCLASSIFIED8


Microsoft’s Virtual Server is different from Virtual PC in that, instead of being designed for simplicity of use for the desktop user, it provides features that support the more complex requirements of enterprise server applications and administration. Virtual Server includes additional features that support greater manageability, scalability, and extensibility. These are important aspects of server management that are not appropriate to the intended uses of Virtual PC.

2.6 Virtualization Support

Virtualization requires support for the host operating system, the virtualization software, and the virtualized guest operating system. Support for the host and guest operating system may vary depending on the vendor chosen. For instance, VMware products support more host and guest operating systems than do Microsoft products. VMware’s Virtual Server has the largest matrix of supported host and guest operating systems. VMware’s Virtual Server may have Linux or Microsoft host and guest operating systems. Microsoft Virtual Server 2005 R2 currently supports only Microsoft operating systems for the host. The guest operating systems may be Microsoft operating systems and some Linux distributions.

The host operating system support from vendors depends on the virtualization platform. Microsoft Virtual Server 2005 R2 only installs on top of a Microsoft operating system. Therefore, the support is no different than the support received from Microsoft on a normal server installation. However, VMware Virtual Server may be installed on a Linux or Microsoft host operating system. The host operating system would be under the same level of support as if there was no virtualization software installed. The ESX Server runs on its own hypervisor/kernel as the host operating system, which is supported by the VMware technical support.

(VCGEN0100: CAT II) The IAO/SA will ensure the virtualization server host operating system is supported and/or approved by the virtualization server vendor.

The guest operating systems on the host operating system must be supported by the virtualization software. Guest operating systems will need to be approved by the virtualization vendor so that if problems are encountered with the guest operating system, the virtualization vendor can assist with the resolution. Unsupported guest virtual machines create problems since no documentation or support is available from the virtualization vendor.

(VCGEN0102: CAT II) The IAO/SA will ensure the virtualization software supports the installed guest operating systems within the virtualization environment.

2.7 Patch Management

Patch Management is an important part of maintaining a secure and stable environment. Virtualization server administrators will subscribe to virtualization vendor security notices, updates, and patches to ensure that all new vulnerabilities are known. New patches and updates should be reviewed for both the host and guest operating systems before moving them into a production environment. All patches will be tested first in a development environment and any issues or special precautions will be documented, as a patch could technically disable all virtual

UNCLASSIFIED9


networks and machines. For instance, minor releases of VMware ESX Server are typically released every 9-12 months, with an occasional maintenance release in between. When a VMware releases a patch or update, it is always accompanied with an installation script. Running the installation script will perform the proper actions to apply the patch. If the patch updates the kernel of the Service Console or the VMkernel, there is a good chance a reboot will be necessary for the changes to take effect. Some patches may only require a service to be restarted.

Also, test and development virtual machines will be located on separate physical virtualization servers. Placing test and development virtual machines on a separate physical virtualization server ensures that any hardware issues with the test and development virtualization server does not affect live production virtual machines. This separation will enhance the availability of the production servers.

(VCGEN0103: CAT III) The IAO/SA will subscribe to virtualization server vendor security, patches, and update notifications.

(VCGEN0104: CAT II) The IAO/SA will ensure all patches for the virtualization server are tested in a development environment before installing them on the production virtualization servers.

(VCGEN0105: CAT II) The IAO/SA will ensure the virtualization software is at the latest patch level.

(VCGEN0106: CAT II) The IAO/SA will ensure all test and development virtual machines are not located on the same physical virtualization server as the production virtual machines.

Virtual machines create a condition where they may be on, off, or suspended. The requirement that machines be on in a conventional approach to patch management, virus and vulnerability scanning, and machine configuration creates an issue in the virtual world. Virtual machines can appear and disappear from the network sporadically. Conventional networks can “anneal” new machines into a known good configuration state very quickly. However, converging virtual machines to a known good state is more challenging since the state may change quickly. For instance, a vulnerable machine can appear briefly and either become infected or reappear in a vulnerable state at a later time. Therefore, vulnerable virtual machines may become infected with a virus and never be detected since the virtual machine may be offline. Suspended and off virtual machines should be patched regularly to ensure patches are up to date. Virtual machines that are on will be kept current with the operating system patches per the appropriate OS STIG.

(VCGEN0107: CAT II) The IAO/SA will ensure all off and suspended guest virtual machines are at the latest patch level for the guest operating system.

2.8 Product Specific Checklists

There are specific product dependent settings and controls that will need to be configured to ensure the secure configuration of the virtualization servers. Because these controls do not apply

UNCLASSIFIED10


to every virtual server product, the specifics are documented in the associated product specific companion checklist.

(VCGEN0108: CAT II) The IAO/SA will ensure the virtualization product is configured in accordance with the product specific companion checklist.

UNCLASSIFIED11



UNCLASSIFIED12


3. VIRTUALIZATION SERVER CONFIGURATION

3.1 Virtualization Server Management

Virtualization servers will be configured and managed to provide confidentiality, integrity, and availability. There are several areas that should be addressed when deploying virtual machines within a virtualization server. These areas include managing master images, authentication, time synchronization, virtual machine moves, owners, deletions, snapshots, and auditing. These areas are discussed below.

3.1.1Master Images

Virtual machines are similar to physical machines in that deployment is made easier when using a set of master installation images. Regardless of what physical hardware the virtual server is installed on, the hardware platform presented to the guest operating systems usually does not change. Virtual servers simplify this by removing the hardware dependency, resulting in less master installations and fewer images to manage.

A master installation is a completely configured operating system installation that is up-to-date on service packs and security fixes. Master installations typically contain the support applications such as antivirus software, management agents, or backup software. Master installations can be deployed to new servers, greatly reducing the time, cost, and likelihood of errors when deploying multiple servers simultaneously.

Master images are created from using operating system CD-ROMs or ISO images of the operating system. ISO operating system images reduce the time in deploying virtual servers since the media is readily available as a file on the hard drive. The other reason for ISO operating system images is it maps easily to the virtual CD-ROM drive of the guest machine once the guest machine is running. One drawback to ISO images is the space required to keep them. ISO images consume a lot of disk space since they are not compressed when they are created. Unauthorized access to the ISO operating system images could potentially allow these images to be corrupted or altered in some way. As a result, access to the ISO operating system images will be restricted to authorized users only.

(VCGEN0109: CAT II) The IAO/SA will ensure ISO images are restricted to authorized users only.

Since ISO operating system images are typically large files, transferring these ISO operating system images over the network may cause corruption to the files. There are simple ways to check the integrity of the file on both the source and destination system using the MD5 algorithm. Users will verify all ISO operating system images on the virtual server before utilizing the ISO operating system image for virtual machines. For Linux systems, use the “md5sum filename”. For windows, there are several free utilities which include http://winmd5sum.solidblue.biz. It has a compare feature integrated.

(VCGEN0110: CAT II) The IAO/SA will ensure all ISO images that are moved are verified for integrity by using the md5sum utility.

UNCLASSIFIED13

http://winmd5sum.solidblue.biz/


Users should create a master image for each guest operating system to be deployed in the virtual server environment. For instance, create one master image for Windows 2003 Standard, Windows 2003 Enterprise, and RedHat Enterprise Linux 4, etc. Master images are primarily used for server provisioning. Server build processes that would normally take several steps for each new operating system can easily be rolled up into a master image. Master images reduce the amount of time from the server provisioning process. Figure 3-1 illustrates the master image installation process.

Figure 3-1. Master Installation Image

Utilizing master installation images can also help keep a networked environment secure. Maintaining approved master images reduces the likelihood that a new server is infected on the network. While deploying new guest virtual machines from master images will not completely eliminate all security risks, it does provide a baseline configuration for new guest virtual machines. The final major benefit of master installation images is that they provide a standardized platform for your server environment. As images are deployed from the master image, administrators will not need to worry about whether a specific registry hack was applied, or whether the password is the same.

NOTE: It is strongly recommended that all guest virtual machines are configured from the appropriate master images for the OS.

The strategies used to store your master installation images will directly impact the deployment scenarios. The master images will be stored in a separate partition (NTFS, VMFS, etc) from the

UNCLASSIFIED14


production guest virtual machines. Partitioning the master installations keeps the images isolated from system, application, and user files. This isolation helps protect the disk space used by the Linux kernel and various applications. Files cannot grow across partitions. Another advantage is that if a bad spot develops on the hard drive, the risk to the data is reduced as is recovery time. Furthermore, separate master installation partitions provide the ability to set up certain directories as read-only filesystems. Lastly, master installations will be restricted to authorized users only. If these images were compromised, all future guest installations could be corrupt or contain malicious code. Master installation images will be restricted to only users that are administering and/or creating guest virtual machines.

(VCGEN0112: CAT III) The IAO/SA will ensure all master installation images are stored on a separate partition.

(VCGEN0114: CAT II) The IAO/SA will ensure all master installation images are restricted to authorized users.

3.1.2Managing Virtualization Servers and Virtual Machines

The two major vendors, VMware and Microsoft offer several ways to manage a virtualization server. The ways include a web-based management interface, command line, virtual server management console, and management applications designed for the virtualization server. Each method to manage virtualization servers is described below.

A web-based management interface is used to manage a single virtualization server or virtual machine at a time. The web-based management interface is used for monitoring and managing the majority of administrative tasks. The web-based management interface for virtualization server platforms will have all management traffic encrypted with an approved FIPS 140-2 encryption algorithm (SSLv3/TLSv1), providing confidentiality.

Virtualization servers or virtual machines may also be administered through the command line. Accessing the command line give much more granular control and options to the administrator for configuring the virtualization servers or virtual machines. Also, the command line is faster than the web-based management interface since all communication is in ASCII text. All remote virtualization server and virtual machine connections for command line configurations will use an encrypted channel. Telnet, ftp, and other clear text protocols will not be used since commands sent in the clear may be sniffed by unauthorized users.

Most virtualization server products provide a virtual machine remote management console (VMRC) utility. This VMRC displays the virtualization server or machine in a window where users may interact with the virtualization server or virtual machine as they would with a physical computer. This traffic needs to be encrypted as well, since commands sent over this remote console could be seen by unauthorized users. Furthermore, VMRC sessions will be configured to timeout after 15 minutes of inactivity. Virtual machines will timeout if the user is at the console, remotely connected via RDP, etc. Therefore, all virtual machine remote console sessions will timeout after 15 minutes of inactivity to protect against sessions remaining open indefinitely, and potentially giving access to unauthorized users.

UNCLASSIFIED15


Management applications provide central administration and connectivity to virtualization server farms. VMware provides VirtualCenter as a management application to manage both VMware Virtual Server and ESX Server hosts running virtual machines. Microsoft recently introduced a management pack add-on to its Microsoft Operations Manager (MOM) solution to help manage and monitor virtual environments. All traffic from the virtualization management application to the virtualization servers will be encrypted if technically possible to ensure that unauthorized users are not able to see or modify this traffic.

(VCGEN0118: CAT II) The IAO/SA will ensure all web-based management (SSLv3/TLSv1), remote console (SSH, SFTP, etc.), and VMRC traffic to virtualization servers or virtual machines are encrypted with an approved FIPS 140-2 algorithm.

(VCGEN0120: CAT II) The IAO/SA will ensure all VMRC traffic with the virtualization server or virtual machines are configured with an idle timeout value of 15 minutes or less.

3.1.3Virtualization Server Authentication

Virtualization servers will be configured to secure the virtualization server system and users. Passwords will be created IAW the policy outlined in DoDI 8500.2, IA controls IAIA-1, and IAIA-2. Passwords are susceptible to compromise while in transit, at rest, and at the server console by unauthorized users. Virtualization servers will ensure all passwords at rest are encrypted, and virtualization server passwords are masked. Virtualization server remote management tools will be configured to prevent anonymous logons, guest logons, and lockout users after three unsuccessful attempts. Locking out users after three unsuccessful attempts make is less likely that password attacks will occur on the virtual system.

(VCGEN0122: CAT II) The IAO/SA will ensure all passwords are created and maintained in accordance with the policy outlined in DoDI 8500.2, IA Controls IAIA-1, and IAIA-2. Passwords will be 9 characters in length with a character mix of upper case letters, lower case letters, numbers, and special characters, including at least two of each (e.g., DemPa3*2!). –Caveat: Password length may vary depending on the INFOCON level.

(VCGEN0124: CAT II) The IAO/SA will ensure all stored passwords at rest are encrypted.

(VCGEN0126: CAT II) The IAO/SA will ensure the maximum number of unsuccessful logons is configured to three by the virtualization server.

3.1.4Virtual Machine Tools

Virtualization server products provide an optional set of virtualization platform extensions to its virtual machines in packages. For instance, VMware has an extension known as VMware Tools, and Microsoft Virtual Server has an extension known as Virtual Machine Additions. These tools contain a set of optimized, guest operating system-specific drivers and services designed to enhance performance of virtual machines and add integrated features. Although virtual machines can operate without these tools, it is highly recommended that these tools are installed into supported guest operating systems to optimize not only overall performance of the virtual

UNCLASSIFIED16


machine, but the usability and manageability. These packages are not automatically installed into guest operating systems.

Virtual machine tools provide a set of system services in addition to the optimized drivers in a component referred to as the virtual server service or the guest service. These services provide such features as including a “heartbeat” signal, time synchronization, optimized drivers, and automated scripts. These services are only available if the virtual machine tools are installed and the service is running.

Some virtual machine tools, such as VMware Tools, include clipboard integration for virtual machines. Accessing the virtual machine via the Virtual Machine Remote Console (VMRC) application enables data to be exchanged between the virtual machine and the machine running the VMRC using the clipboard’s cut, copy, and paste operations. The clipboard copy operation can transfer only text, not files or streams. In a windows environment, where the guest virtual machine and the VMRC machine are both running windows, the normal windows clipboard functions are available between the two computers across the VMRC boundary. When exchanging clipboard data with the virtual machine running a Linux-based operating system, the Linux-based operating system must be running X windows as clipboard integration does not work with Linux-based operating systems running in command-line mode (runlevel 3).

The clipboard copy operation within the VMRC has a vulnerability that needs to be addressed. To enable the clipboard function requires a check of the clipboard, and a reboot of the virtual machine. If the virtual machine is not rebooted, this functionality will not work. When the clipboard feature is turned on and working, the direction of the clipboard content transfer is the same as the direction of the focus change between virtual machine and host operating systems and vice versa. However, when the host operating system clipboard is empty and the focus is moved to the virtual machine clipboard, the virtual machine clipboard is not cleared and left with its current content. Now, when the focus is returned back to the host operating system clipboard, the empty clipboard is now filled with the content of the virtual machine clipboard. Thus the host operating system clipboard is failing to keep itself erased, and its previously cleared content is re-filled from the virtual machine. This behavior may re-fill the host operating system’s clipboard with data that was intentionally erased (like password or credit card number). This behavior does not happen when the process is started from the virtual machine clipboard. So, the issue is only when the process starts from the host operating side.

Several security issues arise as a result of this behavior this clipboard behavior. The first is that the administrator might turn on the clipboard transfer and use it. However, deselecting the clipboard check box will not turn off the function, since a reboot is required. So, the clipboard function is still active. Therefore, transferring text objects, such as a password from one clipboard to another, in any direction between the virtual machine and the host operating system is possible. Secondly, this breaks the virtual machine isolation. This may cause information leakage and potentially infect other operating systems if the text is a string that can be run as a command or URL. As a result, all clipboard capabilities will be disabled within the virtual machine.

UNCLASSIFIED17


(VCGEN0128: CAT II) The IAO/SA will ensure virtual machine tool’s clipboard functions (cut, copy and paste) are disabled within all virtual machines.

Virtual machine tools provide the capability to pass string values to the virtual machine from the host virtual server through a scripting language such as Perl, Windows batch file, Linux Bash shell, etc. The strings must be placed into the virtual machine configuration file and its value must be added to a new line and be set to a relevant string value. Once the virtual machine is booted, a special command can be used inside the virtual machine’s guest operating system to obtain the value. For instance, a VMware script can run inside the virtual machine to execute a command-line to obtain the value: VMwareService –cmd machine.id.get where machine.id = “Jupiter 172.16.100.12”. This string would change the hostname and IP address of the virtual machine. Passing strings to the virtual machine via the virtual machine configuration file will be disabled since this too breaks the virtual machine isolation, and potentially incorrect parameters may be passed causing a denial of service.

(VCGEN0130: CAT II) The IAO/SA will ensure passing strings to the virtual machine from the virtualization server are disabled.

3.1.5Time Synchronization

Generally speaking, PC-based operating systems keep track of time by counting timer interrupts or ticks. When the operating system starts up, it reads the current time to the nearest second from the computer's battery-backed (CMOS) real time clock or queries a network time server to obtain a more precise time. To update the time from that point on, the operating system sets up one of the computer's hardware timekeeping devices to interrupt periodically at a known rate (i.e., 100 or 1000 times per second). The operating system then fields these interruptions and keeps a count to determine how much time has passed.

Supporting this form of timekeeping accurately in a virtual machine is difficult. Virtual machines share their underlying hardware with the virtualization server operating system. Other applications and other virtual machines may also be running on the same virtualization server. Thus, at the moment a virtual machine should generate a virtual timer interrupt, it may not actually be running. In fact, the virtual machine may not get a chance to run again until it has accumulated a backlog of many timer interrupts. In addition, even if a virtual machine is running at the moment when one of its virtual timer interrupts is due, the virtual machine may not check for the interrupt at that moment and deliver it to the guest operating system on time.

As a result, the accuracy of time within the virtualization environment is problematic due to this timer interrupt. Sometimes the time drift may be dramatic such as a loss of 5-10 minutes. Inaccurate time causes other inaccuracies within the virtualization environment, which may include event logs, domain synchronization, session timeouts, etc.

Virtual machine time synchronization may be achieved through an external time source or through the virtualization server operating system. Configuring time synchronization for virtual machines to an external source keeps the virtual machines in synchronization with the rest of the network. However, the external time source is not aware of the unusual clock behavior of virtual

UNCLASSIFIED18


machines. Therefore, it does not synchronize accurately. As a result, synchronizing the virtual machine with the virtualization server is the preferred method for time synchronization.

NOTE: Do not enable both virtual machine tools time synchronization and non-virtual machine clock synchronization software since they will run without knowledge of each other, causing problems.

(VCGEN0132: CAT II) The IAO/SA will ensure all virtual machines are time synchronized with the virtualization server or an approved authoritative time server.

The virtualization server operating system will require time synchronization to an authoritative time server to maintain accurate time. Most environments will have a centralized time server already configured for the network. Time servers are configured in the /etc/ntp.conf file on UNIX systems. Once they are configured with an atomic clock, the ntpd daemon should be configured to start at the runlevels 3, 4, and 5. Windows servers are configured via the command line using the net time /setsntp:clock.isc.org. The w32time service will need to be configured to start after the change.

(VCGEN0134: CAT II) The IAO/SA will ensure the virtualization server is time synchronized to an approved authoritative time server.

3.1.6Deleting Virtual Machine Files

Deleting virtual machines is more involved than simply selecting remove or delete from the web console. Simply removing the virtual machine from the virtualization server will not delete the virtual machine configuration files. The virtual hard disk files and several configuration files may still reside on the system. The administrator must know the location of the virtual hard disk, virtual machine configuration files, and networking files. All these files or settings must be deleted for the virtual machine to be removed from the system. If virtual configuration files are not properly removed, then residual data will be left on the system. This residual data may be viewed by unauthorized users.

(VCGEN0136: CAT III) The IAO/SA will ensure all virtual machine files no longer in use are deleted and purged from the network.

3.1.7Virtual Machine Owners

In traditional computing environments, servers were usually assigned to various personnel for administration. For instance, the data server was administered by the database administrator; the domain controller was maintained by the network administrator, etc. Other methods include assigning the MAC address to specific personnel or identifying machines by Ethernet location or port number. All these approaches are impractical in the virtual machine environment.

In the virtual environment, virtual machines may be moved or have MAC addresses that may change. This mobility makes it difficult to establish who owns the virtual machine running on a particular host. Also, virtual machines may have static or dynamic MAC addresses. The

UNCLASSIFIED19


conventional assignment methods become impractical for virtual machines. Therefore, a policy will need to be implemented to identify and assign virtual machines to the appropriate personnel.

(VCGEN0138: CAT III) The IAO/SA has a policy and procedure in place to identify and assign virtual machines to the appropriate personnel.

3.1.8Snapshots

Traditionally, a physical server’s lifetime can be envisioned as a straight line where the current state of the machine is a static point forward as software executes, configuration changes made, and software is installed. In a virtual environment the virtual machine state is more akin to a tree, where at any point the execution can fork of into N different branches. These different branches are the multiple instances of the virtual machine running or existing at any point in time. Branches are caused by undo-able disks and checkpoint features that allow virtual machines to be rolled back to previous states in their execution.

This execution model conflicts with the assumptions of patch management and maintenance that rely on monotonic forward progress. Rolling back a virtual machine can re-expose patched vulnerabilities, re-enable previously disabled accounts or passwords, remove log files of a machine, use previously retired encryption keys, and change firewalls to expose vulnerabilities. Rolling back virtual machines can also reintroduce malicious code, and protocols reusing TCP sequence numbers that had been previously removed, which could allow TCP hijacking attacks. A subtler issue with rolling back virtual machines is an attackers memory of what has been seen cannot be erased. For instance, a one-time password transmitted in the clear will be seen by the attacker and replayed at a moment of their choice. Therefore, virtual machine rollbacks will be performed when the virtual machine is off or disconnected from the network.

(VCGEN0142: CAT II) The IAO/SA will ensure any rollback of a production virtual machine to a previous state is performed while the virtual machine is off or disconnected from the network.

NOTE: This does not apply to snapshot backups.

(VCGEN0144: CAT II) The IAO/SA will ensure virtual machines log files are saved for auditing purposes before any virtual machine rollback occurs.

3.1.9Moving Virtual Machines

Virtual machines provide mobility similar to a normal file. This portability gives rise to a host of security problems. In the virtual machine world, the trusted computing base consists of all the hosts that the virtual machine has run on. If no history was maintained for each virtual machine, this can make it very difficult to figure out how far a security compromise has extended if the virtual machine has been moved several times. For instance, if a file server has been compromised, any virtual machine that was on the server may have been compromised by an attacker. Determining which virtual machines were exposed can be difficult. Another issue is porting a virtual machine from an employee’s workstation to the production environment. From a theft perspective, virtual machines are easy to copy and move to a person’s USB drive, portable

UNCLASSIFIED20


hard drive, etc. An insider could potentially move the organization’s entire data center on any type of removable media that had sufficient space.

(VCGEN0148: CAT III) The IAO/SA will ensure virtual machines have a log file detailing when the virtual machine was moved from one physical server to another, and who authorized the move.

(VCGEN0150: CAT II) The IAO/SA will ensure all virtual machines are not removed from the site unless it has been approved by the IAO/SA.

3.1.10 Logging and Auditing

Virtualization server sends information about events that occur in the application to the logging daemon or event log. Each event has a unique identifier to track various events with automation tools. The types of events that are sent to the log file include any changes to a virtual machine’s power state, when a virtual machine is powered on or off, suspended or resumed, the addition or removal of virtual machines from the inventory, and the deletion of a virtual machine from the virtualization server system. These events are important for auditing purposes so any unusual power state changes may be reviewed at a later date.

(VCGEN0152: CAT II) The IAO/SA will ensure the virtualization server logs all virtual machine power states, suspends, resumes, deletions, and inventory changes.

(VCGEN0154: CAT II) The IAO/SA will ensure the virtualization server logs are reviewed daily for any anomalies.

(VCGEN0156: CAT III) The IAO/SA will ensure log files for all virtualization servers are stored online for 30 days and offline for 1 year.

3.2 Virtualization Server Operating System Configuration

Managing the virtualization server is possible through web-based access, SNMP, SSH, Application Programming Interfaces (API) interfaces, etc. The virtualization server host operating system is vulnerable to attack depending on the operating system utilized. Therefore, the virtualization server will be configured according to the appropriate operating STIG.

(VCGEN0158: CAT II) The IAO will ensure the virtualization server is configured in accordance with the appropriate operating STIG.

NOTE: The following findings will be open after the UNIX SRR is run against the ESX Server operating system. The executable stack GEN003540 will not be applicable. However, the GEN006640 may be closed by installing Symantec Antivirus software available at the JTG-GNO site.

- Executable StackGEN003540 (CAT II) OPEN

UNCLASSIFIED21


FINDING DESCRIPTION GEN003540: The SA will ensure the executable stack is disabled.SYSTEM CONFIGURATION: VMWare 3.0.1 does not support this configuration.The kernel has executable stack enabled.

- Virus ProtectionGEN006640 (CAT I) OPENFINDING DESCRIPTION GEN006640: An approved DOD virus scan program in not used and/or updated.SYSTEM CONFIGURATION: Unable to install McAfee Virus scan command-line tool on VMware ESX. Some of the prerequisite filesets for this product conflict with the versions required by VMware Operating System filesets.

Once the virtualization server is configured and operating, unnecessary services need to be disabled. Utilizing unnecessary services opens up ports and vulnerabilities that may be exploited to gain access to the server. Furthermore, these unnecessary services consume CPU cycles and memory. Within the virtualization environment where resources are shared, resources should be preserved if possible. For instance, running screen savers on the host or virtual machines consumes a lot of CPU. All unnecessary services will be disabled on the virtualization server.

(VCGEN0160: CAT III) The IAO/SA will ensure the virtualization server has all unnecessary services or daemons disabled.

The virtualization server will have a group structure design that allows the staff to manage the virtual machines they own. Those personnel should have individual logins so that there is an audit trail in the event of problems. The virtualization server may utilize an external directory server (Active Directory, LDAP, etc.) for authentication of virtualization server administrators and virtual machine administrators. Virtual machine users will not have virtualization server logins, since there is no inherent need. Virtualization server users will be created for personnel who will be creating or administering those virtual machines. Virtualization server users who create, access, and modify virtual machines will not have the additional administrative privileges of the root user.

(VCGEN0162: CAT II) The IAO/SA will ensure the virtualization server is configured with a user and group structure for administering all virtual machines.

3.2.1Virtual Machine Creation

The default location for virtual machine files is the home directory of the user that created the virtual machine. Home directories will not be used for storing virtual machine files. Home directories are used for user’s files, not a virtual machine that could be used by many users. Users will create and store virtual machines in a central directory structure to ensure permissions, users, and files are created to share the virtual machines.

(VCGEN0164: CAT II) The IAO/SA will ensure virtual machine files are not stored in the users’ home directory.

UNCLASSIFIED22


Virtual machines by default typically run as the user who created the virtual machine. This implies that the user must have rights to the virtual machine’s files. These rights can easily be configured by assigning them to the directory. Running the virtual machine as the creator of the virtual machine is the default configuration. However, the virtual machine may be powered off when the user logs off. Powering off a virtual machine due to the owner logging off while it is in use could have unforeseen consequences to applications or system users. As a result, the virtual machine will be configured to run as a system user to ensure the virtual machine is always available.

(VCGEN0165: CAT II) The IAO/SA will ensure virtual machines are configured to run as local system accounts, not individual user accounts.

NOTE: The ESX Server virtual machines run as root, and not the user who created it.

Any user with access to the virtualization server has the ability to create a virtual machine or virtual disk file on the virtualization server. The number of users that should be allowed to access and create virtual machines should be limited. Without any controls in place, a user may accidentally consume too much disk space on the host server or add an un-patched virtual machine that could expose other virtual machines or physical machines on the same network. Therefore, users will be restricted on the virtualization server with only the necessary permissions to administer their virtual machines.

(VCGEN0166: CAT II) The IAO/SA will ensure only authorized users have access to the virtualization server to create virtual machines or virtual hard disks.

With the creation of virtual machines, the actual virtual topology becomes increasingly complex. The topology changes when a virtual machine is created, added to a virtual switch or port group, moved to another virtualization server, etc. With the dynamic nature of the virtualization environment, administrators of the virtualization environment will maintain up to date documentation all virtual machines, virtual switches, IP addresses, MAC addresses, etc. This documentation will give a baseline to administrators to use when designing and troubleshooting virtual networking, machines, and other issues with the virtual infrastructure.

(VCGEN0168: CAT II) The IAO/SA will have up to date documentation of all virtual machines running on all physical servers detailing the IP addresses, MAC addresses, virtual switches, virtual operating systems, and virtual applications.

3.2.2Virtual Machine Permissions

Access to a virtual machine is based on the permissions granted to the virtual machines configuration file. Specific permissions let users’ access virtual machines in different ways. These access methods include browsing, interacting, configuring, and administering virtual machines. Browsing a virtual machine connects users to the console displaying only the virtual machine’s power state. The virtual machine display is blank, even if the virtual machine is running. Interaction with the virtual machine is not permitted. To browse a virtual machine, users need

UNCLASSIFIED23


read permission for the virtual machines configuration files on a Windows host, or read (r) permission on a Linux host.

Interacting with a virtual machine lets users change its power state (power on or off,suspend, or resume) and connect or disconnect removable devices. However, changes to the virtual machine’s configuration file are not permitted. Among other restrictions, this means no changes may be made to the virtual hardware. To interact with a virtual machine, users must have read and execute permission for the virtual machines configuration files on a Windows host, and read and execute (r and x) permissions on a Linux host.

Configuring a virtual machine lets you add and remove virtual hardware from the virtual machine. To configure a virtual machine, user’s must have read and write permissions for the virtual machine’s configuration file and virtual machine resources (such as a physical disk or certain devices) on a Windows host, or read and write (r and w) permissions on a Linux host.

Virtual machine administrators will need user permissions to connect, interact, and configure a virtual machine. For virtual machine administrators to configure the virtual machine through the remote console, the user must have read, write and execute access to that virtual machine’s configuration file.

(VCGEN0170: CAT II) The IAO/SA will ensure only authorized virtual machine administrators have read and execute access to the virtual machine’s configuration file and directories.

Some virtualization servers by default create virtual machines as private virtual machines. When a virtual machine is private, it appears in the inventory of the console of the user who created it. The virtual machine does not appear in the inventory of consoles for other users connected to the host. Other users cannot browse to the virtual machine and add it to the inventory. If the virtual machine is made private after it has been created, it disappears from other users’ inventories.

(VCGEN0174: CAT II) The IAO/SA will ensure all virtual machines are configured as private so unauthorized users cannot browse other virtual machines.

NOTE: This is not applicable to the ESX Server platform.

3.2.3Antivirus

The virtualization server will have antivirus and anti-spyware applications installed. These applications consume many resources during their scans. If too many resources are allocated to the anti-virus scan, the virtualization server may deny resources to the virtual machines potentially causing a denial of service. Therefore, all real-time anti-virus or anti-spyware protection scans will run at non-peak times. Non-peak times will be determined by the site based on their performance baselines and activity.

Furthermore, administrators will disable anti-virus real-time scanning on virtual hard disks and virtual memory files. These are the files that contain your virtual machine’s virtual disk and virtual memory. When a virtual machine is active and running, the virtual machine process

UNCLASSIFIED24


regularly reads and writes to those files. If your anti-virus' real-time scanner has to scan every read and write, it can slow down the effective performance of the virtual machine. Therefore, to prevent any potential availability issues due to performance bottlenecks, virtual machine hard drive disks and memory files will be excluded from real-time scans.

(VCGEN0176: CAT III) The IAO/SA will ensure the virtual server has anti-virus and anti-spyware scans scheduled to run at non-peak times.

(VCGEN0178: CAT III) The IAO/SA will ensure the virtual server is not configured to scan virtual machine hard disks and memory for viruses and spyware.

Creating new virtual machines is as easy as copying a file. Copying files is a quick and efficient way to rollout new virtual machines. Virtual machines can grow at an explosive rate and really tax the security systems of an organization. Many administrative tasks that may be automated, but some upgrades and patches require manual tools. For instance, virtual machines may need to be patched, scanned, and purged in response to a virus or worm attack on the network. Therefore, to protect against potential virus and spyware infections, all off and suspended virtual machines will have the latest up to date anti-virus software and signatures.

(VCGEN0180: CAT II) The IAO/SA will ensure all virtual machines in the power state of off or suspend will have the latest up to date anti- virus software and signatures.


3.3 Guest Operating System Configuration

Guest operating systems are configured and installed within virtual machines on the virtualization server. Depending on the virtualization server, the number of hosts may vary. For instance, the ESX Server may host up to 128 virtual machines at one time. Guest operating systems may require different resources depending on the server function. A database or email server will require more resources than a basic Windows Domain Controller. Therefore, proper planning is required to determine what servers will be built within the virtualization server environment.

To properly create virtual machines within the virtualization server environment, a minimal list of requirements will be determined. These requirements are the amount of memory, amount of required disk space, the networking card assignment, required media, and proper disk mode to be used.

(VCGEN0182: CAT II) The IAO/SA will ensure virtual machine requirements are documented before creating virtual machines within the virtualization server environment. These requirements include amount of memory, amount of required disk space, networking card assignment, required media, and the proper disk mode.

UNCLASSIFIED25


3.3.1Guest OS Selection

All guest operating systems must be supported by the virtualization server product. Virtualization servers present physical hardware through a virtualized “layer.” While this hardware is presented as “generic” hardware, it still requires specific drivers and functionality, just as a physical server needs drivers. Virtualization servers provide drivers for its virtual hardware for a number of operating systems that it supports.

In general, unsupported guest operating systems will run slower than supported operating systems with the guest enhancement software installed. There will usually be performance and usability issues when attempting to access virtual machines that have unsupported guest operating system installed using the virtualization platform’s native remote control technology, such as virtual machine remote console. This is primarily due to the lack of optimized keyboard, mouse, and video drivers in the guest operating system usually supplied by the guest enhancement software package.

Selecting the correct guest operating system for each virtual machine is important. Virtualization servers optimize certain internal configurations on the basis of this selection. For this reason, it is important to set the guest operating system correctly. The correct guest operating selection can greatly aid the operating system chosen and may cause significant performance degradation if there is a mismatch between the selection and the operating system actually running in the virtual machine. The performance degradation may be similar to running an unsupported operating system. Selecting the wrong guest operating system is not likely to cause a virtual machine to run incorrectly, but it could degrade the virtual machine’s performance.

(VCGEN0184: CAT III) The IAO/SA will ensure all virtual machine’s guest operating selection correctly matches the operating system installed.

3.3.2Guest OS Logging

Virtual machines may log troubleshooting information into a virtual machine log file stored in the client operating system. Normal, non-root, or non-administrator users and processes in the virtual machine may abuse this logging and cause large amounts of data to be logged. Over time, a log file can consume the file system space designated for the client operating system and cause a denial of service attack. Therefore, rotating logs is critical to prevent file system problems. Also, the virtualization vendor may not offer technical support without virtual machine logging enabled.

(VCGEN0186: CAT II) The IAO/SA will ensure all virtual machines have logging enabled for troubleshooting and auditing purposes.

3.3.3Controlling Guest Resource Usage

Guest virtual machines are allocated hardware resources based on minimums, maximums, and shares. Resources allocated to virtual machines may be memory space, CPU time, network bandwidth, and disk bandwidth. The minimum and maximum resource settings within

UNCLASSIFIED26


virtualization server are absolutes values, whereas shares are used to give preference to a guest operating system when a resource is scarce.

Virtual machines will disable CPU intensive processes since they will directly impact the virtualization server performance and may potentially cause resources to be scarce. Scarce resources are distributed based on availability and priorities assigned to virtual machines. To ensure all virtual machines have sufficient and available resources, screen savers and hibernation within the virtual machines will be disabled.

(VCGEN0188: CAT III) The IAO/SA will ensure guest operating systems have screen savers disabled.

(VCGEN0190: CAT III) The IAO/SA will ensure guest operating systems have hibernation disabled.

Memory minimum and maximum resource settings are specific for each virtual machine. To illustrate settings, a virtual machine may have the maximum memory of 384MB and the minimum to 0MB. This means that the virtual machine may never have more than 384 MB of physical memory or less than 0MB of physical memory. Assuming this virtual machine is idle and other virtual machines require access to physical memory, this virtual machines memory can be redistributed to other active virtual machines all the way down to 0MB. The reported memory to the guest (384MB) is not physical memory but instead swapped memory. To guarantee that the a virtual machine has a certain amount of physical memory at all times, a higher minimum will need to be set other than 0MB.

Minimum and maximum resource settings are applicable to the CPU as well on the virtual machine. CPU or processor resources can be set to the minimum and maximum of 0 and 100% (the default is min 0 and max 100%). For instance, setting the minimum CPU to 10% will allocate the virtual machine 10% processor. However, the drawback to this configuration is that the virtual machine may be idle, and the processor has reserved 10% for just this virtual machine. Conversely, setting the maximum amount of processor utilization on any virtual machine limits the virtual machine even though it might require more processor time. If the virtual machine needs more processing time and cannot get more, then the virtual machine will take longer to complete its tasks. This is basically “processor throttling” at the host level for that virtual machine. The key to understanding resources is the consequences of setting maximums and minimums. Minimums guarantee a specific amount of a resource to the virtual machine but deny that much of the resource to other virtual machines. Maximums deny the virtual machine a portion of a resource while allowing other virtual machines more access to that resource.

Setting the minimum and maximum memory and CPU resources requires some planning for multiple virtual machines on the same virtual server. If one virtual machine has been set to the minimum CPU or memory that the virtual server has available, and then the other virtual machines will not be able to function. If virtualization server is unable to guarantee a virtual machine’s specified minimum percentage, it will not allow the virtual machine to power on.

UNCLASSIFIED27


(VCGEN0192: CAT II) The IAO/SA will ensure all minimum resource settings for memory and CPU for each virtual machine does not equal the total physical amount available.

3.4 Virtualization Server Networking

Virtual networks are used to connect virtual machines to internal or external networks. Virtual network components are utilized to connect virtual machines to internal and external networks. These network components include bridges, NAT, host-only, virtual switches, virtual DHCP server, and virtual network adapters. With these components, virtual servers may offer three types of network configurations. These configurations are bridged, Network Address Translation (NAT), and host-only networking. In order to facilitate communication a virtual switch, a virtual network adapter, and optionally a virtual DHCP server are utilized. Depending on the product the number of virtual network adapters assigned to each virtual machine may vary, but the maximum is usually four per virtual machine. Each network component is described below.

The bridged network configuration allows a virtual machine to access a physical network that the virtualization server is connected to. The virtualization server does not necessarily need to be participating on the network. This is analogous to thinking of the virtual network adapter in the virtual machine being connected to the physical Ethernet adapter in the virtualization server.

The NAT network configuration enables communication between virtual machines and the external network. Using a NAT device becomes advantageous when there is a limited amount of IP addresses available on the physical network and those IP addresses are being used by the physical servers.

The host-only network configuration creates a virtual Ethernet LAN on the virtualization server that allows communication between the virtualization server and virtual machines on that virtualization server. By default, the host virtual adapter is not connected to any external network.

A virtual switch is similar to a physical switch in that is connects networking components together. A virtual switch can be connected to the physical network or it can be completely virtual and therefore isolated from the outside network. Depending on the virtualization server operating system the number of virtual switches that can be created varies.

(VCGEN0194: CAT II) The IAO/SA will ensure virtual switches not in use are deleted or disabled.

NOTE: This is not applicable to Vmware VMotioned virtual machines.

The DHCP server is useful when virtual machines are configured to use host-only or NAT configurations. The DHCP server provides a range of IP addresses to virtual machines that are not bridged to an external network. Using the virtual DHCP server is much faster than using an external one since the addresses will be retrieved at bus speed.

UNCLASSIFIED28


The virtual network adapter is added to each virtual machine for connectivity to internal and external networks. Typically, the maximum number of virtual network adapters allowed per virtual machine is four.

3.4.1Networking Configurations

There are three types of networking configurations that may be implemented within a virtualization server. These types are bridged networking, NAT networking, and host-only networking. In each of these configurations, a Windows host operating system may connect an unlimited number of virtual devices to a virtual switch, while a Linux host operating system can only connect up to 32 devices.

The bridged network configuration allows a virtual machine to access a physical network that the virtualization server is connected to. Bridged networking is the easiest way to connect a virtual machine to the external LAN. With bridged networking, the virtual machine has two-way communication on the LAN. This means it can access other equipment on the network and it can be accessed by other devices on the network. Each virtual machine that is configured to use bridged networking must be assigned a unique IP address. Figure 3-2 illustrates a bridged network.

Figure 3-2. Bridged Network

(VCGEN0196: CAT II) The IAO/SA will ensure all virtual machines bridged to the external network have a valid external network IP address.

NAT networking allows the virtual machine to transcend the private network and communicate with the external LAN. NAT is a good choice if IP addresses are scarce on the external network.

UNCLASSIFIED29


When using this type of networking, the guest operating system does not have an IP address on the external network. Instead, a private network is setup on the virtualization server and the guest operating system receives an internal IP address from the virtual DHCP server. The virtual machines then communicate with a router node, the NAT device, which passes network data between one or more virtual machines and external networks. Communication across the NAT device is recorded in translation table and the traffic is then funneled back to the correct destination. Figure 3-3 illustrates a NAT network.

Figure 3-3. NAT Network

The NAT Service used in some virtualization products have been exploited by buffer overflows. These buffer overflows allow an attacker to execute arbitrary code with the privileges of the NAT service and potentially cause a denial of service. When a virtual machine is using NAT networking, a malicious guest virtual machine could potentially use a specific NAT networking configuration to execute unwanted code on the host virtualization server. Upgrading to the latest version of software should fix the vulnerability.

(VCGEN0198: CAT II) The IAO/SA will ensure NAT networking is not implemented in a production environment.

NOTE: This does not apply to the ESX Server.

Host-only networking provides a network connection between the virtualization server and the virtual machines located on that server. It uses a virtual network adapter that is visible only to the virtualization server. The entire network is virtual and isolated from everything on the

UNCLASSIFIED30


outside of the virtualization server. Only the virtual machines on the virtualization server and the virtualization network adapter are connected to a private TCP/IP network. Communication also is possible between the virtual machines since IP addresses may be provided by the virtual DHCP server. Figure 3-4 illustrates a Host-Only networking.

Figure 3-4. Host-Only Network

Some virtualization server products such as VMware Virtual Server include utilities for troubleshooting virtual networks. These tools are used to sniff packets from the virtual machines virtual network adapters. These tools will only be accessible to the administrator of the host virtualization server, since these tools could expose information about databases, users, passwords, records, etc. Restricting access to these tools to the administrator or root user of the virtualization server provides some level of protection against unauthorized use. Furthermore, these tools should be used only when troubleshooting a virtual infrastructure problem.

(VCGEN0200: CAT II) The IAO/SA will ensure all virtualization server packet sniffing utilities are restricted to only the administrator or root user of the virtualization server.

3.4.2MAC Addresses

Every Ethernet network interface card, whether physical or virtual, has a unique identifier assigned to it know as the media access control or MAC address. Ethernet MAC addresses are typically shown as a string of 12 hexadecimal digits. The first six digits identify the vendor ID or the manufacturer of the card and are known as the Organizational Unique Identifier (OUI). The OUI prefixes are assigned to organizations by the IEEE. The last six digits are assigned by the manufacturer of the card and are known as the burned-in address. The MAC address of the

UNCLASSIFIED31


virtual network adapter will be unique and whether it is dynamic or static. For instance, the MAC address 00:09:6B:BF:6D:31, the OUI would be 00:09:6B, which is IBM. Virtualization vendors do not manufacture any physical network adapters, however, they do provide virtual ones. For instance, VMware has two OUIs. The first VMware OUI is for automatically generated MAC addresses, and the second VMware OUI is used for MAC addresses that are manually set.

Once the virtual MAC address has been generated it becomes static, unless the virtual machine is moved to a different location. For instance, a virtual MAC address could change if the virtual machine has a different path on the same server or moved to a different virtualization server. MAC addresses are saved in the configuration file of the virtual machine. The MAC address of powered-off virtual machines is not checked against running or suspended virtual machines. Therefore it is possible that when a virtual machine is powered on again, it can get a different MAC address.

(VCGEN0202: CAT III) The IAO/SA will ensure all dynamic or static MAC addresses of all virtual network adapters are unique.

3.5 Virtual Machine Hard Drive Disk Management

3.5.1Sharing and Moving Virtual Machine Files

As virtual machines replace real hardware they can undermine the security architecture of many organizations which often assume predictable and controlled change number of hosts, host configurations, hosts location, etc. Some useful mechanisms that virtual machines provide are copying or sharing virtual machine hard disks. Copying or sharing virtual machine hard disks can be done over networks and removable media. Typically, test and development virtual machines will be moved and updated more frequently than production virtual machines. There will be a policy in place to restrict the copying and sharing of production virtual machines over networks and removable media to ensure that administrators do not give unauthorized users access to the virtual machine files.

(VCGEN0204: CAT III) The IAO/SA will have a policy in place to restrict copying or sharing virtual machine files over networks and removable media.

NOTE: This does not apply to snapshot backups, disaster recovery virtual machines, and virtual machines in a cluster.

3.5.2Virtual Hard Drive Disk Types

Virtual hard disks provide storage for virtual machines. Within the virtual machine, the virtual hard disk is represented as a physical disk and is used by the virtual machine as if it were a physical disk. Technically, the virtual hard disk is a file that resides on a physical disk that the virtual server can access. On the physical disk, the virtual hard disk file is stored as a .vhd, .vmdk, etc.

UNCLASSIFIED32


The virtualization server may create a virtual hard disk on any storage that the host file system can access including IDE, SCSI, RAID, SAN, NAS, and so forth. The virtual machine accesses the virtualized IDE/SCSI device and the VMM maps that to the physical device access. Virtual machines can access virtual hard disks on any storage topology, including SAN and NAS configurations, supported by the underlying virtualization server.

Virtual hard disk drives support additional features and functionality that are not usually available with physical hard disk drives. These features are available through the use of different virtual hard disks. A virtual hard disk drive may exist as a fixed disk, dynamic disk, differencing disk, or a raw disk. Additionally, virtual hard disk drives may be configured with a redo or undo log file that alters the persistence behavior of the drive. Each of the hard disk drive types are discussed below.

A fixed disk is a virtual hard disk drive that most closely resembles a physical hard disk drive. The fixed disk is represented by a fixed-extent file residing on the virtualization server. When a fixed disk is created, the file is sized immediately to its maximum size, consuming an equal amount of storage from the virtualization server. The file on the virtualization server will always remain the size it was originally given. Fixed disks will typically provide better performance than dynamic disks because there is no overhead due to growth. Fixed virtual disks will be allocated according to the space requirements for the server. These virtual disk sizes will be verified for integrity. Figure 3-5 illustrates a fixed disk

Figure 3-5. Fixed Disk

(VCGEN0206: CAT III) The IAO/SA will ensure all fixed virtual hard disk drives are allocated according to the space required by the documented virtual machine requirements.

A dynamic disk is a virtual disk drive that begins as a sparse file, consuming only the amount of storage from the virtualization server that is needed, and grows as new data is written to the virtual hard disk drive. When the dynamic disk is created, the maximum size of the disk is specified, but the file representing the disk is sized only to the size needed to store its data at that point in time. Once the process of installing the guest operating system begins, the dynamic disk will grow as new data is written to fill the disk. When files are deleted from a dynamic disk, the file on the virtualization server representing the dynamic disk will not change or shrink. Instead, deleted sectors are marked and are reused as necessary. This type of disk could be problematic if the size specified consumes the entire physical disk. One virtual machine may consume the entire disk space gradually if not configured with the appropriate maximum values. Figure 3-6 illustrates a dynamic disk.

UNCLASSIFIED33


Figure 3-6. Dynamic Disk

(VCGEN0208: CAT II) The IAO/SA will ensure all dynamic virtual hard disk drives have a maximum file size value specified.

A differencing disk is a special type of dynamic virtual hard disk drive that is dependent on the parent disk. Differencing disks share a parent/child relationship with another virtual hard disk drive in which the differencing disk is the child. The combination of the differencing disk and its parent virtual hard disk drive create a single virtual hard disk drive as it appears to the virtual machine to which the differencing disk is attached. Once a differencing disk is linked to a parent disk, the parent virtual hard disk drive becomes read-only. Any changes made thereafter of the differencing disk controls the maximum disk size overall. When a read on the disk occurs, the most recent changes are always returned. First the read on the parent disk and then on the differencing disk, returning the most current data from the differencing disk. If the parent of the differencing disk is ever altered in any way, the differencing disk will become invalid, as its relationship to the parent disk depends on the state of the parent disk at the time they are linked. Differencing disks provide the ability to save changes without actually altering the parent disk. This type of scenario is best suited for development and testing. As a result, differencing disks will not be used for production environments.

(VCGEN0210: CAT II) The IAO/SA will ensure differencing virtual hard disks are not used in production virtual environments.

A raw disk is the mounting of either an entire physical hard disk drive or a logical volume on a physical hard disk drive to a virtual machine. Raw disk are useful in special circumstances where virtual machine may need to access data that already exists on the physical disk or when high disk performance is required. Raw disks are not virtual hard disk drives, but appear as any other virtual hard disk drive to the virtual machine to which it was mounted.

By default, all virtual hard disk drives are persistent. This means that changes made to the virtual hard disk drive occur immediately and are permanent. Most virtualization platforms have a feature to modify the persistence mode of a virtual hard disk drive. A redo or undo disk (redo log file) is a special disk that is created when a virtual hard disk drive’s persistence mode is

UNCLASSIFIED34


changed. The undo disk is similar to a dynamic disk. It is linked to its parent disk when it is created, the parent disk becomes read-only, and all changes are written to the undo disk. A virtual machine must be powered off to enable any of its virtual hard disk drives to have their persistence mode changed. Once changed the virtual machine is powered on and booted normally. At the point of powering on the virtual machine, any virtual hard disk drive attached to the virtual machine that is in undo mode and does not already have an undo disk file on the virtualization server has a new, empty undo disk file created. Undo disks are temporary disks that allow users to decide how the data should be handled. Undo disks may be discarded or deleted, and this reverts the parent disk to the previous known state. Undo disks may be committed into their parent disk which merges the changes permanently. Once the undo disk is committed, it is no longer needed since the parent disk has the changes. Undo disk are well suited for the testing and development, since undo disks may be deleted at the users discretion.

(VCGEN0212: CAT II) The IAO/SA will ensure undo or redo virtual hard disks are not used in production virtual environments.


3.6 Virtualization Server Backup and Recovery

Backup and recovery of the virtual environment includes the virtualization server and the virtual machines. The virtualization server represents the virtual infrastructure while the virtual machines are the servers and applications that utilize it. The virtualization server has three major components required for backup and recovery. These components are virtual disks, virtual machine configuration files, and the configuration of the virtualization server itself. Backing up these components is possible through several approaches. These approaches are traditional agent-based backups, non-agent-based backups, flat-file backups, and snapshots.

Please see Section 2.5.4, Backup and Recovery, of the Enclave STIG for operating system backup and recovery guidance and requirements.

3.5.3Traditional Backups

All the information normally backed up in the enterprise infrastructure, including the operating system, applications, and data, is included in the virtual disks. Because a virtual machine is just like a physical machine, one approach is to back it up in the same manner as a physical machine, using backup software running inside a virtual machine. Furthermore, the virtualization server operating system will be backed up with traditional agent-based backups since the operating system is critical to the virtual environment. Due to the array of products and options available to backup the virtualization infrastructure, procedures will need to be developed to provide guidance to administrators.

(VCGEN0214: CAT III) The IAO/SA will ensure procedures exist for backup and recovery of the virtualization server and virtual machines.

Running backup agents on virtual machines is the preferred method for backing up virtual disks, and the virtualization server operating system according to the virtualization vendors. A backup

UNCLASSIFIED35


agent is software that runs as part of the backup software application, and copying files over the LAN or through a storage network to backup media. Backing up virtual machines includes the configuration files too, and these will be backed up by the virtualization server operating system backup agent.

The virtualization server operating system has several critical components that need backed up on a regular basis. These include the virtualization server operating system configurations, virtualization server configurations, and the virtual machine configurations.

(VCGEN0216: CAT II) The IAO/SA will ensure all virtual machines have backup agents installed and configured to backup the virtual machines on a regular basis.

NOTE: This does not apply for VI3 if online snapshots are utilized or VMware Consolidated

Backup (VCB) is implemented. (VCGEN0218: CAT II) The IAO/SA will ensure the virtualization server has a backup agent

installed and configured to back up all virtual machine configuration files and virtualization server components.

Non-agent-based backups may be performed on the virtualization server operating system and virtual machines as well. The backup tools included with each virtual machine and the virtualization server operating system may be adequate for small environments. As with most commercial backup software, Full, Incremental, and Differential backups are available for backup strategies. Full backups are the most complete as they secure every file on virtual hard disk.

Since backups are critical to the recovery of the virtualization infrastructure, storing these files on the same physical server as the production servers is not recommended. The backup files will be stored on a separate physical server so restoration is possible in case of any hardware failures on the production physical servers.

(VCGEN0220: CAT II) The IAO/SA will ensure virtual machine and virtualization server backups are stored on a separate physical device than the production virtualization server and virtual machines.

NOTE: This does not apply for VI3 if online snapshots are utilized or VMware Consolidated Backup (VCB) is implemented.

3.5.4Flat-File Backups

The third option is to perform off-line backups where files encapsulating virtual machines are accessed and backed up without loading the virtualization server operating system that the virtual machines normally run on. This type of backup is similar to database backups where the database first shuts down (taken offline) and the related files are backed up. When a database is online, the application locks the database’s application files, so normal backups are not possible. The virtual machines do the same thing by locking the virtual machine files, and could be using physical memory as a buffer for writes to the virtual disk. So backing up the virtual machine

UNCLASSIFIED36


while online might be corrupt if all the disk writes are not performed. Therefore, to perform a flat-file backup, the virtual machines must be shutdown first before any backups can be performed. Normally, all virtual machine files reside in a single folder, so backing up the virtual machine while off should include making a copy of the virtual machine’s folder. The files that need to be backed up are the virtual hard disk files, the logs, and the configuration files.

The drawback to implementing flat-file backups is that the virtual hard disk files are large. Backing up these files could take a substantial amount of time depending on the size and number is virtual hard disk files. Also, flat file backups do not allow for file level restores within the virtual machine. These drawbacks do not make flat-file backups a viable choice for a regular backup strategy; however, flat-file backups are beneficial for a disaster recovery strategy.

(VCGEN0222: CAT III) The IAO/SA will ensure flat-file backups are not utilized as the primary backup strategy for production virtual machines.

3.5.5Snapshot Backups

Online snapshots or “saved states” are features that are supported within virtual server applications. An online snapshot is a point-in-time copy of a running virtual machine. With an online snapshot saved, this provides the ability to revert to it at a later time. This gives administrators the ability to create an image of a virtual machine with the ability to roll back to the time of the last snapshot.

Snapshots basically freeze the hard disk file and take a snapshot of it. At the exact moment the REDO log is added, the hard disk file is frozen, whether or not the files are open or closed and databases are running or quiesced. The majority of the time there shouldn’t be a problem recovering data, however, there is a chance that there is corrupted data, broken file systems, and uncommitted transactions.

Snapshots are not a viable option for a backup strategy since the virtual machine is in a running state during the snapshot. This means that all the memory was not committed and possible transactions were pending or in the middle of being committed, which means some application data may be missing. Furthermore, there are limitations to this type of backup. For instance, with the VMware ESX Server, once a REDO log is placed on the disk file, the disk file is now in a crash consistent state. A crash consistent state is when the OS and applications were not made aware of the shutdown, so normal shutdown procedures were not processed. It is similar to pressing the power button on the physical machine. Next, file level restores can take a significant amount of time since the entire disk must be restored. Also, the actual time to backup the entire hard drive file could grow longer and longer as the virtualization server environment grows. Finally, there are performance considerations when running with a redo/undo log. The redo/undo log file can grow quite large very quickly. Redo/undo logs need to be committed after the virtual hard disk is backed up. Rewriting the redo/undo log to the virtual hard disk will require CPU time from the virtualization server.

Snapshots for virtual machines that are powered on will not be utilized as the primary backup strategy. However, virtual machines may be powered off, in which snapshots could be utilized

UNCLASSIFIED37


to aid in disaster recovery, testing, demonstrating, and training. It is important to note that most products only support a single saved snapshot per virtual machine.

It should be noted that some third-party software, such as Vizioncore, is available that utilizes snapshots as a backup strategy providing real-time backups for online virtual machines. Redo logs are utilized while the virtual hard disk is backed up, exported, or simply copied. Most third-party software compresses the virtual machine backups removing the burden on the host operating system.

(VCGEN0224: CAT III) The IAO/SA will ensure virtual machines that are powered on do not use snapshots as the primary backup strategy for production virtual machines.

NOTE: This does not applicable to VI3 or higher.

UNCLASSIFIED38


4. VMWARE ESX SERVER CONFIGURATION

4.1 VMware ESX Server

VMware ESX Server is the datacenter-class virtualization platform used by many organizations to consolidate servers. ESX Server implements isolation so if one virtual machine crashes, the others are isolated from the failure and continue running. ESX Server dynamically allocates system CPU, memory, disk, and networking resources to virtual machines based on need or parameters specified by the administrator. ESX Server provides virtual machines mainframe-class capacity utilization and control of server resources.

The virtualization environment has produced new terminology to address concepts and virtualized hardware. These new terms should be defined to understand the virtualization environment. The following list defines common terms used within the VMware environment.

VMware ESX Server – A production-proven virtualization layer run on physical servers that abstract processor, memory, storage and networking resources to be provisioned to multiple virtual machines.

VMware Virtual Machine File System (VMFS) – A high-performance cluster file system for virtual machines. VMFS: the VMware File System. Virtual disks for virtual machines are stored as files in this high-performance, dedicated file system.

Virtual Machine Configuration File: a text file (*.vmx) that declares the virtual hardware composing a virtual machine. These files are stored in the Service Console’s file systems.

VMware Virtual Symmetric Multi-Processing (SMP) – Enables a single virtual machine to use multiple physical processors simultaneously.

VirtualCenter Management Server – The central point for configuring, provisioning and managing virtualized IT infrastructure.

Virtual Infrastructure Client (VI Client) – An interface that allows administrators and users to connect remotely to the VirtualCenter Management Server or individual ESX Server installations from any Windows PC.

Virtual Infrastructure Web Access – A Web interface for virtual machine management and remote consoles access.

VMware VMotion – Enables the live migration of running virtual machines from one physical server to another with zero downtime, continuous service availability and complete transaction integrity.

VMware High Availability (HA) – Provides easy-to-use, cost effective high availability for applications running in virtual machines. In the event of server failure, affected virtual machines are automatically restarted on other production servers that have spare capacity.

UNCLASSIFIED39


VMware Distributed Resource Scheduler (DRS) – Intelligently allocates and balances computing capacity dynamically across collections of hardware resources for virtual machines.

VMware Consolidated Backup (VCB) – Provides an easy to use, centralized facility for agent-free backup of virtual machines. It simplifies backup administration and reduces the load on ESX Server installations.

VMkernel: The VMkernel is a proprietary micro-kernel for the ESX Server.

Service Console: the administrative interface for the ESX VMkernel. The Service Console supports the Web-based Management User Interface, as well as remote access to a virtual machine’s keyboard, display, and mouse. ESX 2.x versions are based on RedHat AS 2.1. ESX 3.x is based on REL3. (future versions 3.x may be REL4 or REL5).

VMware Infrastructure SDK – Provides a standard interface for VMware and third-party solutions to access VMware Infrastructure.

The ESX Server provides an operating environment dedicated to hosting multiple virtual machines. The ESX Server operates on a physical server and has direct access to the physical hardware of the server. This enables high-speed I/O operations as well as having complete resource management control. The major conceptual components of ESX Server are the irtualization layer, the resource manager, hardware interface components, and the service console.

Virtualization layer— or hypervisor implements the idealized hardware environment and virtualizes the physical hardware devices including the CPU, network and disk controllers, memory subsystem, and hardware interface devices. Each virtual machine sees its own set of virtual devices and is isolated from other virtual machines running on the same physical system.

Resource Manager—partitions the physical resources of the underlying machine and allocates CPU, memory, disk, and network bandwidth to each virtual machine.

Hardware interface components— or drivers enable hardware-specific service delivery while hiding hardware differences from other parts of the system. These components include the device drivers and the VMFS.

Service Console—Runs applications that implement support, management, and administration functions.

Figure 4-1 illustrates the basic architecture of ESX Server. Each virtual machine has its own guest operating system and applications. The virtualization layer is comprised of the VMkernel and Virtual Machine Monitor (VMM). The VMkernel controls and manages the physical resources of the underlying server. The VMM implements the virtual hardware for each virtual machine. The resource manager and hardware interface components are implemented in the

UNCLASSIFIED40


VMkernel as well. Finally, the service console provides management and administration services to the ESX Server system.

Figure 4-1. ESX Server Architecture

4.1.1Service Console

The service console provides a control API that allows the virtual machines and resource allocations to be managed. The administrator may manage the virtual machines and resources through web server running in the service console. In addition to the web server, the following processes and services involved in the management of an ESX Server system run in the service console:

- Server daemon (hostd) — Performs actions in the service console on behalf of the VMware Remote Console and the Web-based VMware Management Interface. VI3 uses hostd for the server daemon and it is not configurable with TCP wrappers. Hostd listens on http/https ports.

- Authentication daemon (authd) — Authenticates remote users of the management interface and remote consoles using the username/password database. Any other authentication store that can be accessed using the Pluggable Authentication Module (PAM) capabilities present in the service console may also be used. This permits the use of passwords from a Windows domain controller, LDAP, RADIUS server, or similar central authentication store to be used with VMware ESX Server for remote access.

- SNMP server (net-snmp) — Implements the SNMP data structures and trapsan administrator can use to integrate an ESX Server system into an SNMP-based

UNCLASSIFIED41


system management tool.

To protect these important services on the service console, access control lists will be utilized to ensure only authorized IP addresses are able to access these services.

(VCESX0502: CAT II) The IAO/SA will ensure the ESX Server service console OS Server daemon (hostd), the OS Authentication daemon (authd), and OS SNMP daemon (snmpd) are configured with IPtables to allow only authorized internal IP addresses.

External USB drives may be inserted into the ESX Server and be loaded automatically on the service console. The USB drive will still need to be mounted, but drivers are loaded to recognize the device. Malicious users may be able to run malicious code on the ESX Server and go undetected since the USB drive is external. As a result, USB drives will not be loaded automatically within the ESX Server. This is disabled by modifying the /etc/modules.conf file and commenting out the alias usb-controller text.

(VCESX0504: CAT II) The IAO/SA will ensure USB drives inserted into the ESX Server are not loaded automatically.

4.1.2Setuid and Setgid Applications

Setuid is a flag that allows an application to temporarily change the permissions of theuser running the application by setting the effective user ID to the program owner’suser ID. Setgid is a flag that allows an application to temporarily change thepermissions of the group running the application by setting the effective group ID tothe program owner’s group ID.

During the ESX Server installation, several applications that include the setuid and setgidflags are installed by default. These applications are initiated by or through the serviceconsole. Some of them provide facilities required for correct operation of the ESX Serverhost. Others are optional, but they can make maintaining and troubleshooting the ESXServer and network easier. Disabling any of the required applications will result in problems with ESX Server authentication and virtual machine operation, but optional applications may be disabled.

Required Application Purpose and Path:

- Pam_timestamp_check - Supports password authentication. Path: /sbin/pam_timestamp_check,

- Passwd - Supports password authentication. Path: /usr/bin/passwd

- Pwdb_chkpwd - Supports password authentication.Path: /sbin/pwdb_chkpwd

- Ssh-keysign Performs host-based authentication for SSH secure shells.

UNCLASSIFIED42


Path: /usr/libexec/openssh/ssh-keysign, Required if you use host-based authentication.Otherwise optional.

- Su - Lets a general user become the root user by changing users.Path: /bin/su

- Unix_chkpwd - Supports password authentication.Path: /sbin/unix_chkpwd

- Vmkload_app - Performs tasks required to run virtual machines. Path for standard use:/usr/lib/vmware/bin/vmkload_app

- Vmware-authd - Authenticates users for use of services specific to VMware.Path: /usr/sbin/vmware-authd

- Vmware-vmx - Performs tasks required to run virtual machines. Path for standard use:/usr/lib/vmware/bin/vmware-vmx

(VCESX0506: CAT II) The IAO/SA will ensure setuid and setgid flags for required applications are not disabled.

4.1.3Memory Requirements

For ESX Server 3.x, the default amount of memory assigned to the service console is 272MB. The service console memory is adjustable up to 800MB. The service console memory will be adjusted as needed to facilitate any third party applications that might be running on the ESX Server. The amount of memory in the service console for ESX Server 3.x has no direct affect on the number of virtual machines running.

For applications running in the service console, increase the amount of memory reserved for the service console. To determine the sufficient amount of memory, add the memory requirements for each application to the above determined amount reserved for the service console.

4.1.4Disks and Partitions

ESX Server uses the VMware ESX Server File System (VMFS) for storage of virtual machines. VMFS is a high-performance file system that is loaded on physical SCSI disks, ISCSI, or Fiber Channel SANs. These disks and partitions are optimized for storing large files such as virtual disk images and the memory images of suspended virtual machines. ESX Server uses VMFS-3 volumes that can span multiple partitions, across the same or multiple (up to 32) LUNs or physical disks. A VMFS-3 volume is a logical grouping of physical disk partitions. Because the files stored on the VMFS may exceed 2GB in size, they cannot always be accessed using the same tools as files on a standard ext2, ext3, FAT, or NTFS file system.

UNCLASSIFIED43


VMFS volumes have a default access type of public, unless the virtual machine is going to be clustered. Clustered virtual machines use the access type shared for VMFS volumes. The difference in a shared VMFS file system is that multiple ESX hosts can access a file concurrently. In public mode, virtual machines can open a file non-exclusively Read-Only, or exclusively Read-Write. File locks are managed through VMFS Volume MetaData on a per-file basis. Public mode may be used if the clustered servers are on the same physical ESX server, but not recommended. As a result, shared VMFS volume types will only be used for clustered servers.

(VCESX0508: CAT III) The IAO/SA will ensure shared VMFS volume types are only enabled for clustered virtual machine servers.

There are two disk modes administrators may use when configuring the virtual disk for the virtual machine. These disk modes will determine what is written to the disk and at what time. The two disk modes are persistent and nonpersistent. All production virtual machines will utilize persistent disk mode to ensure any changes to the disk are committed. Nonpersistent disk mode is ideal for test and development environments. Each disk mode is discussed below.

Persistent disk mode behaves exactly like conventional disk drive on a computer. All writes to a disk in persistent mode are written out permanently to the disk as soon as the guest operating system writes the data. This is the default access mode for all newly-created VMDK files. Persistent disk mode provides the best overall disk performance.

Nonpersistent disk mode discards all changes to a disk when a virtual machine session is powered off. VMDK files configured in non-persistent mode discard all changesmade to the file system at the point in time when the access mode was changed to non-persistent. When the virtual machine is powered off (not rebooted), any changes that were made to the VMDK are ignored. Nonpersistent disk mode is useful when multiple users have access to a machine through some form of remote connectivity. If someone were to delete key files, users instantly return the system to its original state. When using nonpersistent disk mode, users should completely configure the server first and change the disk mode to nonpersistent second. This ensures that the VMDK file will go back to the original configuration on the VMDK file.

(VCESX0510: CAT II) The IAO/SA will ensure all production virtual machines are set to persistent disk mode only.

Virtual machine disk files are stored on the VMFS file system. These files are in a special format and generally use .dsk or .vmdk file extensions. The disk files can comprise all the information the virtual machine stores on the virtual disk or be a symbolic link from a VMFS to a raw LUN when raw device mappings (RDM) are used.

Most implementations will use VMFS with disk files that comprise all the information in a virtual machine. In the default state, a virtual machine disk is simply a single file. All changes to that disk are written directly and immediately to that .vmdk file. However, ESX Server can use what is known as a redo log. When a redo is added to a .vmdk file, that base disk file becomes static and unchanging. All writes are “trapped” in the redo log. This new state is represented by

UNCLASSIFIED44


file name disk.vmdk, and the redo log will be called base.vmdk.REDO. A disk file may have a maximum of two redo logs, which is a total of three files: base. vmdk, base.vmdk.REDO, and base.vmdk.REDO.REDO. The format of the redo log is a bitmap record of changes to the disk. Redo logs are useful for disk snapshots. When a disk is represented by the two files disk.vmdk and base.vmdk.REDO, the disk.vmdk reflects the state of the drive at the time the disk snapshot was taken while base.vmdk.REDO is a bit-by-bit map of changes to the hard drive since that time. Figure 4-2 illustrates VMFS with a REDO Disk File.

Figure 4-2. VMFS with REDO Disk File

Permissions for the virtual machine files will adhere to VMware’s best practices. The configuration file (.vmx), will be read, write, execute (rwx) for owner and read and execute (r-x) for group and others (755). The virtual machine’s virtual disk (.vmdk) will be read and execute (r-x) for owner (550). An important note is that read permissions for a virtual disk file are sufficient if the virtual disk is nonpersistent. Be sure that the same user owns both the virtual machine’s configuration and virtual disk file, and this user has full access privileges for both files.

(VCESX0512: CAT II) The IAO/SA will ensure the permissions on the configuration file (.vmx) are 755, and the virtual disk (.vmdk) file is 500 for all virtual machines.

4.1.5Managing VMDK Files

All VMDK (virtual machine disk) files created for virtual machines must exist in a VMFS partition for ESX Server to utilize them. VMFS is a proprietary file system developed by VMware that stands up to the high amount of I/O that is generated by the ESX Server. To minimize overhead and optimize performance, VMware made the VMFS a flat file system, meaning all files are located on the root of the partition. VMFS does not permit the creation of subdirectories, so a proper naming standard must be implemented for all VMDK files.

A VMDK file represents a physical hard drive that gets presented to the guest operating system. When connecting a new VMDK file to a virtual machine, the operating system sees it as a non-partitioned physical drive. This drive can be partitioned and formatted using a variety of file systems including NTFS and ext3. Up to 56 VMDK files can be assigned to any guest operating

UNCLASSIFIED45


system, allowing for quite a bit of flexibility in creating a partitioning scheme in the guest operating system. The ESX Server has a limitation of 2TB per VMDK. All VMDK files must be stored on a VMFS partition if they're going to run as or within a virtual machine on an ESX Server. Virtual hard disk files redo log files, suspended state files, and virtual machine swap files will all reside within the VMFS volumes.

(VCESX0514: CAT III) The IAO/SA will ensure all vmdk, redo logs, and virtual machine swap files are stored on VMFS volumes.

NOTE: VMFS volumes can store files of other types such as executable script files and ISO images.

Virtual disk files on the VMFS are accessible through ESX Server service console and Virtual Infrastructure Client. From the service console, files can be viewed and manipulated on VMFS volumes under the /vmfs directory with ordinary file commands, such as ls and cp. Although mounted VMFS volumes may appear similar to any other file system, such as ext3, VMFS is primarily intended to store large files, such as disk images. The ftp, scp, and cp commands can be used for copying files to and from a VMFS volume as long as the host file system supports these large files.

Additional file operations are enabled through the vmware-converter utility. This utility supports the creation of a VMware ESX Server file system (VMFS) on a SCSI disk and can be used to create, manipulate, and manage files stored in VMFS volumes. This command is also used to list files on the VMFS volume, add a redo log, commit a redo log, and export .dsk files into other formats. The vmsnap and vmres scripts that automate many of the common backup and restore tasks are run from the service console. Importing and exporting disk files can also be done through the Virtual Infrastructure Client or service console by copying the files from VMFS mount and pasting them to a partition running ext3 file system, the file system for Linux.

There will be situations that require the import or export of VMDK files on the VMFS partition. VMware recommends using vmware-converter utility rather than the Linux command cp to move the VMDK files. Utilizing the vmware-converter utility is required since the VMFS file system utilizes such large files.

(VCESX0516: CAT III) The IAO/SA will ensure all VMDK file imports and exports are done via the vmware-converter utility onto the VMFS partition.

There are actually two different VMDK formats that can be used with various VMware products, monolithic and COW-formatted. Monolithic format is the only format that ESX can use. COW-formatted file types are required for VMware Workstation and VMware Server. Up until recently VMware had utilized two different extensions that were utilized to distinguish these formats (DSK for Monolithic and VMDK for COW). Now, everything uses the VMDK extension. The legacy DSK extension may be substituted for VMDK.

There are differences between the Monolithic and COW disk files. Monolithic disk files are contained in a single file and can only be accessed within the ESX Server. Also, the vmdk file is

UNCLASSIFIED46


filled with zeros to fulfill the full size of the file system. COW disk files may be contained in single file or may be broken up into 2 GB files. COW VMDK files may grow dynamically and can only be accessed by VMware Workstation and Server.

One thing to note with any VMDK file is that the file system that they are stored on must be capable of handling the proper file size. With ESX, monolithic files can only be accessed when running on a VMFS partition, which can handle file sizes up to 2TB in size. Linux file systems such as ext2, ext3, or reiserfs, do not have the luxury of being able to handle file sizes larger than 2GB. This is why COW files may be broken up into several 2GB files instead of self-contained in a single VMDK file.

VMDK files may be transferred between ESX servers. The issue with these transfers is that the files are transmitted in plaintext. Transferring VMDK files will be moved across ESX hosts using a FIPS 140-2 encryption algorithm or through a dedicated VLAN. Typically, scp (secure copy protocol) is used to transfer VMDK files. For instance, to copy a VMDK file the following command would be utilized:

# scp /path/to/sourcefile.vmdk username@remotehost:/path/to/destination.vmdk

(VCESX0518: CAT II) The IAO/SA will ensure all VMDK file transfers between ESX Servers are encrypted with an approved FIPS 140-2 algorithm or transferred over a dedicated VLAN.

(VCESX0520: CAT II) The IAO/SA will ensure all virtual machines are unregistered and registered when moved from one ESX Server to another.

NOTE: Registering and un-registering virtual machines is done automatically when using Virtual Infrastructure Client with VirtualCenter.

It is important to note that more than one user may be connected to the same virtual machine through the VMRC at the same time. Each user will have interactive access to the virtual machine; therefore, a coordinated effort must be made to manage remote users connecting via the VMRC.

4.1.6Renaming and Moving Virtual Machines

It may become necessary to rename a guest virtual machine at some point during the course of testing to production. To rename a guest virtual machine, the virtual machine must be powered down before proceeding with the renaming feature. Also, ensure a solid backup is done before any renaming any virtual machine. The configuration files for VMware will be located on the service console typically in /root/vmware/ directory and the virtual disks will be in the /vmfs/ directory.

(VCESX0522: CAT II) The IAO/SA will ensure all ESX production virtual machine renames are documented and approved by the change control board.

UNCLASSIFIED47

mailto:username@remotehost:/path/to/destination.vmdk


4.1.7Logging and Auditing

The syslog daemon performs the system logging on the ESX Server. The ESX Server creates log files for troubleshooting and support. These log files may be viewed through the Virtual Infrastructure Client by logging in as root and choosing Options followed by System Logs. These logs may also be accessed by the Service Console by going to the /var/log/ directory.

There are several types of log files generated by the ESX Server. These log types include VMkernal warnings, VMkernal logs, virtual machine log files, service console messages, and ESX Server service logs. The VMkernel warning log file records activities with the virtual machines. The log file location is located at /var/log/vmkwarning. The VMkernel logs records activities related to the virtual machines and the ESX Server. The VMkernel logs are located at /var/log/vmkernel. The virtual machine log files contain information when a virtual machine crashes or ends abnormally. This file is located in the same directory as the affected virtual machine’s configuration files and is named vmware.log. The service console messages contain all general log messages used to troubleshoot virtual machines or the ESX Server. The location of the messages file is /var/log/messages. Lastly, the ESX Server service logs contain information regarding connections to virtual machines and the Virtual Infrastructure Client. This log may assist in diagnosing connection problems. The log location is /var/log/vmware/vmware-serverd.log.

(VCESX0524: CAT II) The IAO/SA will ensure the ESX Server is configured to record the following log files: VMkernel warnings, VMkernel logs, virtual machine logs, service console messages, and service logs.

Audit utilities can extract information about specific users and processes from the audit files. The IAO/SA will ensure audit files are only accessible to authorized personnel. Auditing will be configured to immediately alert personnel of any unusual or inappropriate activity with potential IA implications. All users, including root, will be audited. The SA will rotate and compress the audit logs one or more times a day to ease space requirements and to reduce the time required for log searches and reviews. Audit data will be backed up no less than weekly onto a different system or media than the system being audited. The implementation of an audit server will ease the attention required by audit logs and provide compliance with the requirement for the back up of audit data.

Auditing will be configured according to section 3.16 of the UNIX STIG. Audit logs and audit files must be analyzed at regular intervals. Such files can quickly grow to large proportions. To keep the size of log files and audit files within a useful range, the evaluation intervals should not be impractically short, but short enough to allow a clear examination. Collected data will be examined and analyzed daily to detect any compromise or attempted compromise of system security. The IAO/SA will review audit files daily to detect possible system compromise, malicious users, or users that may need more instruction.

4.1.8VirtualCenter

VMware’s primary management tool for managing the Virtual Server/ESX Server environment is VirtualCenter. VirtualCenter provides a single management interface for all the ESX Servers

UNCLASSIFIED48


within the environment. Administrators may create new virtual machines, review performance data, and schedule tasks for virtual machines through a central interface with VirtualCenter. The following discussion reviews the application, components, users, groups, roles, privileges, and network ports utilized by VirtualCenter.

The VirtualCenter Management Server provides a convenient single point of control for the virtual datacenter. It is an application that runs on Windows operating system that provides many essential datacenter services such as access control, performance monitoring, and configuration. It unifies the resources from the individual computing servers to be shared among virtual machines in the entire datacenter. It accomplishes this by managing the assignment of virtual machines to the computing servers and the assignment of resources to the virtual machines within a given computing server based on the policies set by the system administrator. Each ESX server is under the control of a VirtualCenter that communicates through a VirtualCenter agent to the VirtualCenter. The VirtualCenter Management Server automatically executes user-specified scheduled tasks, such as powering on, or moving powered-off virtual machines. Figure 4-3 illustrates the VirtualCenter Interface.

Figure 4-3. VirtualCenter Interface

VirtualCenter with the optional VMotion package enabled supports moving a virtual machine from one ESX Server to another, while the virtual machine continues operation. Migration with VMotion occurs without service interruption on the virtual machine. VirtualCenter provides a visible interface for monitoring system availability and performance of the virtual machines in

UNCLASSIFIED49


the VirtualCenter configuration. VirtualCenter runs as a service on Windows 2000, Windows XP Professional, and Windows 2003.

Since VirtualCenter controls the entire virtualization environment, it must be protected and available to manage the resources. To protect VirtualCenter and its resources, all users will be presented with a warning message that notifies them that monitoring, recording, and auditing will occur while they are logged in. To ensure availability, VirtualCenter will be installed on a dedicated server. Combining several applications on the VirtualCenter server represents a risk for availability. Applications servers such as web or database server require a significant number of installed programs, number of active processes, and number of privileged users defined. Any of these elements may include vulnerabilities exploited in an attack on the VirtualCenter server. Email applications may provide a simple means by which privileged user unintentionally introduces malicious code. To address these issues, the use of applications on VirtualCenter servers will be limited only to the necessary applications required to run VirtualCenter.

(VCESX0528: CAT III) The IAO/SA will ensure VirtualCenter users are presented with a logon banner warning them of the following:

- The system is a DOD system.- The system is subject to monitoring.- Monitoring is authorized in accordance with applicable laws and regulations and conducted for purposes of systems management and protection, protection against improper or unauthorized use or access, and verification of applicable security features or procedures. - Use of the system constitutes consent to monitoring.- This system is for authorized US government use only.

(VCESX0530: CAT II) The IAO/SA will ensure VirtualCenter servers are not utilized to host other applications such as database servers, e-mail servers or clients, dhcp servers, web servers, etc.

NOTE: This does not apply to VirtualCenter components such as the database or license server which may be installed on same server.

VirtualCenter is comprised of five unique components. Each of the components interacts with each other providing a usable interface for managing the ESX Server environment. The five components are the Virtual Infrastructure Client, VirtualCenter Server, VirtualCenter Database, VirtualCenter Agent, and the VirtualCenter Web Service. Each VirtualCenter component provides different functionality. The Virtual Infrastructure Client is an application that provides the user interface to the VirtualCenter. The VirtualCenter Server is a service that is installed on a Windows server that accepts commands from the VirtualCenter client, and interacts with the VMware ESX Servers. It processes all commands and gathers performance data for all ESX Servers. The VirtualCenter database is an ODBC compliant database that stores all VirtualCenter information. Information includes hosts, virtual machine configurations, VirtualCenter security, performance data, tasks, and virtual machine permissions. Three supported databases are available for the VirtualCenter database. These are Microsoft MSDE, SQL 2000/2005 SP1, and Oracle. The VirtualCenter agent is installed on all ESX Servers that are to be managed. The agent coordinates actions received from the VirtualCenter server. The

UNCLASSIFIED50


final component is the VirtualCenter Web Service which is an optional component that exposes the VirtualCenter functions to third party applications.

(VCESX0532: CAT II) The IAO/SA will ensure VirtualCenter has the latest VMware security updates and patches installed.

(VCESX0534: CAT II) The IAO/SA will ensure VirtualCenter server OS is configured according to the Windows OS STIG.

(VCESX0536: CAT II) The IAO/SA will ensure VirtualCenter server database is configured according to the appropriate Database STIG.

(VCESX0538: CAT II) The IAO/SA will ensure VirtualCenter server web server is configured according to the Web STIG.

The VI Console allows a user to connect to the console of a virtual machine, in effect seeingwhat a monitor on a physical server would show. However, the VI Console also provides powermanagement and removable device connectivity controls, which could potentially allow amalicious user to bring down a virtual machine. In addition, it also has a performance impact onthe service console, especially if many VI Console sessions are open simultaneously. To prevent performance issues and potential unauthorized users from accessing the VI Console, users will use remote management services, such as terminal services and ssh, to interact with virtual machines.

(VCESX0540: CAT II) The IAO/SA will ensure the VI Console is not used for the daily administration of virtual machines.

VirtualCenter implements users, groups, permissions, and roles to manage the virtualization environment. VirtualCenter determines the level of access for the user based on the permissions assigned to the user. Users are authenticated to the VirtualCenter through the use of a user name, password. VirtualCenter maintains lists of authorized users and the permissions assigned to each user.

It is important to distinguish that there are two types of users within the virtualization environment. The first type is users who can access the ESX Server through VirtualCenter. The second type is users who can access the ESX Server by directly logging on to the host from VI Client, VI Web Access, a third-party client, or a command shell. These users are authorized users that are included in the Windows domain list referenced by VirtualCenter or local Windows users on the VirtualCenter host. VirtualCenter cannot manually create, remove, or change users. In order to manipulate the user list or change user passwords, this is done through the tools used to manage the Windows domain or server. Any changes made to the Windows domain or server is reflected in VirtualCenter. Direct access users are authorized to work directly on an ESX Server. These users are added to the internal user list by default when ESX Server is installed or by the system administrator after installation.

UNCLASSIFIED51


It is important to note that there are identical users on both the VirtualCenter and the ESX Server. These users however are not the same user even if they appear as the same user. These users should be treated as separate users who have the same name. For instance, the attributes of the devuser in VirtualCenter, including permissions, passwords, and so forth, are separate from the attributes of devuser on the ESX Server host.

VirtualCenter runs as a user that requires local administrator privilege and must be installed by alocal administrative user. To limit the scope of administrative access, a dedicated VirtualCenter administrator account will be created. Users will create a local VirtualCenter administrator account as an ordinary user that will be used to manage VirtualCenter. Within VirtualCenter, log on as the Windows Administrator, and then grant VirtualCenter root administrator access to the newly-created account. Remove the permissions in VirtualCenter for the local Administrators group. Configuring accounts this way avoids automatically giving administrative access todomain administrators, who typically belong to the local Administrators group. This alsoprovides a way of getting into VirtualCenter when the domain controller is down, because thelocal VirtualCenter administrator account does not require remote authentication.

(VCESX0542: CAT II) The IAO/SA will ensure a dedicated VirtualCenter administrator has been created within the Windows Administrator Group on the Windows Server for managing the VirtualCenter environment.

VirtualCenter creates a user called vpxuser that is used to manage ESX Servers. The vpxuser user is a VirtualCenter entity with root rights on the ESX Server host, allowing it to manage activities for that host. The vpxuser is created when the ESX Server host is attached to VirtualCenter. It is not present on the ESX Server host unless the host is being managed through VirtualCenter. System administrators will not change vpxuser and its default permissions. Modifying these permissions may create problems working with the ESX Server host through VirtualCenter.

(VCESX0544: CAT I) The IAO/SA will ensure the VirtualCenter default permissions and attributes for the vpxuser have not been changed or modified.

A group is a set of users that you want to manage through a common set of rules and permissions. When you assign permissions to a group, they are inherited by all users in the group. Using groups can significantly reduce the time it takes to set up user permissions. The group lists in VirtualCenter are drawn from the same sources as the user lists. VirtualCenter uses the group list from the Windows domain or server. Groups should be utilized to create a permissions structure to ensure confidentiality is applied to objects, since granting individual users permissions to objects becomes unmanageable.

Ensuring that the membership in privileged groups is controlled requires the maintenance of baseline documentation and periodic reviews to determine that unauthorized users are not members. If an unauthorized user is able to gain membership to the Database Administrator group, Virtual Machine Administrator group, or the Resource Administrator group, etc., that user would be able to display, add, or change permissions to objects that could impact the confidentiality, integrity, or availability of an entire virtualization structure. Therefore, a

UNCLASSIFIED52


baseline of the groups will be established to ensure proper permissions exist. These will be reviewed on a regular basis to ensure no unauthorized users have been granted access to objects.

(VCESX0546: CAT II) The IAO/SA will document those users assigned to the following security groups: Datacenter Administrators, Virtual Machine Administrators, Resource Pool Administrators, ESX Administrators, Virtual Machine Power Users, and any Custom Roles.

NOTE: VirtualCenter registers any selected Windows domain user or group through theprocess of assigning permissions. By default, all users who are members of theWindows administrators group on the VirtualCenter Server are granted the sameaccess rights as any user assigned to the administrator role. Users who aremembers of the administrators group can log on as individuals and have fullaccess.

Privileges are assigned to users or groups to perform actions. VirtualCenter uses these privileges, or roles, to control which users or groups can access virtualization objects. VirtualCenter provides a set of pre-established roles; however, new roles may be created if necessary. The roles assigned on an ESX Server are separate from the privileges and roles assigned on a VirtualCenter server. Once an ESX Server is added to the VirtualCenter as an object, the privileges and roles assigned through the VirtualCenter override the ESX Server roles. When the ESX Server is removed from the VirtualCenter Server inventory, then the previously set ESX Server privileges and roles are used.

VirtualCenter uses elements to define the virtual environment. These elements are managed entities, related to managed entities, and global entities. Managed entities include virtual machines, folders, datacenters, clusters, hosts, resource pools, and templates. Related to managed entities are networks and datastores. The final element is global entities which include custom fields, licenses, statistical intervals, roles, and sessions.

Each of these elements might have multiple permissions for users and or groups. Permissions are applied hierarchically downward on these elements. Situations may arise that a user is a member of multiple groups with different permissions or multiple group permissions are defined on the same object and a user belongs to two or more of these groups. Users that are a member of multiple groups with different permissions, the same permissions apply as if granted to the user directly. If multiple group permissions are defined on the same object and the user belongs to two or more of those groups then the following apply:

1. If there is no permission defined explicitly for the user on that object, the useris assigned the union of privileges assigned to the groups for that object.2. If there is a permission defined explicitly for the user on that object, thatpermission takes precedence over all group permissions.

The following scenario illustrates a simple example of expanding a user’s permissions.

-Role 1 can power on virtual machines.-Role 2 can take snapshots of virtual machines.

UNCLASSIFIED53


-Group A is granted Role 1 on virtual machine.-Group B is granted Role 2 on virtual machine.

-User 1 is not assigned specific permission:-User 1, who belongs to groups A and B, logs on.-User 1 can both power on and take snapshots of virtual machine.

These situations can create confusion and privileges that were thought to be limited might actually be elevated. Furthermore, all changes take affect immediately not requiring users to log off and log back in. As a result, all users, groups, roles, and permissions will be documented and approved to ensure proper permissions are granted only to authorized users.

(VCESX0548: CAT II) The IAO/SA will ensure a documented configuration management (CM) process exists and is utilized for all VirtualCenter additions, changes, or deletions of objects, groups, users, roles, and permissions.

(VCESX0550: CAT II) The IAO/SA will ensure VirtualCenter objects, groups, users, roles, and permissions have a baseline configuration that is documented.

(VCESX0552: CAT II) The IAO/SA will ensure VirtualCenter logs all object, group, user, role, and permission changes.

(VCESX0554: CAT II) The IAO/SA will ensure all VirtualCenter logs are reviewed on a daily basis for any anomalies.

VirtualCenter uses several TCP/UDP ports in ESX 3 and VC 2. The following tables illustrate the ports required for connectivity to an ESX Server. Figure 4-4 illustrates the ports used for communication in a VirtualCenter environment.

For Connectivity to the ESX Server

Incoming - TCP to ESX ServerPorts Purpose Details

80 User access to ESX Web Center443 Secure user access to ESX Web

Center SDK Access

ESX Server internal services are typically accessible through port 443 via reverse proxy. Services can be accessible directly through HTTP without authorization so that Web Center can be accessed directly at 8080, SDK (web services) at 8085, and mob at 8087.

902 VC Server access to ESX Server VI Client access to ESX Server ESX Server access to other ESX

Servers

For authd traffic (NFC, HA, Vpxa); VM configuration; migration and provisioning between ESX Servers

UNCLASSIFIED54


903 VI Client access to VM consoles VI Web Access Client access to

VM consoles

Console traffic generated by user access to VM’s on a specific ESX Server

2049 Transactions with NAS devices3260 Transactions with iSCSI devices8000 Incoming requests from VMotion27010 License server access to ESX Server

Incoming - UDP to ESX ServerPorts Purpose Details2050-5000

From ESX Server to other ESX Servers

For purpose of DAS

8042-8045


For purpose of DAS

Outgoing - TCP from ESX ServerPorts Purpose Details27000 Licensing (to talk to license server)2049 Transactions with NAS devices2050-5000


For purpose of DAS

3260 Transaction with iSCSI devices8000 Response to VMotion requests8042-8045


For purpose of DAS

Outgoing – UDP from ESX ServerPorts Purpose Details902 From ESX Server to VC Server

ESX Server access to other ESX Servers

Vpxa heartbeats. In this case the vpxa installer should probably upgrade the firewall settings. Also for migration between ESX Servers

2050-5000


For purpose of DAS

8042-8045


For purpose of DAS

For Connectivity to the Virtual Center Management Server

NOTE: This assumes that the VC Server and License Server are running on the same “Management Server” machine.

UNCLASSIFIED55


Incoming - TCP to VC ServerPorts Purpose Details902 VI Client access to VC Server27000 ESX Server access to License Server

Incoming - UDP to VC ServerPorts Purpose Details902 ESX Server to VC Server

Outgoing - TCP from VC ServerPorts Purpose Details902 VC Server access to ESX Server27010 License Server access to ESX Server

Outgoing - UDP from VC ServerPorts Purpose DetailsNone

UNCLASSIFIED56


Figure 4-4.VirtualCenter Ports

UNCLASSIFIED57


VirtualCenter makes use of temporary TCP connections for clone, cold migrate or deploy operations. The port range utilized for these is 49152-65535. These connections are between the VirtualCenter Server and a managed host or between two managed hosts. The port range is hard-coded and to find the exact ports that are being used.

4.2 ESX Server Networking

The ESX Server provides Ethernet-compatible networking components for the service console and the virtual machines. Virtual switches create a logical component referred to as a virtual network. A virtual network is a LAN created by all the virtual network adapters connected to a virtual switch. Virtual switches may or may not be bound to the physical network.

4.2.1Physical and Virtual Network Adapters

A minimum of two physical network adapters is required in each physical server to enable networking for both the service console and the virtual machines. A minimum of two network adapters per ESX Server are required because the first network adapter discovered during the installation of the ESX Server is always dedicated to the service console by default. Up to sixteen physical network adapters are supported per ESX Server. The ESX Server service console network adapter connects to the management user interface, SCP, SSH, and any other tool used to access the ESX Server’s file system. The other physical network adapter will be dedicated to the virtual machines. Figure 4-5 illustrates an ESX Server with two network adapters.

Figure 4-5. ESX Server With Two Network Adapters

UNCLASSIFIED58


(VCESX0556: CAT III) The IAO/SA will ensure the ESX Server hast a minimum two physical network adapters.

NOTE: VMware recommends that at least 4 network adapters are used for production environments (1 for VMotion, 2 for virtual machines, and 1 for the service console).

(VCESX0558: CAT II) The IAO/SA will ensure the service console and virtual machines physical network adapters are configured on separate VLANs.

Each virtual network adapter in a virtual machine has its own initial MAC address assigned when the adapter is created. In addition, each adapter has an effective MAC address that filters out incoming network traffic with a destination MAC address different from the effective MACaddress. Upon creation, a network adapter’s effective MAC address and initial MAC address are the same. However, the virtual machine’s operating system can alter the effective MAC address to another value at any time. If an operating system changes the effective MAC address, its network adapter then receives network traffic destined for the new MAC address. The operating system can send frames with an impersonated source MAC address at any time. Thus, an operating system can stage malicious attacks on the devices in a network by impersonating a network adapter authorized by the receiving network. Users can use virtual switch security profiles on ESX Server hosts to protect against this type of attack by setting two options on virtual switches. These options are MAC address changes and Forged transmissions.

MAC address changes are set to accept by default. This means that the ESX Serveraccepts requests to change the effective MAC address to other than the initial MACaddress. The MAC Address Changes option setting affects traffic received by a virtualmachine. To protect against MAC impersonation this option will be set to Reject. With the Reject setting, the ESX Server does not honor requests to change the effective MAC address to anything other than the initial MAC address. Instead, the port that the virtual network adapter used to send the request is disabled. As a result, the virtual network adapter does not receive any more frames until it changes the effective MAC address to match the initial MAC address. Also, the guest operating system does not detect that the MAC address change has not been honored.

(VCESX0560: CAT II) The IAO/SA will ensure all virtual switches have MAC address changes set to Reject.

Forged transmissions are set to accept by default as well. This means the ESX Server doesnot compare source and effective MAC addresses. The Forged Transmits option settingaffects traffic transmitted from a virtual machine. If this option is set to Reject, the ESX Server compares the source MAC address being transmitted by the operating system with the effective MAC address for its adapter to see if they match. If the addresses do not match, ESX Server drops the packet. The guest operating system does not detect that its virtual network adapter cannot send packets using the impersonated MAC address. With the option set to Reject, the ESX Server intercepts any packets with impersonated addresses before they are delivered, and the guest operating system might assume that the packets have been dropped.

UNCLASSIFIED59


(VCESX0562: CAT II) The IAO/SA will ensure all virtual switches have Forged transmissions set to Reject.

ESX Server has the ability to run virtual and physical network adapters in promiscuous mode. Promiscuous mode may be enabled on virtual switches that are bound to a physical network adapter (public virtual switch) and virtual switches that do not bind to a physical network adapter (private virtual switch). When promiscuous mode is enabled for a public virtual switch, all virtual machines connected to the virtual switch have the potential of reading all packets sent across that network, from other virtual machines and any physical machines or other network devices. When promiscuous mode is enabled for a private virtual switch, all virtual machines connected to the private virtual switch have the potential of reading all packets across that network, meaning only the virtual machines connected to that private virtual switch. Promiscuous mode will be disabled on the ESX Server virtual switches since confidential data may be revealed while in this mode. Promiscuous mode is disabled by default on the ESX Server. However, there might be a legitimate reason to enable it for debugging or troubleshooting reasons. To enable promiscuous mode for a virtual switch, a value is inserted into a special virtual file in the /proc file system. Since the value is in the /proc directory, if the ESX Server is rebooted, the value is removed. One way around this is to enable promiscuous mode during the boot process of the ESX Server. This is done by adding a command to the /etc/rc.local boot script in the service console. Figure 4-6 illustrates promiscuous mode.

Figure 4-6. Promiscuous Mode

UNCLASSIFIED60


(VCESX0564: CAT II) The IAO/SA will ensure the ESX Server public and private virtual switches are not configured to run in promiscuous mode.

(VCESX0566: CAT II) The IAO/SA will ensure the service console /etc/rc.local does not contain a boot script to enable promiscuous mode during the boot process.

The physical network adapters allocated to the VMkernel for the virtual machines are referred to as either outbound adapters or vmnics. Vmnic refers to any one of the physical network adapters allocated to the VMkernel. Specific vmnics are assigned by a name based upon an incremental zero based numbering system using the format, where vmnic <id> is the number assigned to the vmnic. The first vmnic is vmnic0.

Network adapters installed in the virtual machines are referred to as virtual network adapters. Each virtual machine may have zero to four virtual network adapters. ESX Server supports two different types of virtual network adapters, vlance and vmxnet. The vlance virtual network adapter emulates an AMD PCnet network adapter. It is highly compatible with many operating systems, however, not optimized for high-performance networking and will consume more CPU cycles from the guest operating system. The vmxnet virtual adapter is an optimized, high performance virtual network adapter that appears as a VMware PCI Network Adapter within a virtual machine’s guest operating system. The vmxnet is not as compatible as the vlance virtual network adapter, but the drivers are installed when VMware Tools are installed. The higher performance is utilized when virtual machines are connected to gigabit Ethernet physical network adapters.

NOTE: It is strongly recommended that virtual machines use the vmxnet adapter. VMware Tools must be installed.

In the physical infrastructure, two network adapters may be used for fault tolerance. The virtual machine does not require two virtual network adapters for fault tolerance. Instead, the single virtual network adapter is associated with a virtual switch that has its outbound connection configured to a bonded set of vmnics. These physical network adapters provide redundancy for the entire virtual switch. If one of the vmnics should fail, the others in the bond continue to function as normal with no interruption on the guest side.

There are situations that require multiple virtual network adapters in a virtual machine. These are when there are multiple connections to multiple virtual switches or when backup software specifically requires this configuration. Figure 4-7 illustrates multiple virtual network adapters.

UNCLASSIFIED61


Figure 4-7. Multiple Virtual Network Adapters

4.2.2Virtual Switches

ESX Server provides the ability to create abstract network devices called virtual switches. Each virtual switch is a network hub and may be configured to support 1016 virtual network ports and 512 port groups. Each virtual machine may have up to four virtual network adapters that may connect to virtual switches. Each virtual network adapter has a MAC address and may have an IP address or multiple IP addresses as well. A virtual switch can route traffic internally between virtual machines, or link to external networks. Public virtual switches and private virtual switches are the two types of virtual switches that may be implemented within the ESX Server environment. Public virtual switches are virtual switches that are connected to the physical network adapters, whereas the private virtual switches have no physical network adapters associated with them.

Public virtual switches are bound to physical network adapters providing virtual machines connectivity to the physical network. Public virtual switches may be used to combine the bandwidth of multiple physical network adapters and balance communications traffic among them. Public virtual switches may also be configured to maintain persistent network connections despite link failures for physical individual adapters. The master configuration file for virtual switches is the esx.conf file. Once the virtual switches are created, the hostd daemon must be restarted or reloaded.

UNCLASSIFIED62


(VCESX0570: CAT II) The IAO/SA will ensure public virtual switches only allow virtual machines that require access to the physical network adapters.

Private virtual switches do not have a connection to the physical network. In a private virtual switch, all traffic is generated between virtual machines is handled by the ESX Server CPU. A virtual network adapter may connect to a virtual switch port. Traffic that flows between any two virtual network ports on the same virtual switch will actually traverse over the system bus and will not traverse the network. When information is transferred in this manner, no network resources are impacted. The isolation inherent in this design makes private virtual switches especially useful for supporting network topologies that normally depend on the use of additional hardware to provide security and isolation. For Instance, an effective firewall can be constructed by configuring one virtual machine on an ESX Server system with two virtual network adapters, one bound to a public virtual switch (giving it a connection to a physical network) and the other bound to a private virtual switch. Other virtual machines would be connected only to the private virtual switch. By running filtering software in the dual-homed virtual machine, a user can construct an effective firewall without the need for additional hardware and with high-performance virtual networking between the virtual machines. Figure 4-8 illustrates a virtual Ethernet network.

Figure 4-8. Virtual Ethernet Network

UNCLASSIFIED63


Private virtual switches provide many advantages to the virtual machines. Private virtual switches provide isolation from the physical network and may be used for high-speed networking between virtual machines, allowing private, cost-effective connections between virtual machines. Private virtual switches are best utilized in the support of back-end processing among virtual machines. The horsepower needed to deliver packets on private virtual switches comes from the pool of CPU time not used directly by virtual machines. Therefore, these private virtual switches frequently offer faster performance than physical LANs, because no network hardware is involved. Another advantage of private virtual switches is their improved physical security. Because they have no physical ports, it is impossible for virtual machine administrators to patch a system to them without knowing a service console username and password.

Configuring virtual switches is performed by using predefined ESX Sever commands. These commands are located in the /usr/bin of the file system hierarchy. Since these commands can create, disable, and modify existing configurations, they will be restricted to the root user only. If other users were able to access these commands, inadvertent changes could potentially disable a virtual network.

(VCESX0572: CAT II) The IAO/SA will ensure the permissions on the /usr/sbin/esxcfg-* utilities are 500, except for esxcfg-auth which should be544.

Virtual switches within the ESX Server require a field for the name of the switch. This label is important since it serves as a “functional descriptor” for the switch. It is important to note that labeling the virtual switches will not contain the first character as a number. There have been known issues in the past that result from beginning the name with a number has caused erratic behavior, especially when renaming or removing the virtual switch. Labeling virtual switches will indicate the function or the IP subnet of the virtual switch. For instance, labeling the virtual switch with “internal” or some variation will indicate that the switch is only for internal networking between virtual machines private virtual switch with no physical network adapters bound to it.

(VCESX0574: CAT II) The IAO/SA will ensure all private and public virtual switches not in use are disabled.

(VCESX0576: CAT II) The IAO/SA will ensure the all virtual switches are labeled within the ESX Server environment.

(VCESX0578: CAT II) The IAO/SA will ensure the all virtual switches labels do not begin with a number.

NOTE: Virtual machines configured to use virtual switch will not power on if virtual switch label is changed and the hostd is not restarted.

VMotion may be utilized on the ESX Server to move virtual machines between ESX Servers. VMotion traffic provides no confidentiality for the data in transit. All traffic is sent in clear text. Therefore, VMotion will require a dedicated virtual switch and must contain one physical

UNCLASSIFIED64


network adapter (public virtual switch). Also, a dedicated VLAN will be utilized to move virtual machines between ESX Servers. Figure 4-9 illustrates VMotion technology.

Figure 4-9. VMotion Technology

(VCESX0580: CAT II) The IAO/SA will ensure VMotion virtual switches contain at least one physical network adapter and are configured to use a dedicated VLAN.

4.2.3VLANS

Virtual Local Area Networks (VLANs) provide for logical groupings of stations or switch ports, allowing communications as if all stations or ports were on the same physical LAN segment. This includes stations or ports that are physically located on different 802.1D bridged LANs.

Each VLAN is simply a broadcast domain. VLAN broadcast domains are configured through software rather than hardware, so even if a machine is moved to another location, it can stay on the same VLAN broadcast domain without hardware reconfiguration. The ESX server supports VLANs and 802.1Q trunking providing additional security to the virtual environment. The ESX Server VLAN requires one of the elements on the virtual or physical network to tag the Ethernet frames with 802.1Q tag.

4.2.4VLAN Modes

There are three different modes in which the ESX Server can configure VLANs. Each method has a different impact on how the virtual switches are configured and how the guest operating systems interact with the network. These modes each tag and untag the packets with an 802.1Q tag for virtual machine frames. These modes are External Switch Tagging (EST), Virtual Switch Tagging (VST), and Virtual Machine Guest Tagging (VGT).

EST mode utilizes external physical switches for VLAN tagging. This is similar to a physical network where VLAN configuration is normally transparent to each individual physical server. The 802.1Q tag is appended when a packet arrives at a switch port and stripped away when a packet leaves a switch port toward the server. Figure 4-10 illustrates EST mode.

UNCLASSIFIED65


Figure 4-10. EST Mode

EST is the default configuration for all virtual switches within an ESX Server. In this mode, all VLAN configurations are handled by the physical switch. This mode only requires that the ESX Server physical network adapters are plugged into the correct physical switch ports for the proper VLAN assignment. The virtual machines must be configured with the correct IP address that falls within the subnet range that defines the specific VLAN it is connected to. One drawback of this approach is that if port-based VLAN tagging is used the total number of virtual LANs supported would be limited to the number of physical network adapters installed on a given ESX Server system.

EST mode has a one-to-one relationship, the number of VLANs supported on the ESX Server system is limited to the number of physical NIC ports assigned to the VMkernel. EST is enabled when the port group’s VLAN ID is set to 0 or left blank. Due to the integration of the ESX Server into the physical network, the physical network adapters will need to have spanning-tree disabled or portfast configured for external switches, since VMware virtual switches do not support STP. If these are not set, potential performance and connectivity issues could arise.

UNCLASSIFIED66


(VCESX0584: CAT II): The IAO/SA will ensure external switches have spanning-tree disabled or portfast configured for the ports connected to the ESX Server physical adapters running in EST mode.

NOTE: Virtual switch uplinks do not create loops.

(VCESX0585: CAT II) The IAO/SA will ensure trunking is disabled on all virtual switches utilizing EST mode.

Furthermore, the service console and virtual machines will communicate on separate VLANs. The service console is considered administrative or management traffic, while the virtual machines represent production traffic.

(VCESX0586: CAT II): The IAO/SA will ensure the service console and virtual machines physical network adapters are on separate VLANs.

VST mode allows the virtual switch to handle its own VLAN tagging. The processing of the 802.1Q tags is handled by the network adapter hardware, so overhead from these tags never reaches the VMkernel, which has no impact on virtualization processing. In this mode, each physical switch port that connects to a virtual switch is configured in trunk mode. After the trunk is established, specific VLANs are presented down the trunk to the virtual switch.

Once the virtual switch receives the VLAN information from the trunk it must assign virtual network adapters to specific VLANs. In order to achieve this, port groups are configured within ESX Server. Since a virtual switch does not have physical ports that reference to a particular VLAN, port groups are configured on a per virtual switch basis. Each VLAN that is being announced down the trunk to the virtual switch must have its own port group configured before virtual switches can be utilized on the published VLAN. Virtual machines would attach the virtual adapter to the port group instead of the virtual switch directly. The virtual switch port group tags all outbound frames and removes tags for all inbound frames. It also ensures that frames on one VLAN do not leak into a different VLAN. There are two values that must be configured when creating a port group, the Port Group Label and the VLAN ID. The Port Group Label is the name of the VLAN. It is recommended to set Port Group Labels to “VLANX”, where X is replaced by the published VLAN’s ID.

Virtual network adapters must point to the Port Group Label that was defined for the VLAN that the virtual machine will reside in. For instance, one virtual machine network adapter will point to VLAN 101 port group, and another virtual machine network adapter will point to VLAN 102 port group.

The ESX Server does not support port groups configured to the native VLAN (VLAN1) for the virtual switch. The native VLAN is utilized for switch management and does get tagged with a VLAN ID, and is dropped by the ESX Server. The default VLAN value for most physical switches is 1, so any value between 2 and 4094 will work within the ESX Server for a port group number.

UNCLASSIFIED67


(VCESX0588: CAT II): The IAO/SA will ensure physical network adapters in EST or VST mode are not configured to use VLAN1.

VST changes the traditional way of configuring virtual switches. VST can be thought of as physical switch to a virtual switch. Typically, creating multiple virtual switches to utilize multiple VLANs is utilized for the EST mode. With VST, there a limit of 512 port groups or VLANs that can be accessed through a single virtual switch. As a result, with VST one public virtual switch will be utilized and configured that contains all the physical network adapters in the system. This provides additional redundancy throughout the virtual switch, and simplifies the management of an ESX Server by configuring port groups on one virtual switch.

VST has the several benefits to the virtual machines. One benefit is different VLAN frames can be multiplexed and consolidated onto a single physical network adapter regardless of VLAN. VST eliminates the need to run a guest operating system specific VLAN driver inside the virtual machine. Also, there is little performance impact by supporting VLAN tagging in the virtual switches. Finally, once the switch trunk mode is appropriately set up, no additional switch configuration is needed when provisioning additional VLANs. Figure 4-11 illustrates VST Mode.

Figure 4-11. VST Mode

(VCESX0590: CAT II): The IAO/SA will ensure the external switch and virtual switch in VST mode use the non-negotiate option for trunking between them.

UNCLASSIFIED68


NOTE: The desirable and auto settings do not work on virtual switches because the switch expects its peer, the ESX Server virtual switch port, to communicate using Dynamic Trunking Protocol (DTP). The non-negotiate and on options enable VLAN trunking unconditionally. The difference between non-negotiate and on options is that on mode still sends out DTP frames. To minimize the unnecessary network traffic, VMware recommends using the non-negotiate option.

(VCESX0596: CAT II) The IAO/SA will ensure a dedicated VLAN is configured for all trunks between the virtual switch in VST mode and the external physical switch.

(VCESX0598: CAT II) The IAO/SA will ensure virtual machines are not assigned to the port group used for the trunk VLAN between the virtual switch in VST mode and the external physical switch.

(VCESX0600: CAT II) The IAO/SA will ensure port groups on virtual switches that are no longer needed are removed.

The third mode for configuring VLANs for virtual machines is the VGT mode. In VGT mode, the virtual switch no longer reads the 802.1Q tags, but instead forwards them directly to the virtual machine. The virtual machine guest operating system is responsible for properly configuring the VLAN for the virtual network adapter of the virtual machine. This mode requires the 802.1Q VLAN trunking driver inside the virtual machine, which tags the virtual machine Ethernet Frames. Tags are preserved between the virtual machine networking stack and external switch when frames are passed from and to virtual switches. There is extremely limited support for this configuration. Currently, the Windows OS does not support VLAN tagging, however, Linux does support it. VGT mode allows the number of VLANs per virtual machine not to be bound by the number of virtual adapters. Figure 4-12 illustrates VGT Mode.

UNCLASSIFIED69


Figure 4-12. VGT Mode

In VGT mode, server administrators would be able to control and configure which VLANs the virtual network adapters were communicating on. Therefore, virtual switches or external physical switches will perform the VLAN functionality with the ESX Server environment.

(VCESX0602: CAT II): The IAO/SA will ensure VGT mode is disabled for all virtual machines within the ESX Server.

4.2.5Bonded Network Adapters

ESX Server provides the ability to bind physical network adapters together into a single logical network device called a bond. Creating logical network adapters provide redundancy for the ESX Server to the external network. Once a logical network adapter is configured, the virtual machines are not aware of the underlying physical network adapter. This is similar to a hardware RAID configuration for hard drives. Packets sent to the logical network adapter are dispatched to one of the physical network adapters in the bond and packets arriving at any of the physical network adapters are automatically directed to the appropriate logical network adapter. Figure 4-13 illustrates bonded network adapters.

UNCLASSIFIED70


Figure 4-13. Bonded Network Adapters

Binding together identical models of physical adapters ensures that all features of the adapter can be used by ESX Server. ESX allows binding up to ten physical network adapters to each virtual switch. The network connection for a virtual machine is linked to the associated virtual switch. The operation of the virtual machine depends on the configuration of its physical network connection. Thus, binding or detaching physical network adapters of a virtual switch in use by a virtual machine is not allowed.

In ESX 3.x binding physical network adapters is done at the port group or at a virtual switch level through the Virtual Infrastructure Client. The Virtual Infrastructure Client option is called “Route based on IP hash” for IP based, or “Route based on source MAC hash” for MAC based.

4.2.6Bonded Modes

There are three modes for determining how ESX Server distributes traffic among the physical network adapters assigned to a virtual switch. These modes are MAC address balancing, IP address balancing, and Standby. MAC address load balancing distributes networking traffic based on the MAC hardware addresses of the source network adapters. MAC address balancing is the default load balancing mode in ESX Server. IP address load balancing distributes networking traffic based on IP addresses. ESX Server distributes network traffic in this mode based on the destination IP address of the packet. The final mode Standby designates a specific adapter to use as the primary connection. Standby mode is used primarily for redundant connections to physical switches.

MAC address load balancing is the default load balancing mode for the ESX Server. This mode does not require any additional configuration to the external physical switches. The VMkernel has full control over the vmnic physical network adapter that the virtual machine’s MAC address is broadcasted over to the physical switch infrastructure. This control only permits one physical

UNCLASSIFIED71


network adapter to send out the MAC address and prevents announcing duplicate MAC addresses to the physical switches.

MAC address load balancing occurs when a physical network link is detected at the vmnic physical network adapter. The VMkernel automatically re-ARPs the virtual machine’s MAC address down an alternative “known-good” path. This failover usually takes around 10 seconds. After the physical network adapter has established connectivity, traffic may resume.

IP address is the second method for load balancing traffic generated by virtual machines. IP addressing load balancing is based on the destination IP address. Since outgoing virtual machine traffic is balanced based on the destination IP address of the packet, this method provides a much more balanced configuration than the MAC address mode. Once a link failure is detected by the VMkernel, there will be no impact on the connectivity of the virtual machines. To implement IP address load balancing requires some extra configuration on the external switch and ESX Server.

The final load balancing mode is Standby mode. Standby mode is typically utilized when gigabit switch ports are available, but in short supply. To connect several physical network adapters together, one link would be configured to the gigabit (1000Mb) port, and the second link would be configured for a fast Ethernet (100Mb) port. One physical network adapter will be the primary network connection and the other would be the secondary for a virtual switch. In this configuration, ESX Server routes all traffic through the primary network adapter and reserves the other adapters in case of connection failure. This type of redundant connection switch is defined as using a “failover” configuration.

Redundant connection switches use a primary link for a virtual switch that overrides the load balancing mode. ESX Server monitors the primary link for physical connection failures. When the primary adapter loses contact, ESX Server transfers the network traffic to one of the secondary adapters while continuing to monitor the primary adapter. When ESX Server detects that the physical connection of the primary link has been restored, it transfers the connection for the virtual switch back to the primary. This basic failure detection mode passively monitors an adapter for loss of physical connection to an external switch.

4.6.7 ESX Server Beacon Monitoring

The beacon monitoring feature broadcasts beacon packets on the external network linked to the ESX Server to detect distributed connection failures. ESX Server issues beacon packets from one adapter addressed to other adapters assigned to a virtual switch. By monitoring beacon reception, the server can determine the state of connections in a multi-point network route. Beacon monitoring is designed to be used in configurations where the multiple adapters assigned to a virtual switch are connected to more than one external switch. Physical link monitoring only indicates whether an adapter is communicating with one external switch. Beacon failures can detect connection failures between external switches or routing errors among switches in a distributed network domain. ESX Server uses beacon monitoring as a variable indicator of network connection failure. The server indicates a connection loss after it fails to receive a set number of broadcast beacons in succession. Only when the number of failed beacons exceeds the failure threshold will the server identify a link as disconnected and switch to another adapter.

UNCLASSIFIED72


ESX Server uses beacon monitoring as a secondary method to detect network failures. When the ESX Server detects a physical link failure for the primary adapter, it will switch to a secondary adapter without regard to whether beacon monitoring indicates a failed connection.

Beacon monitoring can cause false indications of network connection failure. External switches may trap beacon packets, causing ESX Server to declare a switch failure for a connection that is functioning normally. When the ESX Server switches to a secondary link, traffic from the primary may still be transmitted because the connection has not actually failed. This can result in an external switch receiving duplicate packets from both links. As a result, beacon monitoring will be disabled due to these issues.

(VCESX0606: CAT III): The IAO/SA will ensure beacon monitoring is disabled for all ESX Server virtual switches.

4.2.7ESX TCP/IP Ports

The TCP/IP ports available for management access to the ESX Service Console may vary depending on the security settings specified for the server. There are three default security settings that may be selected, High, Medium, and Low. Setting the system to High ensures that all traffic with the ESX service console is encrypted.

ESX Server 3.x includes a firewall between the service console and the network. To ensure the integrity of the service console, VMware has reduced the number of firewall ports that are open by default. At installation time, the service console firewall is configured to block all incoming and outgoing traffic except for that on ports 902, 80, 443, and 22, which are used for basic communication with ESX Server. This setting enforces a high level of security for your ESX Server host. Medium Security blocks all incoming traffic except on the default ports (902, 433, 80, and 22), and any ports users specifically open. Outgoing traffic is not blocked. Low Security does not block either incoming or outgoing traffic. This setting is equivalent to removing the firewall.

Because the ports open by default are strictly limited, additional ports may need to be open after installation. For instance, Veritas NetBackup™ 4.5 backup agent uses ports 13720, 13724, 13782, and 13783. These are used for NetBackup client-media transactions, database backups, user backups or restores.

(VCESX0608: CAT II) The IAO/SA will ensure the ESX Server is configured at High or Medium Security. Medium Security is used only if additional ports are required to be open. These additional ports are documented and approved by the IAO/SA.

NOTE: High Security if the default ESX Server configuration.

(VCESX0610: CAT II) The IAO/SA will ensure no third party firewall is installed on the ESX Server except for IPtables.

UNCLASSIFIED73


4.3 SNMP

Simple network management protocol (SNMP) is a communication protocol between an SNMP client and an SNMP agent. The SNMP client queries the SNMP agent that provides information to the client regarding the device’s status. The SNMP agent controls a database called the SNMP Management Information Base (MIB), a standard set of statistical and control values. SNMP allows the extension of these standard values with values specific to a particular device.

ESX Server ships with an SNMP agent that monitors the health of the physical machine where the ESX server is running and the virtual machines running on it. This agent is based on Net-SNMP with enhancements to support data specific to ESX Server. ESX Server SNMP support is a module that can be loaded into a daemon based on the Net-SNMP package. The VMware-specific SNMP modules are automatically installed when it is installed on the ESX Server. By default on a fresh install, ESX Server components are enabled in SNMP and VMware traps are always on. The configuration parameter snmp/generateTraps in the /etc/vmware/host/config.xml file determines whether to generate a trap. The ESX Server SNMP agent can be used with any management software that can load and compile a management information base (MIB) in SMIv1 format and can understand SNMPv1 trap messages.

The ESX Server SNMP package takes the simplest approach to SNMP security in the default configuration. It sets up a single community with read-only access. This is denoted by the “ro” community configuration parameter in the configuration file for the master snmpd daemon, snmpd.conf which is configured by running snmpsetup.sh script.

(VCESX0614: CAT II) The IAO/SA will ensure SNMP is only enabled in the read mode for all ESX Servers. Read/Write is not enabled unless approved and documented by the IAO/SA.

NOTE: Read only is the default mode.

UNCLASSIFIED74


APPENDIX A. RELATED PUBLICATIONS

VMware, Inc. Using VMware ESX Server System and VMware Virtual Infrastructure for Backup, Restoration, and Disaster Recovery, 2005

VMware Inc., ESX Server 2 Security White Paper, 2004

VMware Inc., ESX Server 3 802.1Q VLAN Solutions, 2006

VMware Inc., ESX Server 2 Administration Guide, 2005

VMware Inc., Virtualizations: Architectural Considerations and Other Evaluation Criteria, 2005

VMware Inc., Virtualization Overview, 2006

VMware Inc., VMware Infrastructure Architecture Overview, 2006

VMware Inc., VMware ESX Server 2 Architecture and Performance, 2005

VMware Inc., VI3 Introduction, 2006

VMware Inc., VI3 Administrator Guide, 2006

VMware Inc., VI3 Server Configuration Guide, 2006

VMware Inc., VI3 Securing and Monitoring VMware Infrastructure 3, VMWorld 2006

Tal Garfinkel and Mendel Rosenblum, When Virtual is Harder than Real: Security Challenges in Virtual Machine Based Computing Environments, Stanford University Department of Computer Science

John Y. Arrasjid and Jason Mills, Security Management in a VMware Virtual Infrastructure Environment, VMware, Inc., September 2005

David Marshell, Wade A Reynolds and Dave MCrory, Advanced Server Virtualization, 2006

Chris Wolf and Erick M Halter, Virtualization From the Desktop to the Enterprise, 2005

Rob BNastiaansen, Rob’s Guide to Using VMWARE, Second Edition, September 2005

Al Muller, Scripting VMware, 2006

Ron Oglesby and Scott Herold, VMware ESX Server Advanced Technical Design Guide, 2005

John Scott Robin and Cynthia E. Irvine, Analysis of the Intel Pentium’s Ability to Support a Secure Virtual Machine Monitor

UNCLASSIFIED75


APPENDIX B. PRODUCT VMM TYPES

Product VMM Type

VMware ESX Server Type I

VMware Ace Type II

VMware Workstation Type II

VMware Player Type II

VMware Workstation Type II

VMware Server Type II

Microsoft Virtual Server 2005 R2 Type II

UNCLASSIFIED76

http://iase.disa.mil/stigs/draft-stigs/virtual-computing-stig

Technology

esx serversfor purpose

guest operating systems

guest operating system

mac address balancing

client operating system

initial mac address

microsoft operating systems

virtual infrastructure client