tolly212131symantecsep12dot1vmwareavefficacy

8/13/2019 Tolly212131SymantecSEP12dot1VMwareAVEfficacy

1/5

2012 Tolly Enterprises, LLC 1 of 5Toll .com

#212131October 2012

Commissioned by

Symantec Corp

Executive SummaryAs virtualization-aware endpoint security solutions continue to

evolve, more and more functionality is offloaded from a single VM to

its supporting infrastructure in the form of Virtual Appliances (VA). In

addition to considering the performance impact of this re-

architecting, administrators must also ensure that the protectionoffered remains fully-functional, even in virtual environments.

Symantec Corp. commissioned Tolly to evaluate the A/V effectiveness

of its new Symantec Endpoint Protection (SEP) 12.1 within VMware

vSphere 5 virtual environments vs. competitive agentless and agent-

based solutions. Testing focused on evaluating vendor solutions

against a malware source using on-access scanning to determine a

solutions real-time effectiveness in virtual environments.

TEST HIGHLIGHTS

Symantec Endpoint Protection 12.1Anti-virus Effectiveness in VMware vSphere 5 Virtual Environments

1Blocked or neutralized 99% of malware threat

samples

Exhibited 0% false positive rate for non-malicious

samples2

Symantec Endpoint Protection 12.1:

0%

20%

40%

60%

80%

100%

P

ercentofThreats

Blocked Neutralized Compromised

On-Access Virus DetectionAs reported by Solution Consoles

Notes: Windows 7 Professional, 64-bit installation. 870 confirmed malicious samples were used to evaluate effectiveness. OfficeScan used SmartScan method. Tolly created a custom

application to launch and monitor the success or failure of solutions as the machines accessed the malicious URLs. Results reported are best of 5 runs. See test methodology section for

details.

Figure 1Source: Tolly, September 2012

A B C D

(870 Malware Samples)

A Symantec Endpoint Protection B Trend Micro OfficeScanC Kaspersky Security for Virtualization D McAfee MOVE


2/5

Executive Summary (cont)Tolly engineers set up a script to

automatically run each client against a set of870 samples which were confirmed

malicious applications. In order to evaluate

the false positive rate, 50 legitimate

applications were included in a separate

corpus.

Tolly used a bank of 20 VMs per solutions,

scripted using AutoITv3 to download and

execute the samples.

McAfee and Kaspersky were tested using

their vShield-enabled configurations, whileSymantec and Trend Micro were tested on

their agent-based platforms, which had

been optimized for virtual environments.

Test Results

On-Access Virus Detection Rate

Throughout the work day, the endpoint

security solution is invoked to scan files and

other registry/RAM contents as they are

accessed.

For this test, Tolly created a custom

application to launch and monitor the

effectiveness of solutions as the cleanly-

booted machines accessed the malicious

URLs.

When a solution defended against the

threat, the threat was deleted upon

download, whereas a neutralized threat was

downloaded but forbidden to run, or

cleaned of its malicious content.

Tolly engineers observed how solutionsperformed when they accessed the 870

samples of malware- whether the solutions

under test blocked or neutralized the

threats, or whether the threats were

permitted to run, and thus compromised

the system.

Symantecs detection and protection

against viruses was the highest among

solutions tested. SEP 12.1 blocked 96% of

threats, and neutralized another 3.3%, while

only compromising 0.7%, (6 out of 870). See

Table 1.

Kaspersky also performed well, blocking

92.2% of threats and neutralizing another

3.2%. Kaspersky, however, compromised the

system 5.5 times more than Symantec,

allowing 4.6% of the malicious programs to

run.

Trend Micro blocked only 59.3% of threats,

and neutralized another 0.4%, thus allowing

nearly half of the malicious samples to

compromise the system.

McAfee blocked 79.5% of threats and

neutralized another 0.2%, allowing 20.2% of

malicious samples to compromise the

system. See Figure 1.

False Positive

Though the primary function of any

endpoint security solution is to block

malicious applications, an endpoint security

solution must also be able to determine

legitimate software and allow it to run

uninterrupted. When a system blocks alegitimate application from running, it

negatively impacts the user experience.

Many organizations with custom

applications may face this challenge when

deploying updates or custom software

within their organizations. The ideal scenario

with any endpoint security solution is to

recieve zero false positives and a 100%

malicious detection rate, however, this is no

a realistic goal given the ever-expanding

threat of new malware.

Endpoint security solutions handlelegitimate programs differently. Some

merely issue a warning when faced with a

legitimate application that they believe to

be malicious, while others go a step furthe

and block the application.

Symantec Endpoint Protection 12.1 Efficacy #212131

2012 Tolly Enterprises, LLC Page 2 of 5Toll .com

Symantec

Corporation

Symantec

Endpoint

Protection

12.1

Endpoint

Security for

VirtualizationEfficacy

Tested

September

2012

Defended Neutralized Compromised Total

Symantec Endpoint

Protection835 29 6 870

Trend Micro OfficeScan 515 5 350 870

Kaspersky Security for

Virtualization802 28 40 870

McAfee MOVE Agentless 692 2 176 870

On-Access Virus Detection Rate of Endpoint Protection SolutionsUsing 870 Samples of Confirmed Malware

Table 1Source: Tolly, September 2012


3/5

When the endpoint security solution

prohibits legitimate software from running,a false positive is generated.

Tolly engineers used a corpus of 50

legitimate software samples from CNET to

determine which solutions under test

permitted their use.

Symantec Endpoint Protection 12.1 allowed

all legitimate samples and generated no

false positives

McAfee, Kaspersky and Trend Micro alsogenerated no false positives.

Test SetupAll clients were deployed from a clean

Windows 7 64-bit with 1vCPU and 2GB RAM

fully updated as of August 13, 2012. Each

solution was installed per each vendors

best practices on a template VM, with

signatures updated to August 13, 2012 (See

Table 2). Using VMware View 5, theseimages were then deployed into a non-

persistent 20-VM linked clone pool each.

Tolly engineers prepared a script which

emulated a user download of a particular

file. During the setup phase, malware and

clean samples were aggregated from

multiple sources to form the test corpus.

All files (clean and malicious) were hosted

locally to provide an identical environment

between different solutions, as each

malware was delivered via the same static

sample.

Test MethodologyMultiple different scripts were running in the

environment for the duration of the test,

which were all prepared by Tolly. The goal of

the workload was to enable each client VM

to boot up from a fresh (clean) image,

snapshot its file system and running

processes, and proceed to download andexecute a sample from the file server.

The director script kept track of which

samples had already been downloaded for a

particular iteration and test corpus. Each

time a VM booted up, it checked in with this

application and was assigned a file to

download.

The workload script was created using

AutoIT, leveraging the IE_Create

functionality to download the samples

Using process and file system snapshot

taken immediately after the download, the

script determines whether or not the file has

been successfully downloaded to the

system. Then, if the file exists, the scrip

attempts to execute it, followed by anothe

file and process snapshot to determine if the

file was allowed to run.



Table 2

Vendor Product Components Implementation

Symantec Corp.Endpoint Protection

12.1Symantec Endpoint Protection Manager 12.1.1959.1959;

Symantec Shared Insight Cache 12.1.1959.1959

Endpoint client with Shared InsightCache for on-demand scan

optimization

Trend Micro, Inc. OfficeScan 10.6OfficeScan 10.6.1062

VDI plug-inEndpoint client with VDI plug-in for

on-demand scan optimization

Kaspersky Lab Kaspersky Securityfor Virtualization 1.1

Kaspersky Security Center 9.2.69Kaspersky Security for Virtualization (ksv appliance) 1.1.0.54 Single virtual appliance. Agentlessclient communicates via VMwarevShield API

McAfee, Inc. MOVE Agentless 2.5

McAfee ePolicy Orchestrator 4.6.2 (Build: 234)

[McAfee move-sva: McAfee MOVE AV Agentless 2.5.0.228

McAfee VirusScan Enterprise for Linux 1.7.0

McAfee Agent for Linux 4.6.0.2156]

Single virtual appliance. Agentlessclient communicates via VMware

vShield API

Systems Under Test

Source: Tolly, September 2012


4/5

Using this data, the script determines at

what stage each of the samples was allowed

and where it was detected and defended by

the software, reporting this information tothe director.

After a full run-through of the client

workload, the client writes all its data back

to a shared file server, and performs a logoff.

On the View Composer, the last script

performs a client disk refresh as each

workload finishes, deleting the VM state and

cloning out a fresh base disk from which to

boot for the next iteration.

Each sample i terat ion requiredapproximately ten minutes, with all VMs

randomized to avoid excessive resource

consumption. The director logged over

6,000 boots per solution over the one week

duration of the test. Each sample was run

through at least four different VMs to ensure

accuracy.



TOLLY A/V PERFORMANCE

TEST HIGHLIGHTS

1 Lowest single-VM disk I/O and memory demand for on-demand scan withfast per-machine run time

Demonstrates avoidance of anti-virus storms through implementation of

randomization algorithm for resource-intensive functions2

Symantec Endpoint Protection 12.1:

See Tolly Report #212130 : Symantec Endpoint Protection 12.1

Competitive Anti-virus Performance in VMware vSphere 5 Virtual Environments

for complete test findings by scanning the QR code.

Source: Tolly Report #212130 October 2012

Figure 2Source: Tolly, September 2012

Test Bed Diagram


5/5

Terms of Usage

This document is provided, free-of-charge, to help you understand whether a given product, technology or service merits additional

investigation for your particular needs. Any decision to purchase a product must be based on your own assessment of suitability

based on your needs. The document should never be used as a substitute for advice from a qualified IT or business professional. This

evaluation was focused on illustrating specific features and/or performance of the product(s) and was conducted under controlled,

laboratory conditions. Certain tests may have been tailored to reflect performance under ideal conditions; performance may vary

under real-world conditions. Users should run tests based on their own real-world scenarios to validate performance for their own

networks.Reasonable efforts were made to ensure the accuracy of the data contained herein but errors and/or oversights can occur. The test/

audit documented herein may also rely on various test tools the accuracy of which is beyond our control. Furthermore, the

document relies on certain representations by the sponsor that are beyond our control to verify. Among these is that the software/

hardware tested is production or production track and is, or will be, available in equivalent or better form to commercial customers.

Accordingly, this document is provided "as is", and Tolly Enterprises, LLC (Tolly) gives no warranty, representation or undertaking,

whether express or implied, and accepts no legal responsibility, whether direct or indirect, for the accuracy, completeness, usefulness

or suitability of any information contained herein. By reviewing this document, you agree that your use of any information contained

herein is at your own risk, and you accept all risks and responsibility for losses, damages, costs and other consequences resulting

directly or indirectly from any information or material available on it. Tolly is not responsible for, and you agree to hold Tolly and its

related affiliates harmless from any loss, harm, injury or damage resulting from or arising out of your use of or reliance on any of the

information provided herein.

Tolly makes no claim as to whether any product or company described herein is suitable for investment. You should obtain your own

independent professional advice, whether legal, accounting or otherwise, before proceeding with any investment or project related

to any information, products or companies described herein. When foreign translations exist, the English document is considered

authoritative. To assure accuracy, only use documents downloaded directly from Tolly.com. No part of any document may be

reproduced, in whole or in part, without the specific written permission of Tolly. All trademarks used in the document are owned by

their respective owners. You agree not to use any trademark in or as the whole or part of your own trademarks in connection with

any activities, products or services which are not ours, or in a manner which may be confusing, misleading or deceptive or in a

manner that disparages us or our information, projects or developments.

About Tolly

The Tolly Group companies have beendelivering world-class IT services for

more than 20 years. Tolly is a leading

global provider of third-party

validation services for vendors of IT

products, components and services.

You can reach the company by email at

[email protected], or by telephone at

+1 561.391.5610.

Visit Tolly on the Internet at:

http://www.tolly.com

Interaction with Competitors

In accordance with our process for conducting comparative tests, Tolly contacted the

competing vendor, inviting them to review test methodology and their results prior to

publication. Trend Micro and McAfee did not respond to our request. Kaspersky Lab

responded to the invitation, reviewed the proposed test methodology and confirmed the

test results prior to publication.

For more information on the Tolly Fair Testing Charter, visit:

http://www.tolly.com/FTC.aspx



212131-tb-12-mts-yy - 2012-Oct-26-VerJ

tolly212131symantecsep12dot1vmwareavefficacy

Documents