mcquade-open source web vulnerability scanners - the cost effective choice?

8/10/2019 McQuade-Open Source Web Vulnerability Scanners - The Cost Effective Choice?

1/13

2014 Proceedings of the Conference for Information Systems Applied Research ISSN: 2167-1508Baltimore, Maryland USA v7 n3319

_________________________________________________

_________________________________________________2014 EDSIG (Education Special Interest Group of the AITP) Page 1www.aitp-edsig.org

Open Source Web Vulnerability Scanners:

The Cost Effective Choice?

Kinnaird [email protected] Technology Department

Marymount UniversityArlington, VA 22207

Abstract

A plethora of tools are available to software testers so that software vulnerabilities can be mitigatedbefore product deployment. However, some of these tools are less effective than others. In particular,open source dynamic web vulnerability scanners raise concerns including (1) total attack and inputvector support, (2) scan coverage of different application protocols, and (3) rate of required manualdetection versus automated detection. Additionally, what is often most attractive about proprietaryscanners is vendor support and frequent software maintenance bundled with a paid licensingagreement. Indeed, the need for software support will ensure the longevity of proprietary dynamicweb vulnerability scanners on the market. However, a low-cost alternative is available andrecommended for web developers involved in agile development at small to medium sizeddevelopment firms; it is the finding of this research that when a combination of certain open sourcetools are used in conjunction with a specific scanning strategy, there is a greater vulnerabilitydetection accuracy than solely using a single proprietary scanner.

Keywords: web vulnerability scanners, application security, dynamic tools, software developmentlifecycle, quality assurance, open source

1. INTRODUCTION

Serious security vulnerabilities that an attackercan exploit to take control over a website,compromise user accounts, or access sensitivedata occur across every major industry in theUnited States. During 2013, 96% of all webapplications tested by a major applicationsecurity firm had at least one serious securityvulnerability, and the medium number of

vulnerabilities per website was 14 (Cenzic, p. 3,2014). Hackers are concentrating their efforts onweb applications where users enter sensitiveinformation such as forms, login pages, andshopping carts to gain access into servers to findpersonal and corporate information. Insecureweb applications jeopardize corporate andgovernment databases that are critical to thefinancial health and economic stability of theUnited States. The sooner a security

vulnerability is discovered and corrected, theless likely that error will create more errors thatmay result in significantly more effort to correct.

Application security specialists use acombination of static, dynamic, and manualtesting techniques to perform securityassessments (OWASP, p.4, 2009). Reliance onone technique for security testing before thedeployment phase in the software development

lifecycle would be a potentially disastrousmistake. Therefore, application security firmsusually implement a holistic assessment usingthese techniques as well as performingarchitecture risk analysis (Stevens, 2011). Inorder to understand the role of dynamic webvulnerability scanners, it is important tounderstand the difference between dynamic,static, and manual testing.
mailto:[email protected]:[email protected]


2/13


_________________________________________________


To properly explain the difference betweendynamic and static testing: in the case exampleof a Hello World Java application, static toolswould only analyze the Main.java class at theprogram compile time, whereas dynamic tools

would not only scan the Main.java class, butwould also scan the entirety of theaccompanying Java Runtime Environment.Gartner explains that dynamic scanners analyze

applications in their running state duringoperation or testing phases, simulatingprepared attacks and analyzing the response todetermine the existence of vulnerabilities(MacDonald & Feiman, 2014).

Burp Suite Professional is an excellent case

example of manual testing (Portswigger WebSecurity, 2014). The web browser forwards allincoming requests to Burp Proxy, which

intercepts the HTTPS traffic passing in eitherdirection and allows the user to analyze andalter the GET and POST requests to profile ortamper with the web application. Manual testingrequires a notable amount of training andtechnical knowledge (Halfond, Choudhary, &Orso, pg. 18, 2011).

There are certainly limitations to the usefulnessof dynamic web vulnerability scanners; they are

not the cure-all solution for vulnerabilitydetection. Cigital estimates that dynamic testingonly uncovers up to 12% of all discoveredsoftware flaws, whereas Architecture Risk

Analysis can uncover up to 60% of discoveries(Stevens, 2011). Amongst the assessmenttechniques for detecting web application

vulnerabilities, dynamic scanning tools are muchless effective when large-scale security testingoccurs just prior to the deployment phase.However, it has an indispensable role in thesecurity of the session, presentation, andapplication layers of web applications and should

be used in conjunction with secure designprinciples, static testing, and manual codereview. Brian W. Kernighan and Rob Pike explainin The Practice of Programming (1999):

Its tedious and unreliable todo much testing by hand;

proper testing involves lots oftests, lots of inputs, and lotsof comparisons of outputs.Testing should therefore bedone by programs, whichdont get tired or careless.

Mark G Graff and Kenneth R. van Wyk advisethat performing static, dynamic, and manualtesting at every stage of the development cycleis paramount to securing the session,presentation, and application layers of a web

application (Graff & van Wyk, p. 157, 2003).The automation of these scanners increases thetime efficiency and consequently, the cost-effectiveness of vulnerability detection whileoffering the promise of allowing developers tofocus on coding and design rather than having tobecome nearly full-time security testers. Thesescanners can be used in a local environmentbefore deployment. They should be used inconjunction with static scanners, which is notincluded in the scope of this paper. With training

and an understanding of the underlying behaviorin Web Application Vulnerability Scanners,software developers in agile development groups

can perform security testing betweendevelopment cycles so that software errors arediscovered earlier, without sacrificing detectionaccuracy.

Although the liberty of choice when selecting aweb vulnerability scanner (or any software, forthat matter) upon budget flexibility, thescanners vulnerability detection features,accuracy, coverage, and stability should be

considered just as strongly as the reality offinancial pressures. Indeed, the persistence ofsevere software bugs can prove to be a muchhigher cost than commercial licenses but

making the best decision for an organizationinvolves much more than just asking thequestion to spend or not to spend.

In any purchase, high cost can deceive theconsumer into assuming that the product is ofhigher quality. There are high-qualitycommercial scanners on the market, but the factis that there is no silver bullet solution in this

area of Information Technology or any other.Black-box vulnerability scanners do not cover allfeatures identified for comparison purposes byWAVSEP. Proprietary Web Vulnerability Scannerssuch as IBM Appscan and HP WebInspect, when

used in conjunction, cover the most categories.However, they can also cost the user $20,000per year and $10,000 a year per installation,respectively. To complicate the issue, theirdetection accuracy on vulnerable applications inthe 6 main vulnerabilities evaluated by WAVSEPdoes not always have the highest detectionaccuracy.


3/13


_________________________________________________


After analyzing and confirming his findings, thisauthor of this paper confirmed that a low-costcombination of vulnerability scanning tools canbe used to support the same attack vectors witha detection accuracy that is greater than or

equal to proprietary scanners. The finding of thispaper is most highly recommended for agiledevelopers wishing to test their modules forsecurity vulnerabilities before entering anotheriteration cycle.

2. LITERATURE REVIEW

The existing research that will be discussed inthis section examines the cost-effectiveness ofdifferent Web Application Vulnerability Scanners,

the limitations of these dynamic scanners, andperspective regarding the commercial methodsof providing product evaluation for potential

customers.

Analyzing the Accuracy and Time Costs of

Web Application Security ScannersIn Larry Sutos follow-up to his 2007 paper, hegeneralized the web application testingcommunity into two groups. He explains thatGroup One uses scanners in a point and shootmanner, relies on the scanners crawler andautomation to exercise the sites functionality

within minimal or no human guidance, whileGroup Two believes that scanners should be anadjunct to human testing and only used to grabthe easy vulnerabilities, or low hanging fruit

(Suto, p. 13, 2010).

Why Johnny Cant Pentest: An Analysis ofBlack-box Web Vulnerability ScannersIn their 2010 work, Doupe, Cova, and Vignaexplain that web application scanners areessentially comprised of three main modules: acrawler, an attacker, and an analysis module(Doupe et. al, p. 3, 2010). During their

research, they created the WackoPicko Web Sitea purposefully vulnerable picture sharing site which is now distributed on OWASPs BrokenWeb Applications Virtual Machine. All of theirWeb Application Vulnerability Scanners were

commercial tools, with the exception of Grendel-Scan, w3af, and Paros. Their research generatedthree strong conclusions:

(1)

Support for well-known, pervasivetechnology (such as JavaScript, Flash,etc.) should be improved.

(2)

More sophisticated algorithms areneeded to perform deep crawling and

track the state of the application undertest.

(3)

More research is warranted to automatethe detection of application logicvulnerabilities.

Similarly to Sutos 2010 paper, they assert thatvulnerability scanner technology is far frombeing point-and-click tools to be used by

anybody and that web application black-boxsecurity scanners require a sophisticatedunderstanding of the application under test andof the limitations of the tool in order to beeffective (Doupe et. al., p. 20, 2010). Theirfinal conclusion is that there is no strongcorrelation between the cost of the scanner and

functionality provided as some of the free orvery cost-effective scanners performed as wellas scanners that cost thousands of dollars

(Doupe et. al., p. 20, 2010)

The most interesting part about their well-earned conclusion is their perspective that anintimate understanding of the target applicationis mandatory in order to achieve useful results.It is important that developers are involved inthe dynamic testing process for maximumeffectiveness.

Avoiding the Test Site FallacyVeracode, a Massachusetts-based applicationsecurity company known for their static codeanalysis engine, produced a white paper in 2012

that sought to explain why purchasers ofproprietary web vulnerability scanners shouldnot gauge the abilities and effectiveness of a

particular scanner by only looking at the resultsfrom scanning public test sites (Dawson, 2012).In their evaluation, they addressed five sites foranalysis, produced by a few prominent scannervendors. The test sites for IBM Appscan,NTOSpider, HP WebInspect, and Acunetix were

included in their evaluation. Veracode criticizedthese closed-source sites for possessingunrealistic or fake vulnerabilities, unrealisticform validation or checks, missing vulnerabilitycoverage, and for hiding these issues from the

user. From their own testing of the site, theyprovide convincing evidence of theseshortcomings from each individual site.

Although Veracodes alarming attack on thevendors of proprietary scanners raises manyquestions about the true capabilities of thesescanners, it does not lessen the industry-wideattraction to proprietary scanners. It does,


4/13


_________________________________________________


however, indicate the need for third-partyevaluations providing comparative analysis onthe scanners in question.

3. EXISTING EVALUATION METHODS

One of the most popular technologies regardingWeb Application Vulnerabilities is the BrokenWeb Apps Project, produced by the OWASPFoundation and distributed as a Linux VirtualMachine. It is a collection of trainingapplications, intentionally vulnerableapplications, outdated versions of realvulnerable applications, applications for testingtools only (including WAVSEP), and vulnerabilitydemonstration pages. The most well-known of

these web apps is WebGoat, an applicationsecurity training web app that offers over 30lessons dealing with various vulnerabilities.

Gartners Magic Quadrant for ApplicationSecurity Testing offers a qualitative evaluation ofthe commercial vendors in the Web ApplicationSecurity Market. Their evaluation criteria focuseson the support structure between the scannersvendor and their customer, drawing theirevidence from the collection of surveys fromcustomers, responses from vendors, andhundreds of inquiries regarding the scanners

throughout 2013-2014. Customer experience,marketing strategy, and completeness ofcompany vision were among the comparisonpoints. This report is especially helpful for

companies who intend on purchasing a licenseand expect to use this software frequently.

The Web Application Vulnerability ScannerEvaluation Project (WAVSEP) is a yearlybenchmark produced by Shay Chen, a widelyrespected application security researcher, whichtests the vulnerability detection accuracy of 63Black Box Web Application Vulnerability

Scanners and discusses their capabilities. Inaddition to the benchmark, Shay Chen has alsopublished a feature comparison between all thescanners, the numbers and types of VulnerabilityDetection Features, and the detection accuracy

of 6 software weakness types (5 of which are inthe OWASP Top 10). One of the most helpfulwork products from WAVSEP for consumers isthe visual, understandable comparison of all thescanners so that the consumer can use the datafor their own analysis.

Web Application Security Consortium produced aset of evaluation criteria in 2008 (WASSEC) that

grades web application scanners on their abilityto effectively test web applications and identifyvulnerabilities. It covers areas such as crawling,parsing, session handling, testing, and reporting.The goal, similar to Shay Chens WAVSEP

project, is a vendor-neutral document producedto:

(1)

Provide scanner users with the toolsthey need for conducting a detailedevaluation and making an informeddecision about which web applicationscanner(s) to choose;

(2)

Provide scanner developers with a list ofcapabilities to compare their toolsagainst to help them create a roadmap

of future enhancements.

4. MAPPING THE SUBSTITUTIONS

The first requirement for determining the propercombination of scanners is that the maximumamount of input vectors and attack vectorsshould be covered so that most of the targetapplication can be scanned. The recommendedcombination of open source substitutions hasbeen mapped in Appendix 2.

The following tools are recommended as a result

of this evaluation:

1. Burp Suite Professional - $300 per year;low-cost proprietary.

2.

IRONWASP Open Source3.

Zed Attack Proxy (ZAP) Open Source4.

Arachni Open Source

5.

W3af Open Source.

It should be noted that w3af was not includedfor evaluation in this study. Its detectionaccuracy in the WAVSEP evaluation was equal toor less than the detection accuracy of the

highest-performing open source scanners ineach vulnerability category. It is, however, ahighly regarded web vulnerability scanner in theopen source penetration testing community. Itsexploitation features after vulnerability scanning

are more extensible than some of therecommended tools in this study. However, thisstudy focused on vulnerability detection, notexploitation. It was decided for this reason tonot include w3af in the results of this paper.IRONWASP is a powerful open source webvulnerability scanner that offers plugincompatibility with both Python and Ruby,making it an attractive open source scanner for


5/13


_________________________________________________


those wishing to make their own scanners orcustomize the tool with their own plugins(Kuppan, 2014). It is very easy to use and offerssome interactive scan capabilities that aresimilar to Burp Suite.

Zed Attack Proxy (ZAP) is noticeably similar toBurp Suite Professional and doubles as ascanner with the help of the Plug-n-Hack MozillaFirefox plugin (OWASP Zed Attack Proxy Project,2014). The design is clean, the tool islightweight, and it is strong in its detectionaccuracy. It is not as automated as the Arachniweb user interface and does not have thecleanest reporting tool, but it is clearly apowerful open source scanner

Arachni is a very high-performing, easy-to-use,modular web vulnerability scanner written in

Ruby (Laskos, 2014). It offers a web userinterface that permits multiple users to managescans and collaborate afterwards. It is veryquick and its HTML report presentation is veryclean. The web user interface is verycustomizable but unfortunately the interactivecapabilities for manual testing are reserved forthe command line version only. However, theavailable scan options are so customizable thatit would be ideal for a software developer

wishing for a point and click dynamic scanningexperience.

There are multiple attack vectors not supported

by the recommended combination of tools in thisstudy. However, these attack vectors are eitherrarely implemented issues or are so vendor-

specific that manual and static testing is the onlyreliable or realistic way to find a vulnerability. Asindicated by the visually significant gap in inputvector support (see Appendix 1), mostproprietary scanners do not even support theseareas.

The data below highlights his findings in sixareas:

(1) Old, Backup, and Unreferenced Files

(BACKUP)(2) Path traversal/Local File Inclusion (LFI)(3)

Unvalidated Redirects (REDIRECT)(4)

Reflected Cross Site Scripting (RXSS)(5)

SQL Injection (SQLi)(6)

Remote File Inclusion (RFI)

These first three are the areas where opensource stands out above the proprietary

software in detection accuracy; in the last groupof three, open source programs are equal indetection accuracy.

Figure 1. Results adapted from WAVSEP.A visual representation of the vulnerabilitydetection from the WAVSEP 2014 benchmarkcan be seen in Figure 1 above.

Figure 2. Results adapted from WAVSEP.

The recommended open source combinationperforms well in each one of these vulnerabilitydetection results. In the last four vulnerabilitygroups, open source tools have the samedetection accuracy as their proprietarycounterparts 100%. Surprisingly, open source

Arachni I

RONWASP

IR

SP

rachni

IR

SP

ZAP

AppScan

BurpSuitePro c

unetix

AppScan

ebInspect

Acunetix

0.00%10.00%20.00%30.00%40.00%50.00%60.00%70.00%80.00%90.00%

100.00%

Detection Accuracy

Open Source WVS

High-cost Proprietary WVS

WAVSEP Detection Accuracy by tool

OpenSource orLow-cost

scanner

High-costProprietaryscanner

Remote FileInclusion

100%

Arachni

100%

IBM

AppscanSQLDetection

100%

SQLMap,Burp Suite

100%

AcunetixWebInspect

ReflectedXSS

100%

IRONWASP

100%

Acunetix,AppScan,Netsparker

PathTraversal/LFI

100%

Arachni

100%

AppScan

UnvalidatedRedirect

73.33%

IRONWASP

50%

WebInspect

Old, Backup,andUnreferenced

Files

38.04%

ZAP

32.61%

Acunetix


6/13


_________________________________________________


outperforms the highest performing proprietaryscanners in the first two categories Unreferenced Backup files and UnvalidatedRedirects.

The detection accuracy for these vulnerabilitiescan be seen above in Figure 2. The detectionaccuracy is listed in the same cell as the scannername, subdivided by a dashed line. Therecommended open source tools are indicatedby the grey column.

The WAVSEP v1.5 evaluation emulates differentcommon test case scenarios for generictechnologies. Shay Chen explains his reasoningfor the WAVSEP evaluation structure further in

his 2014 evaluation:

A scanner that is not accurate enough

will not be able to identify manyexposures, and might classify non-vulnerable entry points as vulnerable.

These tests aim to assess how good iseach tool at detecting the vulnerabilitiesit claims to support, in a supported inputvector, which is located in a known entrypoint, without any restrictions that canprevent the tool from operating properly(Chen, 2014).

To put it simply while each of these scannerscan detect more flaws than the vulnerabilitiesthat were evaluated, the test results speak to

the overall accuracy of the tool itself when thetests are put on an equal playing field. Tosupport this point, this paper includes an

independent evaluation of the recommendedcombination of tools below using the JavaEnterprise Edition 7 evaluation.

5. EXPERIMENTAL EVALUATION

This research used Dukes Forest, a WebApplication provided by Oracle as part of theirJava EE 7 tutorial, to evaluate the effectivenessof the recommended combination of free or low-cost web vulnerability scanners. Dukes Forest is

an e-commerce application provided by Oracleas a case study for understanding the fullcapabilities of Java Enterprise Edition 7(Jendrock, et. al, 2014). This final example inthe Java EE 7 Tutorial acts as a store thatprovides gardening supplies to online shoppers.The complete web application includes a productcatalog, customer self-registration, and ashopping cart; it also offers shipment and

payment functionalities that can be managedthrough a separate administrative portal.

DesignDukes Forest is a simple e-commerce

application that interacts with the user throughthe Dukes Store interface. A non-administrativeuser of Dukes Store is able to browse theproduct catalog for foresting supplies, add itemsto their shopping cart, specify the shipmentprocess, and manage their basic accountinformation.

Authentication and SecurityDukes Forest uses HTTP Basic Authenticationand JAAS (Java Authentication and Authorization

Service) to authenticate the user. Securityconstraints are built in to differentiate betweencustomers and administrators. Single Sign-On

(SSO) is used to simplify the administratorsbrowsing experience in navigating between theDukes Store and Dukes Shipment portals.

Browsing the product catalogThe foresting product categories include Plants,Food, Services, and Tools. Users can browse theproduct catalog, filter their searches accordingto product categories, and view product details.

Signing up as a new customerWhile the product catalog can be browsedwithout user registration, Dukes Store requiresuser registration in order to add items to the

shopping cart and make purchases. Thepassword value must be at least 7 characters inlength. There are already two user names

included in the database upon websitedeployment [email protected] [email protected] and they both have thepassword, 1234.

Shopping Cart and Checking OutA registered user on Dukes Store can purchaseforesting supplies. The purchase follows aprocess in which a shopping cart is filled with theforesting items to be purchased. After theforesting supplies are added to the cart, the sum

price is calculated, and the order is placed.Orders exceeding $1,000 are not permittedbecause the Payment web service denies ordersover that limit. Once the order is placed, theuser must wait for the administrative user toapprove the shipping process.

Viewing order status
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


7/13


_________________________________________________


After the checkout process is complete, amessage appears to the user: Your order isbeing processed. Check the Orders page to seethe status of your order.

Administrative InterfaceDukes Forest allocates a portion of the websitefor administrators only. The Administrator pageis used to perform back office operations, suchas the creation, editing, updating, and deletionof products, categories, customers, and usergroups.

VulnerabilitiesCross-Site Scripting: The ImageServlet.javaclass in the /com/forest/web/util package is

vulnerable to Cross-Site Scripting (OWASP,2014c). Almost every other java class in theprogram utilizes the data validation method

provided by Java Enterprise Edition, but thisparticular class simply does not include themechanism. This could be due to an error by theprogrammer.

SQL Injection:the JSESSIONID cookie, which isdisplayed in cleartext in the URL, is vulnerable toSQL Injection attacks (OWASP, 2014a). Byappending a basic blind SQL injection statementsuch as 1 OR 1=1 as encoded URL into the

cookie, different responses are returned,indicating that the input is being incorporatedinto the SQL query incorrectly. Upon furthermanual testing, this can be exploited to gain

access to more database information.

Cleartext Credentials: The store page, login

page, customer registration page, andindex.html page contain a login form which issubmitted over clear-text HTTP headers.Manipulating the GET and POST requests with anintercepting proxy such as Zed Attack Proxy orBurp Suite Proxy can disclose passwords to a

hacker performing a man-in-the-middle attacksuch as ARP poisoning (OWASP, 2014b).

Insecure Cookies:Forms with sensitive content,like passwords, must be sent over HTTPS. The

login page, strangely, has enabled the HTTP-only flag for the web cookie exposing anylegitimate user to credential theft, man-in-the-middle attacks, and cross-site request forgery(OWASP, 2014b).

Session token in URL: The session ID istransmitted in the URL when the user logs in tothe website in certain cases. In the case of a

man-in-the-middle attack, an attacker could usethe session ID to create a fake browser cookie inorder to gain administrator or user access to thesystem (OWASP, 2014b).

Password Auto-enabled: Disabling theautocomplete function in the login form will limitthe users browser from remembering usercredentials that are entered into HTML forms.The stored credentials can be captured by anattacker who gains physical or remote access tothe computer. If the administrators computerwere hacked, for example, the security of thecompanys website would be jeopardized(OWASP, 2014b).

Missing anti-Cross-Site Request Forgery token:There is no synchronized token mechanism inthe user form data validation mechanisms for

guaranteeing the freshness of the submitteddata. This unique anti-CSRF token is thesuggested standard for all secure J2EEapplications and should exist in all webapplication elements that can affect businesslogic (Alur, p. 77, 2013). An attacker couldimpersonate a legitimate user of the webapplication by exploiting this vulnerability withthe help of the aforementioned HTTP-only flag-enabled cookies (OWASP, 2014f).

ResultsSQL Injection:Burp Suite was the only scannerto detect the SQL Injection vulnerabilities by

appending an encoded injection to theJSESSIONID cookie and inside the formsubmissions. It detected 4 of these cases.

Cross-Site Scripting:None of the scanners wereable to detect this vulnerability. Thisvulnerability was only discovered by theresearcher after inspecting the code line-by-linefor instances of any code missing proper data

validation mechanisms. Indeed, there is only somuch that an automated black-box vulnerabilityscanner can do to discover vulnerabilities. AsSuto mentioned these dynamic scanners aremeant to discover the low hanging fruit (Suto,

p. 13, 2010.) White-box code inspection, manualblack-box testing, and architecture analysismust all play a part in discovering thesevulnerabilities.

Cleartext Credentials: This was peculiar.Acunetix was not able to detect an instance ofbroken authentication in this case, even whenthe HTTP Sniffer function (separate from the


8/13


_________________________________________________


overall scanner module) was configured to lookfor this vulnerability. Burp Suite and IRONWASPwere able to detect all 19 instances of thisvulnerability.

Insecure Cookies: Burp Suite outperformed allscanners in this case, producing 20 uniquecases; Acunetix discovered only one instance,and none of the other scanners were able todetect the insecure cookies.

Session token in URL: Surprisingly, Acunetix,ZAP, and Arachni did not find this vulnerabilitythat was so easy to spot with the human eye.Burp Suite and IRONWASP were able to see thatthe JESSESSIONID was transmitted in the URL

bar and found all 3 instances.

Password auto-enabled: This simple security

misconfiguration was detected in full by BurpSuite and ZAP 12 instances; Acunetix andArachni discovered 8 apiece and IRONWASPfound none.

Missing anti-Cross-Site Request Forgery token:Acunetix, ZAP, and Arachni all tied for maximumdetection on this vulnerability, finding 8instances of HTML forms without Cross-SiteRequest Forgery protection. Burp Suite

discovered one instance, and IRONWASP foundnone.

Java EE 7 applications rely primarily on security

policy files for configuring authentication,authorization, and encryption. Java EE wasdesigned with the intention of shieldingapplication developers from the complexity ofimplementing security features (Jendrock, p.48, 2014). It also provides standard loginmechanisms so that application developers donot have to implement these mechanisms intheir applications (Jendrock, p. 48, 2014).

Security restraints must be defined to specifyauthorization and authentication controls. DukesForest was implemented with default securitysettings provided by Java EE 7 mechanisms,which ignores the 5thitem on the OWASP Top 10

list of 2013 Vulnerabilities SecurityMisconfiguration (OWASP, 2014). The methodsprovided by Java EE might not be sufficient foreach individual web application. For instance,data validation for ZIP codes should be differentthan forum-like text boxes that are meant toaccept HTML input. In this case, dynamicscanners are helpful to developers that mightnot have extensive experience implementing the

data validation methods provided by Java EE forcustom applications.

The vulnerability detection rate of theserecommended web vulnerability scanners on

other realistic web applications may be different,depending on the application. This is apparentwhen comparing the results of the webvulnerability scanners in the context of Dukes

Forest, and Chens WAVSEP test cases.However, the scan results on the Dukes Forestapplication signify that open source scanners notonly present a cost-effective option todevelopers and testers they are a viable optioninstead of proprietary scanners in the casesmentioned above.

In any software development company, the costof these scanners must be considered along with

the detection accuracy. Burp Suite offers a low-cost solution to companies that are moreunwilling to purchase the expensive softwarelicense that proprietary scanners require for onecomputer. Acunetixs licensing options rangebetween approximately $3,000-12,000 per year.HP WebInspect offers rates starting at $1,500for one IP address and one computer, with risinglicensing costs thereafter. IBM Appscans pricingranges between approximately $10,000-38,000

per year (Chen, 2014).

Dynamic web vulnerability scanners shouldnever be the only solution for discovering

software security flaws, but using open sourceweb vulnerability scanners earlier in thesoftware development lifecycle will increase

early detection rates, lower security assessmentworkloads by using automated tools, anddecrease total cost over the products lifecycleby limiting expensive licensing costs.

6. CONCLUSION

Dynamic web vulnerability scanners shouldnever be the only solution for discoveringsoftware security flaws, but using open sourceweb vulnerability scanners earlier in the

software development lifecycle will increaseearly detection rates, lower security assessmentworkloads performed before applicationdeployment, and decrease total cost over theproducts lifecycle by limiting expensive licensingcosts.

This paper presented a low-cost alternativebased on open source tools to high-cost


9/13


_________________________________________________


proprietary black-box web vulnerability scannersand supported this alternative combination oftools with the results of scans on the DukesForest application and scans performed by ShayChens WAVSEP yearly benchmark. The results

of this papers evaluation clearly show:

The detection accuracy with these toolsis more accurate than the detectionaccuracy with proprietary webvulnerability scanners in the test caseprovided by this evaluation.

The input vector and attack vectorsupport from these scanners can covernearly every area of support byproprietary web vulnerability scanners

We also hope that future research anddevelopment will create an aggregate tool for

integrating select functions from theserecommended web vulnerability scanners usingthe APIs provided by the application developers.An aggregate tool utilizing the strongestcapabilities of these open source products willcreate a web vulnerability scanner that is trulymore powerful than the sum of its parts.

7. ACKNOWLEDGEMENTS

The research for this paper was financiallysupported by the National Science Foundation,under grant no. 1241440. In developing theideas presented here, I have received helpful

input from Neil Bahadur, Ashley Corbett, andLavelle Perry. I also thank my three facultymentors Dr. Diane Murphy, Dr. Ali Bicak, and

Dr. Michelle Liu for guiding me in this process.

8. REFERENCES

Bennetts, S. (2014, May 21). Zed Attack Proxy(Version 2.3.1) [Computer software].

Retrieved July 1, 2014, fromhttps://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project

Cenzic, Inc. (2014). Cenzic Application

Vulnerability Trends Report [White Paper].

Chen, S. (2014). Security Tools Benchmarking:WAVSEP Web Application ScannerBenchmark 2014.

Dawson, I. (2012). Broken Logic: Avoiding theTest Site Fallacy. [White paper].

Doupe, A., Cova, M., & Vigna, G. (2010). WhyJohnny Can't Pentest: An Analysis of Black-box Web Vulnerability Scanners. 7thInternational Conference on Detection ofIntrusions and Malware, and Vulnerability

Assessment, 1-21. Retrieved June 31, 2014.

Graff, M., & van Wyk, K.R. (2003). Securecoding: Principles and practices. SanFrancisco, CA: O'Reilly.

Halfond, W. G., Choudhary, S. R., & Orso, A.(2011). Improving Penetration Testingthrough Static and Dynamic Analysis.Software Testing, Verification and Reliability.doi: 10.1002/stvr.450

Kernighan, B. W., & Pike, R. (1999). Thepractice of programming. Reading, MA:

Addison-Wesley.

Kuppan, L. (2014, April 15). IRONWASP (Version0.9.8.4) [Computer software]. RetrievedSeptember 15, 2014, fromhttps://ironwasp.org/about.html

Laskos, T. (2014, September 7). Arachni WebApplication Security Scanner Framework(Version 1.0.1) [Computer software].

Retrieved September 20, 2014, fromhttp://www.arachni-scanner.com/

Jendrock, E., Cervera-Navarro, R., Evans, I.,

Haase, K., Markito, W., & Oracle Corp.(2014, January 1). Java EE 7 Tutorial.

MacDonald, N., & Feiman, J. (2014). MagicQuadrant for Application Security Testing(Rep.). Stamford, CT: Gartner.

Malks, D., & Crupi, J. (2013). Synchronizer orDeja vu Token. In D. Alur (Author), Core

J2EE Patterns: Best Practices and DesignStrategies (pp. 77-87). Upper Saddle River,NJ: Prentice Hall.

OWASP Foundation. (2009). Code Review Guide

(Vol. 1.1). Bel Air, MD: OWASP Foundation.

OWASP Foundation. (2014a). A1 - Injection.Retrieved September 3, 2014, fromhttps://www.owasp.org/index.php/Top_10_2013-A1-Injection


10/13


_________________________________________________


OWASP Foundation. (2014b). A2 BrokenAuthentication and Session Management.Retrieved September 3, 2014, fromhttps://www.owasp.org/index.php/Top_10_2013- A2-

Broken_Authentication_and_Session_Management

OWASP Foundation. (2014c). A3 Cross SiteScripting. Retrieved September 3, 2014,fromhttps://www.owasp.org/index.php/Top_10_2013-A3-Cross-Site_Scripting_(XSS)

OWASP Foundation. (2014d). A6 SensitiveData Exposure. Retrieved September 3,

2014, fromhttps://www.owasp.org/index.php/Top_10_2013-A6-Sensitive_Data_Exposure

OWASP Foundation. (2014e). A7 MissingFunction Level Access Control. RetrievedSeptember 3, 2014, fromhttps://www.owasp.org/index.php/Top_10_2013-A7-Missing_Function_Level_Access_Control

OWASP Foundation. (2014f). A8 Cross SiteRequest Forgery. Retrieved September 3,

2014, fromhttps://www.owasp.org/index.php/Top_10_2013-A8-Cross-Site_Request_Forgery_(CSRF)

OWASP Foundation. (2014g). WebGoat Project(Version 6.0) [Computer software].Retrieved September 24, 2014, fromhttps://www.owasp.org/index.php/Category%3AOWASP_WebGoat_Project

Open Web Application Security Project. (2013).OWASP Top Ten Project. Retrieved fromhttps://www.owasp.org/index.php/Category%3AOWASP_Top_Ten_Project

Portswigger Web Security. (2014, July 22). BurpSuite Professional (Version 1.6.03)[Computer software].

Stevens, J., & Cigital. (2011). When All YouHave is a Hammer. [White paper].

Suto, L. (2010). Analyzing the Accuracy andTime Costs of Web Application Security

Scanners. Analyzing the Accuracy and TimeCosts of Web Application Security Scanners,1-20. [White paper].

Tung, Y., Tseng, S., Shih, J., & Shan, S. (2013).A cost-effective approach to evaluatingsecurity vulnerability scanner. Paperpresented at the Network Operations andManagement Symposium (APNOMS), 201315th Asia-Pacific.

Web Application Security Scanner EvaluationCriteria. (2009). Web Application SecurityConsortium,1-26.


11/13


_________________________________________________


APPENDIX 1

Dukes Forest Scan Results

Vulnerability Acunetix

Burp

Suite IRONWASP ZAP W3AF Arachni

1 Application Error Message 13

2 HTML Form w/o CSRF

Protection 8 1 8 8

3 User Credentials in Clear text 3 19 194 Clickjacking 1 14 14 14 14

5 OPTIONS method is enabled 1

6 Possible sensitive files 22

7 Session cookie without

secure flag reset 1 20

8 Content type is not specified 7

9 GHDB: Outlook PST File 1

10 GHDB: Outlook Postscript

File 1

11 Password input: auto-

complete enabled 8 12 12 8

12 SQL Injection 0 4

13 Session token in URL 3 314 Cross Domain Referrer

Leakage 1 1

15 Email Addresses disclosed 1

16 Web Browser XSS Protection

not enabled 1

Key:

Burp Suite (low cost)

Open Source

Highest Detection Count

False Positive Vulnerabilities


12/13


_________________________________________________


APPENDIX 2

Attack Vector Support. Adapted from WAVSEP.

Key:

= supported by Burp Suite

= supported by recommended open sourcescanner

= supported by proprietary

= not supported


13/13


_________________________________________________


APPENDIX 3

Input Vector Support. Adapted from WAVSEP.

Key:

= supported by Burp Suite

= supported by recommended open sourcescanner

= supported by proprietary

= not supported

mcquade-open source web vulnerability scanners - the cost effective choice?

Documents