Research ArticlePerformance-Based Comparative Assessment ofOpen Source Web Vulnerability Scanners

Mansour Alsaleh,1 Noura Alomar,2 Monirah Alshreef,2

Abdulrahman Alarifi,1 and AbdulMalik Al-Salman2

1King Abdulaziz City for Science and Technology, Riyadh, Saudi Arabia2College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia

Correspondence should be addressed to Mansour Alsaleh; maalsaleh@kacst.edu.sa

Received 7 January 2017; Revised 25 March 2017; Accepted 16 April 2017; Published 24 May 2017

Academic Editor: Michele Bugliesi

Copyright © 2017 Mansour Alsaleh et al. This is an open access article distributed under the Creative Commons AttributionLicense, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properlycited.

The widespread adoption of web vulnerability scanners and the differences in the functionality provided by these tool-basedvulnerability detection approaches increase the demand for testing their detection effectiveness. Despite the advantages of dynamictesting approaches, the literature lacks studies that systematically evaluate the performance of open source web vulnerabilityscanners. The main objectives of this study are to assess the performance of open source scanners from multiple perspectivesand to examine their detection capability. This paper presents the results of a comparative evaluation of the security features aswell as the performance of four web vulnerability detection tools. We followed this comparative assessment with a case study inwhich we evaluate the level of agreement between the results reported by two open source web vulnerability scanners. Given thatthe results of our comparative evaluation did not show significant performance differences among the scanners while the resultsof the conducted case study revealed high level of disagreement between the reports generated by different scanners, we concludethat the inconsistencies between the reports generated by different scanners might not necessarily correlate with their performanceproperties. We also present some recommendations for helping developers of web vulnerabilities scanners to improve their tools’capabilities.

1. Introduction

The multilayered architectures of web-based systems andtheir sophisticated interactions with different types of sub-systems increase the number of security flaws that can beexploited by attackers. As emphasized by theOpenWeb Secu-rity Application Project (OWASP) [1], attackers might followvarious paths through the digital infrastructure of an orga-nization to find security weaknesses that might cause severeconsequences. According to Symantec [2], over 6500 webvulnerabilities were reported in 2014 and twenty percent ofthem were expected to have critical consequences. Kasperskyalso reported that more than 93millionmalicious URLs weredetected in the first quarter of 2015 [3]. By 2018, web servervulnerabilities are also expected to be ranked as the first inthe list of the most serious cybersecurity attacks [4].

Dynamic security testing and static code analysis areones of the main approaches that are used for detecting webvulnerabilities. Software developers use web vulnerabilityscanners to automate the process of examining the security oftheir web-based applications and conduct large-scale securityanalyses on many different web applications [5]. While auto-mated static code analysis techniques are more effective fordetecting security defects (e.g., SQL vulnerabilities) [6, 7], theadoption of these techniques is limited compared to thewidespread application of automated dynamic security test-ing tools. One of the reasons that contributed to limiting theadoption of static code analysis by software developers is thatit cannot be performed without accessing the source code ofthe evaluated web-based application, whichmight not alwaysbe accessible [5].

HindawiSecurity and Communication NetworksVolume 2017, Article ID 6158107, 14 pageshttps://doi.org/10.1155/2017/6158107

2 Security and Communication Networks

There are many commercial tool-based web vulnerabilitydetection approaches for identifying web security defects.IBM Security AppScan [8], HP webInspect [9], Acunetix webvulnerability scanner [10] are all examples of commercialdynamic web vulnerability detection tools. Although thesescanners are considered state-of-the-art and were proved tobe effective in identifying and analyzing security vulnerabil-ities, their utilization is limited to the developers who canafford their high prices [11–14]. In addition, a large percentageof the web content is developed by either individual develop-ers or small and medium-sized enterprises that do not followa secure software development life cycle and thus pay moreattention to delivering the functionality of their systems thandealing with system security complexities. Such web applica-tions and systems are vulnerable to various attacks compro-mising their web servers’ code integrity [15]. Due to the highimpact of the propagation of security attacks on theweb, thereis a high demand for facilitating the vulnerability detectionactivities by providing effective, efficient, and easy to use webvulnerability scanners. As an attempt to fulfill this demand,our study focuses on evaluating the performance of opensource web vulnerability scanners and determining whethertheir cost-effectiveness negatively correlates with their detec-tion capability and accuracy.

In this paper, we conduct a comprehensive evaluation ofthe performance of a set of open source dynamic web appli-cation security testing tools. These tools work by crawlingor scanning all web pages within a domain and automatingthe security testing process by randomly simulating attackingscenarios [16]. Although web security scanners might beeffective in identifying security flaws and malicious con-tent, previous studies proved that there are inconsistenciesbetween the results reported by different scanners [17–19].Further, these tools vary in their performance characteristics,types of detected vulnerabilities, and crawler coverage [17–19]. This in turn decreases the level of trustworthiness in webvulnerability scanners and increases the demand for conduct-ing experiments that quantitatively evaluate their quality andaccuracy.

Our study started by reviewing some of the currentlyavailable open source dynamic security testing tools andchoosing four of them based on a predefined selection crite-ria. After performing a comparative evaluation of the perfor-mance of the four chosen scanners, two scanners were chosento be included in a case study which utilized these tools todetect the security flaws in 140 unique web-based applica-tions. Wapiti, Skipfish, and two versions of Arachni opensource scanners were studied in our comparative assessmentwhereas the experimental evaluation performed in our casestudy included Wapiti 2.3.0 and Arachni 0.4.3 only.

Of the four evaluated scanners, our results show thatSkipfish 2.1 [20] is the fastest. In terms of crawler coverage,however, Arachni 1.0.2 [21] ranks first with a coverage per-centage of 94%. We utilized the benchmark provided by theWeb Application Vulnerability Scanner Evaluation Project(WAVSEP) [22] to evaluate the accuracy of the four scannersin question. Our findings reveal that the best SQL and XSSvulnerability detection rates were obtained by Arachni 1.0.2which detected 100% of the SQL test cases and 87.67% of

the XSS test cases. After scanning a sample of 140 websitesusing Arachni 0.4.3 andWapiti 2.3.0 [23], we found that bothscanners indicated that at least 21% of the assessed websiteshad high-severity security vulnerabilities. Arachni 0.4.3 andWapiti 2.3.0 also found that 12.86% of the evaluated websiteswere vulnerable to SQL attacks whereas 11.43% of them hadXSS vulnerabilities. Given that the results of our comparativeevaluation did not show significant performance differences(e.g., scanning speed) among the scanners while the results ofthe conducted case study revealed high level of disagreementbetween the reports generated by different scanners, weconclude that the inconsistencies between the reports gen-erated by different scanners might not necessarily correlatewith their performance properties. Further research is there-fore needed to investigate the factors that could degrade theeffectiveness of web vulnerability detection scanners and thuslead them to report inconsistent results.

Contributions. We frame the contributions of our work asfollows.

(1) Comparative Evaluation of Open Source Web Vulner-ability Scanners. After reviewing currently availableopen source web vulnerability scanners, a compara-tive evaluation of the security features as well as theperformance of four web vulnerability detection toolsis conducted. Two versions of Arachni scanner as wellasWapiti and Skipfish scanners are evaluated in termsof speed, crawler coverage, detection accuracy, andextra supported features.The conducted evaluation isbased on rigorous performance indicators as well asstandard security benchmarks.

(2) Case Study on the Security State ofWeb-Based Systems.As a second step for evaluating the performanceof web vulnerability detection tools, a comprehen-sive web vulnerability assessment on 140 web-basedapplications is conducted. Two selected tools wereutilized to identify the types, quantities, and degreesof severity of security vulnerabilities that exist in theevaluated web content. The results reported by thesescanners are then analyzed, aggregated, compared,and linked to the results obtained from our scanners’comparative evaluation experiment to clearly under-stand the detection accuracy of these tools in variousscenarios.

(3) Design Guidelines. We present some recommenda-tions for helping developers of web vulnerabilitiesscanners to improve their tools’ capabilities.

Organization. We first describe the steps followed to review,compare, and select open source web vulnerability scannersin Section 2. We then present the results of the conductedcomparative evaluation in Section 3. In Section 4, we use twoweb vulnerability scanners to detect the security vulnerabil-ities in a large collected dataset. Further discussion, designguidelines for tool developers, and the related literatureare then presented in Section 5, Section 6, and Section 7,respectively. We present the limitations and future work inSection 8 and conclude in Section 9.

Security and Communication Networks 3

(1) Surveying OS webvulnerability scanners

(2) Testing OS webvulnerability scanners

(3) Selecting the mostappropriate scanners

(4) Comparing theperformances and features

of the selected scanners

Figure 1: Web vulnerability scanners’ comparison: research approach.

2. Web Vulnerability Scanners:Evaluation Approach

We followed a structured research methodology that focuseson addressing the problem from multiple perspectives. Westarted by reviewing some popular open source web vulnera-bility scanning tools based on a predefined criteria. We thenrandomly chose a set of web-based applications and scannedthemusing some of the surveyed open source scanners.Threeof themwere then selected to be included in our detailed opensource web vulnerability scanners comparative evaluation.Figure 1 summarizes theweb vulnerability scanners and secu-rity benchmarks utilized throughout each phase of our eval-uation approach. The following subsections further explainthe decisions taken throughout the phases of the comparativeassessment.

2.1. Web Vulnerability Scanners Selection. The first phaseof our study involved conducting an initial survey of themost popular open sourceweb vulnerability assessment tools.Six scanners were evaluated against predefined criteria thatconsider the performance properties of the scanners aswell astheir crawler coverage and types of web vulnerabilities theycan identify (see Table 1). The open source web vulnerabilityassessment scanners that were included in our survey areSkipfish, Arachni, Wapiti, IronWASP, Vega [24], and w3af[25].

For each scanner, we investigated the provided securityscanning and visualization features, the supported operatingsystems, and the scanning scope. In order to evaluate thesescanners and include the most effective ones in our compar-ative evaluation, we chose the best four scanners and testedthemon some randomly selectedwebsites.These scanners areWapiti, Arachni, Skipfish, and IronWASP. In the testing phase,Wapiti, Arachni, and Skipfish were installed on Ubuntu 12.04whereas IronWASP [26] was installed on Windows 8.

Based on the scanning results obtained in the testingphase, two web vulnerability scanners were chosen to beincluded in the conducted comparative assessment: Wapiti2.3.0 and Arachni 0.4.3. In fact, after comparing the resultsreported by the four scanners, we found inconsistencies

Table 1: Web vulnerability scanners’ comparison: scanners’ selec-tion and evaluation criteria.

Scanners’ selection criteriaScanning speedVisualization featuresScanning scopeExport file formatsSupported operating systemsConsistency with other scannersSupported programming languagesAvailability of web-based GUI

Scanners’ evaluation criteriaPerformance

Quantitative measuresTrue positive rate (TPR)True negative rate (TNR)False positive rate (FPR)False negative rate (FNR)Positive predictive values (PPVs)Negative predictive values (NPVs)False omission rate (FOR)Accuracy𝐹-measure

Scanning speedCrawler coverageVulnerability detection accuracy

FeaturesVisualization featuresReporting featuresEase of configurationTypes of vulnerabilities that can be detected

between the types and quantities of security vulnerabilitiesreported by IronWASP and the other three scanners. Giventhis and the limited reporting features of IronWASP, we didexclude IronWASP scanner. Skipfish was also not included inthe security evaluation conducted in our case study. However,

4 Security and Communication Networks

Table 2: The speed and crawler depth results obtained after scanning the Web Scanner Test Site and the two Acunetix sites.

Scanner Version webscantest(HH:MM:SS)

testphp(HH:MM:SS)

testaspnet(HH:MM:SS) Rank (speed) Crawler coverage Rank (crawler coverage)

Arachni 0.4.3 01:23:22 00:22:10 01:05:33 2 19% 4Arachni 1.0.2 12:11:13 00:41:34 01:00:00 3 94% 1Wapiti 2.3.0 06:20:00 01:30:00 03:30:00 4 44% 3Skipfish 2.1 00:11:06 00:27:40 00:11:30 1 48% 2

we included Skipfish in our comparative evaluation of webvulnerability detection tools in order to compare its speedwith those of Wapiti and Arachni scanners.

2.2. Web Vulnerability Scanners Comparison. As a prestep toevaluating the security of a sample of 140 distinct web-basedapplications, we conducted a comparative evaluation of fourweb vulnerability detection tools which are Arachni 0.4.3,Arachni 1.0.2, Wapiti 2.3.0, and Skipfish 2.10. The aim of thisevaluation was to assess the effectiveness of these scannersand decide whether their reported results will accuratelyreflect the security state of the websites examined in our casestudy. We evaluated the scanning speed of each tool, theircrawler depth, and their vulnerability detection accuracy.Our evaluation criteria also considered the extra featuressupported by each scanner (e.g., reporting and visualizationservices) and the types of web vulnerabilities that can be dis-covered by each scanner. Table 1 summarizes the criteria thatwere defined to evaluate the chosen web scanners, whichcovers the essential performance characteristics that allowedus to base our evaluation on rigorous standards and bench-marks.

Three vulnerability test websites were considered forevaluating the speed of each scanner: Web Scanner Test Site[27] and twoAcunetix [28] sites.These test websiteswere fullyscanned using the selected four web vulnerability scanners.We then compared the scanning times of the scanners andranked them accordingly. Similarly, Altoro Mutual [29] andWAVSEP [22] were utilized for assessing the accuracy of thescanners.We also used theWeb InputVector Extractor Teaser(WIVET) [30] benchmarking project to examine the crawl-ing capabilities of the evaluated scanners. To assess theaccuracy of each scanner, we calculated (1) the true positiverates (TPRs); (2) the true negative rates (TNRs); (3) the falsepositive rates (FPRs); (4) the false negative rates (FNRs); (5)the positive predictive values (PPVs); (6) the negative pre-dictive values (NPVs); (7) and the false omission rates(FORs) [31, 32].We alsomeasured the vulnerability detectionaccuracy of the evaluated scanners and their associated 𝑓-measures.

3. Scanners’ Comparative Evaluation:Results and Analysis

After using Arachni 0.4.3, Arachni 1.0.2, Wapiti 2.3.0, andSkipfish 2.10 to scan the chosen test websites, we observed thescanning behaviors of the evaluated scanners and quantita-tively compared their speed, detection accuracy, and crawler

coverage. We then ranked these scanners accordingly andchose Arachni 0.4.3 and Wapiti 2.3.0 to be included in ourcase study (as explained in Section 4). In this section, wepresent and discuss the results of our web vulnerabilityscanners comparative assessment.

In general, our results show that the speed of the evaluatedscanners was relatively acceptable. However, in terms ofcrawler coverage, the obtained WIVET scores for the fourscanners show that 75% of them covered less than 50% ofthe crawled web-based systems. By considering the detectionaccuracy of the evaluated scanners, we found variationsbetween the scanning results obtained from Altoro Murualand WAVSEP. For example, all the scanners detected at least77% of WAVSEP’s SQL test cases whereas 75% of themdetected at most 38% of AltoroMutual’s SQL test cases. How-ever, this might be due to differences between the securityproperties examined by WAVSEP’s and Altoro Mutual’s testcases. We also observed variations in number and types ofweb vulnerabilities that each scanner can detect. However,all the chosen web scanners were able to discover basicsecurity defects including database injection attacks, XSSvulnerabilities, and misconfiguration attacks. Further, nodifficulties were faced in understanding the functionality ofthe evaluated scanners, as all of them provide sufficient doc-umentation in their corresponding websites. The followingsubsections detail the results of the conducted comparativeassessment.

3.1. Speed. After scanning three test sites and recording theelapsed scanning times (see Section 2.2), we observed thatSkipfish took at most 28 minutes to scan a test websitewhereas Wapiti took at least an hour and a half to performa full vulnerability scan. Similarly, the vulnerability assess-ments conducted using Arachni scanner consumed at least23 minutes. Thus, Skipfish is considered the fastest scannerwhereas Wapiti is the slowest one in detecting web-basedvulnerabilities. Table 2 shows the scanning times for eachevaluated scanner.

3.2. Crawler Coverage. We based our evaluation of thecrawler coverage of the evaluated scanners onWIVET scores.The best crawling coverage percentage was 94% which wasobtained by Arachni 1.0.2. The crawlers of the remainingthree scanners covered at most 48% of the scanned webapplications. As shown in Table 2, WIVET results show thatSkipfish 2.1, Wapiti 2.3.0, and Arachni 0.4.3 covered 48%,44%, and 19% of the crawled web applications, respectively.

By linking the crawling coverage with the speed of theevaluated scanners, note that the excellent crawling ability of

Security and Communication Networks 5

Table 3: WAVSEP and Altoro Mutual: test cases detected by each scanner.

Scanner Version WAVSEP test cases IBM AppScan Test Site (Altoro Mutual) test casesSQL (135) SQL FP (10) XSS (73) XSS FP (7) SQL (13) Blind SQL (1) XSS (10)

Arachni 0.4.3 134 2 47 0 9 1 8Arachni 1.0.2 135 0 64 0 2 0 4Wapiti 2.3.0 131 2 45 3 5 0 7Skipfish 2.1 104 0 63 0 1 0 3

Table 4: Vulnerability scanners’ comparative assessment: extra features.

Scanner Version Web-basedInterface Features Live

feedbackDetailedreports

Userfriendliness Configuration

Arachni 0.4.3 Yes More than 10 active and 20 passive checks Yes Yes Easy FlexibleArachni 1.0.2 Yes More than 10 active and 20 passive checks Yes Yes Easy FlexibleWapiti 2.3.0 No 10 modules for 9 vulnerability types Yes Yes Easy BasicSkipfish 2.1 No Different scanning modes and dictionaries No Yes Easy Basic

Arachni 1.0.2 could had negatively affected its speed. Aftercomparing the crawler coverage of Arachni 0.4.3 andArachni1.0.2, we noted that the newer release was slower due to thesignificant improvement of its crawler. However, this does notnecessarily imply that the advanced crawling capability of agiven web vulnerability scanner is correlated with its poorperformance. For example, although Skipfish was the fastestscanner, the results show that the coverage of its crawler wasbetter than the coverage of both Wapiti 2.3.0 and Arachni0.4.3.

3.3. Accuracy. We scanned IBM’s AppScan test site (AltoroMutual) to evaluate the level of accuracy of the scanningresults reported by the chosen tools. We also used 225 ofthe test cases included in the Web Application VulnerabilityScanner Evaluation Project (WAVSEP) to clearly understandthe accuracy weaknesses of the four evaluated scanners (seeTable 3). Altoro Mutual site helped us to test our scannersagainst 13 SQL test cases, one blind SQL test case and tenXSS test cases. Similarly, we scanned the cases presented byWAVSEP benchmark which included 135 SQL true positivecases, 10 SQL false positive cases, 73 XSS true positive cases,and 7XSS false positive cases. Based onWAVSEP benchmark,our results show that the four scanners were able to detect atleast 104 SQL injection true positives and 45 XSS true positivecases. However, the majority of Altoro Mutual’s SQL caseswere not detected byWapiti 2.0.3, Arachni 1.0.2, and Skipfish2.1. Similarly, all the evaluated scanners were able to report atleast 62% of WAVSEP’s XSS true positive test cases whereas50% of the evaluated scanners detected at most 40% AltoroMutual XSS vulnerabilities. ForWAVSEP’s SQL and XSS falsepositive test cases, the four scanners reported at most 20%SQL cases and 43% XSS cases. Table 3 details the number ofcases detected by each scanner and Section 5 discusses theseobservations.

3.4. Additional Features. To comprehensively evaluate thechosen web vulnerability scanners, we manually experi-mentedwith all the features offered by these tools and utilized

the extra information documented in their correspondingwebsites. While we did not conduct a user study, we didnot observe any notable problems with the graphical userinterfaces of these scanners as we found them user friendly,easy to use, and understandable. Further, the tools providedlive feedback which helped us to easily identify the vulnerableURLs and understand the types of detected vulnerabilitiesand their degrees of severity. Furthermore, we were satisfiedwith the vulnerability reporting features of these tools as allof them provided reports that included detailed statistics andexplanations about the detected security vulnerabilities. Fur-ther, Arachni vulnerability assessment scanners offer extrafeatures that allowed us to identify redundant URLs, classifythe scanned URLs according to their security levels, andmodify the crawling scope to include or exclude some webpages.

The four scanners varied in the types and number of secu-rity checks that can be performed. For example, Arachni 0.4.3and Arachni 1.0.2 havemany plugins that can be used for per-forming more than 30 active and passive vulnerability checksfor a wide range of web-based vulnerabilities. Similarly,nine different types of web vulnerabilities can be checkedusing Wapiti 2.3.0 scanner including SQL and XSS injectionattacks, file disclosure attacks, and XPath attacks. In addition,Skipfish has an extra feature that helps in improving thecrawling coverage and including unlinked web pages in thevulnerability scanning process. This feature performs brute-force based security checks and allows specifying the crawlingscope as minimal, medium, or complete. Table 4 summarizessome of our scanners’ additional features.

3.5. Quantitative Measures. Based on WAVSEP test cases,nine metrics were calculated to review the effectiveness ofthe four scanners in detecting SQL and XSS attacks. Foreach scanner, the true positive and true negative rates weremeasured to determine whether the results reported by theevaluated scanners truly reflect the security state of thescanned websites. We also measured the rates of false pos-itives and false negatives reported by the scanners as well as

6 Security and Communication Networks

Table 5: Scanners’ detection accuracy of SQL attacks.

Scanner Version TPR TNR FPR FNR PPV NPV FOR Accuracy 𝐹-measureArachni 0.4.3 99.26% 80% 20% 0.74% 98.53% 88.89% 11.11% 97.93% 98.89%Arachni 1.0.2 100% 100% 0% 0% 100% 100% 0% 100% 100%Wapiti 2.3.0 97.04% 80% 20% 2.96% 98.50% 66.67% 33.33% 95.86% 97.76%Skipfish 2.1 77.04% 100% 0% 22.96% 100% 24.39% 75.61% 78.62% 87.03%

Table 6: Scanners’ detection accuracy of XSS attacks.

Scanner Version TPR TNR FPR FNR PPV NPV FOR Accuracy 𝐹-measureArachni 0.4.3 64.38% 100% 0% 35.62% 100% 21.21% 78.79% 67.50% 78.33%Arachni 1.0.2 87.67% 100% 0% 12.33% 100% 43.75% 56.25% 88.75% 93.44%Wapiti 2.3.0 61.64% 57.14% 42.86% 38.36% 93.75% 12.50% 87.50% 61.25% 74.38%Skipfish 2.1 86.30% 100% 0% 13.70% 100% 41.18% 58.82% 87.50% 92.65%

Arachni 0.4.3 Arachni 1.0.2 Wapiti 2.3.0 Skipfish 2.1Web vulnerability scanners

SQL accuracySQL F-measure

TPRFPR

0

50

100

Perc

enta

ge

Figure 2: Scanners’ performance: detection of SQL vulnerabilities.

the positive predictive and negative predictive values for eachscanner. To clearly understand the detection accuracy of SQLand XSS attacks, the rate of true positives and true negativesto the total number of test cases and the 𝑓-measures werecalculated for each scanner (see Section 2.2 and Table 1). Theresults of our analysis are presented in Tables 5 and 6.

Based on the calculated 𝑓-measures, the detection accu-racy of SQL vulnerabilities was over 97% in Arachni 0.4.3and Arachni 1.0.2 (see Figure 2). Although Skipfish’s SQLdetection accuracy was the lowest, it has a false positive rateof 0%, a false negative rate of 23%, and a true negative rateof 100%. Furthermore, by considering the accuracy of eachtest which is measured through dividing the sum of truepositives and true negatives by the total number of test cases,the rates of Arachni and Wapiti scanners were higher than95%. For SQL vulnerability detection, the accuracy measuresof Arachni 1.0.2 were the best relative to the remainingthree scanners. Further, Skipfish has a true positive rateof 77%, which was the minimum observed TPR. For SQL

attacks, the positive predictive values of all the scannerswere very high ranging from 98.50% to 100%. Also, the falsenegative rates of three of the evaluated scanners were lessthan 3%. After ranking the four scanners according to theirSQL vulnerability detection rates, Arachni 1.0.2 took the firstposition whereas Arachni 0.4.3, Wapiti 2.3.0, and Skipfish 2.1took the second, third, and fourth positions, respectively.

The scanners’ performance related to their effectivenessin detecting XSS vulnerabilities was also evaluated. AlthoughSkipfish’s accuracy rates for detecting SQL vulnerabilitieswere the lowest (see Figure 2), our results show that itsdetection rate for XSS vulnerabilities was very high incomparison with the values recorded for Wapiti 2.3.0 andArachni 0.4.3. For all the evaluated scanners, we also recordeda false positive rate of 0% except for Wapiti which had aFPR of 42.86%. Similarly, for all the four scanners exceptWapiti, we also obtained true negative rates and positivepredictive values of 100%. The lowest true positive rates,𝑓-measures, and accuracy values related to detecting XSSvulnerabilities were recorded for Wapiti 2.3.0 and Arachni0.4.3 (see Figure 3). While the highest FNR related toSQL vulnerability detection was recorded for Skipfish (seeTable 5), the number of XSS false negatives recorded forSkipfish was lower than the ones calculated for Arachni0.4.3 and Wapiti 2.3.0. Similar to our observations for SQLvulnerability detection, the best accuracy measures relatedto the detection of XSS attacks were the ones recorded forArachni 1.0.2. However, in contrast to our observations forSQL attacks, the results show that Skipfish 2.1 was moreaccurate thanWapiti 2.3.0 and Arachni 0.4.3 in detecting XSSvulnerabilities.

We observed that the newer version of Arachni was themost effective one in detecting SQL and XSS vulnerabilities.As presented in Tables 5 and 6, Arachni 1.0.2 has the highesttrue positive rate, true negative rate, accuracy value, and 𝑓-measure. In most of the cases, we did not note significantdifferences in performance among all the evaluated scanners.Generally, the overall performance of the four scanners wasgood as the lowest obtained 𝑓-measure was 74.38% andthe highest FNR was 38.36%. Further, the recorded truepositive rates for most of the evaluated scanners are above

Security and Communication Networks 7

Arachni 0.4.3 Arachni 1.0.2 Wapiti 2.3.0 Skipfish 2.1Web vulnerability scanners

XSS accuracyXSS F-measure

TPRFPR

0

50

100

Perc

enta

ge

Figure 3: Scanners’ performance: detection of XSS vulnerabilities.

86% whereas the worst FPR is 42.86%. Based on the resultsobtained after scanning IBM AppScan test site (see Table 3),the highest number of detected test cases was scored byArachni. During our evaluation, scanning IBM AppScanusing Arachni 1.0.2 unexpectedly stopped at the end ofthe experiment. The results shown in Table 3 are thereforeaccording to the number of vulnerabilities detected beforeexperiencing this technical problem. We also note that it waschallenging to determine if the calculated false negative rateswere actual false negatives or resulted from some crawlercoverage issues.

4. Case Study: Evaluating the Security State ofPart of the Web Content

We followed our performance evaluation of the scanners (seeSection 3) with an evaluation of the overall security state ofpart of the web content as a case study. Two scanners werechosen for this purpose:Wapiti 2.3.0 and Arachni 0.4.3. First,we collected a sample of 140 unique URLs to popular web-based systems and used the chosen scanners to assess thesecurity state of the websites in question (the collected URLdataset is available in [33]). Based on the quantitative analysisof the security scan reports generated by Wapiti 2.3.0 andArachni 0.4.3, a security holistic view of part of the web con-tent was then analyzed. In the following sections, we describethe conducted security scans, analyze the generated scanreports, and compare the results obtained from the twochosen scanners.

4.1. Dataset andMethodology. In this section, we describe themethodology followed to collect, scan, and analyze a sampleof unique 140 URLs. Figure 4 summarizes the steps followedin our experiment.The following sections detail the decisionstaken throughout each step in this case study and present theresults of the conducted security vulnerabilities assessment.

4.1.1. Dataset Collection. One of the main objectives of thiscase study is to evaluate the overall security state of a part ofthe web content. The sample of the examined URLs includesdifferent websites that vary in size, complexity, and topics.We used Google Trends [34] to select trending search termsfrom the period of 2004 to 2013. This was followed byperforming Google searches on the trending keywords andusing SEOquake [35] Firefox extension to harvest the firstthirty URLs obtained from each search (the collected URLdataset is available in [33]). Figure 5 describes the followedsteps to collect our dataset.

4.1.2. Methodology. To scan the collected URLs, two virtualmachines were installed on Amazon EC2 [36] cloud-basedservice. The two instances were accessed using PuTTY [37]client andTightVNC [38] remote desktop controller.Thiswasfollowed by installing Wapiti and Arachni web vulnerabilityscanners on the cloud instances and running the securityvulnerabilities scans for each URL in our dataset. Note thatArachni was utilized to detect a wider range of securitydefects relative to Wapiti which was restricted to detectingSQL injection and XSS attacks.

We then analyzed and compared the results generated byeach scanner to accurately understand the overall securitystate of part of the web content. Our analysis is basedon the overall performance of each scanner as discussedin Section 3. For each website, we recorded the types ofdetected vulnerabilities by each scanner and their degree ofseverity.We were therefore able to identify themost commonvulnerabilities in the examined URLs, rank them accordingto their degree of severity, and then obtain an idea about thegeneral security state of the evaluated URLs.

4.2. Detected Vulnerabilities. The findings of the conductedexperiments and the level of agreement between the webvulnerability scanners evaluated in this study are discussedin this section.

4.2.1. Arachni. After scanning 140 unique URLs usingArachni, we found that 98% of them had security vulnerabili-ties. 60%of the vulnerable web servers included high-severityvulnerabilities and 96% had informational vulnerabilities.Arachni reports also show that 72% of the vulnerable webservers included low-severity vulnerabilities and 65% hadmedium-severity vulnerabilities.

Based on Arachni results, our sample had seven high-severity, four medium-severity, three low-severity, and threeinformational web vulnerability types. Table 7 summarizesthe types, numbers, and severity ratings of the web vulnera-bilities that were detected by Arachni and ranks them accord-ing to their frequency of occurrence. According to Arachni,around 64% of the web server with high-severity vulnerabili-ties were also vulnerable to cross-site request forgery (CSRF)attack. Further, cross-site scripting vulnerabilities (XSS) werethe second most frequently observed security issues in oursample (36% of the scanned URLs that had high-severityvulnerabilities). Arachni also reported that 15% of the samplewere vulnerable to SQL injection attacks. Also, 4% of theevaluated websites had backdoor files and file inclusion

8 Security and Communication Networks

Create 2 AmazonEC2 instances

Install the scannerson the cloud VMs Analyze scan reports Compare the results of

Wapiti and ArachniScan each URL in

the collected dataset

Figure 4: Arabic websites security evaluation: the followed approach.

Harvest 40 URLs per keyword

Result: 40 URLs per file Collect 10 CSV files

Search for 10 trending keywords

2004-2013 SEOquakeAdd-on

Figure 5: Collecting the sample URLs.

Table 7: Arachni’s reported vulnerabilities: a summary.

Severity Vulnerability # of vulnerable URLs

High

CSRF 53XSS 30SQL 21

Source code Disclosure 6Backdoor File 5File Inclusion 5

X-Forwarded-For 1

Medium

Common directory 68Unencrypted passwords 43

Backup files 9Unvalidated redirect 2

LowCommon sensitive files 77Password autocomplete 42

Directory listing 4

InformationalInteresting response 123Insecure cookie 90Email disclosure 45

vulnerabilities. Source code disclosure vulnerabilities werealso observed with a percentage of 7% of our URLs that hadhigh-severity vulnerabilities. The most spread high-severityvulnerabilities in the evaluated web-based applications wereCSRF, XSS, and SQL injection attacks.

As presented in Table 7, 90 of the scanned webservers were vulnerable to four types of medium-severity

vulnerabilities. Among medium-severe vulnerabilities, atleast 75% were vulnerable to the attacks related to the useof common directories on web servers. Arachni’s resultsalso showed that at least 30% of our evaluated websitesdid not implement proper encryption mechanisms (e.g.,users’ credentials were sent without encryption). Arachnialso reported that 6% had stored their backup files inthe same web servers. Furthermore, unvalidated redirectsecurity defects were observed in 2% of the scanned URLswith medium-severity vulnerabilities. In short, 65% of theevaluated websites had medium-severity vulnerabilities thatcould have been caused by incorrect software developmentpractices.

Three types of low-severity web vulnerabilities weredetected by Arachni in at least 71% of the evaluated URLs.In at least 77% of these vulnerable web servers, Arachni wasable to locate some files (e.g., robot.txt, sitemap.xml, andconfig.php) that might reveal sensitive information about theweb servers in question. For instance, attackers might usesuch files to view the names of some hidden directories ordisclose information about the configurations of the respec-tive servers. In 30% of the sample, Arachni was also able tofind some form fields with password autocompletion optionenabled. Similarly, Arachni reports show that a directorylisting option was permitted in 4% of the URLs that includedlow-severity vulnerabilities.

Informational vulnerabilities were found in 95% of oursample in which more than 92% were related to receiving“Not Found” error message (i.e., 404 HTTP responses) fromthe requested web servers. Arachni also reported that over67% of the evaluated URLs had informational vulnerabilities

Security and Communication Networks 9

Table 8: Arachni: highest number of occurrences of each vulnera-bility type.

Vulnerability Severity Highest numberCommon directory Medium 54XSS High 33SQL High 29Backup file Medium 27CSRF High 26PW autocomplete Low 25Interesting response Informational 25Backdoor file High 24Source code disclosure High 18File inclusion High 10Common sensitive files Low 10Email disclosure Informational 8Insecure cookie Informational 7Unencrypted password Medium 5Directory listing Low 1X-Forwarded-For High 1

related to sending data using unencrypted cookies. In 32%of the collected URLs, Arachni was also able to discoversome email disclosure vulnerabilities in which some emailaddresses that werewritten in some scripts or code commentswere accessible by the scanner.

This experiment enabled us to identify the most spreadweb vulnerabilities and rank them accordingly. Withoutconsidering the severity degrees of the detected vulnera-bilities, our results show that the average number of webvulnerabilities found in one web server was 21. For eachweb vulnerability type, we recorded the highest number ofvulnerabilities found in one web server. For example, thehighest number of SQL injection, XSS, and CSRF vulnera-bilities found in a website was 29, 33, and 26, respectively.For each vulnerability type, Table 8 ranks the number ofweb vulnerabilities detected in one web server based on theirfrequency of occurrence.

4.2.2. Wapiti. We used Wapiti to detect two types of webvulnerabilities: SQL injection and XSS attacks (see Figure 7for a sample scan result). Based on Wapiti results, 86 outof our collected 140 URLs had either SQL injection orXSS vulnerabilities. Around 92% of the 86 vulnerable webservers had high-severity SQL injection or XSS vulnerabil-ities. Wapiti also reported that at least 19% of our collectedsample was vulnerable to low-severity risks. Wapiti’s scanresults also indicated that at least 81% of the URLs that hadhigh-severity vulnerabilities were vulnerable to SQL injectionattacks. Similarly, Wapiti detected high-severity cross-sidescripting vulnerabilities in 28% of the evaluated websites.Thehighest numbers of SQL injection and cross-side scriptingvulnerabilities discovered in one web-based application were600 and 234, respectively. Additionally, Wapiti found thatthe highest number of HTTP internal server error responsesreceived from one web server is 759. Table 9 summarizesthe number of URLs that were reported as vulnerable to

Table 9: Wapiti: detection of SQL and XSS vulnerabilities.

Vulnerability Number of vulnerable URLs Highest numberXSS 64 600SQL 39 234

1630

XSSWapitiArachni

6421 18

SQL injectionWapitiArachni

39

Figure 6: Wapiti versus Arachni: detection of vulnerable URLs.

Figure 7: Wapiti web vulnerability scanner: sample scan result.

high-severity XSS or SQL attacks as well as the highestnumbers of SQL or XSS vulnerabilities that were found in oneweb server.

4.2.3. Arachni and Wapiti: A Comparison. After scanningthe same sample using Arachni and Wapiti, we comparedtheir results and quantified the number of URLs that werereported by the two scanners as vulnerable to SQL injectionor XSS attacks. Based on the reports generated by Arachni,15% of our collected sample was vulnerable to SQL injectionattacks. However, the number of URLs that were reported asvulnerable to SQL injection attacks by Wapiti scanner wasat least two times higher than Arachni’s one. According toWapiti, around 46% had SQL injection vulnerabilities. Bothscanners indicated that 12.86% of the evaluated sample wasvulnerable to SQL injection attacks. On the other hand,the findings of our comparison also showed that 11.43% ofthe scanned URLs were reported as vulnerable to cross-sidescripting attacks by both scanners. For XSS vulnerabilities,there was a little difference between those reported positiveby the two scanners (i.e., Wapiti reported 27.86% relative to21.43% reported by Arachni’s). Both Wapiti and Arachniindicated that at least 12.86%were vulnerable to SQL injectionattacks whereas 11.43% had XSS vulnerabilities (see Figure 6).

10 Security and Communication Networks

5. Further Discussion

We combined the results of our first experiment, whichcompared the performances of four scanners based on objec-tive measures, with the findings of the second experimentwhich evaluated the performance of two scanners on alarge set of selected URLs. Before conducting our casestudy and based on the results obtained from our scanners’comparative evaluation, we expected Arachni to report lessnumber of security vulnerabilities compared toWapiti, as thecrawler coverage ofArachniwas 19%whereasWapiti’s crawlercoverage was 44%. In our case study, we observed that thenumber of SQL and XSS vulnerabilities reported by Wapitiwas higher than the results reported by Arachni. As shownin Table 7, Arachni’s reports showed that 15% of our samplewas vulnerable to high-severity SQL attacks and 21% of thecollected URLs were vulnerable to high-severity XSS attacks.According to Wapiti, around 46% of the URLs examinedin our case study were vulnerable to SQL attacks and XSSvulnerabilities existed in 28% of our sample. However, thismight not necessarily correlate withWapiti’s crawler coveragecapabilities as our observations also showed that Wapiti hadthe highest FPR for XSS and SQL attacks (as shown in Tables5 and 6). Additionally, the results showed that Arachni’svulnerability detection rates were higher than Wapiti’s ones.Hence, because Arachni’s crawler coverage was the lowest,there is a possibility thatmany web pages were not reached byArachni’s crawler.

Although there were little differences in 𝑓-measures anddetection accuracy rates for SQL and XSS vulnerabilitiesbetween Wapiti and Arachni, we observed that Wapiti hadhigher false negative rates and lower true negative rates forboth SQL andXSS attacks (see Tables 5 and 6).Thus, althoughWapiti’s crawler coverage rate was higher than Arachni’sone, there is a possibility that Wapiti had not detected somevulnerable URLs that were examined in the conducted casestudy. For XSS attacks, we also observed thatWapiti 2.3.0 andArachni 0.4.3 had the lowest true positive rates compared toSkipfish 2.1 and Arachni 1.0.2. The percentage of URLs thatwere vulnerable to XSS attacksmight therefore be higher thanthose reported by the two scanners included in our case study.This finding might also be supported by the number ofWAVSEP XSS true positive test cases that were detected byArachni 0.4.3 and Wapiti 2.3.0 (see Table 3). Based on thecomparative evaluation, both scanners had detected at most47 XSS cases out of WAVSEP’s 73 true positive XSS testcases. Further, although both scanners had high true positiverates for SQL vulnerabilities, Wapiti had detected only 5 outof 13 Altoro Mutual’s SQL true positive cases whereas thenumber of cases detected byArachni 0.4.3 was 9 (see Table 3).Therefore, the number of URLs that were vulnerable to SQLattacks might also be higher than the numbers reported byWapiti and Arachni.

We observed notable differences between the scan resultsof Wapiti and Arachni. Although Arachni’s reports showedthat 21 of the examined web servers had SQL vulnerabilities,Wapiti’s scan results indicated that 64 of our 140 URLs werevulnerable to SQL attacks. However, only 18 URLs werereported as vulnerable to SQL attacks by the two scanners. For

XSS vulnerabilities, we note that the two scanners indicatedthat 16 web servers were vulnerable although each scannerflagged at least 30 URLs as vulnerable to XSS attacks (seeFigure 6). Although the results of our scanners’ comparativeassessment did not reveal notable differences between theperformances of the four evaluated scanners, there was ahigh level of disagreement between the scan results of Wapitiand Arachni. This suggests that web vulnerability detectionscanners have limitations that might be related to their crawl-ing capabilities, detection accuracy, or other implementationdetails that need to be researched. Our observations alsoshow that the slowperformance of the studied scannersmightnot necessarily imply that the respective scanner has a goodcrawling coverage. We observed that Wapiti’s long scanningtime resulted from scanning web pages containing similar orduplicate content. This means that the there was a consider-able increase in the scanning time that resulted from sendinga high number of unnecessary HTTP requests and repeatedlyfilling many form elements without guaranteeing the cover-age of all web pages belonging to the websites in question.

By observing the highest numbers of XSS and SQL injec-tion vulnerabilities that existed in one website, we also notevariations between the scan results reported by Arachni andWapiti. Although Arachni’s reports showed that the highestnumber of SQL injection vulnerabilities that were found inone website was 29 (as shown in Table 8),Wapiti’s scan resultsindicated that 600 SQL vulnerabilities were detected in onewebsite. Similarly, Wapiti found 234 XSS vulnerabilities inone website whereas the highest number of XSS vulnerabil-ities detected in one website by Arachni was 33. There aretherefore considerable differences between the scan results ofWapiti and Arachni although our quantitative comparativeperformance assessment did not reveal notable differencesbetween the detection accuracy rates of these scanners. Thisvariation can be explained either by the crawler coveragewhich is presented in Table 2 or by the false alarm ratesof Wapiti and Arachni.

Our comparative assessment results showed that Wapiti’sobserved false alarm rate for XSS vulnerabilities was 42.86%whereas no false positives were recorded for Arachni 0.4.3(see Table 6). For SQL injection vulnerabilities, our obser-vations also showed that a false positive rate of 20% wasobserved for the two scanners; however, our case studyshowed that there was a notable difference between thenumbers of SQL vulnerabilities detected by the two scanners.Because Arachni’s crawler coverage percentage was poor,there is a possibility that Arachni’s crawler did not scan allthe pages that were scanned by Wapiti. However, this doesnot imply that Wapiti’s results were more accurate since weobserved that Wapiti had repeatedly scanned a considerablenumber of URLs containing similar web content.

6. Guidelines and Recommendations

Our experimental findings highlighted limitations related toscanners’ crawling capabilities, reporting and visualizationfeatures, efficiency, and vulnerability detection accuracy. Inthis section, we note some areas that should be addressed bydevelopers in order to increase the effectiveness of their tools.

Security and Communication Networks 11

We recommend system designers to utilize the availablesecurity benchmarks and test sites (e.g., the ones shown inFigure 1) to evaluate the performance of their open sourcevulnerability detection solutions. This could contribute toimproving the capabilities of open source scanners as it wouldallow comparing the quantities and types of vulnerabilitiesthat were detected by open source scanners with thosediscovered by well-known proprietary scanners (e.g., IBMSecurity AppScan [8]). We also recommend tool developersto improve the crawling capabilities of their tools by ensuringthat all the clickable elements in a crawled web page are iden-tified and thus all the web pages that belong to the examineddomains are traversed. For increasing the coverage of attackclasses checked by web vulnerability scanners, tool develop-ers could also enhance the comprehensiveness of their tests byvalidating all possible combinations of inputs to web formswhile ensuring that the efficiency of their tools is not com-promised. The elimination of duplicate links before startingthe vulnerability scanning process is also recommended forimproving the efficiency of vulnerability detection.

Designers of open source web vulnerability scanners arealso recommended to improve the accuracy of the securityverification checks incorporated in their tools to simplifythe task of manually checking whether the vulnerabilitiesreported by these scanners are false positives or could actuallybe exploited by attackers [39]. That is, reporting web pagesas vulnerable to exploitable attacks when in fact they donot exist increases the effort spent by penetration testers onvalidating the accuracy of the results reported by these tools.For instance, web vulnerability scanners might sometimesreport security vulnerabilities related to having cookies thatare not flagged as secure when in fact they store nonsensitivedata (e.g., user preferences) that is of no value to attackers.Platform designers could therefore address this problemby identifying the characteristics of nonsensitive cookies toavoid reporting them as potential security vulnerabilities.Web vulnerability scanners might also sometimes report vul-nerabilities relating to locating server files without verifyingwhether the content of these files could actually disclosehidden information about the configurations of the webservers in question or not.We also recommend improving thereportingmechanisms implemented in open source scannersby presenting the user with more details about detected vul-nerabilities and highlighting potential false positives. Reportsgenerated by penetration testing tools should also includemore guidance to draw the testers’ attention to the factors thatcould characterize false positives and provide remediationadvice onhow to address detected vulnerabilities. Penetrationtesters should also be made aware of the circumstances thatcould cause a scanner to not detect certain vulnerabilities ormisinterpret the observed behavior of the scanned websiteand report a vulnerability that does not exist.

From a usability viewpoint, tool designers are recom-mended to implement web-based interfaces of their opensource scanners to allow web developers to access these scan-ners and run their scans easily.They are also recommended toimprove the vulnerability reporting and visualization featuresof these tools to ensure that users are able to monitor thescanning processes in real time.Users should also be provided

with various options allowing them to customize and controlthe scope of their scans [40].

With the emergence of new client and server side tech-nologies, the developers of open source vulnerability scan-ners should continuously update their solutions to providesupport to a variety of web platforms and protocols. Asignificant attention should be given to resolving the com-patibility and interoperability issues that prevent softwaredevelopers from scanning their web-based systems. Althoughthe inclusion of all attack classes in one database is challeng-ing, tool developers should ensure that their vulnerabilitydatabases are regularly updated to include newly discoveredattacks. In order for a vulnerability scanning process to besuccessful and comprehensive, all applications states shouldalso be thoroughly examined to ensure that all complex attackscenarios are handled by the correspondingweb vulnerabilityscanner [5].

7. Related Work

Prior work addressed web security vulnerabilities by eitherresearching the ways to improve dynamic testing techniques[41], proposing or evaluating static analysis testing mecha-nisms [42–45], or using these testing approaches to evaluatethe security of web-based systems [46–49]. The majorityof research studies that evaluated the performance of webvulnerability scanners suggested that they have limited crawl-ing capabilities and high false positive rates. The findingsreported in this paper show that there is a low level of agree-ment between the results reported by different open sourceweb vulnerability scanners. In the following subsection, wesummarize the literature related dynamic testing tools andthe detection of web vulnerabilities.

7.1.Web Security Scanners: Detection Effectiveness. Evaluatingthe security ofweb-based applications anddetecting the secu-rity vulnerabilities in thesemultisourced systems is a complexproblem that can be addressed in many different ways [50–52]. With the increasing complexity of web-based systemsand the emergence of new development technologies thatvary in their support for security, the demand for tools thatautomatically detect security vulnerabilities is growing [53–55]. Penetration testing is one of the widely used techniquesthat is applied to detect security vulnerabilities in a cost-effective way and without requiring technical expertise [6, 53,56, 57].

In spite of the advantages of tool-based web penetrationtesting approaches, previous research attempts proved thatweb vulnerability scanners have poor performance and foundthat they are often unable to detectmany security vulnerabili-ties [6, 53, 58, 59].The limited discoverability capability is alsoone of the drawbacks of web vulnerability scanners (i.e.,security vulnerability cannot be detected until the crawler ofthe corresponding scanner reaches the URL in question) [5].Although the results generated by these scanners are mostlyregarded by software developers as comprehensive and accu-rate [56], prior work found that these scanners only detect asubset of the security vulnerabilities in the evaluated systemsand give a significant number of false alarms [53, 56].

12 Security and Communication Networks

Prior work has also addressed the performance of websecurity vulnerability scanners by either evaluating the detec-tion effectiveness of a set of scanners [18, 19, 32, 58] or devel-oping techniques that can be incorporated into these tools toincrease their detection accuracy [5, 60, 61]. Other researchattempts focused on comparing the effectiveness of dynamictesting with other security testing approaches (e.g., statictesting or manual code review) [62]. Vieira et al. used somecommercial web vulnerability detection tools for quantifyingthe number of security vulnerabilities in publicly accessibleweb services and observed a high false positive rate [17].Consistent with our findings, the experimental results pre-sented in [17] highlight limitations in the vulnerability detec-tion and crawling capabilities of dynamic testing tools andshow that there are major differences between the types andquantities of security vulnerabilities detected by differentscanners.

Their findings also suggested that there are variationsbetween the results reported by different web vulnerabil-ity scanners. Similarly, three commercial web vulnerabilitydetection scanners were evaluated in [18] by injecting SQLand XSS vulnerabilities into web-based systems and testingthe scanners’ effectiveness in detecting these security flaws.Their results furthermore suggest that these scanners leavesome vulnerabilities undetected and produce inconsistentresults. Although the observations reported in [17, 18] showthat the percentages of false positives were high, the FPRs of75% of the evaluated scanners in this study were less than orequal to 20%. This implies that the inconsistencies betweenthe scan results of different scanners might not necessarily belinked to their poor performance.

Doupe et al. [32] also highlighted the limitations inthe crawling capabilities of web vulnerability scanners afterevaluating eight commercial and three open source scanners.Although some open source scanners were included in theevaluation conducted in [32], their comparative assessment isbased on scanning oneweb-based system that was specificallydeveloped for the purpose of their study. In our study, weevaluate the selected scanners based on a systematic approachthat considers scanning a dataset that includes more than 140distinct web-based systems. Clearly, there is a lack of studiesthat thoroughly analyze the effectiveness of open source webvulnerability scanners based on well-known security bench-marks and standards.

A number of researchers have focused on identifying theroot causes of web vulnerabilities and proposed solutionsto mitigate the effects of web security attacks [63–66]. Forinstance, as an attempt to explore the correlations betweensecurity vulnerabilities and software faults, Fonseca andVieira analyzed a large collection of security patches andfound that the majority of SQL injection and XSS attackswere not caused by software bugs [63]. Soska and Christinaddressed themaliciousness of web content in a different wayby implementing a classification system that predicts whethera given web server will be compromised in the future [64].In this study, we focused on comparing the effectiveness oftwo open source web vulnerability scanners in detecting SQLinjection and XSS attacks. Our results show that there isa high level of disagreement between the results reported

by the two scanners which implies that there is a demandfor improving the efficiency, effectiveness, and detectionaccuracy of these scanners.

8. Limitations and Future Work

While our web vulnerability scanners’ comparative evalua-tion included three scanners that are among the widely usedopen source web vulnerabilities detection tools, our resultsmight not be generalized to all other similar open sourcetools.However, our evaluation approach can be applied to anyset of open source or commercial web vulnerability scanners.Although the dataset collected for our case study included 140URLs only, the approach followed to collect, scan, analyze,and evaluate the security state of these web applications canbe applied to other web-based systems as well. The sampleexamined in our case study had only 140 URLs which werecollected based on a predefined approach that allowed us tocollect a sample of websites that vary in their size, complexity,and development technologies. Our case study can thereforebe repeated to include a larger URL dataset. In our case study,we compared the results of the evaluated scanners based ontheir detection of SQL and XSS vulnerabilities only. Futurework might consider other types of web vulnerabilities (e.g.,CSRF and source code disclosure). Future research effortsmight also utilize the methodology followed in this paperto evaluate commercial web vulnerability scanners. Otherresearch efforts might also be directed toward investigatingthe level of agreement between open source and commercialweb vulnerability scanners. Our experiment could also beapplied on a large set of open source web vulnerability scan-ners to further understand the factors that might contributeto improving the performance of these scanners. In futurework, special attention should also be given to thoroughlyinvestigating the effect of false positives on the quantitiesand types of security vulnerabilities reported by the scan-ners. For example, repeating the tests multiple times couldreveal inconsistencies in the scanners’ reports and thus helpresearchers in drawing more accurate conclusions on thedetection accuracy of the scanners.

9. Concluding Remarks

The widespread adoption of web vulnerability scannersand the differences in the functionality provided by thesetool-based vulnerability detection approaches increase thedemand for testing their detection effectiveness. Althoughthere are many previously conducted research studies thataddressed the performance characteristics of web vulnerabil-ity detection tools by either quantifying the number of falsealarms ormeasuring the corresponding crawler coverage, thescope of themajority of these studies is limited to commercialtools.

Despite the advantages of dynamic testing approachesin detecting software vulnerabilities at any stage in thesoftware development process, the literature lacks studies thatcomprehensively and systematically evaluate the perfor-mance of open source web vulnerability scanners based onsound measures. The main objectives of this study are

Security and Communication Networks 13

to assess the performance of open source scanners frommultiple perspectives and to examine whether the cost-effectiveness of these tools negatively correlates with theirdetection capability. We expect the results of this researchwork to guide tool developers in enhancing the software pro-cesses followed while designing these tools, which in turn isexpected to encourage software engineers to effectively utilizeweb vulnerability detection scanners during and after releas-ing their software products.

The results of our comparative evaluation of a set of opensource scanners highlighted variations in the effectivenessof security vulnerability detection and indicated that thereare correlations between different performance properties ofthese scanners (e.g., scanning speed, crawler coverage, andnumber of detected vulnerabilities).There was a considerablevariance on both types and numbers of detected web vulner-abilities among the examined tools.

We followed our comparative assessment with a casestudy in which we evaluate the level of agreement betweenthe results reported by two open source web vulnerabilityscanners. These scanners were utilized to examine the websecurity vulnerabilities in 140 URLs and quantify the numberof low, medium, and high degree security vulnerabilities thatexisted in the evaluated URLs. By considering the fact thatthe examined scanners indicated that only 18 URLs werevulnerable to SQL injection attacks, this might suggest thatthe inconsistencies between the scan results of different toolsmight had been caused by factors other than performance dif-ferences.This findingmight also be supported by the fact thatalthough our scanners’ comparative assessment did not showsignificant performance differences among the evaluatedscanners, we observed a high level of disagreement betweenthe scan results. Therefore, additional research should bedirected toward identifying the factors that lead differentwidely known software verification tools to report inconsis-tent results.

Conflicts of Interest

The authors declare that there are no conflicts of interestregarding the publication of this paper.

Acknowledgments

This project was supported by NSTIP Strategic TechnologiesProgram (no. 11-INF1855-02) in theKingdomof SaudiArabia.The authors thank Monirah Alshreef for preparing the URLdataset and conducting the experiments reported in thepresent study. This work was supported in part by KACST.

References

[1] “OWASP: the ten most critical web application security risks,”The Open Web Application Security Project, 2013.

[2] “Internet security threat report,” Tech. Rep., Symantec, 2015.[3] M. Garnaeva, V. Chebyshev, D. Makrushin, and A. Ivanov, “IT

threat evolution in Q1 2015,” Tech. Rep., Kaspersky, 2015.[4] Gartner, “The next three years in security threats,” 2015 http://

www.gartner.com/smarterwithgartner/the-next-three-years-in-security-threats/.

[5] A. Doupe, L. Cavedon, C. Kruegel, and G. Vigna, “Enemy of thestate: a state-aware black-boxweb vulnerability scanner,” in Pro-ceedings of the USENIX Security Symposium, pp. 523–538, 2012.

[6] N. Antunes andM.Vieira, “Comparing the effectiveness of pen-etration testing and static code analysis on the detection of SQLinjection vulnerabilities in web services,” in Proceedings of the15th IEEE Pacific Rim International Symposium on DependableComputing (PRDC ’09), pp. 301–306, November 2009.

[7] F. Elberzhager, J. Munch, and V. T. N. Nha, “A systematic map-ping study on the combination of static and dynamic qualityassurance techniques,” Information and Software Technology,vol. 54, no. 1, pp. 1–15, 2012.

[8] IBMAppScan, 2015 http://www-03.ibm.com/software/products/en/appscan.

[9] HPWeb Inspect, 2015 http://www8.hp.com/us/en/software-solu-tions/webinspect-dynamic-analysis-dast/.

[10] AcunetixWeb Vulnerability Scanner, 2015 http://www.acunetix.com/vulnerability-scanner/.

[11] K.McQuade, “Open source web vulnerability scanners: the costeffective choice?” in Proceedings of the Conference for Informa-tion Systems Applied Research, vol. 2167, p. 1508, 2014.

[12] S. Chen, “SECTOOL Market: price and feature comparison ofweb application scanners,” 2015 http://goo.gl/ZUKaK8.

[13] D. Geer, “Are companies actually using secure development lifecycles?” Computer, vol. 43, no. 6, pp. 12–16, 2010.

[14] J. Witschey, “Secure development tool adoption in open-source,” in Proceedings of the 2013 Companion Publication forConference on Systems, Programming, & Applications: Softwarefor Humanity, pp. 105-106, ACM, 2013.

[15] G. A. Di Lucca and A. R. Fasolino, “TestingWeb-based applica-tions: The state of the art and future trends,” Information andSoftware Technology, vol. 48, no. 12, pp. 1172–1186, 2006.

[16] E. Fong and V. Okun, “Web application scanners: definitionsand functions,” in Proceedings of the 40th Annual HawaiiInternational Conference on SystemSciences (HICSS’07), January2007.

[17] M. Vieira, N. Antunes, and H. Madeira, “Using web securityscanners to detect vulnerabilities in web services,” in Proceed-ings of the IEEE/IFIP International Conference on DependableSystems & Networks, pp. 566–571, IEEE, 2009.

[18] J. Fonseca, M. Vieira, and H. Madeira, “Testing and comparingweb vulnerability scanning tools for SQL injection and XSSattacks,” in Proceedings of the 13th Pacific Rim InternationalSymposium onDependable Computing, pp. 365–372, IEEE, 2007.

[19] L. Suto, Analyzing the effectiveness and coverage of web appli-cation security scanners. San Francisco, October, 2007.

[20] Skipfish scanner, 2012 https://code.google.com/p/skipfish/.[21] Arachni Web Application Security Scanner Fremework, 2015

http://www.arachni-scanner.com/.[22] The Web Application Vulnerability Scanners Evaluation

Project, 2013 https://code.google.com/p/wavsep/.[23] Wapiti-web application vulnerability scanner, 2014 http://wapiti

.sourceforge.net/.[24] Vega Vulnerability Scanner, 2014 https://subgraph.com/vega/.[25] w3af: Open Source Web Application Security Scanner, 2013

http://w3af.org/.[26] IronWASP, 2014 https://ironwasp.org/.[27] Web Scanner Test Site, 2015 http://webscantest.com/.[28] Test Website for Acunetix Web Vulnerability Scanner, 2005

http://testaspnet.vulnweb.com/.

14 Security and Communication Networks

[29] AltoroMutual, 2015 http://demo.testfire.net/.[30] Web Input Vector Extractor Teaser, 2014 https://github.com/

bedirhan/wivet.[31] N. Antunes and M. Vieira, “Benchmarking vulnerability detec-

tion tools for web services,” in Proceedings of the IEEE 8th Inter-national Conference on Web Services (ICWS ’10), pp. 203–210,IEEE, July 2010.

[32] A. Doupe, M. Cova, and G. Vigna, “Why Johnny can’t pentest:an analysis of black-box web vulnerability scanners,” in Detec-tion of Intrusions and Malware, and Vulnerability Assessment,vol. 6201, pp. 111–131, Springer, 2010.

[33] Performance-Based Comparative Assessment of Open SourceWeb Vulnerability Scanners List of Scanned Websites, 2016https://goo.gl/XjlFMl.

[34] GoogleTrends, 2015 https://www.google.com/trends/.[35] SEOquake Firefox Extension, 2012 http://www.seoquake.com/.[36] Amazon Elastic Compute Cloud, 2015 http://aws.amazon.com/

ec2/.[37] PuTTY, 2015 http://www.putty.org/.[38] TightVNC Software, 2014 http://www.tightvnc.com/.[39] A. Austin and L. Williams, “One technique is not enough: a

comparison of vulnerability discovery techniques,” in Proceed-ings of the International Symposium on Empirical SoftwareEngineering and Measurement (ESEM ’11), pp. 97–106, IEEE,2011.

[40] H.-Z. Shi, B. Chen, and L. Yu, “Analysis of web security compre-hensive evaluation tools,” in Proceedings of the 2nd InternationalConference on Networks Security, Wireless Communications andTrusted Computing (NSWCTC ’10), pp. 285–289, April 2010.

[41] N. F. Awang andA. A.Manaf, “Detecting Vulnerabilities inWebApplications Using Automated Black Box and Manual Pen-etration Testing,” in Advances in Security of Information andCommunication Networks, vol. 381, pp. 230–239, Springer, 2013.

[42] V. B. Livshits andM. S. Lam, “Finding security vulnerabilities injava applications with static analysis,” in Proceedings of the 14thUsenix Security Symposium, pp. 271–286, 2005.

[43] Y. Xie and A. Aiken, “Static detection of security vulnerabilitiesin scripting languages,” in Proceedings of the USENIX SecuritySymposium, vol. 6, pp. 179–192, 2006.

[44] K. Goseva-Popstojanova and A. Perhinschi, “On the capabilityof static code analysis to detect security vulnerabilities,” Infor-mation and Software Technology, vol. 68, pp. 18–33, 2015.

[45] G. Dıaz and J. R. Bermejo, “Static analysis of source code secu-rity: assessment of tools against SAMATE tests,” Informationand Software Technology, vol. 55, no. 8, pp. 1462–1476, 2013.

[46] A. Alarifi, M. Alsaleh, and A. M. Al-Salman, “Security analysisof top visited Arabic Web sites,” in Proceedings of the 15th Inter-national Conference on Advanced Communication Technology(ICACT ’13), pp. 173–178, IEEE, 2013.

[47] M.Alsaleh andA.Alarifi, “Analysis of web spam for non-englishcontent: toward more effective language-based classifiers,”PLOS ONE, vol. 11, no. 11, Article ID e0164383, 2016.

[48] A. Alarifi and M. Alsaleh, “Web spam: a study of the pagelanguage effect on the spamdetection features,” inProceedings ofthe 11th IEEE International Conference onMachine Learning andApplications (ICMLA ’12), vol. 2, pp. 216–221, IEEE, December2012.

[49] A. Alarifi, M. Alsaleh, A. Al-Salman, A. Alswayed, and A.Alkhaledi, “Google penguin: evasion in non-english languagesand a new classifier,” in Proceedings of the 12th International

Conference on Machine Learning and Applications (ICMLA ’13),vol. 2, pp. 274–280, IEEE, December 2013.

[50] M. R. Stytz and S. B. Banks, “Dynamic software security testing,”IEEE Security & Privacy, no. 3, pp. 77–79, 2006.

[51] M. Alsaleh, A. Alqahtani, A. Alarifi, and A. Al-Salman, “Visu-alizing PHPIDS log files for better understanding of web serverattacks,” in Proceedings of the 10thWorkshop on Visualization forCyber Security (VizSec ’13), pp. 1–8, October 2013.

[52] M. Alsaleh, A. Alarifi, A. Alqahtani, and A. Al-Salman, “Visual-izing web server attacks: patterns in PHPIDS logs,” Security andCommunication Networks, vol. 8, no. 11, pp. 1991–2003, 2015.

[53] M. Curphey and R. Arawo, “Web application security assess-ment tools,” IEEE Security & Privacy, vol. 4, no. 4, pp. 32–41,2006.

[54] Z. Alshaikh, A. Alarifi, and M. Alsaleh, “Christopher Alexan-der’s fifteen properties: toward developing evaluation metricsfor security visualizations,” in Proceedings of the 11th IEEEInternational Conference on Intelligence and Security Informatics(IEEE ISI ’13), pp. 295–300, IEEE, June 2013.

[55] N. Alomar, M. Alsaleh, and A. Alarifi, “Social authenticationapplications, attacks, defense strategies and future researchdirections: a systematic review,” IEEE Communications Surveys& Tutorials, vol. PP, no. 99, 2017.

[56] B. Arkin, S. Stender, and G. McGraw, “Software penetrationtesting,” IEEE Security & Privacy, vol. 3, no. 1, pp. 84–87, 2005.

[57] J. Bau, E. Bursztein, D. Gupta, and J. Mitchell, “State of the art:automated black-box web application vulnerability testing,” inadd Proceedings of the 31st IEEE Symposium on Security andPrivacy (SP ’10), pp. 332–345, IEEE, May 2010.

[58] N. Suteva, D. Zlatkovski, and A.Mileva, “Evaluation and testingof several free/open source web vulnerability scanners,” in Pro-ceedings of the 10th Conference for Informatics and InformationTechnology (CIIT ’13), pp. 221–224, 2013.

[59] A. Austin, C. Holmgreen, and L. Williams, “A comparison ofthe efficiency and effectiveness of vulnerability discovery tech-niques,” Information and Software Technology, vol. 55, no. 7, pp.1279–1288, 2013.

[60] V. Felmetsger, L. Cavedon, C. Kruegel, and G. Vigna, “Towardautomated detection of logic vulnerabilities in web applica-tions,” in Proceedings of the USENIX Security Symposium, pp.143–160, 2010.

[61] Y.-W. Huang, C.-H. Tsai, T.-P. Lin, S.-K. Huang, D. T. Lee, andS.-Y. Kuo, “A testing framework for Web application securityassessment,” Computer Networks, vol. 48, no. 5, pp. 739–761,2005.

[62] M. Finifter and D.Wagner, “Exploring the relationship betweenWeb application development tools and security,” in Proceedingsof the USENIX Conference on Web Application Development,2011.

[63] J. Fonseca and M. Vieira, “Mapping software faults with websecurity vulnerabilities,” in Proceedings of the IEEE InternationalConference onDependable Systems andNetworks with FTCS andDCC, pp. 257–266, IEEE, June 2008.

[64] K. Soska and N. Christin, “Automatically detecting vulnerablewebsites before they turn malicious,” in Proceedings of theUSENIX Security, 2014.

[65] R. Sekar, “An efficient black-box technique for defeating webapplication attacks,” in Proceedings of the 16th Annual Networkand Distributed System Security Symposium (NDSS ’09), 2009.

[66] A. Alarifi, M. Alsaleh, and N. Alomar, “A model for evaluatingthe security and usability of e-banking platforms,” Computing,vol. 99, no. 5, pp. 519–535, 2017.

RoboticsJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Active and Passive Electronic Components

Control Scienceand Engineering

Journal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

International Journal of

RotatingMachinery

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Hindawi Publishing Corporation http://www.hindawi.com

Journal of

Volume 201

Submit your manuscripts athttps://www.hindawi.com

VLSI Design

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 201

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Shock and Vibration

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Civil EngineeringAdvances in

Acoustics and VibrationAdvances in

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Electrical and Computer Engineering

Journal of

Advances inOptoElectronics

Hindawi Publishing Corporation http://www.hindawi.com

Volume 2014

The Scientific World JournalHindawi Publishing Corporation http://www.hindawi.com Volume 2014

SensorsJournal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Modelling & Simulation in EngineeringHindawi Publishing Corporation http://www.hindawi.com Volume 2014

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Chemical EngineeringInternational Journal of Antennas and

Propagation

International Journal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

Navigation and Observation

International Journal of

Hindawi Publishing Corporationhttp://www.hindawi.com Volume 2014

DistributedSensor Networks

International Journal of