ory segal, tsvika klein akamai technologies
DESCRIPTION
Big Data Intelligence Harnessing Petabytes of WAF statistics to Analyze & Improve Web Protection in the Cloud. Ory Segal, Tsvika Klein Akamai Technologies. About Us. Ory Segal Principal Product Architect, Cloud Security Tsvika Klein Product Manager, Cloud Security. - PowerPoint PPT PresentationTRANSCRIPT
Big Data IntelligenceHarnessing Petabytes of WAF statistics to Analyze & Improve Web Protection in the Cloud
Ory Segal, Tsvika KleinAkamai Technologies
About Us
• Ory Segal– Principal Product Architect, Cloud Security
• Tsvika Klein– Product Manager, Cloud Security
Hosted by OWASP & the NYC Chapter
Topics to Cover
Hosted by OWASP & the NYC Chapter
Akamai & OWASP ModSecurity CRS Relationship
Security Big Data @ Akamai
Measuring WAF Accuracy @ Akamai
CRS through the Big Data Prism (Lessons Learned)
About UsBut we only have 45 minutes…
And too much data to cover…
Akamai & OWASP CRS
This is not an Akamai marketing presentation
Akamai has been offering its cloud-based WAF since 2009. Kona Site Defender:
– OWASP CRS (Akamai Kona Rules)– DDoS Protection– DNS Protection– Bot Detection– Site Shield / Site Cloaking
OWASP CRS was ported to Akamai MD, and does not run directly on ModSecurity
SECURITY BIG DATA @ AKAMAI
Akamai’s cloud platform enables secure, high-performing user experiences on any device, anywhere
Highlights: 100 million page views per second and
500 billion hits per day 734 Million IP addresses seen quarterly 260+ Terabytes of compressed daily logs 30% of all internet traffic
120,000+Servers
2,000+Locations
82Countries
1,100+Networks
750+ Cities
Akamai Intelligent Platform
CSI Platform Statistics
10 Terabytes of daily attack data
2 Petabytes of security data stored
45 days retention
140K concurrent connections (incoming data)
600K log lines / sec. indexed by 30 dimensions
8000 queries daily scanning terabytes of data
CSI High Level Architecture
HADOOP
YODALOG AGENT
HBASE
AKAMAI EDGE SERVERS
YODA ADAPTER
BE Applications
FE Applications
Yoda (Distributed Query Engine)
Interactive
Multiple data streams
Intuitive query language
High cardinality aggregation
Security Big Data Challenge #1
Security Big Data Challenge #2
Sample Data App - SARA
Interactive Tool to Analyze Security Events
BACK TO WAF & OWASP CRS…
WAF Accuracy Lingo
• Imagine a WAF that protects against 100% of all possible attack vectors
…by blocking 100% of all HTTP requests
• Accurate WAF testing requires you to measure:• How many real attacks got blocked (TP)• How much valid requests were allowed through (TN)• How much valid traffic was inappropriately blocked (FP)• How many attacks were allowed through ((FN)
Lets talk about measuring Precision, Recall, Accuracy, MCC…
Things You Need to Know
% of blocked requests that were actual attacks
% of attacks that were actually blocked
% of decisions that were good decisions
* MCC: http://en.wikipedia.org/wiki/Matthews_correlation_coefficient
Correlation between WAF decisions and actual nature of requests
Lets Look at Some Examples
A WAF’s accuracy needs to be measured both in its ability to block attacks, as well as it’s ability to allow good traffic through…
WAF Type Requests Valid Attacks Blocked TP TN FP FN P R A MCC
Real 1000 990 10 11 8 987 3 2 0.73 0.8 0.995 0.76
Off 1000 990 10 0 0 990 0 10 N/A 0 0.99 0
Always Block 1000 990 10 1000 10 0 990 0 0.01 1 0.01 0
Noisy 1000 990 10 31 8 967 23 2 0.26 0.8 0.975 0.45
Conservative 1000 990 10 2 2 990 0 8 1.00 0.2 0.992 0.45
Introducing:
Akamai WAF Testing Framework
Akamai WAF Testing (AWT) Framework
• Ability to send both valid & attack traffic
• Easily create or add new test cases:• 3 methods: Text files, Burp Extender, Wireshark .pcaps
• Easily import test cases from Akamai’s Big Data platform
• Configurable and can work with any WAF• Easily define success / fail criteria
• Intuitive XML & HTML reports
• Easy debugging of FP/FN w/ Anomaly Scoring (rule comb.)
AWT Built-In Test Cases
In order to accurately assess WAF, we collected test cases from the following sources:
Web interaction recordings of Alexa Top 100 internet sites – Commerce, Health, Consumer Electronics, Reference, Finance, …
Ported common False Positive cases from Akamai customers (Big Data)
Recorded commercial web application scanner traffic
Attacks from Akamai CSI big data platform
Havij & SQLMap attacks
Exploits from the internet (fuzzers, exploit-db, …*
Tens of Thousands of HTTP Requests, divided 95% - 5%
Ported “valid” test cases from other tools*
AWT Reports – High Level Statistics
AWT Reports – Protection Statistics
AWT Reports – False Positives Analysis
OWASP CRS – LESSONS LEARNED
CRS Issue #1 – Risk Groups
• CRS 2.2.x uses a single anomaly score– Visibility (granularity) issues – What really happened?
• Separate anomaly score “accounting” to smaller risk groups (attack types)– Clear understanding of which attack took place
• Challenge: – requires rule mapping to risk groups– Some rules contribute to more than 1 risk group– Requires to put some more thought into anomaly scoring – it’s not just
one pile of rules/scores
XSS = 35, SQLi = 10, RFI = 0, LFI = 0, …
CRS Issue #2 –Multiple Thresholds
<script>alert('xss')</script>=> Score 30
; /bin/sh cat /etc/passwd=> Score 5
<book> Hello World </book>=> Score 10
Different risks require different anomaly thresholds
25
5
Threshold
<xss>
XSS Attack:
CMDi Attack:
Valid XML:
<xml>
CRS Issue #2
XSS SQLi CMDi HTTP RFI0
5
10
15
20
25
30
TH
CRS Issue #3 – HTTP Violations
“BLOCK HTTP PROTOCOL VIOLATIONS ?!???
THAT’S LIKE 1.21 PETABYTES OF LOGS PER DAY!!!!!”
CRS Issue #3 – HTTP Violations
• HTTP RFC Enforcement?! Good Luck!– APIs, REST services, RSS feeds, Good Bots – most don’t adhere to
HTTP RFC– Prior to system tuning:
• Missing Accept Header (960015): 14%• Missing User-Agent Header (960009 ): 3%
• Can’t trust HTTP violation rules on their own– “Invalid HTTP” risk group with its own threshold
• Blocks only seriously-damaged HTTP requests– Build more focused tool fingerprints
• See next slide for an explanation on 960015
960015 – Research into 3 hours of triggers
Which URLs trigger this rule?
85% Static Media Files
Perhaps a Unique User-Agent?
95.1K “Unique” UAsAnything in Common?
“Android” String found in 50%
Can You Give Me Something Else?
Common: Android (50%), AppleWebKit (19%), News (21%), App (20%)
CRS Issue #4: Cookies
YEAR: 2003
SESSID = 12f0a0193b4d93e9s92a39af;
Quite easy to spot a SQLi or XSS payload in a cookies
CRS Issue #4: Cookies
YEAR: 2013C1state = 24~1~-1~-1~E~6~6~6~10~10~0~0~|~37A1B34A~2EBA820B~0AEBA380~130959B9~0327C30B~7617CC73~21B797A5~C6392AF5~5FE036DB~|~8A173E13~7F5D33BF~30DFEF65~|~~|~0~1~2~3~4~5|3~4~6~7~8||0~1~2|4~4~6||~|~0~0~0~0~0~0~|~0~0~0~0~0~|~~|~~|~~|~~|; C2state = PC#1382573257902-104085.19_06#1384742638|cat#true#1383533098|session#1383533019933-203317#1383534898; C3data = {"v":1,"rid":"1371546489873_699561","to":5,"c":"http://www.some.site/page.aspx?a=5","pv":2,"lc":{"d0":{"v":2,"s":true}},"cd":0,"sd":0,"f":1371546904751} ; Cinfo = 1403D3394_232#scroll on "//<![CDATA[(function() { var f5_cspm = { pass_params: '1102912_0394939_19210_24253..."
CRS Issue #5: Score Spreading Across Selectors
In many FP scenarios, score spreads across “selectors”
c1 = 1384044727071|ABCD:2::|AC:1::|PSD:0:AKFJ~MOBILE^CLAK_KOL:1385149290276 [950901 - 5]
c2 = bn:Samsung|mn:GT-I9300 Galaxy S III|tb:false|mb:true|dos:Android|dosv:4.1|bos:KJSKKL|bosv:9 [981172 - 3]
c3 = PC#1383939352901-916004.20_14#1386636727|check#true#1384044787|session#1384044726390-399957#1384046587 [981231 - 3]
c4: = ”” [981318, 981242 – 2, 5] (Total Score: 18)
Consider a FP reduction heuristics that reduces the total score when spread across selectors? There are security implications,…
CRS Issue #6: Rule Inefficiency
During our big data analysis & AWT usage, we noticed a few troubling rule issues:
– Many rules have redundancies in expressions• This tends to push the anomaly score up in many
scenarios (“reinforcing a FP”)• Forces pushing the threshold much higher than really
needed– Some rules combine weak & strong signatures
• FP-prone rules generate high score – reducing their “weight” hurts the accurate signatures in them
– Some rules seemed almost useless – e.g. 981172
Summary
• Big Data:– OWASP / ModSecurity should consider collecting anonymized
trigger information– CRS would greatly benefit from a much larger sample set
• CRS Future:– Akamai has already contributed to the CRS project, and would
continue to contribute back to the community– We highly recommend adopting some of the major changes done
@ Akamai – mainly the “risk groups” model & multiple thresholds• WAF Testing:
– Now that the WAF industry has matured, it is time that WAF deployments will be measured for accuracy using tools & methods mentioned here – Precision, Recall and MCC
THANK YOU