security testing at mozilla - issta 2016 · 2016. 7. 21. · security testing at mozilla christian...

SECURITY TESTING AT MOZILLA

CHRISTIAN HOLLER, M.SC. SENIOR SECURITY ENGINEER

MOZILLA CORPORATION

303,221 commits made by 3,957 contributors

approx. 13.5M LoC

303,221 commits made by 3,957 contributors

PLATFORM SECURITY

Platform

PLATFORM SECURITY

Platform

PLATFORM SECURITY

Gecko Layout Engine

Platform

PLATFORM SECURITY

Gecko Layout Engine

JS Engine

Platform

PLATFORM SECURITY

Gecko Layout Engine

JS EngineCodecs and 3rd party libraries

…

FUZZ TESTING

FUZZ TESTING

Program

FUZZ TESTING

Program

00110001 00110011 00110011 00110111

FUZZ TESTING

Program

00110001 00110011 00110011 00110111

?

FUZZ TESTING

Program

00110001 00110011 00110011 00110111

?Crashes

AssertionsHangs

FUZZ TESTING

Program ?Crashes

AssertionsHangs

Generator

FUZZ TESTING

Program ?Crashes

AssertionsHangs

Generator

Random API usage

FUZZ TESTING

Program ?Crashes

AssertionsHangs

Mutator

FUZZ TESTING

Program ?Crashes

AssertionsHangs

Mutator

SampleTest

SampleTest

SampleTest

SampleTest

FUZZ TESTING

Program ?Crashes

AssertionsHangs

Mutator

SampleTest

SampleTest

SampleTest

SampleTest

MutatedTest

FUZZ TESTING

Program ?Crashes

AssertionsHangs

Mutator

SampleTest

SampleTest

SampleTest

SampleTest

MutatedTest

Fuzzing is effective: Quick to get, cheap to use, finds tricky bugs

LANGFUZZ

MutatedTest

SampleTest

SampleTest

SampleTest

SampleTest

Grammar-based mutations for interpreter testing

LANGFUZZ

Master Thesis in 2011

USENIX Security 2012

LANGFUZZ

In use at Mozilla since 2011, 2397 bugs in the JS engine.

On Google ClusterFuzz since 2014, ~170 Bugs found.

LANGFUZZ


On Google ClusterFuzz since 2014, ~170 Bugs found.

LANGFUZZ


43 Chrome Security Bug Bounties

until today

OTHER FUZZING AT MOZILLAJSFUNFUZZ


Generator-based JS Fuzzer, written by Jesse Ruderman



Supports most ECMA features as well as Mozilla extensions


2869 bugs found since 2006


Supports most ECMA features as well as Mozilla extensions

OTHER FUZZING AT MOZILLADOMFUZZ

Generator/Mutator DOM Fuzzer, written by Jesse Ruderman



Modular structure, different modules for different DOM features, e.g.

CSS, Events, WebAudio, SVG, Unicode …



Modular structure, different modules for different DOM features, e.g.

CSS, Events, WebAudio, SVG, Unicode …

Open-Source: https://github.com/MozillaSecurity/funfuzz

https://github.com/MozillaSecurity/funfuzz

OTHER FUZZING AT MOZILLADHARMA

Generation-based API fuzzer using a custom language

OTHER FUZZING AT MOZILLADHARMA

Generation-based API fuzzer using a custom language

Open-Source: https://github.com/MozillaSecurity/dharma/

http://mozillasecurity.github.io/dharma/

A NEW APPROACH: COVERAGE


A

B

C


A

B

CA B C


A

B

CA B CA C


A

B

CA B CA C

Public tools: AFL (american fuzzy lop) and libFuzzer

HELPFUL TOOLS

FUZZ TESTING

Program ?CrashesAssertionsHangs

Mutator

SampleTest

SampleTest

SampleTest

SampleTest

MutatedTest

HELPFUL TOOLS

FUZZ TESTING


Mutator

SampleTest

SampleTest

SampleTest

SampleTest

MutatedTest

CrashesAssertions

Hangs

HELPFUL TOOLS

FUZZ TESTING


Mutator

SampleTest

SampleTest

SampleTest

SampleTest

MutatedTest

CrashesAssertions

Hangs

Out-of-bounds readUse-after-free

?

…

HELPFUL TOOLS

FUZZ TESTING


Mutator

SampleTest

SampleTest

SampleTest

SampleTest

MutatedTest

CrashesAssertions

Hangs


?

…

Valgrind: Too slow for fuzzing

HELPFUL TOOLS

FUZZ TESTING


Mutator

SampleTest

SampleTest

SampleTest

SampleTest

MutatedTest

CrashesAssertions

Hangs


?

…

Valgrind: Too slow for fuzzing

Sanitizers: AddressSanitizer, ThreadSanitizer, UndefinedBehaviorSanitizer

AddressSanitizer (ASan)

AddressSanitizer (ASan)

Compile-Time Instrumentation

Average Slowdown: 1.93x

A SIMPLE EXAMPLE

A SIMPLE EXAMPLE

CVE-2014-0160 (Heartbleed)Critical out-of-bounds read in OpenSSL

A SIMPLE EXAMPLE

CVE-2014-0160 (Heartbleed)Critical out-of-bounds read in OpenSSL

Hanno Böck combined: OpenSSL library AddressSanitizer

Coverage Guided Fuzzer

A SIMPLE EXAMPLE

A SIMPLE EXAMPLE

Found Heartbleed in only 6 hours, no TLS-specific fuzzer required

SCALING FUZZING

SCALING FUZZING

More CPU time = More bugs found

SCALING FUZZING


Deployment?

SCALING FUZZING


Deployment? Management?

SCALING FUZZING



Results?

SCALING FUZZING



Results?

A set of generic, open-source tools would be nice

FUZZMANAGER

LIFE OF A BUG

LIFE OF A BUG

FuzzManager

LIFE OF A BUG

FuzzManager

deploy machines

LIFE OF A BUG

provide signatures

FuzzManager

deploy machines

LIFE OF A BUG

provide signatures

FuzzManager

deploy machines

submit crashes

LIFE OF A BUG

provide signatures

TriageFuzzManager

deploy machines

submit crashes

LIFE OF A BUG

provide signatures

Triage

File Bug Report

Bug Database (e.g. Bugzilla)

FuzzManager

deploy machines

submit crashes

GETTING INVOLVED

GETTING INVOLVED

https://github.com/MozillaSecurity


GETTING INVOLVED


irc://irc.mozilla.org/#security



GETTING INVOLVED


https://www.mozilla.org/security/bug-bounty/





GETTING INVOLVED




3,000 USD for every critical/high security vulnerability(… and a t-shirt)




THE FUTURE

THE FUTURE

?

THE FUTURE

? Experiment: Come up with hypotheses, confirm them

THE FUTURE


Derive a model of how things work

THE FUTURE


Derive a model of how things work

Fuzzing tools should learn and evolve

THANKS

security testing at mozilla - issta 2016 · 2016. 7. 21. · security testing at mozilla christian...

Documents