how risky is the - cyber itl · 2017-12-25 · how risky is the software you use? cyber independent...
TRANSCRIPT
How risky is the software you use?
https://34c3.cyber-itl.org
Cyber Independent Testing Lab
{ Tim Carstens
, Parker Thompson
, Patrick Stach
, Sarah Zatko
, mudge
} @ CITL
We are CITL
• A non-profit organization based in USA
• Founded by Sarah Zatko & mudge
• Mission: to improve the state of software security by providing the public with accurate reporting on the security of popular software
• Funding from the Ford Foundation
• Partners with Consumer Reports https://www.consumerreports.org
& The Digital Standardhttps://thedigitalstandard.org
Something like this, but for software
security.
How do you do thisfor software security?
Scores &Histograms
Gentoo
Ubuntu 16 LTS
Samsung UN55KS9000
LG 55UH8500
LG Samsung
Hardened
Linux
# binaries 2196 4243 6526
% with ASLR 69% 80% 99%
stack DEP 99%* 99%* 99%*
% 64 bit 0% 0% 88%
% partial relro 3% 9% 98%
%full RELRO 0% 0% 1%
% with stack guards 1% 57% 83%
partially fortified 1% 37% 72%
fully fortified 0% 6% 10%
unfortified 80% 26% 7%
5th percentile 20 40 75
50th 50 70 100
95th 65 90 100
Security Today:
You can lead the pack by mastering the fundamentals.
LG 55UH8500 vs Samsung UN55KS9000 vs Gentoo
For more info about our work and partnerships, watchSarah Zatko @ DEF CON 25
https://www.youtube.com/watch?v=BufzX-zeZvQ
Our goals
1. Remain independent of vendor influence
2. Automated, comparable, quantitative analysis
3. Act as a user watchdog
• Non-goal: find and disclose vulnerabilities
• Non-goal: tell software vendors what to do
• Non-goal: perform free security testing for vendors
Three big questions
1. What works?
2. How do you recognize when it’s being done?
3. Who’s doing it?
The basic idea
Zatko’s Question
• Given a piece of software, we can ask
1. Overall, how secure is it?
2. What are its vulnerabilities?
• (1) appears to ask for less-info than (2)
• Zatko’s Question:Develop an heuristic which can efficiently answer (1) but not necessarily (2)
I’d like more information.
Zatko’s Question:hacker perspective
• Artisan craft
• A good hacker is: resourceful, patient, clever
• Lots of niche tools
• In principle, fuzzing can find every vulnerability*
Give me the unique crashes!
Zatko’s Question:computability perspective
• Turing-completeness means most security-related questions are undecidable, including:
• Does this software have “hidden” functionality? A backdoor?
• Does data generated by function X ever get passed to function Y?
• If the program handles passwords, key material, etc, does it do so properly? Does it sanitize memory when it is done?
• Can a given bug result in arbitrary code execution?
• Thus being exploitable is recognizable but not co-recognizable
• Thus secure = not exploitable implies being secure is not recognizable
• The best anyone can do: change the problem and/or use heuristics
Ich fühledeinenSchmerz.
Zatko’s Question: Bayesian perspective (1/3)
• Let S be some corpus of software.
• For random s in S, consider the probability
P(s is secure)
• In practice, no good way to compute P(s in S is secure)
• Instead, we consider the surrogate security scores
Ph,k = P(h units of fuzzing against s yields < k unique crashes)
• Note that limh→∞
Ph,1 = P(s in S cannot be made to crash)
Zatko’s Question: Bayesian perspective (2/3)
• Fuzzing is expensive, so we “go Bayesian”
• Let M be an observable property of software• Examples: is compatible with RELRO, has “low complexity,” etc
• For random s in S, consider the conditional probabilities
Ph,k(M) = P(h fuzzing on s yields < k unique crashes | M is true of s )
• Zatko’s Question (CITL variant):
Which M have Ph,k(M) > 0.5 for large log(h)/k ?
We got one person online and
the workload is enough for like 10 users – I think we
got a hacker
Zatko’s Question: Bayesian perspective (3/3)
Indicators might not be causal, and that’s OK:
• There are lots of ways for Ph,k(M) > 0.5 to hold for large log(h)/k
• It could be that M’s presence literally prevents crashes
• But it could also be that M is mostly only found in software written by teams who ship reliable software
• If you’re looking for security, what difference does it make?
Zatko’s Question& CITL:By Analogy
Want to find:
• Diamond (US Geological Survey)
Look for:
• Garnet(Moha112100 @ Wikipedia)
• Diopside(Rob Lavinsky)
• Chromite(Weinrich Minerals, Inc.)
An InterestingCoincidence
Browser “Underground” Exploit Price
Microsoft Edge $80,000
Google Chrome $80,000
Apple Safari $50,000
Mozilla Firefox $30,000
Doing it
photo by elasticsoul @ flickr
The Progression of CITL Tech
Static(Prototype)
Static(Extensible)
AFL CITL-fuzz NEW FUZZER
Today
First Data First reports NEW REPORTS
Our Analytic Pipeline, Today
HDFSData Science
Spark + Zeppelin
Ignite / Spark
Yarn
Compute Compute . . .
Report Generation
FrontendAPI / Submission
Applied Static Analysis
• Lots of architectures: x86-*, ARM-*, MIPS-*
• Lots of operating systems: Windows, Linux, OS X
• Lots of binary formats: Elf, PE, MachO
• Each with their own app-armoring features
• Lots of versions of each of the above!
Static Analysis (Architecture)
Input Binary(PE/ELF/MachO)
Format Analyzers
Function Discover
Control Flow Recovery
Switch Table Parsing
Basic Block Lists and links
Section Properties
Symbol Resolver
Memory Map
Observables& Metadata
CodeAnalyzers
Static Analysis (Observables)
ASLR
Format Structure
Observables
Code Structure
linked libraries
Functionality
Fortify Source
DEP
Stack Guards
RELRO
Control Flow Graphs are Key
Next-generation Static Observables
• Instruction Abstractions• Patterns in usage of arithmetic instructions• Data Manipulation (load/store)• Control flow (branches)
• Artifacts• Widgets of abstractions• Units of functionality
• Data and Control flow of Artifacts
• Further models• Symbolic Representations
Applied Fuzzing
• Fuzzing is at the heart of the surrogate security scores (Ph,k) and thus our model
• … and crash traces are useful in their own right
Reminder: Why Fuzzing?
Bugs
Vulnerabilities
Exploitable
Vulnerabilities
Recognizable
Not Recognizable
Harvesting Observables with Fuzzing
Load
Arithmetic 1 Arithmetic 2
Branch
Exit
Store
Faulting Path Path
Load
Arithmetic 1 Arithmetic 2
Branch
Exit
Store
From then on, when anything went wrong with a computer, we said it had bugs in it.
Amazing Grace
CITL: Impact
• We’ve been reporting bugs• Firefox on OSX was missing ASLR (they fixed it quick!)
• Several patches & bugs submitted to LLVM & Qemu
• We’ve inspired others• Big shout-out to the Fedora Red Team
• We’ve partnered to cover broader domains• Consumer Reports
https://www.consumerreports.org
• The Digital Standardhttps://thedigitalstandard.org
CITL: Today and Tomorrow
• We are building the tooling necessary to compute the surrogate security scores at-scale
• In the mean time, our static analyzers are already making surprising discoveries: see Sarah Zatko’s recent talks at DEFCON/Blackhat
• Advice to software vendors:Make sure your software employs every exploit mitigation mudge has ever heard of!
https://34c3.cyber-itl.org
Cyber Independent Testing Lab
{ Tim Carstens
, Parker Thompson
, Patrick Stach
, Sarah Zatko
, mudge
} @ CITL