Beyond the Tip of the IceBergFuzzing Binary Protocol for Deeper Code Coverage

Mrityunjay Gautam . Alex Moneger

Who we are?

• Security Engineers at Citrix Systems, Inc.

• Interest in low level topics (crypto, fuzzing, exploit dev)

Disclaimer

The views expressed herein are personal and stated in our individual capacity and in no way a statement or position of

Citrix Systems, Inc.

Agenda

1. State of Fuzzers and Fuzzing Technology

2. Code Coverage based Fuzzers – AFL, et al

3. Binary Code Tracing: Gate Function

4. Applications to fuzzing – Feedback Loop

5. PoC Demo – Toy Example

6. Heuristic based Protocol Analysis

FUZZING AS WE KNEW IT

Fuzzing: Myths Vs Reality

• Myth: “fuzzing is easy”:– flip some bits

– Collect bugs

• Reality: “fuzzing is complex”:– Identifying target functions & writing wrapper code

– Building and minimizing a corpus

– Minimizing test-cases

– Instrumentation:• Is input X better then Y?

• Did my application crash on input X or Y?

File format fuzzing

• Lots of focus on parsers:

– American Fuzzy Lop

– Honggfuzz

• Handling network code with them is tricky

Network fuzzing

• Still stuck modeling protocols

• Still slow

• Still requires some sort of agent to detect crashes

• We’re still blind fuzzing• Yet, network stack is a target of choice

• We need more balance

Historically…

• 2 approaches:

– Mutate data forever (randomly, byte flip, …)

– Model data, mutate fields separately (Spike, Peach, Codenomicon, …): Anyone written a complex Peach pit?

• Run for some iterations or until all states are modeled

• Hope for the best

• Claim that you have covered 10n iterations and feel good about it

FUZZING TODAY

Today

Genetic algorithms => retain only best input for further ‘mutation’

1. Mutate best input

2. Send to target

3. Measure fitness() based on Heuristics

4. Discard or prioritize input, back to 1.

We know how inputs affect target!

Fitness Heuristic: Code coverage

• Code coverage is the most used metric

• Tells you if an input has triggered new code paths

• All tools try to measure code coverage one way or another

• Can be achieved :

– Binary instrumentation (PIN, DynamoRIO)

– Static rewriting (Dyninst)

– Kernel probing (perf)

– HW (intel BTS => branch trace store)

How does it work

• Model control flow using basic blocks:

• Discard unconditional edges (JMPs)

• Retain edge count

• Provides an unordered code coverage map

• Code coverage are sets which can be compared:– 0x08040302 => 08040301 : 1

– 0x0804030b => 08040404 : 5

0x08040302 0x08040301

Thanks AFL

• AFL revolutionized fuzzing

• “Batteries included” fuzzer

• Perfect balance between:– Using build systems

– Speed

– Functionality

• Caveat: compares traces across runs:– Target has to exit

– Has to get data off stdin

PROBLEM???

Limitations

• If you have source code, get AFL to work on packets

• Write wrappers, handle state, exit, … not pretty, but kind of works

• Tight coupling may force to stub out function

– using LD_PRELOAD (see preeny)

– using linker -Wl,-wrap

Network daemons

• A solution could be to change the model a bit

• Keep successful AFL concepts:

– Code coverage

– Genetic algorithm

• But avoid restarting the target

• This breaks the deterministic nature of AFL

Requirements

• Improve traditional fuzzers:

– Get rid of the “try single input then check” cycle

• By borrowing from feedback driven fuzzers:

– Code coverage

– Genetic algorithm

• Do this during runtime

• Without re-spawning the target between inputs

OUR APPROACH

Observations

• Network daemon operations:1. Do startup stuff, 2. Wait for connection

1. Connection establishment2. Wait for input (read)3. Process input packet4. Send something based on input (write)5. Loop from 2.2 till connection closes

3. Close (close) and go to 2.

• What code coverage do we care about?

• Trace code between first read (2.2) and last write (2.4)?

Startup

Read

Write

Close

Parse

Gate functions

• Here read()/write() can be considered gates

• When you enter a gate, trace

• When you exit a gate, stop trace

• Transfer code coverage to decision maker

Generalized approach

• Trigger code coverage collection at runtime

• Based on defined “gate” syscalls, say X and Y

• When syscall X is triggered, start recording edge transitions

• When syscall Y is triggered, stop recording

• Dump trace

• Repeat

1000 feet view

• Track only network file descriptors

• Ignore I/O FDs

• Generate a hitmap at runtime through “gate” syscalls

• Dump it to fuzzer for analysis

• Fuzzer elects best input

Filtering file descriptors

• Accept() syscall returns FD

• Track FDs returned

• Checked if they’re passed in to:

– Read

– Write

• Stop tracking on close()

Accept 6,7,86

Read(6)

Write(6)

Read(9)

Write(9)

Aggregate m

apCoverage map

• Coverage maps are per read/write gate

• You get several maps for one connection

• Allows fuzzing a specific state

• Can also aggregate code coverage between gate functions

Accept

Read(6)

Write(6)

Read(6)

Write(6)

Read(6)

Close(6)

Map 1

Map 2

Map 3

Ugly diagram

Accept => 6

6, 7, …, fd

Read(6)

Write(6)

Close(6)

Heat Map

Network FD list

Do stuff

UDP

• Exact same thing, but track:

– Recvfrom/recvmsg

– Sendto/sendmsg

• Generalization is possible to any syscall sequence

• Could use similar grammar to seccomp BPF

Netcov

• “Simple” pintool: https://github.com/alexmgr/netcov

• Generate code coverage maps at runtime

• Write them to a pipe

• Reverse of fuzzing talks, here fuzzing is up to you ;)

• Sidekick: netcallgraph:

– Generates runtime callgraph

• A dummy fuzzing example: https://github.com/alexmgr/netcov-client

It’s a PoC…

• Limitations:– Read hangs– Select/poll– No crash detection– No ASAN to catch memory errors– Hit map format is text based

• Works well:– Multithreaded daemons– Heatmap is per FD=> allows concurrent fuzzing– Mutation independent– Source code independent

• It’s a demo, not a tool

Netcov flow

Netcov

Daemon

Client (Fuzzer, …)

Coverage

Protocol

Demo

• Demo daemon, magic packet: “ABC1234567890i”:if (read(conn_desc, buff, sizeof(buff) - 1) > 0) {

printf("Received %s\n", buff);

if (buff[0] == 'A') { printf("Took first branch\n");

if (buff[1] == 'B') { printf("Took second branch\n");

if (buff[2] == 'C') { printf("Took third branch\n");

if (strncmp(buff + 3, "1234567890", 10) == 0) {

printf("Good job!\n");

char *num = buff + 13;

printf("Got num: %d\n", atoi(num));

int i = 0;

for (i = 0; i < atoi(num); i++) {

printf("%d..", i);

} write(conn_desc, "Good job!", 10);

Example netcallgraph

Fuzzing demo

• Start with an input value

• Byteflip it

• Measure coverage

1. If coverage increases, keep as best input

2. Mutate

3. Repeat 1.

REAL WORLD EXAMPLE – RDP PROTOCOL

RDP – Remote Desktop Protocol

• TCP Protocol on port 3389

• Originally on Windows variants

• Ported to most Unix Environments – XRDP

• Clients available on all Linux, Mac, Windows flavors

Weaponizing the ‘netcov’ PoC

Send Next Mutated Packet XRDP Server

Netcov Binary Tracing

/tmp/netcovmap

Receive Binary Trace between (recv, send)

Fitness function(Unique Code Coverage)

Feedback on Packet Quality

Load RDP Wireshark Trace

Identify Packet to Play With

Mutation Strategy – Based on Feedback

Process Feedback

Result Generation

Synchronization Problem

XRDP Packet Analysis Results

Restricting the trace to libxrdp ONLY

Base Pkt:0300002621e00000000000436f6f6b69653a206d737473686173683d0d0a0100080003000000

Baseline:write:8=libxrdp.so.0+14816->libxrdp.so.0+14840:1;libxrdp.so.0+14840->libxrdp.so.0+14881:1;libxrdp.so.0+14881->libxrdp.so.0+47232:1;libxrdp.so.0+14904->libxrdp.so.0+14908:1;libxrdp.so.0+14908->libxrdp.so.0+14924:1;libxrdp.so.0+14924->libxrdp.so.0+14949:1;libxrdp.so.0+14949->libxrdp.so.0+14989:1;libxrdp.so.0+14989->libxrdp.so.0+15369:1;libxrdp.so.0+15348->libxrdp.so.0+15352:1;libxrdp.so.0+15352->libxrdp.so.0+14816:1;libxrdp.so.0+15369->libxrdp.so.0+15424:1;libxrdp.so.0+15424->libxrdp.so.0+15434:1;libxrdp.so.0+15434->libxrdp.so.0+47152:1;libxrdp.so.0+15446->libxrdp.so.0+15450:1;libxrdp.so.0+15450->libxrdp.so.0+47344:1;libxrdp.so.0+47152->libxrdp.so.0+47165:1;libxrdp.so.0+47165->libxrdp.so.0+15446:1;libxrdp.so.0+47232->libxrdp.so.0+47249:1;libxrdp.so.0+47249->libxrdp.so.0+47280:1;libxrdp.so.0+47280->libxrdp.so.0+14904:1;libxrdp.so.0+47280->libxrdp.so.0+15348:1;

Results

Packet 0: (To RDP Server)

[(0, 0, 'CONTROL'),

(1, 1, 'DATA'),

(2, 3, 'MAGIC'),

(4, 4, 'DATA'),

(5, 5, 'CONTROL'),

(6, 37, 'DATA')]

Results

Packet 0: (To RDP Server)

[(0, 0, 'CONTROL'),

(1, 1, 'DATA'),

(2, 3, 'MAGIC'),

(4, 4, 'DATA'),

(5, 5, 'CONTROL'),

(6, 37, 'DATA')]

Results

Packet 0: (To RDP Server)

[(0, 0, 'CONTROL'),

(1, 1, 'DATA'),

(2, 3, 'MAGIC'),

(4, 4, 'DATA'),

(5, 5, 'CONTROL'),

(6, 37, 'DATA')]

XRDP Implementation Analysis

• Analysis of the 1st Packet:

– Byte (1) mutation leads to control flow change

– Bytes (3,4) are length of the packet. Verified before further processing.

– Byte (5) is length of x224CRQ Header. Not verified before processing or may lead to over-read.

– Byte (6) mutation leads to control flow change

– Bytes (7,38) is DATA. Fuzzable with different Control Flow bits.

Who in the room cannot write a fuzzer now ?

CONCLUSION

Conclusion

• Much to do in the world of network fuzzing

• Still stuck with:

– Dumb mutation fuzzers

– Model based fuzzers

– Slowness

• We present “just” a glimpse of what CAN be achieved

Thank You