I Know What Your Packet Did Last Hop: Using
Packet Histories to Troubleshoot Networks
Nikhil Handigol, Brandon Heller, Vimalkumar Jeyakumar, David Mazières, and Nick McKeown, Stanford University
NSDI 2014 Speaker:Cheng-Da Tsai
1
OutlineMotivation
Netsight architecture
Implementation
Debug tool(API + Application)
Compression
Evaluation
Scaling Netsight
Summary
2
Motivation
Provide direct evidence to diagnose network problem
Bug example:
3
Motivation
Provide direct evidence to diagnose network problem
Bug example:
4
Motivation
Provide direct evidence to diagnose network problem
Bug example:
After hours of debugging…..
5
Motivation
Provide direct evidence to diagnose network problem
Bug example:
forwarding rules were improperly update in wired switch…..QQ
6
Motivation
Use Netsight: you can only ask
“Show me all packet histories for packets to the client when the handover occurred. “
This packet go to the wrong AP.You can also check which switch flow table updated error.
7
Motivation
Packet History Definition:(3W1H)
What the packet looked like as it entered the network (headers)
Where the packet was forwarded (switches + ports)
How it was changed (header modifications)
Why it was forwarded that way (matched flow/actions + flow table).
Goal:
Complete visibility: every event that ever happened to every packet
8
Netsight architecture
9
Topology 保證正確
Implementation
Life Of a Postcard:
10
Implementation
Postcard Generation:Control Plane
Flow table state table recorder
Postcard
collector
Packet headerSwitch
id outport
version #
copy
tag
11
Implementation
Postcard Collection:Control Plane
Flow table state table recorder
Postcard
collector
All postcards for a packet to one server send by VLAN ID
12
Implementation
Postcard Collection:
Postcard
collectorhash based on flow key
(5-tuple)
13
Implementation
History Assembly:
Topo-Sort: assemble to a flow
14
Debug tool(API)
Postcard Filters: --bpf [packet description] -- dpis [switch id] --inport [port#] … EX:--bpf "ip src A" --dpid S --inport not P.
Packet History Filter: start at X: ^{{X}}
end at X: {{X}}$
go through X: {{X}}
go through X, and later Y: {{X}}.*{{Y}}
start at X, never reach Y: ^{{X}}[^{{Y}}]*$
experience a loop: (.).*(\1)
15
Debug tool(Application)
ndb:Interactive Network Debugger
netwatch:Live Invariant Monitor
netshark:Network-wide Path-Aware Packet Logger
netprof: Hierarchical Network Profiler
16
Implementation
Filter triggers:
PHFPHFPHFPHFPHFnotify
Application17
Compression
Compress in two places:
Before shuffling postcards to servers.
Before archiving assembled histories to disk.
18
Compression
Huge redundancy in packet header fields
19
Evaluation
Compression
20
Evaluation
Matching latency
21
Scaling Netsight
Basic Netsight (No Compress):
extract 31% traffic.
Netsight-SwitchAssist(Compress in Switch side):
extract 7% traffic.
Netsight-HostAssist(Compress in Host side):
extract 3% traffic.
22
Summary
Complete visibility: every event that ever happened to every packet is possible.
Exact traffic can be resolved by compression.
Speed of generating Postcard can be resolved by map-reduce method.
23