![Page 1: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/1.jpg)
I Know What Your Packet Did Last Hop: Using
Packet Histories to Troubleshoot Networks
Nikhil Handigol, Brandon Heller, Vimalkumar Jeyakumar, David Mazières, and Nick McKeown, Stanford University
NSDI 2014 Speaker:Cheng-Da Tsai
1
![Page 2: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/2.jpg)
OutlineMotivation
Netsight architecture
Implementation
Debug tool(API + Application)
Compression
Evaluation
Scaling Netsight
Summary
2
![Page 3: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/3.jpg)
Motivation
Provide direct evidence to diagnose network problem
Bug example:
3
![Page 4: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/4.jpg)
Motivation
Provide direct evidence to diagnose network problem
Bug example:
4
![Page 5: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/5.jpg)
Motivation
Provide direct evidence to diagnose network problem
Bug example:
After hours of debugging…..
5
![Page 6: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/6.jpg)
Motivation
Provide direct evidence to diagnose network problem
Bug example:
forwarding rules were improperly update in wired switch…..QQ
6
![Page 7: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/7.jpg)
Motivation
Use Netsight: you can only ask
“Show me all packet histories for packets to the client when the handover occurred. “
This packet go to the wrong AP.You can also check which switch flow table updated error.
7
![Page 8: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/8.jpg)
Motivation
Packet History Definition:(3W1H)
What the packet looked like as it entered the network (headers)
Where the packet was forwarded (switches + ports)
How it was changed (header modifications)
Why it was forwarded that way (matched flow/actions + flow table).
Goal:
Complete visibility: every event that ever happened to every packet
8
![Page 9: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/9.jpg)
Netsight architecture
9
Topology 保證正確
![Page 10: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/10.jpg)
Implementation
Life Of a Postcard:
10
![Page 11: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/11.jpg)
Implementation
Postcard Generation:Control Plane
Flow table state table recorder
Postcard
collector
Packet headerSwitch
id outport
version #
copy
tag
11
![Page 12: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/12.jpg)
Implementation
Postcard Collection:Control Plane
Flow table state table recorder
Postcard
collector
All postcards for a packet to one server send by VLAN ID
12
![Page 13: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/13.jpg)
Implementation
Postcard Collection:
Postcard
collectorhash based on flow key
(5-tuple)
13
![Page 14: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/14.jpg)
Implementation
History Assembly:
Topo-Sort: assemble to a flow
14
![Page 15: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/15.jpg)
Debug tool(API)
Postcard Filters: --bpf [packet description] -- dpis [switch id] --inport [port#] … EX:--bpf "ip src A" --dpid S --inport not P.
Packet History Filter: start at X: ^{{X}}
end at X: {{X}}$
go through X: {{X}}
go through X, and later Y: {{X}}.*{{Y}}
start at X, never reach Y: ^{{X}}[^{{Y}}]*$
experience a loop: (.).*(\1)
15
![Page 16: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/16.jpg)
Debug tool(Application)
ndb:Interactive Network Debugger
netwatch:Live Invariant Monitor
netshark:Network-wide Path-Aware Packet Logger
netprof: Hierarchical Network Profiler
16
![Page 17: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/17.jpg)
Implementation
Filter triggers:
PHFPHFPHFPHFPHFnotify
Application17
![Page 18: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/18.jpg)
Compression
Compress in two places:
Before shuffling postcards to servers.
Before archiving assembled histories to disk.
18
![Page 19: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/19.jpg)
Compression
Huge redundancy in packet header fields
19
![Page 20: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/20.jpg)
Evaluation
Compression
20
![Page 21: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/21.jpg)
Evaluation
Matching latency
21
![Page 22: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/22.jpg)
Scaling Netsight
Basic Netsight (No Compress):
extract 31% traffic.
Netsight-SwitchAssist(Compress in Switch side):
extract 7% traffic.
Netsight-HostAssist(Compress in Host side):
extract 3% traffic.
22
![Page 23: I know what your packet did last hop using packet histories to troubleshoot networks](https://reader036.vdocument.in/reader036/viewer/2022062514/55a552581a28ab0d4b8b4666/html5/thumbnails/23.jpg)
Summary
Complete visibility: every event that ever happened to every packet is possible.
Exact traffic can be resolved by compression.
Speed of generating Postcard can be resolved by map-reduce method.
23