![Page 1: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/1.jpg)
+bash-4.3$ echo 'PCAP cant scale’
PCAP cant scale
+bash-4.3$ echo 'PCAP cant scale'| sed 's/cant/does/’
PCAP does scale
FIRST Amsterdam – April 2017
+bash-4.3$ whoami
Erik Waher (Security Engineer @ Facebook)
Matt Moran (Security Engineer @ Facebook)
![Page 2: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/2.jpg)
Incident Response Life Cycle
![Page 3: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/3.jpg)
Specifically they need indicators of compromise (IOC)
to track down badness…
IOC live in packet captures
Detection Problem:
IR needs network visibility
![Page 4: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/4.jpg)
• Physical space limitations
• Retention time
• High network throughput
• Commercial $olution$
Difficulty to Solve: High
![Page 5: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/5.jpg)
Solution: build something better
• Flexible storage size
• Low Cost
• High throughput
• Network accessible
storage
• PCAP as a service
We set out to
build:• 1PB – 3.6PB
• OCP hardware
• Speed - 44Gbps per host
• NFS backed by GlusterFS
• PCAP as a service
• Open platform anyone can
build
We built:
![Page 6: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/6.jpg)
Demo
![Page 7: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/7.jpg)
Demo
![Page 8: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/8.jpg)
How does all that work?
![Page 9: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/9.jpg)
54 GFS/NFS Hosts
Storage Tier
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFS/
GFS
Brick
NFSd
GlusterFSd
Pandion
Capture2Disk
Napatech Accelerator
Network
Anue
SPAN
Network
![Page 10: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/10.jpg)
Network
![Page 11: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/11.jpg)
• 4 SFP+ capture interfaces to
Napatech accelerator
• PCAP writer/reader
• NIC Teaming - 6 interfaces
(SFP+) for NFS traffic
PCAP ServerNapatech PandionFlex
![Page 12: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/12.jpg)
• Bricks run NFSd and GFSd
• Brick = 30 x 4TB HD ~100TB
useable
• Storage Tier = 54 Bricks = 1.3PB
useable
• GFS performs file hashing, load
Storage TierGlusterFS & NFS on Open Compute
Project Hardware
![Page 13: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/13.jpg)
GlusterFS Introduction
Replication TranslatorReplication Translator Replication TranslatorReplication Translator Replication TranslatorReplication Translator
FUSEClient
NFS
DistributeTranslator
GF APIs
High Level
![Page 14: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/14.jpg)
Replicate Translator
Client IO
Brick 1 Brick 2 Brick n…
Update IO “Matrix”
Heal from “wise” bricks
Brick 2
GlusterFS Introduction
![Page 15: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/15.jpg)
• Didn’t want to create or support a pcap application
• Existing Napatechs already in fleet, building features
on accelerator cards
• Facebook GlusterFS active development branch
https://github.com/gluster/glusterfs/tree/release-3.8-
fb
• NFS = FUSE = easy to take existing products to
network storage
• Network storage provides flexibility to deploy anywhere
Why Napatech, Why Gluster? Why
X?
![Page 16: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/16.jpg)
Things to watch out for
• Long Fat Network (LFN’s)
• Bandwidth Delay Product
• Multi-Homing pcap hosts
• Playing nice with your
network
• File System best practices
(directory structures, reads)
• NFS host failures
• PCAP buffers
• Microbursts
• Fault Tolerance
• Gluster Servers
• PCAP Services
• Reading
• Writing
![Page 17: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/17.jpg)
• Probably internal HDFS infra
• MAP/Reduce function
• Every time you touch a file, can you improve it?
• Slices packets, run yara sigs over it? Generate new meta?
• Publisher/subscriber model
• De-couples reader/writer
• Readers can determine if file is ready to read without
querying writer
• Writer tasked with writing only
Next Generation Architecture
![Page 18: FIRST Amsterdam April 2017 · PDF fileFIRST Amsterdam –April 2017 +bash-4 ... GlusterFS & NFS on Open Compute Project Hardware. GlusterFS Introduction Replication Translator Replication](https://reader030.vdocument.in/reader030/viewer/2022020108/5aaab8707f8b9a86188e60b4/html5/thumbnails/18.jpg)
• Questions?
• https://github.com/gluster/glusterfs/tree/release-3.8-fb
• [email protected] & [email protected]
Thanks