a look at the network: searching for truth in distributed applications
DESCRIPTION
A talk by C. Scott Andreas (@cscotta) of Boundary on "the network" and designing / deploying distributed applications.This session offers a deep-dive into how application-level problems manifest at the network level. Some of these cases range from basic network partitions and node outages to sophisticated application-level changes such as garbage collections on managed runtimes, classes of bugs which evade conventional monitoring but constitute partial failures, changes in network activity based on database partitioning, load balancing, and sharding, and other warning signs that crop up at layer three long before wreaking havoc at layer seven as customer-visible failures begin to occur. Combining application-level metrics with network analytics is a powerful cocktail for identifying hot spots quickly, and connecting the dots out to the client closes the loop.TRANSCRIPT
![Page 1: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/1.jpg)
taco.cat/oscon12
A Look at the Network
Searching for Truth in Distributed Applications
c. scott andreas (cscotta)oscon 2012 - portland oregon
![Page 2: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/2.jpg)
taco.cat/oscon12
![Page 3: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/3.jpg)
taco.cat/oscon12
![Page 4: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/4.jpg)
taco.cat/oscon12
![Page 5: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/5.jpg)
taco.cat/oscon12
THE NETWORKIS RELIABLE
![Page 6: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/6.jpg)
taco.cat/oscon12
THE NETWORK IS SECURETHERE IS ONE ADMINISTRATOR
![Page 7: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/7.jpg)
taco.cat/oscon12
LATENCY IS ZEROBANDWIDTH IS INFINITE
![Page 8: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/8.jpg)
taco.cat/oscon12
THE NETWORK IS HOMOGENOUS
TOPOLOGY DOESN’T CHANGE
![Page 9: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/9.jpg)
taco.cat/oscon12
TRANSPORTCOST IS ZERO
![Page 10: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/10.jpg)
taco.cat/oscon12
where can i buy that?
![Page 11: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/11.jpg)
taco.cat/oscon12
[ another approach ]
![Page 12: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/12.jpg)
taco.cat/oscon12
![Page 13: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/13.jpg)
taco.cat/oscon12
![Page 14: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/14.jpg)
taco.cat/oscon12
![Page 15: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/15.jpg)
taco.cat/oscon12
![Page 16: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/16.jpg)
taco.cat/oscon12
NETWORK + YOUR APPSARE A GRAPH
![Page 17: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/17.jpg)
taco.cat/oscon12
![Page 18: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/18.jpg)
taco.cat/oscon12
YEAH BUT...WHAT DO WE DO WITH IT?
![Page 19: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/19.jpg)
taco.cat/oscon12
graphs change constantlytheir edges can be
represented as a time series
![Page 20: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/20.jpg)
taco.cat/oscon12
GRAPH TIME SERIES
+
![Page 21: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/21.jpg)
taco.cat/oscon12
LEMMA 1:there exists no possible way for
applications to communicate except via the network
![Page 22: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/22.jpg)
taco.cat/oscon12
LEMMA 2:applications are unable to fulfill their purpose without communic-
ating and participating in a cluster
![Page 23: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/23.jpg)
taco.cat/oscon12
LEMMA 3:the network can be represented
as a time series and a graph
![Page 24: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/24.jpg)
taco.cat/oscon12
LEMMA 4:nearly all modes of failure in distributed systems can be
identified and predicated this way
![Page 25: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/25.jpg)
taco.cat/oscon12
WHAT CAN ONE OBSERVE?
network partitions
impaired nodesGC pauses
poor load balancing
good deploys
bad deploys
security breachesvariance
![Page 26: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/26.jpg)
taco.cat/oscon12
![Page 27: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/27.jpg)
taco.cat/oscon12
![Page 28: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/28.jpg)
taco.cat/oscon12
![Page 29: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/29.jpg)
taco.cat/oscon12
![Page 30: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/30.jpg)
taco.cat/oscon12
![Page 31: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/31.jpg)
taco.cat/oscon12
TOOLS
– NProbe / NTop
– CFlowd / flow-tools
– TCPDump / TCPReplay
– CollectD
– R / RStudio
– Esper
– Dynamic Time Warping (algo)
– Python / NumPy / SciPy
![Page 32: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/32.jpg)
taco.cat/oscon12
FURTHER READING– Network Flow Analysis (No-Starch Press)
– Eamonn Keogh (“Atomic Wedgie”)
– K-Snap (“Efficient Aggr. for Graph Summ.”)
– “Medians and Beyond” (Shrivastava)
– Exponential Smoothing:The State Space Approach (Hyndman)
– Gigascope (AT&T Research)
– Dynamic Time Warping (algo)
– Studying Complex Adaptive Systems (Holland)
– HyperLogLog / Count-Min Sketch
![Page 33: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/33.jpg)
taco.cat/oscon12
BONUS!a spec for thai chili salsa and reference implementation
– 28oz peeled tomatoes
– Half a bag of hot thai chilis
– Half a cucumber
– Handful of radishes
– One green bell pepper
– A small red onion
– Half a thing of cilantro
– 6 cloves of garlic
– 2 tablespoons of salt
– 2 tablespoons of white vinegar
– Juice from half a lime
– Bit of parsley
– Some lemongrass
![Page 34: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/34.jpg)
taco.cat/oscon12
BONUS!a spec for thai chili salsa and reference implementation
– 28oz peeled tomatoes
– Half a bag of hot thai chilis
– Half a cucumber
– Handful of radishes
– One green bell pepper
– A small red onion
– Half a thing of cilantro
– 6 cloves of garlic
– 2 tablespoons of salt
– 2 tablespoons of white vinegar
– Juice from half a lime
– Bit of parsley
– Some lemongrassMIX IT ALL UP
![Page 35: A Look at the Network: Searching for Truth in Distributed Applications](https://reader033.vdocument.in/reader033/viewer/2022051817/5484ae3f5906b5a9438b4690/html5/thumbnails/35.jpg)
taco.cat/oscon12
A Look at the Network
Searching for Truth in Distributed Applications
oscon 2012 - portland oregon