with traffic control testing applications · in containers alban crequy testing applications with...
TRANSCRIPT
In containers
Alban Crequy
Testing applicationswith traffic control
ContainerCon North America - August 2016https://goo.gl/ZLLlv3
Alban Crequy
∘ Worked on rkt the last 1.5 years∘ Currently tech lead on rkt∘ In 2014, worked on traffic control for multimedia
applications in cars (tcmmd)
https://github.com/alban
Berlin-based software company building foundational Linux technologies
Some examples of what we work on...
OSTreegit for operating system binaries
Find out more about us…
Blog: http://kinvolk.io/blog
Github: https://github.com/kinvolk
Twitter: https://twitter.com/kinvolkio
Email: [email protected]
∘ What is traffic control and how does it work on Linux∘ How it can be used for testing a microservices application∘ Demo
∘ With CoreOS Linux, Kubernetes, Weave Scope
Plan
What is traffic control?
How does it work on Linux?
Traffic control, why?
web server client
client
client
THEINTERNET
∘ fair distribution of bandwidth
∘ reserve bandwidth to specific applications
∘ avoid bufferbloat
∘ Network scheduling algorithm∘ which packet to emit next?∘ when?
∘ Configurable at run-time:∘ /sbin/tc∘ Netlink
∘ Default on new network interfaces: sysctl net.core.default_qdisc
Queuing disciplines(qdisc)
eth0 THE INTERNETqdisc
Stochastic FairnessQueueing (sfq)
eth0
THE INTERNET
FIFO n
FIFO 1
FIFO 0
...
round robin
Traffic control for testing?
Network emulator(netem)
eth0 THE INTERNETnetem
bandwidth
latency packet loss
corrupt...
Testing with containers
container 1 container 2
eth0eth0
Testing framework
configure “netem” qdiscs:bandwidth, latency, packet drop...
The demo application
microservices-demo
https://github.com/microservices-demo/microservices-demo
Some micro-services
front-end Firefox
catalogue
ordersorders-db
payment
Kubernetes
Kubernetes objects- Kubernetes “Pods”
- Group of container(s) running together
- Kubernetes “Replication Sets”- Control the number of “Pods” replicas running
- Kubernetes “Service”- Directing the traffic to “Pods”
pod
pod
pod
service
Testing with traffic control in Kubernetes
Kubernetesnode 1
pod
pod
Kubernetesnode 2
pod
pod
tc tccontrols∘ Latency∘ Bandwidth∘ Packet drop
∘ configure network simulator
∘ play scenarios
Weave Scope
Weave Scope
Testing with Weave Scope
Kubernetesnode 1
tcScopeProbe
pod pod
pod pod
KubernetesNode 2
tcScopeProbe
pod pod
pod pod
ScopeApp
Demo
Reproduce the demo yourself: https://github.com/kinvolk/demo
Plugins in Scope
tc pluginScopeProbe
ScopeApp
report
reportcontrol
control
- Unix socket in /var/run/scope/plugins/- Protocols: report and control- Write your own plugins for your testing needs
Testing frameworkfor web apps
Selenium
Testing more complex scenarios
(my “wishlist”)
Add latency on a specific connection
front-end Firefox
catalogue
ordersorders-db
payment
latency=100ms
How to define classes of traffic
eth0
netem
interface
latency=100ms
dest_ip=10.0.4.* dest_ip=10.0.5.* other
u32: filter on contenteth0
HTB
HTB
HTBHTB HTB
netemnetem netem
interface
root qdisc (type = HTB)
root class (type = HTB)
leaf qdiscs (type = netem)
leaf classes (type = HTB)
filters (type=u32)
otherip=10.0.5.*ip=10.0.4.*
latency=10ms
Filtering with cBPF/eBPF
eth0
BPF
netemnetem
kernel
userspace
BPF_JMP...BPF_LD...BPF_RET...
if (skb->protocol…) return TC_H_MAKE(TC_H_ROOT, mark); compilation
clang... -march=bpf
uploadin the kernel:
- bpf()- Netlink
x86_64 codeJIT compilation
eBPF maps
eth0
BPF
netemnetem
kernel
userspace
x86_64 code
eBPF map
tc
∘ Build statistics∘ Make them available to
the testing framework
The EndTry the demos yourself: https://github.com/kinvolk/demo
Read more: https://kinvolk.io/blog/
The slides: https://goo.gl/ZLLlv3
Questions?