the analysis of microburs (burstiness) on virtual switch
Post on 14-Apr-2017
50 Views
Preview:
TRANSCRIPT
Copyright 2016 FUJITSU LABORATORIES LIMITED
The Analysis of Microburst (Burstiness) on Virtual Switch
Chunghan Lee Fujitsu Laboratories
09.19.2016
Background
What is Network Function Virtualization (NFV) ?
NFV virtualizes network functions (e.g. Firewall, Load balancer) on IA servers
•The function is called as virtualized network function (VNF)
•Virtual switch is one of major components for virtual network on NFV Infrastructure (NFVI)
•QoS property is also important for NFV
1 Copyright 2014 FUJITSU LABORATORIES LIMITED
An image is from http://www.etsi.org/deliver/etsi_gs/NFV/001_099/002/01.01.01_60/gs_NFV002v010101p.pdf
Virtual network*
*Virtual switch is included in virtual network
Microburst – (1) What is microburst ?
Spikes in shot time period, but causing decreased performance
Impact of microburst on network
Although traffic with QoS property is generated, the spikes can be occurred by massive traffic in very short time period
•This phenomenon cannot catch the ordinary monitoring (SNMP)
•When packet drop is occurred by the spike, it is changed as the microburst
2 Copyright 2016 FUJITSU LABORATORIES LIMITED
Sudden spikes in throughput
Packet drop !!
SNMP (Coarse-grained)
Packet (Find-grained)
Spike
Microburst – (2)
Impact of microburst on NFVI
Although traffic with QoS property is generated on VNF, the sudden spikes would be occurred due to resource state
Packet loss can be occurred by the spikes (microbursts)
3 Copyright 2016 FUJITSU LABORATORIES LIMITED
Traffic with QoS
Sudden spikes !!
NFVI (tx) NFVI (rx)
Problem
Microburst on NFVI
Ordinary application measurement cannot catch the microbursts on NFVI
There are queues and buffers between NFVIs
A deep understanding of packet processing issued by virtual switch and Linux kernel is required to clarify the cause of microburst
4 Copyright 2016 FUJITSU LABORATORIES LIMITED
Traffic with QoS
Sudden spikes !!
NFVI (tx) NFVI (rx)
GOAL
Investigate the occurrence of microburst on NFVI
Find the cause of microbursts on NFVI
5 Copyright 2014 FUJITSU LABORATORIES LIMITED
Approach
Prepare two types of UDP traffic to observe the microbursts on an OvS bridge
Foreground : Target with QoS property
Background : Occur a lack of CPU resource at kernel
Generate fore/background traffic and capture their packets with tcpdump
Analyze them in packet-level to observe the microburst
Profile kernel functions to clarify the cause of microbursts
6 Copyright 2016 FUJITSU LABORATORIES LIMITED
Overview of testbed Foreground traffic (UDP)
Sending rate : 2Gbps [Datagram size : 1400 bytes (No fragmentation)]
vport tx queue : default value (0), UDP buffer : default value (200 KBytes)
Background traffic (UDP)
8 vports are used on the same OVS bridge
iperf using UDP mode with 10 parallel flows per vport (iperf option –p)
Sending rate of UDP flow : 1Gbps (configured bandwidth by iperf)
7 Copyright 2016 FUJITSU LABORATORIES LIMITED
Server and switch spec.
Overview of spec.
All servers have the same spec.
Switch spec.
•Fujitsu SR-X 526 (10G switch)
8
Server type : Fujitsu RX100S7
CPU (4 cores) Intel(R) Xeon(R) CPU E31220 @ 3.10GHz
Memory 16GB (Speed: 1333 MHz)
OS CentOS 7.2
Kernel version 3.18.25
NIC 10G:Intel X710
iperf 2.0.8
Open vSwitch 2.4.0 (release version)
vport queue Default value (txqueuelen : 0)
UDP buffer Default value (200 Kbytes)
Copyright 2016 FUJITSU LABORATORIES LIMITED
Investigation of microburst
9 Copyright 2016 FUJITSU LABORATORIES LIMITED
Measurement points
6 throughput measurement points
Sender : iperf, Entering OVS, Leaving OVS
Receiver : iperf, Entering OVS, Leaving OVS
10 Copyright 2016 FUJITSU LABORATORIES LIMITED
iperf User
Kernel
NIC driver
< Sender > < Receiver >
iperf
OVS
(bridge)
NIC driver
OVS
(bridge)
Network
iperf (RX) iperf (TX)
Entering OVS
(tcpdump)
Leaving OVS
(tcpdump)
Throughput measurement points
Entering OVS
(tcpdump)
Leaving OVS
(tcpdump)
Throughput at 6 measurement points Throughput of foreground per second
Leaving OVS (sender), the throughput is decreased
At receiver, the throughput is also decreased at iperf (RX)
11 Copyright 2016 FUJITSU LABORATORIES LIMITED
< Sender >
< Receiver >
Decreased throughput
Decreased throughput
Throughput of foreground at sender Throughput of foreground per 1 millisecond
Entering OVS, the sending rate is fluctuated due to a lack of CPU
Leaving OVS, sudden spikes in throughput are found
•Throughput with QoS property is changed
12 Copyright 2016 FUJITSU LABORATORIES LIMITED
< Entering OVS >
< Leaving OVS >
Unit time : 1 millisecond
NIC driver
iperf
OVS (bridge)
< Sender >
Entering
OVS
Leaving
OVS
Network
Packet drop and throughput at receiver A relation between throughput and packet drop
The timing of packet drop is similar to the timing of sudden spikes in throughput at receiver
The overflow of socket buffer is frequently occurred by the sudden spikes (spikes → microbursts)
13 Copyright 2016 FUJITSU LABORATORIES LIMITED
< Packet drop at socket >
< Leaving OVS >
NIC driver
iperf
OVS (bridge)
< Receiver >
Leaving
OVS
Network
Socket
Packet drop and spacing at receiver A relation between packet drop and packet spacing
The packet spacing with moving average (MA)* is decreased while the number of drop packets on socket buffer is increased
14 Copyright 2016 FUJITSU LABORATORIES LIMITED
*MA leg : 1000
Packet spacing Burstiness Packet loss
↓ ↑ ↑
↑ ↓ ↓
Microburst
NIC driver
iperf
OVS (bridge)
< Receiver >
Leaving
OVS
Network
Socket
Cause of microburst
15 Copyright 2016 FUJITSU LABORATORIES LIMITED
Find the cause of microburst
Profiling kernel functions using perf
Profiling rate (sampling rate) is 1 millisecond
Focus on process (iperf with 2Gbps) only for the profiling
16 Copyright 2016 FUJITSU LABORATORIES LIMITED
Profiling Kernel by perf
Common function call graph
Packet processing at Linux kernel
17 Copyright 2016 FUJITSU LABORATORIES LIMITED
Net I/F
Enqueue
(qdisc, tx-ring)
TCP/UDP layer (TX)
iperf
OVS (bridge)
IP layer (TX)
Kernel
User
From vport
to OVS
Copy data
From OVS
to qdisc
Measurement points with TCP/IP stack
Leaving OVS, the packet capture is occurred after qdisc
18 Copyright 2016 FUJITSU LABORATORIES LIMITED
Entering OVS
Leaving OVS Entering OVS
Leaving OVS
Throughput measurement points
Summary Throughput of foreground per second
Leaving OVS (sender), the throughput is decreased
At receiver, the throughput is also decreased at iperf (RX)
19 Copyright 2016 FUJITSU LABORATORIES LIMITED
< Sender >
< Receiver >
Packet drop at qdisc
Packet drop at socket buffer by microbursts
Conclusion
We investigated the occurrence of microbursts on NFVI A major cause of microburst is packet queuing on qdisc, and
the packet loss at socket buffer on receiver is occurred by the microbursts
At qdisc, the throughput is decreased to 63% and the queue size is not enough to absorb the packets
At socket buffer (rx), the throughput is decreased to 41% and the buffer capacity is also not enough
We found the cause of microbursts using kernel profiling Although the total sending rate is 10 Gbps, qdisc at the sender
is frequently full
20 Copyright 2016 FUJITSU LABORATORIES LIMITED
Future work
Clarify the cause why qdisc is frequently full although the total sending rate is 10 Gbps
Analyze the profiling results with kernel trace
Modify Linux kernel to change the packet capture point
Extend the experiments
Without OVS (Clarify the overhead of OVS)
With TCP, DPDK OVS, VMs based on vhost
21 Copyright 2016 FUJITSU LABORATORIES LIMITED
Copyright 2015 FUJITSU LABORATORIES LIMITED 22
top related