os*: osv on *
TRANSCRIPT
OS*: OSv on *
@ysmoo
BitVisor
BitVisor
Hardware
OS
App App App
http://www.justis.as-1.co.jp
https://bitbucket.org/bitvisor/bitvisor
OSv
VMM
Hardware
OSvApp
http://medical-care.feed.jp
https://github.com/cloudius-systems/osv
BitVisor
BitVisor OSv
OSv on BitVisor
BitVisor
Hardware
OSvApp
OSv on BitVisor
BitVisor
Hardware
OSvApp
OSb
OSb Design
BitVisor
Hardware
OSvApp
Virtio NIC
With a small fix,OSv can handle chipsets
OSv has an AHCI driver
BitVisor provides Virtio NIC
OSb Design
BitVisor
Hardware
OSvApp
Virtio NIC
With a small fix,OSv can handle chipsets
OSv has an AHCI driver
BitVisor provides Virtio NIC
One thing to fix
Skipped the code!(drivers/acpi.cc:early_init())
OSb Design
BitVisor
Hardware
OSvApp
Virtio NIC
With a small fix,OSv can handle chipsets
OSv has an AHCI driver
BitVisor provides Virtio NIC
Para Pass-throughVirtio
BitVisor
Hardware
Virtio Ring
PRO/1000 Ring
OSvApp
MSI-X etc.
Handle minimal operations.
PCI Function
Pass-through!
For more details, see:http://www.slideshare.net/yushiomote/osb-osv-on-bitvisor-2
for (12-billion times) val++;
0.00
6.50
13.00
19.50
26.00
OSb Linux (Host, Bare-metal)
25.51
20.75
Elap
sed
Tim
e (s
ec)
Netperf TCP_STREAM
0.00
237.50
475.00
712.50
950.00
OSb Linux (Host, Bare-metal)
941.39921.83
Thr
ough
put
(Mbp
s)
Ping
0.00
0.04
0.08
0.12
0.16
OSb Linux (Host, Bare-metal)
0.16
0.12
Late
ncy
(mse
c)
Thinner and thinner
Hardware
KVM/Linux
OSv
Hardware
BitVisor
OSv12.8M LOC
0.3M LOC
virtio
sched.
driver
http://cdn-ak.f.st-hatena.com/images/fotolife/j/jovi1kamiya/20140607/20140607114847.jpg
Forultimate thinness…
OSv on Physical Machine
Hardware
KVM/Linux
OSv
Hardware
BitVisor
OSv
Hardware
OSv
OSp
OSp Design
Hardware
OSv
App
Physical NICDriver
ImplementedIntel X540 10GbE Driverin C++11
With a small fix,OSv can handle chipsets
OSv has an AHCI driver
About implementation
• Intel NIC TX/RX Basics
• Device initialization
• TX initialization/operations
• RX initialization/operations
Intel NIC TX/RX Basics
Descriptor 0
Descriptor 2
Descriptor 3
Descriptor 4
Descriptor 5
Descriptor 6
Descriptor 7
Ring Buffer
Descriptor 1
Producer I/O Register Consumer I/O Register
OSv
I/O Registers
Intel NIC TX Basics
Descriptor 0
Descriptor 2
Descriptor 3
Descriptor 4
Descriptor 5
Descriptor 6
Descriptor 7
Ring Buffer
Descriptor 1
Producer I/O Register Consumer I/O Register
OSv
I/O Registers
Intel NIC TX Basics
Descriptor 0
Descriptor 2
Descriptor 3
Descriptor 4
Descriptor 5
Descriptor 6
Descriptor 7
Ring Buffer
Descriptor 1
Producer I/O Register Consumer I/O Register
OSv
I/O Registers
Intel NIC TX Basics
Descriptor 0
Descriptor 2
Descriptor 3
Descriptor 4
Descriptor 5
Descriptor 6
Descriptor 7
Ring Buffer
Descriptor 1
Producer I/O Register Consumer I/O Register
OSv
I/O Registers
Xmit!
Intel NIC RX Basics
Descriptor 0
Descriptor 2
Descriptor 3
Descriptor 4
Descriptor 5
Descriptor 6
Descriptor 7
Ring Buffer
Descriptor 1
Consumer I/O Register Producer I/O Register
OSv
I/O Registers
Recv!
Device Init.
TX Init.
TX Interface(OSv => Driver)
• OSv’s upper layer (TCP/IP) calls:
• int try_xmit_one_locked(void *mbuf_etc)
• Then, driver puts mbuf on ring buffer.
• bool kick_hw()
• Then, driver updates producer reg.
• …
EOP EOP
mbuf
try_xmit_one_locked()to put mbuf on ring buffer
kick_hw()to update producer reg.
RX Init.
RX Interface(Driver => OSv)
• struct ifnet *_ifn = if_alloc (IFT_ETHER);
• On init., driver creates ifnet and…
• ether_ifattach(_ifn, macaddr);
• …passes it to the upper layer. (Then, the upper layer puts an callback function on ifnet.if_input)
• (*_ifn->if_input)(_ifn, mbuf);
• On packet reception, driver calls the callback function (if_input) by passing mbuf as an argument.
Under improvement(A lot of TODOs)
• No interrupts… (bad polling impl.)
• No TSO, LRO, checksum assist…
• No advanced descriptor… (legacy descriptor seems slow)
• …
Netperf TCP_STREAM
0.00
2000.00
4000.00
6000.00
8000.00
OSp OSv
7229.00
4200.00
Thr
ough
put
(Mbp
s)
Future Work
0.00
2000.00
4000.00
6000.00
8000.00
OSp OSv
7229.00
4200.00
Thr
ough
put
(Mbp
s)Revenge of the OSp
One more thing
KVM/Linux
Hardware
OSvApp
./scripts/bindctrl bind 8086 1528 0000:07:00.0
./scripts/run.py -a 07:00.0
Additional scripts & optionsto start OSv withPCI pass-through config.
Output$ sudo ./scripts/run.py -n -v -c 8 -a 07:00.0ixgbe [00:04.0] 8086:1528Address: a0:36:9f:14:5f:84PHY Status: State: Connected Type: MDI-X Speed: 10Gbps Duplex: FullMAC Status: Link: UP Speed: 10Gbpsixgbe [00:04.0] 8086:1528eth0: 192.168.122.15ixgbe0: 192.168.100.2/#/#/#/# exitGoodbye
Summary
OSv → OSb → OSp
References
• The Ugly Patch of Intel X540 10GbE Driver and Additional scripts for PCI Pass-through Configuration
• http://goo.gl/0yRcV4
• git diff -r 5e5c227b2aec23418e5e3fd104cc21e98f31729c
• Usage & True Motivation
• http://www.yushiomote.org/blog/?p=428