march 5, 2008 kangho kim system software team internet server group vee ‘08

Inter-domain Socket Communications Supporting High Performance

and Full Binary Compatibility on Xen

March 5, 2008

Kangho KimSystem Software TeamInternet Server Group

VEE ‘08

Outline

Motivation Design & Implementation

– Socket creation– Connect– Accept– Send/recv– Close– Live migration

Evaluation results Conclusion

Motivation

Low bandwidth (Xen 3.0.3)

We need XWAY!!

Motivation(cont’d)

Low latency (Xen 3.0.3)

Existing socket applications

Design & Implementation

Shared memory

Direct communication TCP/IP bypass Direct data copy

Shared memory

Hardware

VM VM VM

Ker

nel

Use

r

Domain 0 Domain U Domain U

XWAY Shared memory

TCP

XWAY channel

Head Tail

SQ

Head Tail

RQ

Head Tail

RQ

Event channel

Head Tail

SQ

Shared memoryfor one XWAY channel

Domain A Domain B

XWAY architecture

socket application

library

VFS

INET

UNIX X25…XWAY switch

TCP

IP

XWAYprotocol

Network device driverXWAY driver

BSD socket

user

kernel

UDP

Dual channel

XWAY channel for data TCP/IP channel for socket options and socket state

socket application

XWAY socket

socket application

XWAY socket

TCP/IP channel

XWAY channel

xway_sock

Socket data structure for XWAY

llseek: no_llseekpoll: sock_pollopen: sock_no_openrelease: sock_closereadv: sock_readvwritev: sock_writevsendpage: sock_sendpage

release: inet_releasebind: inet_bindconnect: xsw_stream_connectaccept: xsw_acceptpoll: xsw_polllisten: inet_listenshutdown: inet_shutdownsendmsg: inet_sendmsgrecvmsg: sock_common_recvmsgsendpage: inet_sendpage

close: xsw_closeconnect: xsw_v4_connectdisconnect: xsw_disconnectaccept: xsw_csk_acceptioctl: tcp_ioctlinit: xsw_v4_init_sockshutdown: xsw_shutdownsetsockopt: tcp_setsockoptgetsockopt: tcp_getsockopt

remote_dom_ideventchn_porteventchn_irqxring_sendxring_recvxring_send_hxring_send_txring_recv_hxring_recv_t…

inode

socket

sock

Create

Sock

xway_sock

llseek: no_llseekpoll: sock_pollopen: sock_no_openrelease: sock_closereadv: sock_readvwritev: sock_writevsendpage: sock_sendpage

release: inet_releasebind: inet_bindconnect: inet_stream_connectaccept: inet_acceptpoll: tcp_polllisten: inet_listenshutdown: inet_shutdownsendmsg: inet_sendmsgrecvmsg: sock_common_recvmsgsendpage: inet_sendpage

close: xs_closeconnect: xs_v4_connectdisconnect: xs_disconnectaccept: xs_csk_acceptioctl: xs_ioctlinit: xs_v4_init_sockshutdown: xs_shutdownsetsockopt: xs_setsockoptgetsockopt: xs_getsockopt

inode

socket

sock

Connect

TCP channel

Sock Sock

Connectionhelper

XWAY channelSQ

RQ

SQ

RQ

Accept

TCP channel

XWAY channel

Sock Sock

XWAY channel

XWAY channel

XWAY channel

Send/Recv

TCP channel

Sock Sock

XWAY channelSQ

RQ

SQ

RQ

CloseSock Sock

TCP channelTCP channel

XWAY channelSQ

RQ

SQ

RQ

XWAY channel

Poll

TCP session

XWAY session

Sock Sock

Live migration

Cut XWAY channels underneath XWAY socket– Redirect all data send/receive calls to TCP channels

Ready to live migration

Hardware

VM VM VM

Ker

nel

Use

r

Domain 0 Domain U Domain U

TCP/IP

Hardware

VM

Domain 0

Live migration (cont’d)

TCP channel

Sock Sock

XWAY channelSQ

RQ

SQ

RQ

Evaluation

Bandwidth

Netperf TCP_STREAM

Latency

Netperf (TCP_RR)

Application performance

DBT-1 performance

Execution time of API

Binary compatibility

Scp Ssh Apache – TCP_DEFER_ACCEPT Vsftpd – connect, accept, MSG_PEEK Proftpd – shutdown, connect, SO_LINGER Telnet MySQL Netperf

Conclusion

XWAY design and implementation– Full binary compatibility for socket interface

– High performance

– Live migration support Remaining work

– UDP, …

– Domain crash resiliency

– Reducing CPU usage Code release

– April 20, 2007: xway ver 0.6

– June 4, 2007: xway ver 0.6.1

– By the end of March, 2008: xway ver 0.7 (live migration included) Download

– http://sourceforge.net/projects/xway

Questions or Comments

Kangho Kim– khk@etri.re.kr

Cheiyol Kim– gauri@etri.re.kr

Hyun-Sup Shin– superstarsup@etri.re.kr