high performance user-level sockets over gigabit ethernet
DESCRIPTION
High Performance User-Level Sockets over Gigabit Ethernet. Pavan Balaji Ohio State University [email protected]. Piyush Shivam Ohio State University [email protected]. Pete Wyckoff Ohio Supercomputer Center [email protected]. D.K. Panda Ohio State University - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/1.jpg)
High PerformanceUser-Level Sockets over
Gigabit Ethernet
Pavan BalajiOhio State University
Piyush ShivamOhio State University
D.K. PandaOhio State University
Pete WyckoffOhio Supercomputer Center
![Page 2: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/2.jpg)
Presentation Overview
Background and Motivation
Design Challenges
Performance Enhancement Techniques
Performance Results
Conclusions
![Page 3: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/3.jpg)
Background and MotivationSockets
Frequently used APITraditional Kernel-Based Implementation
Unable to exploit High Performance NetworksEarlier Solutions
Interrupt Coalescing Checksum Offload Insufficient
It gets worse with 10 Gigabit NetworksCan we do better
User-level support
![Page 4: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/4.jpg)
Kernel Based Implementation of Sockets
NIC
IP
TCP
Sockets
Application or Library
Hardware
Kernel
User Space Pros• High Compatibility
Cons• Kernel Context Switches• Multiple Copies• CPU Resources
![Page 5: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/5.jpg)
Alternative Implementations of Sockets (GigaNet cLAN)
“VI aware” NIC
IP
TCP
Sockets
Application or Library
Hardware
Kernel
User Space Pros• High Compatibility
Cons• Kernel Context Switches• Multiple Copies• CPU Resources
IP-to-VI layer
![Page 6: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/6.jpg)
Sockets over User-Level Protocols Sockets is a generalized protocol
Sockets over VIA Developed by Intel Corporation [shah98] and ET Research
Institute [sovia01] GigaNet cLAN platform
Most networks in the world are Ethernet Gigabit Ethernet
Backward compatible Gigabit Network over the existing installation base MVIA: Version of VIA on Gigabit Ethernet
Kernel Based A need for a High Performance Sockets layer over
Gigabit Ethernet
![Page 7: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/7.jpg)
User-Level Protocol over Gigabit Ethernet
Ethernet Message Passing (EMP) Protocol Zero-Copy OS-Bypass NIC-driven User-Level
protocol over Gigabit Ethernet Developed over the Dual-processor Alteon NICs Complete Offload of message passing functionality to
the NIC
• Piyush Shivam, Pete Wyckoff, D.K. Panda, “EMP: Zero-Copy OS-bypass NIC-driven Gigabit Ethernet Message Passing”, Supercomputing, November ’01
• Piyush Shivam, Pete Wyckoff, D.K. Panda, “Can User-Level Protocols take advantage of Multi-CPU NICs?”, IPDPS, April ‘02
![Page 8: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/8.jpg)
EMP: Latency
0
50
100
150
200
250
4 8 16 32 64 128 256 512 1K 2K 4K
Message Size (bytes)
Late
ncy
(us)
TCPEMP
A base latency of 28s compared to an ~120 s of TCP for 4-byte messages
![Page 9: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/9.jpg)
EMP: Bandwidth
0
200
400
600
800
1000
1200
4 8 16 32 64 128 256 512 1K 2K 4K 8K
Message Size (bytes)
Ban
dwid
th (M
bps)
EMPTCP
Saturated the Gigabit Ethernet network with a peak bandwidth of 964Mbps
![Page 10: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/10.jpg)
Proposed Solution
Gigabit Ethernet NIC
Sockets over EMP
Application or Library
Hardware
Kernel
User Space
• Kernel Context Switches• Multiple Copies• CPU Resources• High Performance
OS Agent
EMP Library
![Page 11: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/11.jpg)
Presentation Overview
Background and Motivation
Design Challenges Performance Enhancement Techniques
Performance Results
Conclusions
![Page 12: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/12.jpg)
Design Challenges
Functionality Mismatches
Connection Management
Message Passing
Resource Management
UNIX Sockets
![Page 13: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/13.jpg)
Functionality Mismatches and Connection Management
Functionality MismatchesNo API for buffer advertising in TCP
Connection ManagementData Message Exchange
Descriptors required for connection
management
![Page 14: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/14.jpg)
Message PassingMessage Passing
Data StreamingParts of the same message can be read
potentially to different buffersUnexpected Message Arrivals
Separate Communication Thread• Keeps track of used descriptors and re-posts• Polling Threads have high Synchronization cost• Sleeping Threads involve OS scheduling granularity
Rendezvous ApproachEager with Flow Control
![Page 15: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/15.jpg)
Rendezvous Approach
Sender ReceiverSQ
RQ
SQ
RQ
send()
receive()
Request
ACK
Data
![Page 16: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/16.jpg)
Eager with Flow Control
Sender ReceiverSQ
RQ
SQ
RQ
send()
Data
ACK
Data
receive()
![Page 17: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/17.jpg)
Resource Management and UNIX Sockets
Resource ManagementClean up unused descriptors (connection
management)Free registered memory
UNIX SocketsFunction OverridingApplication ChangesFile Descriptor Tracking
![Page 18: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/18.jpg)
Presentation Overview
Background and Motivation
Design Challenges
Performance Enhancement Techniques Performance Results
Conclusions
![Page 19: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/19.jpg)
Performance Enhancement Techniques
Credit Based Flow Control
Disabling Data Streaming
Delayed Acknowledgments
EMP Unexpected Queue
![Page 20: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/20.jpg)
Credit Based Flow Control
Sender ReceiverSQ
RQ
SQ
RQCredits Left: 4Credits Left: 3Credits Left: 2Credits Left: 1Credits Left: 0Credits Left: 4
• Multiple Outstanding Credits
![Page 21: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/21.jpg)
Non-Data Streaming and Delayed Acknowledgments
Disabling Data Streaming Intermediate copy required for Data Streaming Place data directly into user buffer
Delayed Acknowledgments Increase in Bandwidth
Lesser Network Traffic NIC has lesser work to do
Decrease in Latency Lesser descriptors posted Lesser Tag Matching at the NIC
550ns per descriptor
![Page 22: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/22.jpg)
EMP Unexpected Queue
EMP Unexpected QueueEMP features unexpected message queue
Advantages: Last to be checkedDisadvantage: Data Copy
Acknowledgments in the Unexpected Queue
No copy, since acknowledgments carry no dataAcknowledgments pushed out of the critical
path
![Page 23: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/23.jpg)
Presentation Overview
Background and Motivation
Design Challenges
Performance Enhancement Techniques
Performance Results Conclusions
![Page 24: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/24.jpg)
Performance Results
Micro-benchmarksLatency (ping-pong)Bandwidth
FTP ApplicationWeb Server
HTTP/1.0 SpecificationsHTTP/1.1 Specifications
![Page 25: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/25.jpg)
Experimental Test-bed
Four Pentium III 700Mhz Quads
1GB Main Memory
Alteon NICs
Packet Engine Switch
Linux version 2.4.18
![Page 26: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/26.jpg)
Micro-benchmarks: Latency
0
50
100
150
200
250
Message Size (bytes)
Late
ncy
(us) TCP
Data StreamingNon-Data StreamingEMP
Up to 4 times improvement compared to TCP
Overhead of 0.5us compared to EMP
![Page 27: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/27.jpg)
Micro-benchmarks: Bandwidth
0100200300400500600700800900
4 8 16 32 64 128
256
512
1K 2K 4K 8K 16K
32K
64K
128
Message Size (bytes)
Ban
dwid
th (M
bps)
Data StreamingNon-Data StreamingTCPEnhanced TCP
An improvement of 53% compared to enhanced TCP
![Page 28: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/28.jpg)
FTP Application
0
2
4
6
8
10
12
Tran
sfer
Tim
e (s
ecs)
1 4 16 64 256File Size (Mbytes)
Data StreamingNon-Data StreamingTCP
Up to 2 times improvement compared to TCP
![Page 29: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/29.jpg)
Web Server (HTTP/1.0)
0
2000
4000
6000
8000
10000
12000
Response Size (bytes)
Tran
sact
ions
per
sec
ond
TCPData StreamingNon-Data Streaming
Up to 6 times improvement compared to TCP
![Page 30: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/30.jpg)
Web Server (HTTP/1.1)
02000400060008000
1000012000140001600018000
Response Size (bytes)
Tran
sact
ions
per
sec
ond
TCPData StreamingNon-Data Streaming
Up to 3 times improvement compared to TCP
![Page 31: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/31.jpg)
Conclusions Developed a High Performance User-Level
Sockets implementation over Gigabit Ethernet Latency close to base EMP (28 s)
28.5 s for Non-Data Streaming 37 s for Data Streaming sockets 4 times improvement in latency compared to TCP
Peak Bandwidth of 840Mbps 550Mbps obtained by TCP with increased Registered space
for the kernel (up to 2MB) Default case is 340Mbps with 32KB Improvement of 53%
![Page 32: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/32.jpg)
Conclusions (contd.)
FTP Application shows an improvement of nearly 2 times
Web Server shows tremendous performance improvement HTTP/1.0 shows an improvement of up to 6 times HTTP/1.1 shows an improvement of up to 3 times
![Page 33: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/33.jpg)
Future WorkDynamic Credit Allocation
NIC: The trusted component Integrated QoS
Currently on Myrinet ClustersCommercial applications in the Data
Center environmentExtend the idea to next generation
interconnects InfiniBand 10 Gigabit Ethernet
![Page 34: High Performance User-Level Sockets over Gigabit Ethernet](https://reader036.vdocument.in/reader036/viewer/2022062323/56816386550346895dd4706f/html5/thumbnails/34.jpg)
For more information, please visit the
http://nowlab.cis.ohio-state.eduNetwork Based Computing Laboratory,
The Ohio State University
Thank You
NBC Home Page