performance evaluation of gigabit ethernet-based interconnects for hpc clusters
Post on 08-Jan-2016
36 Views
Preview:
DESCRIPTION
TRANSCRIPT
Cracow, 16 October 2006
Performance Evaluation of Gigabit Ethernet-Based
Interconnects for HPC Clusters
Paweł Pisarczyk pawel.pisarczyk@atm.com.pl
Jarosław Węgliński jaroslaw.weglinski@atm.com.pl
Agenda
Introduction
HPC cluster interconnects
Message propagation model
Experimental setup
Results
Conclusions
Who we are
joint stock company
founded in 1994, earlier (since 1991) as a departmentwithin PP ATM
IPO in September 2004 (Warsaw Stock Exchange)
major shares owned by founders (Polish citizens)
no state capital involved
financial data
stock capital about €6 million
2005 sales €29,7 million
about 230 employees
Mission
building business value through innovative information & communication technology initiatives creating new markets in Poland and abroad
ATM's competitive advantage is based on combining three key competences:
integration of comprehensive IT systems
telecommunication services
consulting and software development
Achievements
1991 Poland’s first company connected to Internet
1993 Poland’s first commercial ISP
1994 Poland’s first LAN with ATM backbone
1994 Poland’s first supercomputer on the Dongarra’s Top 500 list
1995 Poland’s first MAN in ATM technology
1996 Poland’s first corporate network with voice & data integration
2000 Poland’s first prototype Interactive TV system over a public network
2002 Poland’s first validated MES system for a pharmaceutical factory
2003 Poland’s first commercial, public Wireless LAN
2004 Poland’s first public IP content billing system
Client base
telecommunications40,4%
finance17,7%
academia9,6%
media5,9%
utilities5,7%
manufacturing4,9%
public4,4%
other11,4%
(based on 2005 sales revenues)
HPC clusters developed by ATM
2004 - Poznan Supercomputing and Networking Center238 Itanium2 CPU, 119 x HP rx2600 nodes with Gigabit Ethernet interconnect
2005 - University of Podlasie 34 Itanium2 CPU,17 x HP rx2600 nodes with Gigabit Ethernet interconnect and Lustre 1.2 filesystem
2005 - Poznan Supercomputing and Networking Center86 dual core Opteron CPU, 42 Sun SunFire v20z and 1 Sun SunFire v40z with Gigabit Ethernet interconnect
2006 - Military University of Technology Faculty of Engineering, Chemistry and Applied Physics
32 Itanium2 CPU, 16 x HP rx1620 with Gigabit Ethernet interconnect
2006 Gdansk University of Technology: Department of Pharmaceutial Technology and Chemistry
22 Itanium2 CPU (11 x HP RX1620) with Gigabit Ethernet interconnect
Selected software projects related to distributed systems
Distributed Multimedia Archive in Interactive Television (iTVP) Project
scalable storage for iTVP platform with ability to process the stored content
ATM Objectsscalable storage for multimedia content distribution platform
system for Cinem@n company (founded by ATM and Monolith)
Cinem@n will introduce high-quality movies, news and entertainment digital content distribution services
Spread Screens Managerplatform for POS TV
system is currently used by Zabka (shopping network) and Neckermann (travel service)
about 300 of terminals presenting the multimedia content located in many polish cities
Selected current projects
ATMFSdistributed filesystem for petabyte scale storage based on COTS
based on variable-sized chunks
advanced replication and enhanced error detection
dependability evaluation based on software fault injection technique
FastGigRDMA stack for Gigabit Ethernet-based interconnect
message passing latency reduction
increases the application performance
Uses of computer networks in HPC clusters
Exchange of messages between cluster nodes to coordinate distributed computation
requires high maximal throughput and also low latency
inefficiency observed when the time consumed in single computation step is comparable to the message passing time
Access to shared data through network or cluster file system
requires high bandwidth when transferring data in blocks of defined size
filesystem and storage drivers are trying to reduce number of i/o operations issued (by buffering data and aggregating transfers)
Comparison of characteristics of interconnect technologies
* Brett M. Bode, Jason J. Hill, and Troy R. Benjegerdes “Cluster Interconnect Overview” Scalable Computing Laboratory, Ames Laboratory
Gigabit Ethernet interconnect characteristic
Popular technology for low cost cluster interconnects
Satisfied throughput for long frames (1000 bytes and longer)
High latency and low throughput for small frames
Those drawbacks are mostly caused by construction of existing network interfaces
What is the influence of the network stack implementation for the communication latency?
Message propagation model
Latency between transferring message to/from MPI library and transferring data to/from stack
Time difference between sendto/recvfrom function and driver start_xmit/interrupt functions
Execution time of driver functions
Processing time of the network interface
Propagation latency and latency introduced by active network elements
Experimental setup
Two HP rx2600 servers2 x Intel Itanium2 1.3 MHz 3MB cache
Debian GNU/Linux Sarge 3.1 operating system (kernel 2.6.8-2-mckinley-smp)
Gigabit Ethernet interfacesBroadcom BCM5701 chipset connected using PCI-X device bus
In order to eliminate possibility of additional delays, which may be introduced by external active network devices, servers were connected using crossover cables
Two NIC drivers were tested: tg3 (polling NAPI dirver), bcm5700 (interrupt driven driver)
Tools used for measures
NetPipe package for measuring throughput and latency for TCP and several implementations of MPI
For low level testing test programs working directly on Ethernet frames were developed
Testing programs and NIC drivers were modified to allow measuring, inserting and transfer of timestamps
Throughput characteristic for tg3 driver
Latency characteristic for tg3 driver
Results for tg3 driver
The overhead introduced by MPI library is relatively low
There is a big difference between transmission latencies in the ping-pong and streaming mode
The latency introduced for small frames is similar to latency introduced by 115kbps UART (in the case of transmitting one byte only)
We can deduce that there is some mechanism in the transmission path that delays transmission of single packets
What is the difference between NAPI and interrupt driven driver?
Interrupt driven driver vs NAPI driver (throughput characteristic)
Interrupt driven driver vs NAPI driver (latency characteristic)
Interrupt driven driver vs NAPI driver (latency characteristic) - details
Comparison of bcm5700 and tg3 drivers
Using default configuration, BCM5700 driver has worse characteristics than tg3
Interrupt driven version (default configuration) cannot achieve more than 650Mb/s of throughput for frames of any size
After interrupt coalescing disabling, the performance of BCM5700 driver have exceeded the results obtained by tg3 driver
Disabling of the polling can improve characteristics of the network driver, but NAPI is not the major cause of the transmission delay
Tools for message processing time measurement
Timestamps were inserted into the message eat each processing stage
Processing stages on the transmitter sidesendto() function
bcm5700_start_xmit() interrupt notifying frame transmit
Processing stages on the receiver sideinterrupt notifying frame receipt
netif_rx()
recvfrom() function
As high precision timer CPU clock cycle counter was used, (precision of 0.77ns = 1/1.3GHz)
Transmitter latency in streaming mode
Send Answer
17 us 17 us2 us
Distribution of delays in transmission path between cluster nodes
Conclusions
We estimate that RDMA based communication can reduce MPI message propagation time from 43μs to 23μs (doubling the performance for short messages)
There is also possibility of reducing T3 and T5 latencies by changing the configuration of the network interface (transmit and receive thresholds)
In the conducted research we didn’t consider differences between network interfaces (T3 and T5 delays may be longer or shorter than measured)
Latency introduced by switch is also omitted
FastGig project include not only communication library, but also measurement and communication profiling framework
top related