rdma vs tcp experiment. goal environment test tool - iperf test suits conclusion
Post on 20-Dec-2015
228 views
TRANSCRIPT
![Page 1: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/1.jpg)
RDMA vs TCP experiment
![Page 2: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/2.jpg)
• Goal
• Environment
• Test tool - iperf
• Test Suits
• Conclusion
![Page 3: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/3.jpg)
Goal
• Test maximum and average bandwidth usage in 40Gbps(Infiniband) and 10Gbps(iWARP) network environment
• Compare CPU usage between TCP and RDMA data transfer mode
• Compare CPU usage between RDMA READ and RDMA WRITE mode
![Page 4: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/4.jpg)
Environment
40 Gbps Infiniband
10 Gbps iWARP
Netqos03/client Netqos04/server
![Page 5: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/5.jpg)
Tool - iperf• Migrate iperf 2.0.5 to the RDMA environment
with OFED(librdmacm and libibverbs).• 2000+ Source Lines of Code added.• From 8382 to 10562.
• iperf usage extended– -H: RDMA transfer mode instead of TCP/UDP– -G: pr(passive read) pw(passive write)
– Data read from server. – Server writes into clients.
– -O: output data file, both TCP server and RDMA server
• Only one stream to transfer
![Page 6: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/6.jpg)
Test Suits• test suits 1: memory -> memory
• test suits 2: file -> memory -> memory– test case 2.1: file(regular file) -> memory -> memory– test case 2.2: file(/dev/zero) -> memory -> memory– test case 2.3: file(lustre) -> memory -> memory
• test suits 3: memory -> memory -> file– test case 3.1: memory -> memory -> file(regular file)– test case 3.2: memory -> memory -> file(/dev/null)– test case 3.3: memory -> memory -> file(lustre)
• test suits 4: file -> memory -> memory -> file– test case 4.1: file ( regular file) -> memory -> memory -> file( regular file)– test case 4.2: file(/dev/zero) -> memory -> memory -> file(/dev/null)– test case 4.3: file(lustre) -> memory -> memory -> file(lustre)
![Page 7: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/7.jpg)
File choice• File operation with Standard I/O library– fread, fwrite, Cached by OS
• Input with /dev/zero wants to test the maximum application data transfer include file operation – read, which means disk is not the bottleneck
• Output with /dev/null wants to test the maximum application data transfer include file operation – write, which means disk is not the bottleneck
![Page 8: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/8.jpg)
Buffer choice
• RDMA operation block size is 10MB– RDMA READ/WRITE one time
– Previous experiment shows that, in this environment, if the block size more than 5MB, there is little effect to the transfer speed
• TCP read/write buffer size is the default
• TCP window size: 85.3 KByte (default)
![Page 9: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/9.jpg)
Test case 1: memory -> memory CPU
![Page 10: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/10.jpg)
Test case 1: memory -> memory Bandwidth
![Page 11: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/11.jpg)
Test case 2.1: (fread)file(regular file) -> memory -> memory CPU
![Page 12: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/12.jpg)
Test case 2.1: (fread)file(regular file) -> memory -> memory
Bandwidth
![Page 13: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/13.jpg)
Test case 2.2 (five minutes) file(/dev/zero) -> memory -> memory CPU
![Page 14: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/14.jpg)
Test case 2.2 (five minutes) file(/dev/zero) -> memory -> memory Bandwidth
![Page 15: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/15.jpg)
Test case 3.1 (200G file are generated): memory -> memory -> file(regular file) CPU
![Page 16: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/16.jpg)
Test case 3.1 (200G file are generated): memory -> memory -> file(regular file)
Bandwidth
![Page 17: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/17.jpg)
Test case 3.2: memory -> memory -> file(/dev/null) CPU
![Page 18: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/18.jpg)
Test case 3.2: memory -> memory -> file(/dev/null)
Bandwidth
![Page 19: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/19.jpg)
Test case 4.1:file(r) -> memory -> memory -> file(r) CPU
![Page 20: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/20.jpg)
Test case 4.1:file(r) -> memory -> memory -> file(r)
Bandwidth
![Page 21: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/21.jpg)
Test case 4.2:file(/dev/zero) -> memory -> memory ->
file(/dev/null) CPU
![Page 22: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/22.jpg)
Test case 4.2:file(/dev/zero) -> memory -> memory ->
file(/dev/null) Bandwidth
![Page 23: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/23.jpg)
Conclusion
I. For one data transfer stream, the RDMA transport is twice as fast as TCP, while the RDMA has only 10% of CPU load compare with the CPU load under TCP, without disk operation.
II. FTP includes two components: Networking and File operation. Compare with the RDMA operation, file operation (limited by the disk performance) takes most of the CPU usage. Therefore, a well-designed file buffer mode is critical.
![Page 24: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/24.jpg)
Future work
• Setup Lustre environment, and configure Lustre with RDMA function
• Startup FTP migrationo Source controlo Bug databaseo Documento etc (refer to The Joel Test)
![Page 25: RDMA vs TCP experiment. Goal Environment Test tool - iperf Test Suits Conclusion](https://reader036.vdocument.in/reader036/viewer/2022081511/56649d4d5503460f94a2c286/html5/thumbnails/25.jpg)
Memory Cached Cleanup
# sync
# echo 3 > /proc/sys/vm/drop_caches