the dds benchmarking environment james edmondson [email protected] vanderbilt university...
TRANSCRIPT
![Page 2: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/2.jpg)
2
Topics
• What is the DBE?• What technologies does the DBE use?• How does the DBE setup the environment? • What problems had to be dealt with? • What features will be added? • Demonstration• Conclusion and Wrap Up
![Page 3: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/3.jpg)
3
What is the DBE?
• DDS Benchmarking Environment• Sets up the testing environment and framework• Configures and launches implementation specific services• Launches Publishers and Subscribers• Coordinates interaction, logging info, and stats collection• Allows for easily scaling the number of participants• Sets up the statistics for collection by Bagatel
Thro
ughp
ut Publi
shers
NetworkingSubscribers
![Page 4: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/4.jpg)
4
What is the DBE?
• May be used to test other middleware or technologies• Focus is on documented, reusable resources
Thro
ughp
ut Publi
shers
NetworkingSubscribers
Versatility Dependability
![Page 5: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/5.jpg)
5
What does the DBE do?
Node 0
DBE
Node 1 Node 2 Node N
![Page 6: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/6.jpg)
6
Node 0
DBE
Node 1 Node 2 Node N
What does the DBE do?
![Page 7: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/7.jpg)
7
Node 0
DBE
Node 1 Node 2 Node N
Setup TestEnvironment
What does the DBE do?
![Page 8: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/8.jpg)
8
Node 0
DBE
Node 1 Node 2 Node N
Setup TestEnvironment
What does the DBE do?
![Page 9: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/9.jpg)
9
What does the DBE do?
Node 0
DBE
Node 1 Node 2 Node N
Setup TestEnvironment
ConfigureImplementation
![Page 10: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/10.jpg)
10
What does the DBE do?
Node 0
DBE
Node 1 Node 2 Node N
Setup TestEnvironment
ConfigureImplementation
Start theSubscribers
![Page 11: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/11.jpg)
11
What does the DBE do?
Node 0
DBE
Node 1 Node 2 Node N
Setup TestEnvironment
ConfigureImplementation
![Page 12: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/12.jpg)
12
What does the DBE do?
Node 0
DBE
Node 1 Node 2 Node N
Setup TestEnvironment
ConfigureImplementation
![Page 13: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/13.jpg)
13
What does the DBE do?
Node 0
DBE
Node 1 Node 2 Node N
Setup TestEnvironment
ConfigureImplementation
![Page 14: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/14.jpg)
14
What does the DBE do?
Node 0
DBE
Node 1 Node 2 Node N
Subscriber Subscriber Subscriber
Setup TestEnvironment
ConfigureImplementation
![Page 15: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/15.jpg)
15
What does the DBE do?
Node 0
DBE
Node 1 Node 2 Node N
Subscriber Subscriber Subscriber
Setup TestEnvironment
ConfigureImplementation
Start thePublisher
![Page 16: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/16.jpg)
16
What does the DBE do?
Node 0
DBE
Node 1 Node 2 Node N
Subscriber Subscriber Subscriber
Setup TestEnvironment
ConfigureImplementation
![Page 17: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/17.jpg)
17
What does the DBE do?
Node 0
DBE
Node 1 Node 2 Node N
Publisher Subscriber Subscriber Subscriber
Setup TestEnvironment
ConfigureImplementation
![Page 18: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/18.jpg)
18
What technologies does the DBE use?
• Scripts are written in Perl and use PerlAce module• Remote execution is accomplished via SSH• Logs, stats and test info are stored on a NFS partition• All tests are currently ran on Fedora Core Linux machines• Configured to test NDDS, TAO DDS, and Splice• Synchronization between tests is maintained by using
lockfilesDBE
Linux SSH NFS Perl lockfiles
![Page 19: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/19.jpg)
19
How does the DBE setup the environment?
• Most path information is updated using the ~/.ssh/environment file
• Some custom environment variables are set by DBE scripts on individual clients
• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs
DBE
![Page 20: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/20.jpg)
20
How does the DBE setup the environment?
• Most path information is updated using the ~/.ssh/environment file
• Some custom environment variables are set by DBE scripts on individual clients
• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs
DBE
![Page 21: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/21.jpg)
21
How does the DBE setup the environment?
• Most path information is updated using the ~/.ssh/environment file
• Some custom environment variables are set by DBE scripts on individual clients
• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs
DBE
Change $ENV
![Page 22: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/22.jpg)
22
How does the DBE setup the environment?
• Most path information is updated using the ~/.ssh/environment file
• Some custom environment variables are set by DBE scripts on individual clients
• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs
DBE
Change $ENV
![Page 23: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/23.jpg)
23
How does the DBE setup the environment?
• Most path information is updated using the ~/.ssh/environment file
• Some custom environment variables are set by DBE scripts on individual clients
• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs
DBE
Change $ENV Generate ID
![Page 24: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/24.jpg)
24
How does the DBE setup the environment?
• Most path information is updated using the ~/.ssh/environment file
• Some custom environment variables are set by DBE scripts on individual clients
• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs
DBE
Change $ENV Generate ID
![Page 25: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/25.jpg)
25
How does the DBE setup the environment?
• Most path information is updated using the ~/.ssh/environment file
• Some custom environment variables are set by DBE scripts on individual clients
• Individual Test IDs are assigned to each test• Directories are created on the NFS partition for stats and logs
DBE
Change $ENV Generate ID Create Directories
![Page 26: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/26.jpg)
26
How does the DBE setup the environment?
• Publishers and subscribers are notified of test settings– Result file locations
– QoS settings on a per subscriber basis
– Network settings
– Test specific settings (number of messages to send, etc.)
DBE
Repos Publishers Subscribers
Pass Test Info
![Page 27: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/27.jpg)
27
What problems had to be dealt with?
• Lack of a terminal with remote command execution with SSH• TCP and UDP buffer sizes were too small
• Synchronization amongst publishers and subscribers
Subscriber
Subscriber
PublisherW
ait
PublisherBuffer
PublisherBuffer
Next Test
![Page 28: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/28.jpg)
28
What problems had to be dealt with?
• Flock() doesn’t work as expected with stateless NFS
• Shared libraries are referenced locally• Block buffering of stdout
flock(“myfile”) touch(“myfile.lock”)
stdout
stderr
file
fileimmediate
block buffering
![Page 29: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/29.jpg)
29
What problems had to be dealt with?
• More problems on request ;)
![Page 30: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/30.jpg)
30
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
![Page 31: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/31.jpg)
31
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Create Publisher
![Page 32: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/32.jpg)
32
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Create Publisher
![Page 33: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/33.jpg)
33
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher
Create Publisher
![Page 34: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/34.jpg)
34
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher
![Page 35: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/35.jpg)
35
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher
Create Subscriber
![Page 36: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/36.jpg)
36
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher
Create Subscriber
![Page 37: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/37.jpg)
37
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher Subscriber
Create Subscriber
![Page 38: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/38.jpg)
38
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher Subscriber
![Page 39: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/39.jpg)
39
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher Subscriber
Create Subscriber
![Page 40: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/40.jpg)
40
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher Subscriber
Create Subscriber
![Page 41: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/41.jpg)
41
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher Subscriber Subscriber
Create Subscriber
![Page 42: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/42.jpg)
42
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher Subscriber Subscriber
![Page 43: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/43.jpg)
43
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher Subscriber Subscriber
KillSubscriber 1
![Page 44: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/44.jpg)
44
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher Subscriber Subscriber
KillSubscriber 1
![Page 45: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/45.jpg)
45
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher Subscriber
KillSubscriber 1
![Page 46: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/46.jpg)
46
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher Subscriber
KillSubscriber 1
![Page 47: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/47.jpg)
47
What features will be added?
• New tests will require new features• One such feature is per node scripting of subscribers and
publishers• Feature will allow dynamic starting and stopping of
participantsDBE
Publisher Subscriber
![Page 48: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/48.jpg)
48
What other additions are planned?
• The ability to launch logging services– Memory and CPU usage will need to be recorded
• The ability to test other middleware and applications
![Page 49: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/49.jpg)
49
Demonstration
Primer messages = 100
Stats messages = 1000
Test type = throughput
Implementation = NDDS
![Page 50: The DDS Benchmarking Environment James Edmondson jedmondson@gmail.com Vanderbilt University Nashville, TN](https://reader035.vdocument.in/reader035/viewer/2022062621/551c330e5503469e4f8b643b/html5/thumbnails/50.jpg)
50
Conclusion and Wrap Up
• The DBE is an excellent suite of tools for automating DDS testing on a networked environment
• New test types will be added to gain further insight into the DDS implementations
• Many new features will also be added to better reveal problems and benefits of the various DDS products
• We would like to thank our AFRL and Vanderbilt Sponsors for their continued support