tuning and troubleshooting opensplice dds applications
TRANSCRIPT
Ope
nSpl
ice
DD
S
Angelo Corsaro, PhDChief Technology Officer OMG DDS Sig Co-Chair
Tuning and Troubleshooting OpenSplice DDS Applications
Hans van’t Hag.Product Manager
OMG DDS Co-AuthorPrismTech
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
US OSUM 2013
http://www.prismtech.com/events/opensplice-user-meeting-usa-2013
Ope
nSpl
ice
DD
S Part I -- Setting the Context
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Kinds of IssuesWhen building distributed applications there are several kinds of issues you can encounter. Some of the most common issues are often:
☐ Connectivity: “things” don’t seem to talk to each other
☐ Performance:the latency is too high or throughput is too low
☐ Scalability: your system does not scale as necessary
☐ Resource Utilization: your application is using way too much CPU, Memory, and/or Network
This webcast explores common issues in DDS applications and demonstrates how the tools provided by OpenSplice DDS can their diagnose and resolution
Ope
nSpl
ice
DD
S
Connectivity
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Connectivity Issues
☐ DDS provides dynamic discovery thus as there is network connectivity applications that are intended to communicate should discover each other and just work!
☐ What may be the cause of connectivity issues in DDS?☐ The network is not working (did you plug the wire?)☐ Your applications does not share the same domain (thus your mistake)☐ The QoS of the DataWriter and the DataReader don’t match
Ope
nSpl
ice
DD
S
QoS Mismatch?
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Understand the QoS Model☐ DDS defines 22 QoS
policies that can be applied to communication entities to control their local as well as end-to-end behaviour
Tip #8
HISTORY
LIFESPAN
DURABILITY
DEADLINE
LATENCY BUDGET
TRANSPORT PRIO
TIME-BASED FILTER
RESOURCE LIMITS
USER DATA
TOPIC DATA
GROUP DATA
OWENERSHIP
OWN. STRENGTH
LIVELINESS
ENTITY FACTORY
DW LIFECYCLE
DR LIFECYCLE
PRESENTATION
RELIABILITY
PARTITION
DEST. ORDER
RxO QoS Local QoS
Most of the QoS Policies that control an end-to-end property follow the so-called Request vs. Offered (RxO) Model based on which the QoS requested by the Consumer should not exceed the QoS Provided by the Producer. If there is an RxO mismatch no communication will take place
Ope
nSpl
ice
DD
S
Ope
nSpl
ice
DD
S
Performance
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Latency / Throughput☐ DDS provides several QoS Policies to control latency and
throughput
☐ OpenSplice provides extensions such as Streams to efficiently distribute small data (e.g. several millions samples per second)
Throughput
TimeBasedFilter
[Inbound]
[Outbound]Latency
Deadline
TransportPriority
LatencyBudget
☐ OpenSplice Tools such as the Tuner can help find the right QoS setting for achieving the desired level or latency/throughput
Ope
nSpl
ice
DD
S
Scalability
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Federated / Standalone Deployment
Federated Deployment Standalone Deployment
Operating Systems
Application
OpenSplice DDS
Application
OpenSplice DDSOpenSplice DDS
Application Application
Operating Systems
☐ Performance ☐ Scalability☐ Determinism
☐ Usability ☐ Simple Configuration☐ Simple Deployment
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Deployment Configuration
☐ The deployment mode can be changed by a simple configuration parameter. No recompilation or re-linking!☐ <SingleProcess>true</SingleProcess>
☐ The deployment options can be mixed at will (even within a single computing node)
☐ The same application can be deployed in federated and standalone mode (even on the same system)
“Simple when Sufficient.Performant when Required”
With OpenSplice DDS you can choose the
deployment architecture that makes
the most sense for your use case.
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Deployment Configuration
☐ The deployment mode can be changed by a simple configuration parameter. No recompilation or re-linking!☐ <SingleProcess>true</SingleProcess>
☐ The deployment options can be mixed at will (even within a single computing node)
☐ The same application can be deployed in federated and standalone mode (even on the same system)
“Simple when Sufficient.Performant when Required”
With OpenSplice DDS you can choose the
deployment architecture that makes
the most sense for your use case.
Ope
nSpl
ice
DD
S
Resource Utilization
Ope
nSpl
ice
DD
S
Shared Memory Sizing
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Shared Memory Size☐ OpenSplice DDS shared memory size is defined in its
configuration file. The size defined by the default configuration file is 10MBytes
☐ Beware that different OS have different limitations w.r.t. the maximum shared memory segment that can be allocated
☐ If you want to go beyond the OS limits you need to change the configuration of your kernel
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Linux☐ The default value for the maximum shared memory segment is
32MBytes
☐ This default can be changed in several ways
(1) Adding this line to your /etc/rc.d/rc.local file:
(2) Changing the settings for the sys-limits (save changes on /etc/sysctl.conf to maintain them across reboots):
echo “your_max_shared_memory_size” > /proc/sys/kernel/shmmax
$ sysctl -w kernel.shmmax=yourMaxValue
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Windows
☐ The default maximum size for Shared Memory segments on Windows is 2GB
☐ To exend it, say to 3GB, add the /3GB the boot.ini as shown below:
[boot loader]timeout=30default=multi(0)disk(0)rdisk(0)partition(1)\WINDOWS[operating systems]multi(0)disk(0)rdisk(0)partition(1)\WINDOWS="Windows NTWorkstation Version 4.00" /3GB
Ope
nSpl
ice
DD
S
Understand Topic Keys
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Topic Types & Keys
☐ Topic Types can define some of their attributes as keys
☐ Yet, even when a Topic type does not define a key the keylist directive has to be provided -- just to tell the IDL compiler that this is a topic
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Topic Types & Keysenum TemperatureScale { CELSIUS, FAHRENHEIT, KELVIN};
struct TempSensorType { short id; float temp; float hum; TemperatureScale scale;};#pragma keylist TempSensorType id
struct UniversalConstants { double PI; double e; // Nepero Number double g; // Gravitational Constant double NA; // Avogadro Number double F; // Faraday Constant double K; // Boltzman Constant double c; // Light Speed };#pragma keylist UniversalConstants
KeylessKeyful
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Topic Instances
Writer Reader
1 26.0 70.0 CELSIUS
2 26.0 70.0 CELSIUS 1 26.0 70.0 CELSIUS
2 26.0 70.0 CELSIUS
enum TemperatureScale { CELSIUS, FAHRENHEIT, KELVIN};
struct TempSensorType { short id; float temp; float hum; TemperatureScale scale;};#pragma keylist TempSensorType
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Topic Instances
Writer Reader
1 26.0 70.0 CELSIUS
2 26.0 70.0 CELSIUS1 26.0 70.0 CELSIUS 2 26.0 70.0 CELSIUS
enum TemperatureScale { CELSIUS, FAHRENHEIT, KELVIN};
struct TempSensorType { short id; float temp; float hum; TemperatureScale scale;};#pragma keylist TempSensorType id
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Instance Management☐ Each topic instance takes
resources (e.g. memory)
☐ If you don’t choose carefully your key, you may end-up creating an instance per sample...
☐ In addition, instances can be explicitly disposed in order to reclaim resources
Writer Reader
1 26.0 70.0 CELSIUS
2 26.0 70.0 CELSIUS1 26.0 70.0 CELSIUS 2 26.0 70.0 CELSIUS
Ope
nSpl
ice
DD
S
Example
struct Counter { short cid; long count;};#pragma keylist Counter ???
Ope
nSpl
ice
DD
S
Example
struct Counter { short cid; long count;};#pragma keylist Counter cid
Ope
nSpl
ice
DD
S
Example
struct SpatialSensor { short x; short y; float v;};#pragma keylist SpatialSensor ???
Ope
nSpl
ice
DD
S
Example
struct SpatialSensor { short x; short y; float v;};#pragma keylist SpatialSensor
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Topic Instances Lifecycle
com.myco.VPos
com.myco.VPos
struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition plate
struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Topic Instances Lifecycle
com.myco.VPos
com.myco.VPos
“A01” 100 200
“A01” 100 200
New
New
struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition plate
struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
Topic Instances Lifecycle“A01” 100 200
“B41” 57 31
com.myco.VPos
“A01” 100 200“B41” 57 31
com.myco.VPos
New
New
struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition plate
struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
“A01” 100 200“B41” 57 31
com.myco.VPos
“A01” 110 210
Topic Instances Lifecycle“A01” 100 200
“B41” 57 31
com.myco.VPos
“A01” 110 210New
New
struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition plate
struct VehiclePosition { string plate; long x; long y; };#pragma keylist VehiclePosition
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
“A01” 100 200“B41” 57 31
com.myco.VPos
“A01” 110 210
Topic Instances Lifecycle“A01” 100 200
“B41” 57 31
com.myco.VPos
“A01” 110 210“A01” 120 220
“A01” 120 220
New
New
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
“A01” 100 200“B41” 57 31
com.myco.VPos
“A01” 110 210
Topic Instances Lifecycle“A01” 100 200
“B41” 57 31
com.myco.VPos
“A01” 110 210“A01” 120 220
“A01” 120 220
“B41” 47 19
“B41” 47 19
New
New
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
“A01” 100 200“B41” 57 31
com.myco.VPos
“A01” 110 210
Topic Instances Lifecycle“A01” 100 200
“B41” 57 31
com.myco.VPos
“A01” 110 210“A01” 120 220
“A01” 120 220
“B41” 47 19
“B41” 47 19
New
New“B41” - -Disposed
Disposed
Ope
nSpl
ice
DD
S Part II -- Demoes
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
Content☐ Tools
☐ Configuration: OpenSplice Configurator☐ Black-box System (regression) Testing: OpenSplice Tester☐ White-box application Inspection & Tuning: OpenSplice Tuner☐ Recording & Replay: OpenSplice RnR☐ Shared-memory and Network monitoring: OpenSplice mmStat and ‘Wireshark’ ™
☐ Common Issues☐ System Configuration (deployment architecture, networking-setup)☐ System Deployment (resource usage/watermarks, statistics, reports, tracing)☐ Tool Deployment (intrusiveness: local versus remote operation)
☐ Demo’s☐ Tester: system browser (SHMEM vs. SP) and QoS conflict detection (Reliability)☐ Tuner: Statistics monitoring & QoS Tuning (Latency Budget)☐ Tester: 1-click capturing & real-time Charting (virtual attributes, scripting)☐ RnR: scripted recording & replay with RnR-service and Tester☐ RnR: graphical control with RnRManager
OpenSplice Tools
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
Total Control: OpenSplice Configurator™
☐ The reference tool for configuring OpenSplice DDS
☐ Rich online guide to configuration options
☐ Context help and parameter validation
☐ 100% Java
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
OpenSplice Tester™ ☐ “Black-Box” DDS system testing
☐ Automated testing of DDS systems☐ Dynamic Discovery of DDS Entities
☐ Domain-specific scripting languages
☐ Batch execution of regression tests
☐ Debugging of distributed DDS systems☐ System browser of DDS participants
☐ Connectivity & QoS conflict monitoring
☐ One-click definition of monitoring timeline☐ Analysis/comparison of topic data
☐ Virtual topic-attributes to ease analysis
☐ Statistics monitoring
☐ 1-click spawning of Tuner to ‘attach’ to a remote process / federation
☐ Integrated IDE☐ Syntax highlighting editor
☐ One-click relations between script/logs/timeline
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
☐ “Whitebox” debug/tuning Tool☐ Looking ‘inside’ a federation and/or
application
☐ Different perspectives (participant, topic, partition)
☐ Monitoring & Tuning☐ Inspect and Tune the app’s DDS entities
☐ Make snapshots of reader-caches
☐ Detect and resolve QoS Mismatch
☐ Inspect Statistics
☐ Reading & Writing☐ Read/Write data for arbitrary topics
☐ Import & Export☐ Inject Topic Definitions
☐ export and import XML-based reader/writer snapshots
Total Control: OpenSplice Tuner™
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
RnRService
OpenSplice DDS
RnRService(s)
Any topic Record/Replay command & status topics
RnRManager
OpenSplice RnR
Record/Replay command & status topics
☐ Dynamic recording of any topic-data in a DDS system
☐ Selective replay with variable speed
☐ Distributed control by topic-based API
☐ Seamless integration with OpenSplice Tester (topic-based API)
☐ Dedicated RnR-Manager graphical GUI for scenario-definition and data import/analysis
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
Wireshark ☐ WireShark Packet
Dissector
☐ Watch what goes on the wire
☐ Inspect DDSI-RTPS and/or RT-Networking packets being exchanged between applications
☐ Native RTnetworking dissector support available in source-distribution
☐ OpenSpliceDDS\V6.2.1\HDE\x86.win32\tools\wireshark-plugins\ospl (see README)
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
mmStat
mmstat -h mmstat [-M|m] [-e] [-a] [-i interval] [-s sample_count] [URI] mmstat [-t|T] [-i interval] [-s sample_count] [-l limit] [-f filter_expression] [URI]
Mode: -m Show memory statistics (default mode) -M Show memory statistics difference -t Show meta object references -T Show meta object references difference
-h Show this help -e Extended mode, shows bar for allocated memory -a Show pre-allocated memory as well. -i interval Display interval (in milliseconds) -s sample_count Stop after sample_count samples -l limit Show only object count >= limit -f filter_expr Show only meta objects which name passes the filter expression
Common Issues: Examples & Demo’s
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
11
☐ Selecting the proper deployment architecture☐ Federated / shared-memory (for scalability and data/application lifecycle decoupling)☐ Standalone / single-process (for simplicity yet at the cost of ‘network-scheduling determinism’)
☐ Selecting the proper networking architecture☐ RTNative networking: ‘push’ based architecture using pre-configured (multicast) network-partitions
☐ Pro : zero discovery times (optional topology-discovery only needed for fast disconnect-reaction times)☐ Con : push implies data might show-up at places where there’s no actual/current interest .. Yet tunable in OpenSplice
☐ DDSI networking: ‘pull’ based architecture using advertized locators☐ Pro : selective distribution to ONLY the set of discovered participants, using unicast whenever just 1 destination☐ Con : (standardized) discovery-protocol has scalability limitations (amount of heartbeats) .. Yet tunable in OpenSplice
☐ Configuring your domain☐ Using “OpenSplice Configurator” to aid in correct configurations☐ Assure system-wide consistent values
☐ Domain-ID’s and in case of RTNetworking the related port-numbers (channels and discovery)☐ RTNetworking only: NetworkPartition definitions (names and related [multicast-] addresses)
Common issues: System Configuration
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
13
☐ How to Prevent Issues☐ Using the Configurator to validate configurations [demo]☐ Assuring system-wide consistent networking/durability setup☐ Using ResourceLimits policies in your applications
☐ How to detect Issues☐ Setting resource-usage ‘watermarks’ with the configurator [demo]☐ Checking ospl_info.log and ospl_error.log files [demo]☐ Checking QoS RxO conflicts with the Tester and/or Tuner [demo]☐ Capturing behavior with Tester (and [auto-]creation of scripts to reproduce)☐ Recording of information using RnR (manually and/or scripted) [demo]
☐ How to resolve issues☐ Runtime Tuning of QoS policies with the Tuner [demo]☐ Exploring reader/writer/service statistics with Tester & Tuner [demo]
Common issues: System Deployment
Demo’s
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
Demo’s☐ Environment
☐ Perf1/2/3 reference-system (shmem versus SP)☐ Ishapes demo-application☐ Configurator / Tester / Tuner / RnR tools
☐ Configurator☐ Used configuration-files (SHMEM/RnR, SHMEM/Statistics, SP/Resource-limits☐ Demo-config: perf1: RnR added, perf2: statistics added perf3: resource-limits
☐ Tuner/Tester ‘ishapes’ demos☐ Tester: system browser (SHMEM vs. SP)☐ Tester: QoS conflict detection (Reliability)☐ Tuner: QoS Tuning (Latency Budget)☐ Tuner: Status & Statistics (Writer/Reader, perf2: Network-queues, perf3: Network receiving)☐ Tester: real-time Charting (virtual attributes, scripting)
☐ Recording & Replay☐ Scripting with Tester☐ Controlling with RnRManager
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
Perf3Perf2
Splice-Hans
Switch
Perf1
Switch
Router
Office-net
Test-net
Perf-net
Shared Memory
soap RT-Net RnR / Dur.
Tester-1/ RnRManager
DDS libs
IshapesstandaloneSOAP
NETW
Remote Tuner 1
remote Tester-2
Demo setup
Shared Memory
Soap RT-Net Durability
Ishapes Federated
Ishapes Federated
Remote Tuner 2
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
Teaser: ‘Streams’☐ Bundled Example
☐ Streaming 10 mln. Samples
☐ Throughput: up to 29 mln. Samples/s ! (shmem/i7)
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
Configuration
☐ Perf1☐ Ospl_shmem_native_rnr.xml
☐ Perf2☐ Ospl_shmem_native_statistics.xml
☐ Perf3☐ Ospl_sp_native_resource_limits.xml
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
QoS conflict detection
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
QoS Tuning
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
Statistics Monitoring
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
Data-Capture
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
Scripted RnR
Copyrig
ht 2013, PrismTech – A
ll Rights Reserved.
Graphical RnR
Ope
nSpl
ice
DD
S
Copyrig
ht 2011, PrismTech – A
ll Rights Reserved.
Ope
nSpl
ice
DD
S
References
¥Fastest growing JVM Language¥Open Source¥www.scala-lang.org
¥ #1 OMG DDS Implementation¥ Open Source¥ www.opensplice.org
OpenSplice | DDS¥Scala API for OpenSplice DDS¥Open Source¥github.com/kydos/escalier
Escalier
¥Simple C++ API for DDS¥Open Source¥github.com/kydos/simd-cxx
¥DDS-PSM-Java for OpenSplice DDS¥Open Source¥github.com/kydos/simd-java
¥ DDS-based Advanced Distributed Algorithms Toolkit
¥Open Source¥github.com/kydos/dada
Ope
nSpl
ice
DD
S ¥@prismtech
¥@acorsaro
¥youtube.com/opensplicetube ¥slideshare.net/angelo.corsaro
¥ [email protected]¥[email protected]
¥opensplice.com ¥forums.opensplice.org
¥opensplice.org ¥[email protected]
:: Connect with Us ::