applying model-driven engineering to evaluate the qos of ... · applying model-driven engineering...
TRANSCRIPT
![Page 1: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/1.jpg)
OMG's Workshop on Distributed Object Computing for Real-time and Embedded SystemsJuly 10-13, 2006 - Arlington, VA USA
Applying Model-Driven Engineering to Evaluate the
QoS of Distributed, Real-time, Embedded Systems
John M. Slaby, James Hill, Sumant Tambe, Dr. Douglas Schmidt
![Page 2: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/2.jpg)
7/11/2006 Page 2
• Historically, distributed real-time & embedded (DRE) systems were built directly atop OS & protocols
Operating System & Communication Protocols
Hardware Devices
Applications
Motivation: Service-Oriented Architectures
![Page 3: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/3.jpg)
7/11/2006 Page 3
Applications
• Traditional methods of development have been replaced by middleware layers to reuse architectures & code
• Viewed externally as Service-Oriented Architecture (SOA) Middleware
Operating System & Communication Protocols
Hardware Devices
Domain-Specific Services
Common Services
Distribution Middleware
Infrastructure Middleware
Service-Oriented Architecture Middleware
Motivation: Service-Oriented Architectures
• Historically, distributed real-time & embedded (DRE) systems were built directly atop OS & protocols
![Page 4: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/4.jpg)
7/11/2006 Page 4Development Timeline
Leve
l of A
bstr
actio
nSerialized Phasing in Large-scale Systems
• Infrastructure developed
• Applications developed
• System integrated
![Page 5: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/5.jpg)
7/11/2006 Page 5Development Timeline
Leve
l of A
bstr
actio
n Complexities• System infrastructure cannot be
tested adequately until applications are done
• Entire system must be deployed & configured properly to meet QoS requirements
• Existing evaluation tools do not support “what if” evaluation
• Infrastructure developed
• Applications developed
• System integrated
Serialized Phasing in Large-scale Systems
![Page 6: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/6.jpg)
7/11/2006 Page 6Development Timeline
Leve
l of A
bstr
actio
n
Key QoS concerns• Which D&C’s meet the QoS
requirements? • What is the worse/average/best
time for various workloads?• How much workload can the system
handle until its QoS requirements are compromised?
• Infrastructure developed
• Applications developed
• System integrated
Serialized Phasing in Large-scale Systems
![Page 7: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/7.jpg)
7/11/2006 Page 7
ValidateDesign
Conformance
Conduct“What If”Analysis
ValidateDesignRules
Our Solution: New Generation of System Execution Modeling Tools! A Toolset which enables early
testing and evaluation of SOA-based distributed, real-time applications
–Modeling tools to build complex distributed applications
–A component-based workload emulator which can be made to look and behave as an arbitrary SOA application component
–A test harness to capture and analyze performance data
![Page 8: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/8.jpg)
7/11/2006 Page 8
The System Execution Modeling Toolset! CCM – CORBA Component Model (SOA Middleware)! GME – Generic Modeling Environment – for Domain-Specific
Modeling Language (DSML) development! PICML – Platform-Independent Component Model DSML
! CUTS - CoWorkEr Utilization Test Suite– CoWorkEr – Component Workload Emulator – CCM Component
capable of emulating an arbitrary application component– WML – Workload Modeling Language - DSML for characterizing
behavior of CoWorkErs– BDC – Benchmark Data Collector – BMW – Benchmark Manager Web – browser and web service interface
to test control and test results
![Page 9: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/9.jpg)
7/11/2006 Page 9
The Big Picture
IDL
A A
A
Component ModelsCoWorkEr Models
Distributed Application Models
System Modeling
Workload Modeling
Network Models
Deployment Models
System Execution
Application Emulation Components
![Page 10: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/10.jpg)
7/11/2006 Page 10
Building Test Scenarios
![Page 11: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/11.jpg)
7/11/2006 Page 11
The Component Workload Emulator
To Other CoWorkeErs
From Other CoWorkErs
Event Handler
Background Load
DB Worker
Memory Worker
CPU Worker
Event Producer
Benchmark Agent
To BDC To/From Test DB
The CoWorkEr is comprised of a set of generic capabilities representing the behavior of an application – CPU load, memory use, database interactions, background load, and event publications and subscriptions
![Page 12: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/12.jpg)
7/11/2006 Page 12
CoWorkEr Interface Refinement
PICML is used to refine the interfaces – transforming the generic CoWorkEr into a component that looks from an interface perspective like the application it is emulating
BDC
BDC
Assess Assess
Assess
Generic CoWorkEr Assembly
![Page 13: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/13.jpg)
7/11/2006 Page 13
Workload Characterization
startupStatusEventStartup Workload Allocate 10 MB Execute 50 CPU Repetitions
Perform 10 750 byte DB Ops
Publish 50 byte Status Event
Deallocate 10MB
Event-Based Workload Allocate 100KB Execute 30 CPU Repetitions
Perform 2 128 byte DB Ops
Publish 275 byte Command Event
Deallocate 100KB
The Workload Modeling Language (WLM) is used to describe behavior – transforming the generic CoWorkEr into an application that looks from a resource consumption and message handling perspective like the application it is emulating
![Page 14: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/14.jpg)
7/11/2006 Page 14
SOA Application Emulation
A test scenario is created by building an assembly of these characterized CoWorkErs. PICML and WML are used to model the scenario, and CCM is used to deploy the CoWorkErs based on those models.
‘Distributed Application’ RT1High
BDC
![Page 15: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/15.jpg)
7/11/2006 Page 15
Test Metrics and Analysis
![Page 16: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/16.jpg)
7/11/2006 Page 16
Timing MeasurementsNode clocks synchronized using NTP0.
Data periodically collected from CoWorkErs and written to DB
2.
BMW
Data read, processed and made available to both browser and web service clients
3.
Time measured between and within CoWorkErs1.
BDC
![Page 17: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/17.jpg)
7/11/2006 Page 17
Timing Measurements
![Page 18: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/18.jpg)
7/11/2006 Page 18
Timing Variation Analysis
![Page 19: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/19.jpg)
7/11/2006 Page 19
Critical Path Analysis
BMW
BDCThe critical path is defined as part of the BMW test data to allow analysis of these time-sensitive threads
The critical path is a processing and message flow through a distributed set of applications which represents a time-critical thread of execution
![Page 20: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/20.jpg)
7/11/2006 Page 20
Critical Path Analysis
![Page 21: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/21.jpg)
7/11/2006 Page 21
Summary! Some of the biggest challenges in effective use of these tools
involve getting reasonable values for workload. Engineers are generally reluctant to make (even educated) guesses.
! We have found the ability to run continuous, fully automated test suites for more extensive evidence collection to be a major benefit of using CUTS.
! There has been significant interest shown by programs within Raytheon in the toolset since we have begun to socialize our work.
![Page 22: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/22.jpg)
7/11/2006 Page 22
Next Steps! Enhanced CoWorkErs to include timing data in message
payload to get actual deadline measurements rather than worst-case trends.
! A benchmark node agent to aggregate node test data and support remote test control (node, app failures etc.)
! Extend metrics collection to include additional items –application availability, middleware overhead, etc.
![Page 23: Applying Model-Driven Engineering to Evaluate the QoS of ... · Applying Model-Driven Engineering to Evaluate the QoS of Distributed, Real-time, Embedded Systems John M. Slaby, James](https://reader034.vdocument.in/reader034/viewer/2022042105/5e82febc76f7e506d77a14ba/html5/thumbnails/23.jpg)
7/11/2006 Page 23
Questions?