ensuring deployment predictability of distributed real

Ensuring Deployment Predictability ofDistributed Real-time and Embedded Systems

Gan Deng, Douglas C. Schmidt, Aniruddha GokhaleDept. of EECS, Vanderbilt University, Nashville, TN{dengg,schmidt,gokhale}@dre.vanderbilt.edu

Abstract

The dynamic deployment and configuration (D&C)of components in response to environmental changesor system mission mode changes is essential to facil-itate runtime resource allocation for component-baseddistributed real-time and embedded (DRE) systems. Thispaper provides several contributions to the study of pre-dictable D&C for component-based DRE systems. First,we describe how the predictability of component-basedD&C can be affected by application dependency relation-ships and priorities. Second, we describe how a multi-graph algorithm called partial priority inheritance viagraph recomposition (PARIGE) can improve D&C pre-dictability. Third, we empirically evaluate the effective-ness of PARIGE on a representative DRE system basedon NASA Earth Science Enterprise’s MagnetosphericMulti-Scale (MMS) mission system. The results showthat PARIGE can avoid unbounded deployment time pri-ority inversion when component assemblies with differ-ent priorities have complex dependencies among eachother, thereby significantly improving the responsivenessof mission-critical tasks with higher priorities.

Keywords: Component middleware, DistributedReal-time and Embedded systems, Deployment andConfiguration.

1. Introduction

Emerging trends and challenges. Develop-ing distributed real-time and embedded (DRE)systems whose quality of service (QoS) can be as-sured even in the face of changes in available resourcesor QoS requirements is an important and challeng-ing R&D problem. Systems with such characteristicsare called open DRE systems [1] since they oper-ate in open environment and must be prepared toaccommodate changing operating conditions or re-quirements, such as power levels, CPU/network band-width or mission modes. Examples of open DRE sys-tems include shipboard computing environment [2],

and intelligence, surveillance and reconnaissance sys-tems +[3].

Open DRE system are often large and complex, e.g.,a shipboard computing system may consist of thou-sands of software components that run a wide rangeof missions, such as ship navigation, ship structuralhealth monitoring, vision-based object tracking and ob-ject characterization. To manage the overall complex-ity of such systems, the missions are often decomposedinto many domain-related tasks that can be modeledas operational strings [4], which are assemblies of soft-ware components that capture the partial order andworkflow of a set of executing software capabilities forparticular domain tasks.

Operational strings have the following propertiesthat make them useful building blocks for open DREsystems:• Distributable, i.e., operational strings can be de-

ployed onto multiple nodes of the target runningenvironment, and different components in opera-tional strings can communicate remotely with eachother.• Concurrent, i.e., operational strings can run con-

currently in the same target environment andshare many system resources, such as CPU, mem-ory, and network bandwidth.• On-demand, i.e., operational strings can be dy-

namically populated at system runtime and thendeployed into the target running environment on-demand to accommodate changing mission goals.• Cooperative, i.e., to achieve certain mis-

sion goals different operational strings can co-operate with each other through their ports,which delegate to the ports of monolithic compo-nents that consist of the operational strings.• Prioritized, i.e., different operational strings can

be assigned with different priorities by system ar-chitects or online planners to reflect their impor-tance to certain mission tasks or to the overall sys-tem.

The dynamic nature of open DRE systems requiresthe deployment and configuration (D&C) of scores ofoperational strings at runtime to ensure that execut-ing systems keep in sync with changing mission goalsand resource availability. The runtime management ofoperational strings in DRE systems is hard since theD&C framework must be scalable and predicable. It istherefore essential that D&C frameworks be able to dy-namically deploy and configure operational strings in atimely and predictable manner.

In complex DRE systems, many operational stringsmay be deployed dynamically, e.g., in response to mis-sion mode or environmental changes. If dependenciesexist among these operational strings, deployment pri-ority inversions can occur at runtime. A deploymentpriority inversion occurs when a higher priority opera-tional string cannot be deployed before lower priorityoperational string(s) because of the dependencies be-tween them. Existing D&C frameworks [5, 6, 7] onlyconsider the dependency between operational stringsand ignore their priorities, which can cause unboundeddeployment priority inversions for DRE systems.Solution Approach → Partial Priority Inheri-tance via Graph Recomposition.

To address the challenges of open DRE systems de-scribed above, we developed a technique based on analgorithm called partial priority inheritance via graphrecomposition (PARIGE). This algorithm analyzes thedependencies between operational strings and removesall the dependencies from higher priority operationalstrings to lower priority ones by promoting1 compo-nents from lower priority operational strings to higherpriority ones. By applying our technique, a D&C frame-work can avoid potential priority inversions when mul-tiple operational strings are deployed at runtime.

Figure 1 shows the three three main steps of our ap-proach:

• Step 1 converts a deployment descriptor (whichcontains metadata describing a set of operationalstrings) into an in-memory directed graph repre-sentation. Each vertex in the graph represents acomponent in the operational string and each edgerepresents a connection between two components.• Since a deployment plan may have multiple op-

erational strings with different priorities havingdependencies among each other, step 2 analyzesthe dependency relationship between all the opera-tional strings by perform a graph-based algorithm

1 In the context of this paper, promoting a component meansthat before this component is deployed it is temporarily movedfrom a lower priority operational string to a higher priority op-erational string for deployment purpose only.

Component

Required Interface Provided Interface Event SinkEvent Source

Operational String

1

2

3

Figure 1: Overview of Our Approach

called partial priority inheritance via graph recom-position. This algorithm removes all the priorityinverted dependencies between operational stringsby promoting component(s) from the lower prior-ity operational string to the higher priority string.• After graphs are recomposed, step 3 converts them

back to deployment descriptor format and fed tothe D&C framework for deployment. For the oper-ational strings with dependencies with each other,the D&C framework can then deploy the opera-tional strings from the highest priority to the low-est priority.

When a DRE system has many operationalstrings with complex dependencies it is hard to de-termine manually which components in which op-erational strings should be migrated and whichoperational string to migrate. This paper there-fore makes the following three contributions to theresearch on D&C for component-based DRE sys-tems:• Analyze dependency relationships among opera-

tional strings to determine how each relationshipcan affect deployment predictability.• Present a multi-graph algorithm called “partial

priority-inheritance via graph recomposition” toavoid deployment priority inversion.• Empirically evaluate the multi-graph algorithm to

determine how effective it is on a representativeDRE system.

Paper organization. The remainder of this paper isorganized as follows: Section 2 describes a represen-tative DRE system case study that elicits key chal-lenges to ensure the predictability of operational stringD&C; Section 3 presents the PARIGE algorithm basedon a multi-graph recomposition technique that resolvesthese challenges; Section 4 presents results of exper-iments that evaluate our techniques empirically; Sec-tion 5 compares our work with related research; andSection 6 presents concluding remarks and lessonslearned.

2. Motivating Case Study

This section describes different configurations of op-erational strings in DRE systems that can cause de-ployment priority inversion to occur due to the de-pendencies among the strings. To make our discussionconcrete, we use NASA’s Magnetospheric Multi-Scale(MMS) mission system [8] as a case study. We firstpresent the case study and then identify key challengesthat must be addressed to ensure D&C predictabilityfor the case study.

2.1. Overview of NASA MMS Mission Sys-tem

The NASA Earth Science Enterprise’s MMS missionsystem system uses five satellites with multiple sensorson each satellite to perform solar-terrestrial probe task.The satellites orbit the earth in formation and collectelectromagnetic and particle data in the earth’s mag-netosphere. The MMS mission operates in three datamodes: slow, fast, and burst. These data modes mayalso include different goals, orbits, and data priorities.Each satellite must be capable of determining the nec-essary task sequences to achieve prescribed goals basedon the current environmental and system conditions, aswell as revising task sequences in response to changingconditions.

To achieve autonomy, an automated planner is de-ployed within the MMS system to handle autonomousmode changes driven by the satellite position and theresults of analyzing collected data. The task sequencesare implemented by components for coordinating thetrajectory and orientation of satellites, sensor selec-tion and data collection for individual satellites, and

data integration and compression to create telemetrystreams that are beamed down to earth stations.

Figure 2 shows three operational strings that a plan-ner generates for a mission task of one of the satel-lites. Each operational string has different deployment

Operational String C(Low Deployment Priority)

Operational String A(High Deployment Priority)

Operational String B(Medium Deployment

Priority)

Filtering Trace Detection

State EstimateGizmo

ControlTask

Comm

Data Analysis

Monitor

ScienceAgent

Science Agent

FilteringData

Fusion MessagingGizmo

FilteringGizmo

Figure 2: Operational Strings Generated by Planner

priority (i.e., high, medium, and low) that are deter-mined by how each operational string is accessed bythe overall MMS system. The three operational stringsare briefly described as follows:• Operational string A defines a mission-critical

task that collects field data when a satellite movesto particular locations. To ensure this task is per-formed properly, the operational string must bedeployed as fast as possible to avoid loss of data.Operational string A can store the collected datain its own data store, but can also send the data toother operational strings through its event sources.• Operational string B is designed for a domain-

centric data analysis. Different scientific analy-sis tasks can be configured through the facets ofcomponents of this operational string. For exam-ple, Science Agent components can be configuredto achieve scientific objectives, such as analyz-ing models of complex phenomena like extendedweather forecasting.• Operational string C is for less essential data

analysis task and can collect auxiliary field data,such as Sun zenith, satellite view zenith [9], whichcan be served as additional input for analysis.This operational string only operates occasionally,e.g., when the data analysis component in oper-ational string B explicitly issues a request to re-quest such data as additional input for scientificanalysis models. The components in operationalstring C are driven by events exchanged throughtheir event sources and sinks.

Operational strings are organized from domain per-spective, e.g., each operational string is designed to ac-complish certain domain tasks, such as collecting cer-tain field data, or perform certain analysis on different

data models. In our MMS scenario, operational stringA services (e.g., collecting essential field data for sci-entific analysis) are most important for the MMS sys-tem, so it has the highest deployment priority amongthe three operational strings. Conversely, operationalstring C has the lowest deployment priority among thethree since it is designed a less essential, i.e., collect-ing auxiliary data only when necessary. Finally, opera-tional string B is designed to have medium priority be-cause its scientific analysis role is less important thanoperational string A, but more important than opera-tional string C. Operational string B, however, needsto send events to operational C to notify it to collectauxiliary field data and perform analysis when neces-sary.

As shown in Figure 2, there are two dependencies be-tween operational strings: from A to B and from B toC. These dependencies cross the boundary of an indi-vidual operational string. We therefore call them ex-ternal dependencies, in contrast to those dependencieswithin an operational string.

2.2. Challenges of Ensuring D&C Pre-dictability in the MMS Case Study

Below we describe four challenges that arise whenoperational strings are deployed dynamically in openDRE systems, such as the NASA MMS mission casestudy described above.

Challenge 1: Avoid deployment priority inver-sion between two operational strings. In conven-tional D&C technologies, such as the OMG D&C spec-ification [10, 6], when a component of an operationalstring has a connection (either facet/receptacle or eventsink/source) to another component in a separate opera-tional string, an external reference must be specified toindicate the remote component and the provided portin the other operational string upon which it depends.To deploy this operational string successfully, the exter-nal reference endpoint of the other operational stringmust be activated before the deployment of source op-erational string can occur. When such a dependency isfrom a higher priority operational string to a lower pri-ority operational string, however, the low priority op-erational string must be deployed before the high pri-ority operational string can be deployed to avoid de-ployment failure caused by the dependency, which re-sults in a priority inversion at deployment-time.

For example, in our MMS system case study de-scribed in Section 2.1 the dependency from operationalstring B (medium priority) to operational C (low prior-ity) can cause a deployment priority inversion betweenoperational strings B and C. This dependency requiresthe deployment of operational string C before opera-

tional string B to resolve the dependency. Not all com-ponents in operational string C need be deployed toresolve the external dependency between B and C.

Subsection 3.2.1 describe how we address this chal-lenge by promoting components from the lower prior-ity string to the higher priority string.

Challenge 2: Avoid deployment priority inver-sion propagation effect. A more general priority in-version situation involves multiple operational strings.In this case, to resolve a dependency from a higher pri-ority string to a lower priority string, not only must thelower priority string be deployed before the high prior-ity operational string, but also the operational stringsthe lower priority string depends on. When these oper-ational strings have lower priority than the high prior-ity string, however, deployment priority inversion willoccur between operational strings.

For example, in our MMS system case study opera-tional string A has a high priority and an external de-pendency on operational string B. More specifically, itis the Data Analysis component of operational stringB that A depends on. The Data Analysis componentfurther depends on the Messaging component in op-erational string C, however, which can cause anotherdeployment priority inversion between A and C.

Subsection 3.2.2 describes how we address this chal-lenge by recursively tracing dependencies from the highpriority string to all lower priority strings.

Challenge 3: Avoid deployment failure whencircular dependency exists among multi-ple operational strings. Conventional D&C tech-niques [10, 6] will fail if a circular dependency existsamong multiple operational strings. The central co-ordinated phased deployment technique applied inconventional D&C technologies can effectively ad-dress the circular dependency problem within a sin-gle operational string. Conventional D&C technologiescannot handle cases, however, where a D&C re-quest involves multiple operational strings and ifa circular dependency exists among these opera-tional strings.

Limitations with circular dependencies arise becausethe deployment of an operational string is treated asan indivisible process, i.e., conventional D&C technolo-gies treat operational strings as primitive units. If a de-pendency exists between two operational strings, there-fore, one must be deployed before another to resolvesuch dependency requirements. Such treatment, how-ever, makes it hard to handle circular dependenciesamong different operational strings.

For example, in our MMS system case study a circu-lar dependency will exist between A, B, and C if the lessessential operational string C depends on operational

string A because it needs to access a type of serviceprovided by A. Since conventional D&C approachestreat each operational string deployment as an indi-visible process, such deployments cannot be handledproperly.

Subsection 3.2.3 describes how we address this chal-lenge by promoting components in the circular depen-dency trace among operational strings.

Challenge 4: Improve the overall utility of theoperational strings being deployed. The dynamicnature of open DRE systems require on-demand de-ployment of many operational strings that cooperatewith each other to ensure the system is kept in syncwith changing mission goals or environmental changes.Since multiple operational strings with different impor-tance to the entire DRE system may be deployed atthe same time, the goal of a D&C framework is to de-ploy these operational strings in an effective way to im-prove the overall QoS of DRE systems in the followingtwo dimensions:

• Since operational strings with the highest prior-ity are the most important to the entire DRE sys-tem, these operational strings should be deployedas early as possible to ensure the DRE system re-sponsiveness due to the changing environment ormission modes. For example, in our MMS systemcase study, operational string A should always bedeployed immediately since it has the highest pri-ority.

• Since each operational string has a utility value as-sociated with it, a D&C framework should try tofinish deployment of each individual operationalstring as early as possible by taking its utilityvalue into account. For example, in our MMS sys-tem case study there is a dependency from opera-tional string A to operational string B. Althoughthe deployment of operational string A should fin-ish first due to its higher priority, the time to fin-ish deploying operational string B is also an con-tributing factor to the overall system utility andshould thus be considered.

Subsection 3.2.4 describes how we address this chal-lenge by selectively applying the PARIGE algorithm tothe input of operational strings.

3. An Algorithm for Partial Priority In-heritance via Graph Recomposition

This section describes how we resolved the chal-lenges described in Section 2.2 using an algo-rithm called partial priority inheritance via graphrecomposition (PARIGE). This algorithm converts

each operational string into a graph, where each ver-tex and edge of the graph represent a componentand a connection/dependency between two compo-nents, respectively. If there is an external dependencybetween operational strings, then the graph con-verted from one operational string will have a specialtype of vertex that represents the external depen-dency. This special vertex type contains informa-tion about the actual refereed operational string andthe component in the operational string that it de-pends on.

The PARIGE algorithm migrates components fromone graph to another based on operational string char-acteristics, including their priorities and their depen-dency relationships with other operational strings inthe same deployment request. After graphs for all theoperational string are recomposed to account for thecomponent promotion, a new set of operational stringswill be populated from these recomposed graphs. Thesenew strings avoid deployment priority inversion be-tween operational strings and break the circular depen-dency among all the operational strings. These new setof operational strings can then be deployed by conven-tional D&C technologies, such as J2EE or OMG D&Cmodels.

To demonstrate the effectiveness of PARIGE, weintegrated the algorithm into the central coordinatorExecutionManager of OMG D&C model to performexperimental analysis. The ExecutionManager is runsas a daemon and is used to manage the deploymentprocess for a domain. In accordance with the D&Cspecification, a domain is a target environment com-posed of nodes, interconnects, bridges, and resources.An ExecutionManager plays the role of central coordi-nator that manages the nodes in the DRE system en-vironment. On each node, a NodeManager runs asa daemon process and manages the deployment of allcomponents that reside on that node, irrespective ofwhich application they are associated with.

3.1. Overview of the PARIGE Algorithm

Although the PARIGE algorithm recompose opera-tional strings by promoting components from one oper-ational to another, it has also the following propertiesthat makes it well-suited for D&C of DRE systems:

1. The PARIGE algorithm does not af-fect the functional behavior of component-based DRE systems. Component-based DRE sys-tems are deployed in the form of operational stringsthat consist of multiple monolithic components con-nected with each other via their ports. Two opera-tional strings can have dependencies with each other.A dependency between two operational strings is es-

sentially a connection from a monolithic component inone operational string to a port of a monolithic com-ponent in the other operational string.

The PARIGE algorithm evaluates the componentdependency relationships and their priorities and re-composes these operational strings to avoid deploymentpriority inversion. From the perspective of all opera-tional strings to deploy, however, the individual mono-lithic components and their connections among eachother are not modified by the algorithm. In particu-lar, the effect of the PARIGE algorithm on operationalstring recomposition is only visible for the D&C frame-work, which does not affect the running system’s func-tional behavior. This algorithm thus does not affect thefunctionally of operational strings because the topologyof all the operational strings (including all the mono-lithic components and connections) that fulfills func-tional behavior of the system remains unchanged.

2. The PARIGE algorithm only migratescomponents that can improve QoS behav-ior of operational strings. When components aremigrated from a lower priority operational stringto a higher priority operational strings, the prior-ity of the components is also bumped up to matchthe priority of the higher priority string, which is es-sential for a task to avoid priority inversion atdeployment-time [11]. Since the PARIGE algo-rithm only migrates components that one or morehigher priority operational strings have dependen-cies on—and does not migrate components withoutsuch dependencies—the QoS behavior of the opera-tional strings will only be improved and never wors-ened.

Figure 3 presents an overview of the PARIGE algo-rithm by showing an example with three operationalstrings having priorities high, medium, and low. Thedotted and solid arrows represent dependencies be-tween operational strings. In particular, the dotted ar-rows in the figure represent priority inverted dependen-cies, i.e., dependencies from higher priority operationalstrings to lower priority operational strings. Likewise,the solid arrows represents external dependencies with-out causing priority inversion.

The numbered vertices in Figure 3 denote the ver-tices migrated from one graph into another. For exam-ple, in the first iteration of the algorithm, one vertex ismigrated from the medium priority operational stringto the high priority operational string and another ver-tex is migrated from the low priority operational stringto the high priority string. In the second iteration, an-other vertex is migrated from the low priority opera-tional string to the medium priority string.

The PARIGE algorithm recomposes the graphs by

Figure 3: An Overview of the PARIGE Algorithm

parsing the input set of graphs and removing dottedarrows by promoting some component(s) from a lowerpriority operational string to a higher priority string.This process may introduce some new dependencies be-tween operational strings due to component promotion.The algorithm, however, only introduces solid arrows,i.e., only dependencies from lower priority operationalstrings to higher priority strings exist after the recom-position.

When the algorithm finishes, all dotted arrows inthe graphs will be removed and there will be no de-pendencies from higher priority operational strings tolower priority operational strings. As a result, both pri-ority inversion at run-time and deployment-time can beavoided. Moreover, all circular dependencies among op-erational strings, if any, will also be removed in the re-composition process.

3.2. Analysis of Operational String Depen-dencies with Deployment Priority In-versions

The goal of the PARIGE algorithm is to remove alldependencies from higher priority operational stringsto lower priority operational strings. To accomplishthis, the algorithm starts with the operational stringhaving the highest priority and processes all the exter-nal dependencies of this operational string. After all

external dependencies from the highest priority opera-tional string are removed, the algorithm then processesthe operational string with the next highest priority.When multiple operational strings have the same pri-ority, we apply the following tie-breaking policies se-quentially: (1) evaluating the second metric of each op-erational string, if given, (2) evaluating the number ofexternal dependencies to the same priority operationalstrings and treat the operational string with the leastnumber of external dependencies as the higher prior-ity than others, and (3) break the tie randomly if sucha tie still exists.

When processing an external dependency from ahigher priority operational string to a lower priorityoperational string, the algorithm must trace the de-pendency into other operational strings and migratecomponents from them if the lower priority operationalstring has dependencies to them. For example, if a highpriority operational string depends on a component Xin a medium priority operational string, and if compo-nent X also has dependency on a component in a lowpriority operational string, then the component in thelow priority string also must be migrated into the highpriority string.

We define a dependency trace as a totally ordered se-quence S. Each element in the sequence is a componentof an operational string that has a priority value asso-ciated with it. The starting element of the sequenceis the source component of the external dependencyof interest. The PARIGE algorithm analyzes all thedependency traces in the operational strings and re-composes the operational strings based on dependencytrace characteristics. To analyze the dependency tracewe classify the operational dependency relationshipsinto the three categories described below, which willbe used to address the first three challenges in Sec-tion 2.2.

3.2.1. Handling Challenge 1 → Promotion ofComponents Between Two Operational Strings(Non-circular) In this case, a dependency occurs be-tween two operational strings, where a high priorityoperational string has a dependency to a lower prior-ity operational string, as shown in Figure 4. We assumeno circular dependency exists in this case (circular de-pendencies will be discussed in Section 3.2.3).

As shown in Figure 4, the unique characteristic ofthis category is that the dependency trace does notcross the boundary of the lower priority operationalstring. Since no other operational strings are involvedbesides the two operational strings of interest, remov-ing such a priority inverted external dependency onlyrequires promoting all components in the dependency

Figure 4: A Dependency Trace Spanning Two Opera-tional Strings Only

trace from the lower priority operational string to thehigh priority one.

In the context of MMS case study, a priority invertedexternal dependency exists between operational stringB and operational string C, as illustrated in the upperhalf of Figure 5. Our solution removes this priority in-



Priority)

Data Analysis

Monitor

ScienceAgent Science

Agent

FilteringData


FilteringGizmo


Operational String B(Medium Deployment Priority)

Data AnalysisMonitor

ScienceAgent

Science Agent

Filtering

DataFusion Messaging

Gizmo

FilteringGizmo

Figure 5: A Dependency Trace across Two OperationalStrings

verted external dependency by promoting componentsMessaging and Data Fusion from operational C tooperational B, as illustrated in the bottom half of Fig-ure 5. After the promotion, operational string B doesnot have any dependencies to operational string C. Thetwo newly created external dependencies from the twoFiltering components in operational string C to op-erational B are essential to make sure the functional be-havior of the MMS system is not changed. Since thesetwo newly created external dependencies are not prior-ity inverted, deployment priority inversion between the

two operational strings can be avoided.

3.2.2. Handling Challenge 2 → Promotionof Components Across Multiple OperationalStrings (Non-circular) This more general case in-volves multiple operational strings, with a dependencytrace that spans across the operational strings. As be-fore, we assume no circular dependency exists sincethat will be discussed as a separate case in Sec-tion 3.2.3.

A dependency trace that spans across multiple oper-ational strings can be further categorized into the fol-lowing two situations.

1. Ordered dependency trace. Figure 6 showsan ordered dependency trace. In an ordered depen-

Can have other external dependencies.


Medium PriorityHigh Priority Low Priority

Dependency from Higher Priority to Lower Priority Dependency from Lower Priority to Higher Priority

Figure 6: An Ordered Dependency Trace

dency trace the priorities of each element in the se-quence have a non-increasing order, i.e., all externaldependencies in the sequence are priority-inverted. Asa result, all the priority-inverted external dependen-cies must be removed through the component promo-tion mechanism described in Section 3.1. The categorydescribed in Section 3.2.1 where only two operationalstrings are involved is a special case of an ordered de-pendency trace. To remove all priority-inverted exter-nal dependencies, the PARIGE algorithm simply mi-grates all components in the dependency trace into theoperational string where the first component of the de-pendency trace is located.

2. Unordered dependency trace. Figure 7 showsan unordered dependency trace, where the priorities ofthe elements in the dependency trace do not have aparticular order, i.e., some external dependencies arepriority-inverted (shown as dotted lines), whereas oth-ers are not (shown as solid lines). The PARIGE al-gorithm always starts with the highest priority opera-tional string and removes all external dependencies onit before moving to the next operational string. The al-gorithm therefore ensures that in an unordered depen-dency trace, the elements whose priorities are higherthan that of the starting element will have no external


Cannot have other external dependencies.

Low PriorityMedium Priority High Priority


Figure 7: An Unordered Dependency Trace

dependencies, which ensures the convergence of the al-gorithm.

For example, given the 3 operational strings fromSection 2, if the high priority operational string has anexternal dependency to a component in low priority op-erational string and this component must be migratedinto the medium priority operational string. When thispromotion happens, the high priority string will dependon the medium priority string, which introduces addi-tional priority-inverted external dependencies.

To remove all priority-inverted external dependen-cies of an unordered dependency trace, we break the en-tire dependency trace into two concatenated segments.As shown in Figure 8, the first segment is a priority un-ordered subsequence, where all the priorities of opera-tional strings are lower than the priority of the sourceof the dependency trace. The second segment is a prior-

Unordered subsequence, but all the priorities are lower than the priority of the source of the entire dependency trace.

Ordered subsequence, but all the priorities are higher than the priority of the source of the entire dependency trace.

Priority YPriority X Priority U Priority V

Dependency from Higher Priority to Lower PriorityDependency from Lower Priority to Higher Priority

Figure 8: Two Partitions of An Unordered DependencyTrace

ity ordered subsequence, where all priorities are higherthan the priority of the source of the dependency trace.For the first segment, we can migrate all the compo-nents in the subsequence into the operational stringwhere the first component of the dependency trace islocated, which will ultimately result in an ordered de-pendency trace.

In the context of MMS case study, a priority invertedexternal dependency exists between operational string

B and operational string C, and another priority in-verted external dependency exists between operationalstring A and operational string B, as illustrated in theupper half of Figure 9. Our solution traverses the de-




Priority)


State EstimateGizmo

ControlTask

Comm

Data Analysis

Monitor

ScienceAgent

Science Agent

FilteringData


FilteringGizmo




Priority)


State EstimateGizmo

ControlTask

Comm

Data Analysis

Monitor

ScienceAgent

Science Agent

Filtering

DataFusion Messaging

Gizmo

FilteringGizmo

Figure 9: A Dependency Trace across Three Opera-tional Strings

pendency trace across all the three operational strings,and removes both priority inverted external dependen-cies. By doing this, components Messaging and DataFusion are migrated from operational C to operationalA, and components Data Analysis and Monitor aremigrated from operational B to operational A, as il-lustrated in the bottom half of Figure 9. In our MMScase study, only one priority inverted external depen-dency exists from operational string A, so after travers-ing it, operational string A does not have any morepriority inverted external dependency remaining. TheD&C service can then deploy operational string A be-fore other operational strings due to its highest prior-ity, therefore avoiding priority inversion between oper-ational string A and other operational strings.

3.2.3. Handling Challenge 3 → Promotion ofComponents in the Circular Dependency TraceAmong Operational Strings Circular dependenciesmay exist among two or more operational strings,as shown in Figure 10. When we discussed how thePARIGE algorithm processed an unordered depen-dency trace in Section 3.2.2, we showed that a depen-dency trace can be divided into two subsequences. Thepriority of each element in the first subsequence is less

Unordered subsequence – cycles may exist Ordered subsequence – no cycles can exist

Priority YPriority X Priority U Priority V


Figure 10: Circular Dependencies in a DependencyTrace

than or equal to the priority of the source element ofthe dependency trace. Likewise, the priority of each el-ement in the second subsequence is greater than thatof the source elements of the dependency trace.

Since no components in the second subsequence canhave priority inverted external dependencies, cyclesamong operational strings can only occur in the firstsubsequence, as shown in Figure 10. To remove cyclesacross multiple operational strings, only components inthe first subsequence must be migrated. The PARIGEalgorithm therefore only needs to migrate componentsexisting the first subsequence to bring cycles into thesame operational string.

In the context of MMS case study, a circular depen-dency exists between operational string A and opera-tional string B, as illustrated in the upper half of Fig-ure 11. Our solution traverses the dependency trace



Priority)


State EstimateGizmo

ControlTask

CommData

Analysis

Monitor

ScienceAgent

Science Agent



Priority)


State EstimateGizmo

ControlTask

Comm

Data Analysis

Monitor

ScienceAgent

Science Agent

Figure 11: A Circular Dependency Trace Case

across from the operational string A, and promotingcomponents Data Analysis and Monitor from oper-ational B to operational A, as illustrated in the bot-tom half of Figure 11. After such promotion, the circu-lar decadency trace still exists but it is contained withinthe operational string A rater than across the bound-ary operational strings, thereby avoiding deploymentfailure.

3.2.4. Handling Challenge 4 → Selectively Ap-plying the Algorithm to Operational StringsThe deployment latency of an operational string is af-fected by the size of the operational string. The de-pendency trace technique described above will migratecomponents from lower priority operational strings tohigher priority operational strings. Although this tech-nique can ensure operational strings with the highestpriority be deployed as early as possible, it may re-duce the overall utility by delaying the deployment oflower priority operational strings in the worst case, asdescribed as Challenge 4 in Section 2.2. This delay oc-curs when all components in the lower priority oper-ational strings must be migrated into the higher pri-ority operational string, which makes both operationalstrings merge together and hence finish their deploy-ment at the same time. Merging operational stringscan adversely affect the responsiveness of lower prior-ity operational strings since these strings could havebeen deployed first to satisfy the dependency require-ments without affecting the responsiveness of higherpriority operational strings.

To overcome this difficulty, we apply an optimizationtechnique that is based on the results produced by allthe dependency traces of an operational string, as de-scribed in Subsections 3.2.1, 3.2.2, and 3.2.3. If the de-pendency traces of an operational string migrate all thecomponents of an lower priority operational string intothe higher priority operational string, the PARIGE al-gorithm simply chooses the cached graph representa-tion of these two operational strings before the depen-dency traces techniques are applied. This optimizationimproves the overall utility of the operational strings byfinishing the deployment of lower priority operationalstring first. The deployment of lower priority stringcan thus be finished earlier rather than at the momentwhen all components in both operational strings are de-ployed.

3.3. Design of the PARIGE AlgorithmThe PARIGE algorithm uses multi-graph breadth

first search (BFS) to trace dependencies and graph re-construction to migrate components and connectionsbetween components. Each graph corresponds to anoperational string and can have two types of vertices:

• A regular vertex, which refers to a componentwithin the operational string.

• A reference vertex, which refers to a componentin another operational string on which it has a de-pendency.

For example, two operational strings have dependen-cies shown in Figure 12. Based on the dependency re-

High Priority

High Priority Low Priority

Low Priority

Dependency from Higher Priority to Lower Priority

Dependency from Lower Priority to Higher Priority

Figure 12: Convert External Dependencies into Refer-ence Type Vertices

lationship between these two operational strings, thePARIGE algorithm converts them into two graphs withone reference type vertex in each graph, with refer-ence type shown as shaded vertices in the figure. Givena reference type vertex, in O(1) time we can findthe referred operational string and referred componentwithin the operational string.

In our algorithm, we define the following two typesof operators against vertices in the graph:

• Promote(V ): Promote a reference type vertex toa regular type vertex.

• Demote(V1, G1, V2, G2): Demote a regular typevertex V1 in graph G1 in to a reference type ver-tex, and the referred graph and vertex will be G2

and V2, respectively.

Algorithm 1 presents the PARIGE algorithm, whichuses a recursive procedure defined in Algorithm 2 toprocess each dependency trace.

Algorithm 1 PARIGE AlgorithmInput: Set of Operational Strings (Represented

as Graphs)Output: Set of Operational Strings (Repre-

sented as Graphs)Sort strings with decreasing priority;foreach String Gi in the sorted array do

foreach External Dependency j of String Gi

doGet the source vertex Vj1 of j;Get the destination vertex Vj2 of j;if priority(Vj1) > priority(Vj2) then

Get the referenced string Ge and ver-tex Ve;process dependency edge (Gi, Vj2,Ge, Ve);

endend

end

Algorithm 2 A Recursive Procedure to Process a De-pendency Trace

Input: G1: source graph with higher priorityG2: destination graph with lower priorityV1: source vertex (reference type)V2: destination vertex (regular type)

Output: Recomposed graphs G1 and G2

Promote (V1);Demote (V2, G2, V1, G1);Do a BFS (G2, V2) and foreach visited edge Ei

doGet the source vertex Vi1;Get the destination vertex Vi2;if Vi2 is regular type then

Remove Vertex (G2, V2);Add Vertex (G1, V2);Remove Edge (G2, E);Add Edge (G2, E);

endelse

Get the referred graph Gie of Vi2;Get the referred vertex Vie of Vi2;process dependency edge (G1, Vi1, Gie,Vie);

endend

3.4. Analysis of the AlgorithmTo show that it is possible to apply the PARIGE al-

gorithm at run-time to deploy operational strings dy-

namically, we now analyze the time complexity of thePARIGE algorithm and evaluate different effects of ap-plying this algorithm to different configurations of op-erational string.

3.4.1. Time Complexity Analysis The input ofthe PARIGE algorithm is defined as follows:

N : Total number of operational stringsC: Total number of dependencies between opera-

tional strings|V |: Average number of components within an op-

erational string|E|: Average number of connections within an oper-

ational stringAs shown in Algorithm 1, the sort of all external de-

pendencies has a complexity of O(C ∗ log(M)). Therecursive procedure shown in Algorithm 2 shows thatprocessing each dependency trace has a time complex-ity of O(N ∗ (|V |+ |E|)). The overall time complexityof the PARIGE algorithm exhibits a linear complex-ity of O(C ∗N ∗ (|V |+ |E|). In practice, C tends to bemuch smaller compared with |V | and |E|.

In summary, the linear property of our algorithmmakes it possible to apply it dynamically at run-time.Section 4 evaluate the performance overhead empiri-cally in the context of the NASA MMS mission systemcase study.

3.4.2. PARIGE Algorithm Effects on Opera-tional String Deployment Two effects that thePARIGE algorithm could have on the predictability ofoperational string deployment are described below.

Operational string growth effect. This effectmeasures the cost of promoting a number of compo-nents from lower priority operational strings to higherpriority operational strings. Since the deployment ofeach component takes time and consumes resources,the fewer components that are migrated, the more ben-efits the algorithm can provide since priority-inverteddependencies can be satisfied without deploying manycomponents in lower priority operational strings.

In the worst case, all components from lower prior-ity operational strings could be migrated to higher pri-ority operational strings, which essentially merges dif-ferent operational strings together. In production DREsystems, such worst cases happen rarely, i.e., all thecomponents in all operational strings have just onlyone dependence trace. Even in such situations, thePARIGE algorithm still performs the same as a con-ventional approach that does not take priority into ac-count and only accounts for dependencies among oper-ational strings.

Component host distribution effect. This ef-fect means that due to the promotion of compo-

nents, components that can be deployed by contact-ing the NodeManager once now contacts the sameNodeManager multiple times during deployment. Forexample, if an operational string A has 1 compo-nent to deploy on NodeA and operational string Bhas 2 components to deploy on NodeB these two op-erational strings can be deployed by contacting theNodeManager on each node only once. If the algorithmmoves one component from operational string B to op-erational string A, then the NodeManager on NodeB

must be contacted twice, once when deploying the mi-grated component in operational string A and oncewhen deploying the remaining component in opera-tional string B.

Such an effect can increase the overall deploy-ment time due to the increasing number of round tripdelays. One way to alleviate this problem is to in-crease the parallelism among different nodes by usingasynchronous techniques between the Execution-Manager and NodeManagers, such as the Asyn-chronous Method Invocation (AMI) messaging pol-icy provided by CORBA [12]. For example, AMI cancoordinate all the NodeManagers in the domain par-allelism deployment can be achieved among all thenodes, therefore alleviating the component host distri-bution effects.

4. Empirical Results

To evaluate the benefits of our PARIGE algorithm,we applied it to a representative DRE system proto-type of the NASA MMS mission system described inSection 2. This section first describes the characteris-tics of the hardware and software testbed and explainsour experiment design. We then empirically evaluatethe effectiveness of our PARIGE algorithm in three di-mensions: (1) Sections 4.2 and 4.3 empirically measurethe effectiveness of PARIGE to address challenges 1, 2,and 3 in Section 2, Section 4.4 empirically measures theeffectiveness of PARIGE to address challenge 4 in Sec-tion 2, which uses higher level metrics to measure theDRE system D&C QoS, and (3) Section 4.5 empiricallymeasures the performance overhead of the PARIGE al-gorithm.

4.1. Hardware and Software Testbed

We used the ISISlab open testbed (www.dre.vanderbilt.edu/ISISlab) for all our experi-ments. Our experiments used up to 6 nodes run-ning Linux FC4 with Ingo Molnar’s real-time kernelpatches. When operational strings are deployed we useone node to run the central coordinator Execution-

Manager and the rest of the nodes as the deploymenttargets.

The NASA MMS mission system prototyped usedfor our experiments was developed using the CIAO [13]and DAnCE [7] component middleware. This appli-cation consists of 45 components grouped togetherinto 3 operational strings Figure 13 shows an exampleoperational string consisting of a science agent com-

Comm Ground

Gizmo 1 Filter 1 Analysis 1


Science Agent



Figure 13: A Sample Operational String of the Exper-iments

ponent that decomposes mission goals into navigation,control, data gathering, and data processing applica-tions. Multiple gizmo components are connected to thescience agent and are also connected to different pay-load sensors. Each gizmo component collects data fromthe sensors, which have varying data rate, data size,and compression requirements.

The collected data is passed through filter compo-nents, which remove noise from the data. The filtercomponents pass the data onto analysis components,which compute a quality value indicating the likeli-hood of a transient plasma event. Finally, the analyzeddata from each analysis component is passed to a comm(communication) component, which transmits the datato the ground component at an appropriate time.

An operational string can span across multiple phys-ical nodes. The assignment of components to nodesis determined by a planner using high-level resourceplanning algorithms, and was available as input to ourPARIGE algorithm. We intentionally choose differentassignments for our experiments to compare how theycould affect the predictability and performance differ-ent operational strings deployments.

4.2. Effects of Operational String Recom-position

Hypothesis. The hypothesis of this experiment isthat the PARIGE algorithm should not change thefunctional correctness of the input operational stringsbut should produce correct dependencies between op-erational strings. In particular, the PARIGE algorithmmust ensure (1) all the dependencies between compo-nents of original operational strings should remain the

same after the recomposition and (2) the dependen-cies between operational strings must be changed suchthat no dependencies exist from higher priority opera-tional strings to lower priority operational strings.

Experimental design. The experiments consist of3 operational strings with each operational string hav-ing 15 components. The high priority operational stringhas one dependency to the medium priority operationalstring and the medium operational string has one de-pendency to the low priority operational string. Wemeasure the total number of components, number ofnodes, and number of dependencies (both internal andexternal) of each operational string before and after ap-plying the PARIGE algorithm.

Empirical results and analysis. Table 1 summa-rizes the number of components, number of nodes, andnumber of dependencies for each operational string be-fore and after we run the PARIGE algorithm.

High Medium Low Total

Components Before 15 15 15 45

Components After 18 14 13 45

Dependencies Before (H-¿L / L-¿H)

1/0 1/0 1/0 3

Dependencies After (H-¿L/ L-¿H)

0/0 0/3 0/3 3

Dependencies Before (In-ternal + External)

63 (60+3)

Dependencies After (Inter-nal + External)

63 (57+6)

Table 1: PARIGE Effect on Operational Strings

The results in the figure indicate that the numberof components of high priority operational string in-creases from 15 to 19, while that of both medium prior-ity and low priority operational strings decreases from15 to 13, so the total number of components do notchange. In addition, before the experiment, the 3 op-erational strings have 60 internal dependencies and 3external dependencies, with 63 dependencies in total.After the experiment, the number of internal depen-dencies decreases by 1 to 59 and the total number ofdependencies increase by 1 to 4, with the total num-ber still remains the same, which is in accord with thefirst part of our hypothesis. Finally, after applying thePARIGE algorithm, all dependencies from higher pri-ority to lower priority operational strings are removed,which validates the second part of our hypothesis.

4.3. Deployment Latency vs. DeploymentPriority

Hypothesis. The hypothesis of this experiment isthat the PARIGE algorithm can avoid priority inver-sion when deploying multiple operational strings wherehigher priority operational strings have dependencieson lower priority operational strings.

Experimental design. We conducted two experi-ments on different configurations of operational stringdependencies. Our first experiment consisted of 3 op-erational strings, each of which having 15 componentsevenly distributed into 5 nodes. Therefore, each nodehas 9 components. The high priority operational stringhas one dependency on the medium priority opera-tional string, which in turn has one dependency onthe low priority operational string. The dependency be-tween two operational strings is shown in Figure 14.

Comm Ground



Science Agent



Comm



Science Agent



Ground

Figure 14: Operational String Configuration with LowGrowth Rate

Next we conducted another experiment with the ex-ternal dependency between two different componentsin the operational strings, as shown in Figure 15. Wemeasured how the end-to-end deployment latency ofeach operational string can be affected in this config-uration. In this experiment, there are only two oper-ational strings, each having 15 components. The highpriority operational string has one dependency on thelow priority operational string, as shown in Figure 15.

Both experiments first measured the end-to-end la-tency for deploying each operational string withoutapplying the PARIGE algorithm. We then measuredthe end-to-end latency for deploying each operationalstrings with the PARIGE algorithm to see how latencyrelates to their priorities.

Comm Ground



Science Agent



Comm



Science Agent



Ground

Figure 15: Operational String Configuration with HighGrowth Rate

Empirical results and analysis.Figures 16 and 17 shows the end-to-end latency of

D&C request for each operational string in the two ex-periments described above. As shown in the Figure 16,

0

500000

1000000

1500000

2000000

2500000

3000000

3500000

4000000

4500000

5000000

Latency(us)

LowPrio

MediumPrio

HighPrio

Total LowPrio-AMI

MediumPrio-AMI

HighPrio-AMI

Total-AMI

End-to-End Deployment Latency of Operational String Deployment

Without PARIGE

With PARIGE

Figure 16: D&C Latency Changes by the PARIGE Al-gorithm

without applying the PARIGE algorithm, the high pri-ority operational string yields the highest latency whilethe low priority operational string yields the lowest la-tency, while the latency of medium priority operationalstring lies in between.

In our experiments, there is one dependency fromhigh priority operational string to medium prior-ity operational string and another dependency frommedium priority operational string to low priority op-erational string. Without applying the PARIGE al-gorithm, therefore, the low priority operationalstring must be deployed first among the three, fol-lowed by medium priority and high priority opera-tional strings, respectively. The PARIGE algorithm re-

moves the priority inverted dependencies which avoidsdeployment priority inversion, as illustrated in the fig-ure.

Figures 16 also shows how the component host dis-tribution effect introduced by the PARIGE algorithmis masked by applying AMI messaging policy, as de-scribed in Section 3.4.2. In our experiment, applyingAMI improves the performance of the deployment intwo aspects. First, the deployment latency of each op-erational string is reduced because of the Execution-Manager can coordinate the NodeManagers to do de-ployment in parallel. Second, it masks the componenthost distribution effect, which results in a reduced to-tal latency of all operational strings, as shown in thefigure.

Operational String Growth Rate Effect

0

100

200

300

400

500

600

700

Latency (miliseconds)

Low Priority

High Priority

Low Priority 216942 0

High Priority 636233 629922

Without Algorithm With Algorithm

Figure 17: D&C Latency Changes by the Algorithm

High Operational String Growth Figure 17 showsthat after applying the PARIGE algorithm the highpriority operational string has the lowest latency sinceit has no external dependency on any other operationalstrings. The size change of each operational string isalso minimal since the number of migrated componentsis small due to the dependency trace characteristics.

On the other hand, the dependency between the twooperational strings in our second experiment caused allcomponents from the low priority operational string tomigrate to the high priority string, essentially mergingthe two operational strings together. As a result, the la-tency of deploying the high priority operational stringis nearly the same as deploying it without applying thePARIGE algorithm. However, in a DRE system withmultiple operational strings to deploy, it is rare thatall components have only one dependence trace, as de-scribed in Section 3.4.2.

4.4. Effectiveness of PARIGE AlgorithmBased on High Level Metrics

Hypothesis. The hypothesis of this experiment isto reduce the quiescence time of operational stringsduring the period when these operational strings aredeployed. This experiment measures the effectivenessof the PARIGE algorithm based on human-perceivablemetrics. As described in Section 1, the dynamic natureof open DRE systems require on-demand injection ofcertain operational strings to ensure systems are keptin sync with changing mission goals. When operationalstrings with different importance to the entire DREsystem must be deployed at the same time, the D&Cframework must deploy these operational strings in ef-fectively to ensure the overall system QoS. In this ex-periment, therefore, each operational string is assignedwith a mission effectiveness value (MEV) to quantita-tively represent its utility value for the entire system,using two utility metrics described below.

Metric M1 : average MEV loss. The followingutility function M1 measures the average MEV losscaused by the operational string deployment cost.

M1 =m∑

i=1

DeploymentT ime(Si)TotalDeploymentT ime

×MEV (Si), i = 1..m

(1)Since each operational string has an associated deploy-ment cost (measured by its deployment latency), thiscost will necessarily cause a period of quiescence time,which results in a MEV loss.

M1 is computed by first dividing the deploymenttime of each operational string by the total end-to-enddeployment time for all operational strings and thenmultiply by the MEV of that operational string to pro-duce the weighted MEV loss for that operational string.Next, we sum up all weighted MEV loss of each opera-tional string. In the context of our MMS case study asdescried in Section 2, M1 measures the weighted quies-cence time of the the three operational strings and theirpeer components during the period when the three op-erational strings are deployed.

Metric M2 : Highest priority operationalstring MEV loss. The following utility func-tion M2 is similar to M1.

M2 =∑ DeploymentT ime(Smax)

TotalDeploymentT ime(2)

Rather than measuring the weighted MEV loss of alloperational strings, however, M2 measures the MEVloss the highest priority operational string(s) causedby the deployment.

M2 is computed by dividing the total deploymenttime of the highest priority operational string by total

end-to-end deployment time for all operational strings.In the context of our MMS case study as descried inSection 2, M1 measures the quiescence time of thehighest priority operational string A and its peer com-ponents during the period when the three operationalstrings are deployed.

Experimental design. These experiments con-sisted of 3 operational strings having priorities high,medium, and low, respectively, with each string having15 components. The high priority operational stringhas one dependency on the medium priority opera-tional string, which in turn has one dependency on thelow priority operational string. The mission effective-ness values of the three operational string are summa-rized in Table 2. In general, higher priority operationalstrings provide higher mission effectiveness values toDRE system than lower priority operational strings.

String Priority High Medium Low

MEV 3 2 1

Table 2: Mission Effectiveness Values of OperationalStrings

The first experiment has a high promotion growtheffect, and the second experiment has a low promo-tion growth effect, as described in Section 3.4. The ex-act configuration of external dependencies in the ex-periment are the same as those in Section 4.3.

Empirical results and analysis. Table 3 showsthat when the operational growth effect is low, M1was reduced by 40% and M2 was reduced by 69%,which indicates that the PARIGE algorithm can sig-nificantly reduce the mission effectiveness loss for alloperational strings. In the high growth effect situation,

M1 M2

Baseline (without PARIGE algorithm) 4.71 1

Low growth effect 2.83 0.31

High growth effect 5.79 1

High growth effect w/ optimization 4.71 1

Table 3: Impact of Operational String Growth Effect

however, the worst case scenario happens when opera-tional strings are merged together. The result in suchworst case scenario shows that M2 is the same as ourbaseline case, i.e., without applying the PARIGE al-gorithm. This operational string merge effect indicatesthat the utility to the entire DRE system by the highestpriority operational string(s) remains the same, but M1

(which measures the weighted MEV of all operationalstrings) is worse than before. To overcome this short-coming, we applied an optimization technique that se-lectively chooses the deployment descriptor before ap-plying the PARIGE algorithm, as described in Sec-tion 3.2.4, thereby avoiding the degradation of M1, asshown in Table 3.

4.5. Performance Overhead of thePARIGE Algorithm

Hypothesis. The hypothesis of this experiment isthat the performance overhead of the PARIGE algo-rithm is small enough so it can be applied to deployoperational strings at run-time. In contrast to off-lineanalysis techniques, the PARIGE algorithm must bedeployed by ExecutionManager to handle requests atruntime, therefore, the PARIGE algorithm should notincur excessive performance overhead to the end-to-endlatency of deployment of operational strings.

Experimental design. The experiments consist of3 operational strings each having 15 components and2 external dependencies in total. The high priority op-erational string has one dependency on the mediumpriority operational string, which in turn has one de-pendency on the low priority operational string. Wefirst measured the end-to-end latency for deploying allthe operational strings without applying the PARIGEalgorithm. We then measured the end-to-end latencyfor deploying increasing number of operational stringswith the PARIGE algorithm to measure how much la-tency overhead was contributed by running the algo-rithm.

Empirical results and analysis. We first mea-sure the PARIGE algorithm performance itself to de-termine how its performance is affected by the sizeof the problem, i.e., number of components (deter-mined by number of operational strings) and number ofpriority-inverted external dependencies. We then mea-sure its performance overhead against an actual exam-ple with 3 operational strings and 2 external dependen-cies, as described above.

Figure 18 shows the performance result of PARIGEalgorithm itself with increasing number of componentsand number of external dependencies. The results showthat the performance of PARIGE algorithm is roughlylinear to both the number of components and num-ber of external dependencies, which is consistent withthe analysis performed in Section 3.4.1. The linear run-time performance characteristics of PARIGE algorithmmakes it suitable for dynamically deploying operationalstrings online at runtime because the deployment la-tency of all operational strings exhibits a linear time

15 30 45 60 75 90 105 1201

4

7

0

500

1000

1500

2000

2500

3000

3500

4000

4500

5000

time

(us)

# of components

# of

dependencies

4500-5000

4000-4500

3500-4000

3000-3500

2500-3000

2000-2500

1500-2000

1000-1500

500-1000

0-500

Figure 18: Running Time of the PARIGE Algorithm

complexity to the number of components in the oper-ational strings.

As long as the performance overhead of thePARIGE algorithm is acceptable to deploy one com-ponent, therefore, it should be acceptable to deployany number of components. To validate this claim, weconducted an experiment that deployed up to 64 op-erational strings with 960 total components. Theresults in Figure 19 shows that the deployment la-tency of all operational strings with and without thePARIGE algorithm. The experiment measures differ-ent number of operational strings and different num-ber of components, ranging from 1 operational stringwith 15 components to 64 operational strings with 960components. These results show that the actual per-

End-to-end Deployment Latency of All Operational Strings

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

45,000

1(15)

2(30)

4(60)

8(120)

16(240)

32(480)

64(960)

Number of Operational Strings

(Number of Components)

Latency (miliseconds)

Without PARIGE

With PARIGE

Figure 19: Performance Overhead of the PARIGE Al-gorithm

formance overhead of PARIGE algorithm for ourexperiment is consistently less than ∼1%, which fur-ther validates our earlier analysis.

5. Related Work

This section compares our work on the PARIGE al-gorithm with related work. We classify the related work

into two categories. The first category includes formalmodels and using various formal method techniques toensure consistency and dependency correctness issuesof software deployment. The second category includesrelated work that addresses various QoS issues withsoftware deployment.

5.1. Dependency Management Models

Much prior research has been conducted on depen-dency management of software components. Most liter-ature in this area analyzes deployment installability viaformal models that explicitly express dependencies ofsoftware components or software packages. For exam-ple, [14] present a model to formalize deployment de-pendencies of software components. These dependen-cies are expressed in a logical language associated witha D&C framework that allows proving properties (suchas whether the dependency is mandatory, optional, ornegative) of the deployment plan. [15] defines an ap-plication buildbox as a software deployed environmentand defines a formal Labeled Transition System (LTS)on the buildbox with transitions for deployment op-erations that include build, install, ship, and update.Formal properties of the LTS are introduced, includ-ing version dependency and software component de-pendencies. This Prior work, however, does not addressdeployment predictability, which makes it unsuited forDRE systems.

Some software frameworks have been devel-oped to support software component deploymentor redeployment. For examples, [16] and [17] usesframework-guided reconfiguration mechanism forcomponent-based distributed systems. These frame-works offer mechanisms to analyze dependenciesbetween peer components to deal with reconfigura-tion consistency, which is similar to our work. Unlikeour approach, however, this work does not con-sider how to improve the predictability of the de-ployment process, but instead focuses only on consis-tency.

5.2. QoS Assurance of Software Deploy-ment

Prior related work has focused on improving the QoSof software D&C frameworks. Existing literature in thisarea focuses on performance related issue, but ignorespredictability related issues, which are important toensure DRE systems QoS. For example, The work re-ported in [18] relies on pre-allocating resource (e.g.,pre-load components in .NET dynamic reconfigurationframework) to make the D&C process of componentsmore predictable. Unlike our approach, however, [18] isonly concerned with how to improve the response time

within a single deployment unit, rather than of man-aging multiple deployment units at the same time.

The approach described in [19] uses a virtual ma-chine (VM) technique that provides automated D&C offlexible VMs that can be configured to meet applicationneeds and then subsequently cloned and dynamicallyinstantiated to improve the predictability of deploy-ments. This approach supports a graph-based modelfor the definition of customized VM configuration ac-tions. Our work on PARIGE is complementary to thiswork and can further improve the predictability of com-ponent deployment by differentiating services of multi-ple operational strings.

5.3. Comparison between PARIGE algo-rithm and Priority Inheritance Proto-col

The PARIGE algorithm has many similarities to thepriority inheritance protocol [20] used for synchroniza-tion in real-time systems. The priority inheritance pro-tocol ensures that when a thread blocks one or morehigh priority threads, it executes its critical section atthe highest priority level of all the threads it blocks,i.e., it inherits the highest threads priority. After exe-cuting its critical section, the thread returns to its orig-inal priority level.

In the PARIGE algorithm lower priority operationalstrings execute at (i.e., “inherit”) the priority of higherpriority operational strings to avoid deployment pri-ority inversions. The “critical section” in the prior-ity inheritance protocol is thus similar to the “de-ployment and configuration” activities in the PARIGEalgorithm. Our work differs from the priority inheri-tance protocol in several ways, however. First, the pri-ority inheritance protocol does not avoid deadlocks, butPARIGE algorithm does not have such limitation be-cause the algorithm removes all priority inverted de-pendencies between operational strings and then de-ploy operational strings from the highest priority tothe lowest priority sequentially. Second, only part of theoperational string is affected, i.e., the PARIGE algo-rithm just increases the deployment priorities of com-ponents with dependencies from higher priority opera-tional strings. Third, the PARIGE algorithm is muchmore complex than the priority inheritance protocolbecause it needs to traverses multiple graphs to iden-tify which components require promotion.

6. Concluding Remarks

The predictability and scalability of D&C frame-works is essential to support the QoS requirementsof open DRE systems. This paper describes a multi-graph algorithm that helps ensure the predictability of

deploying multiple operational strings. We first ana-lyze how deployment priority inversion can occur whenoperational strings have various dependency relation-ships. We then empirically show how the partial priorityinheritance via graph recomposition (PARIGE) algo-rithm can effectively avoid deployment priority inver-sions and thus improve the predictability of componentdeployment in DRE systems.

The following summarizes our lessons learned thusfar from developing and applying the PARIGE algo-rithm to ensure the predictability of deployment of op-erational strings in DRE systems:

1. The overlap of deployment-time with run-time makes D&C frameworks essential to en-sure system QoS. The benefits provided by compo-nent middleware significantly change the lifecycle ofDRE system development. Due to the complexities ofopen DRE systems, D&C frameworks assume more re-sponsibilities to ensure system QoS because deploy-ment of system services/components occurs through-out the lifecycle of the systems. By using informationavailable at deployment time, D&C frameworks can ef-fectively identify the complex dependency relationshipsamong operational strings and perform various on-lineoptimizations, such as the operational string recompo-sition presented in Section 3.

2. Automated Recomposition of operationalstrings can help ensure deployment predictabil-ity of DRE systems. Although operational stringscan simplify the design of DRE systems, it is hardto manually ensure deployment predictability of alloperational strings due to the complex dependenciesamong many operational strings. The PARIGE algo-rithm presented in this paper enhances the deploymentpredictability of different operational strings by recom-posing operational strings automatically based on theinput to the D&C framework and transparently to sys-tem deployers.

3. Recomposition of operational strings bythe PARIGE algorithm is orthogonal to latermanagement of operational strings. As describedin Section 1, component-based DRE systems are of-ten designed, deployed and managed in the form ofoperational strings that are first class entities whoselifecycles are managed by D&C frameworks. Althoughour PARIGE algorithm recomposes operational stringsand modifies their topologies to avoid deployment pri-ority inversion, it does not change the behavior of theoperational strings once the algorithm-modified oper-ational strings are deployed. In particular, the D&Cframework’s ExecutionManager only recomposes op-erational strings for its initial deployment, but oncedeployed the metadata descriptors for the operational

strings are still the original ones without any modifi-cation. Subsequent management of operational stringswill thus not be affected by the PARIGE algorithm.

4. The effectiveness of the PARIGE algo-rithm depends on operational string character-istics and their dependencies. The PARIGE algo-rithm can improve deployment predictability of oper-ational strings with negligible performance overhead,as demonstrated in Section 4.5. The effectiveness ofthe PARIGE algorithm varies for different configura-tions of operational strings and the external dependen-cies among them. As shown by the empirical results inSection 4.3, the PARIGE algorithm is most effectivelywhen operational string growth is minimal and least ef-fectively when growth is large. In the worst case sce-nario, however, when there exits only a single depen-dency trace across all the operational strings, all the op-erational strings are merged into a single operationalstring. In this case, no predictability improvement canbe made by the PARIGE algorithm. Since the perfor-mance overhead of the PARIGE algorithm is negligi-ble, however, the PARIGE algorithm based D&C ap-proach will at least performs approximately the same aswithout the algorithm being applied. Our future worktherefore will investigate how to quantify the gain ofthe PARIGE algorithm by measuring the operationalstring growth effect and the deployment cost of differ-ent components based on the input of the operationalstrings.

5. Advanced OS and middleware features areimportant complements to the PARIGE algo-rithm. The PARIGE algorithm can incur certain ef-fects when recomposing operational strings, such asthe component host distribution effect discussed in Sec-tion 3.4. Our experience shows that modifying themulti-graph based PARIGE algorithm itself alone is in-sufficient to address this undesired effect because thealgorithm introduces constraints on host collocation/-distribution, which affects its performance. One way toalleviate this problem is to apply asynchronous methodinvocations, as presented in Section 4.3.

Our future work will also determine whether/howthe results from the PARIGE algorithm runs can pro-vide feedback to system designers. For example, theD&C framework can analyze the input and output tothe PARIGE algorithm for each deployment request.Using this historical input/output information, a D&Cframework can potentially identify those operationalstrings responsible for most deployment priority inver-sion. We conjecture that this approach will help im-prove DRE system D&C by reorganizing operationalstrings more effectively.

The PARIGE algorithm is an integral part

of DAnCE, and both DAnCE and CIAO areopen-source and available for download atwww.dre.vanderbilt.edu/ciao.

References

[1] Gill, C., Gossett, J., Loyall, J., Schmidt, D., Corman, D.,Schantz, R., Atighetchi, M.: Integrated Adaptive QoSManagement in Middleware: A Case Study. In: Proceed-ings of the 10th IEEE Real-time and Embedded Tech-nology and Applications Symposium (RTAS), Toronto,Canada, IEEE (May 2004)

[2] Schmidt, D.C., Schantz, R., Masters, M., Cross, J.,Sharp, D., DiPalma, L.: Towards Adaptive and Reflec-tive Middleware for Network-Centric Combat Systems.CrossTalk - The Journal of Defense Software Engineer-ing (November 2001)

[3] Sharma, P., Loyall, J., Heineman, G., Schantz, R.,Shapiro, R., Duzan, G.: Component-Based DynamicQoS Adaptations in Distributed Real-time and Embed-ded Systems. In: Proc. of the Intl. Symp. on Dist. Ob-jects and Applications (DOA’04), Agia Napa, Cyprus(October 2004)

[4] Lardieri, P., Balasubramanian, J., Schmidt, D.C.,Thaker, G., Gokhale, A., Damiano, T.: A Multi-layeredResource Management Framework for Dynamic Re-source Management in Enterprise DRE Systems. Jour-nal of Systems and Software: Special Issue on DynamicResource Management in Distributed Real-time Sys-tems 80(7) (July 2007) 984–996

[5] Desertot, M., Cervantes, H., Donsez, D.: Frogi: Fractalcomponents deployment over osgi. In: Software Compo-sition. (2006) 275–290

[6] Quema, V., Balter, R., Bellissard, L., Feliot, D.,Freyssinet, A., Lacourte, S.: Asynchronous, hierarchi-cal, and scalable deployment of component-based appli-cations. In: Component Deployment, Second Interna-tional Working Conference, CD 2004, Edinburgh, UK,May 20-21, 2004, Proceedings. Volume 3083 of LectureNotes in Computer Science., Springer (2004) 50–64

[7] Deng, G., Balasubramanian, J., Otte, W., Schmidt,D.C., Gokhale, A.: DAnCE: A QoS-enabled Compo-nent Deployment and Configuration Engine. In: Proc.of the 3rd Working Conf. on Component Deployment(CD 2005), Grenoble, France, Springer-Verlag (Novem-ber 2005) 67–82

[8] Suri, D., Howell, A., Schmidt, D.C., Biswas, G., Kin-nebrew, J., Otte, W., Shankaran, N.: A Multi-agent Ar-chitecture for Smart Sensing in the NASA Sensor Web.In: Proceedings of the 2007 IEEE Aerospace Conference,Big Sky, Montana (March 2007)

[9] Rigole, P., Clerckx, T., Berbers, Y., Coninx, K.: Pos-sibilities to improve ground-based cloud cover observa-tions using satellite application facility (safnwc) prod-ucts. GEOGRAFIJA. 43(1) (2007) 21–29

[10] Object Management Group: Deployment and Configu-ration Adopted Submission. OMG Document mars/03-05-08 edn. (July 2003)

[11] Bettati, R., Liu, J.W.S.: End-to-end scheduling to meetdeadlines in distributed systems. In: International Con-ference on Distributed Computing Systems. (1992) 452–459

[12] Schmidt, D.C., Vinoski, S.: Programming Asyn-chronous Method Invocations with CORBA Messaging.C++ Report 11(2) (February 1999)

[13] Wang, N., Gill, C., Schmidt, D.C., Subramonian, V.:Configuring Real-time Aspects in Component Middle-ware. In: Proc. of the International Symposium onDistributed Objects and Applications (DOA’04). Vol-ume 3291., Agia Napa, Cyprus, Springer-Verlag (Octo-ber 2004) 1520–1537

[14] Belguidoum, M., Dagnat, F.: Dependency managementin software component deployment. Electron. NotesTheor. Comput. Sci. 182 (2007) 17–32

[15] Liu, Y.D., Smith, S.F.: A formal framework for com-ponent deployment. In: OOPSLA ’06: Proceedings ofthe 21st annual ACM SIGPLAN conference on Object-oriented programming systems, languages, and applica-tions, New York, NY, USA, ACM (2006) 325–344

[16] Chen, X., Simons, M.: A Component Framework for Dy-namic Reconfiguration of Distributed Systems. In: CD’02: Proceedings of the IFIP/ACM Working Conferenceon Component Deployment, London, UK, Springer-Verlag (2002) 82–96

[17] Kon, K., Campbell, R.: Dependence Management inComponent-based Distributed Systems. IEEE Concur-rency 8(1) (2000) 26–36

[18] Rasche, A., Polze, A.: Configuration and dynamic re-configuration of component-based applications with mi-crosoft .net. In: ISORC ’03: Proceedings of the SixthIEEE International Symposium on Object-OrientedReal-Time Distributed Computing (ISORC’03), Wash-ington, DC, USA, IEEE Computer Society (2003) 164

[19] Krsul, I., Ganguly, A., Zhang, J., Fortes, J.A.B.,Figueiredo, R.J.: Vmplants: Providing and managingvirtual machine execution environments for grid com-puting. In: SC ’04: Proceedings of the 2004 ACM/IEEEconference on Supercomputing, Washington, DC, USA,IEEE Computer Society (2004) 7

[20] Sha, L., Rajkumar, R., Lehoczky, J.P.: Priority In-heritance Protocols: An Approach to Real-time Syn-chronization. IEEE Transactions on Computers 39(9)(September 1990)

ensuring deployment predictability of distributed real

Documents