West Virginia
University
Architectural-Level Risk Analysis for UML Dynamic Specifications
Dr. Sherif M. YacoubDr. Sherif M. [email protected]
Hewlett-Packard LaboratoriesPalo Alto, CA
Alaa Ibrahim, and Hany H. Alaa Ibrahim, and Hany H. AmmarAmmar
{ibrahim,ammar}@csee.wvu.eduDepartment of Computer Science
and Electrical Engineering
West Virginia University99thth International Conference on Software Quality International Conference on Software Quality Management, SQM2001Management, SQM2001
1818thth-20-20thth April, 2001 April, 2001Loughborough University, Loughborough, EnglandLoughborough University, Loughborough, England
West Virginia
University
Outline
Research Objectives
Methodology
Towards an Automated Methodology
Process
Case Study: The Pacemaker example
Conclusions
West Virginia
University
Architectural-Level Risk Assessment
Methodology at the early stages of
development(S. Yacoub, H. Ammar. ISSRE'00, IEEE Comp. Soc., October,
2000)
Automated Environment
Automated Risk Assessment
Research Objectives
West Virginia
University
Automated Risk Assessment(continued) Architectural-Level Risk
Assessment Methodology (S. Yacoub, H. Ammar. ISSRE'00, IEEE Comp. Soc., October, 2000)
Utilizes:
• Dynamic Metrics: Component Complexity cpxi Connector
Complexity cpxij (S. Yacoub, H. Ammar, and T. Robinson. Metrics'99,
November 1999)
• Failure Mode Effect Analysis FMEA (MIL_STD 1629A to define
Component Severity svrtyi Connector Severity svrtyij)
• Component Dependency Graphs CDG (adopted from: S. Yacoub, B.
Cukic, and H. Ammar. ISSRE'99 November 1999)
Defines:
• Heuristic Component Risk Factor hrfi = cpxi x svrtyi
• Heuristic Connector Risk Factor hrfij = cpxij x svrtyij
• Risk Aggregation Algorithm that produces HRFappl
West Virginia
University
• Model the architecture of the system using simulation models (UML-RT).
• Perform complexity analysis using simulation traces.
• Perform severity analysis using FMEA and simulation runs.
• Develop heuristic risk factors for components and connectors.
• Develop Components Dependency Graph for risk assessment purposes. (System/Subsystems)
• Aggregate the risk factors using the graph traversal algorithm.
Automated Risk Assessment Architectural-Level Risk Assessment
Methodology (continued) 6 Steps
West Virginia
University
Component Complexity Factors
Connector complexity Factors
CDG “hrfi and hrfij unidentified”
Formatted Excel charts
Violation Tables
UML Simulation Environment
SimulationSettings
SimulationLog andViolationReport
AnalysisTool
Timing Diag.
Violation Table
Analyst
Rose Real Time toolText File
MS Excel
ProcessingMacro
Inspection
Viewing Macro
UML Model
Observer
Sub RunSettings
AnalysisTool HRF
MS Excel
RiskMacro
Excel sheets
SeverityRanking
Severity Analysis (Failure/Effect analysis)
Automated Risk Assessment(continued) Automated Environment
CARA Tool
West Virginia
University
Model the architecture of the system together with the risk logging capability using Rose RealTime.
Adjust the simulation runs in the observer as desired.
Run the simulation and get two log files containing:
• Component complexities.
• Component Execution Time.
• A log of all the messages exchanged.
Automated Risk Assessment Automated Environment (continued)
Process
West Virginia
University
Process the log with Excel Risk Macro and get:
• Transition Probabilities.
• Connector complexities.
• CDG “where Risk Factors = Severity Factors * Complexity Factors (hrfi = cpxi x svrtyi )”
Perform severity analysis using FMEA and simulation runs.
Traverse the CDG using the Excel traversal macro.
Automated Risk Assessment Automated Environment
Process (continued)
West Virginia
University
Example: Pacemaker Main Use Case Diagram
Programming Mode
Operational Modes
DoctorsProgramer
Operating_in_AVIOperating_in_ AAT
Operating_in_ VVIOperating_in_ VVT
PatientsHeart
1
1
1 1
1
1
1
1
1
111
1
Operating_in_ AAI
1
1
1
1
Programming
1
1
1
1
«extend»«extend»«extend»«extend»
«extend»«extend»«extend»«extend»
«extend»«extend»
West Virginia
University
1) Develop a Simulation Model Capsule Diagram
Example: Pacemaker
West Virginia
UniversityAtrial statechartCase Study: Pacemaker (continued)
Idle
A_AVI
A_Self_inhibited
A_Self_triggered
ToOn ToOff
ToTriggered
ToInhibited
ToAVI
ToOnToOnToOn ToOffToOff
ToTriggered
ToInhibited
ToAVI
ToOn
West Virginia
UniversityAtrial statechartCase Study: Pacemaker (continued)
Refractory
Wait
Pacing
ToAVI
A_Pace_Pulse_Done
V_Refract_Done_Received
ini tial ize
V_Sense_Received
Time_Out
ToAVIToAVI
A_Pace_Pulse_Done
V_Refract_Done_Received
ini tial ize
V_Sense_Received
Time_Out
A sequence diagram for the AVI scenarioCommunication
GnomeAtrial Ventricular Heart
ToON
ToON
ToAVI
ToAVI
Refactoring
V SenseGot V Sense
RefTimeOut
Waiting
V Refract Done
Waiting
SensTimeOut
A Pace Start
A Pace Start
Pacing
Pacing
PaceTimeOut
A Pace Done
Pace
Refactoring
Refactoring Refactoring
A sequence diagram for the Programming scenarioProgrammer ReedSwitch CoilDriver Communication
GnomeAtrial Ventricular
ApplyMagnetEnableComm
EnableComm
Pulse
Receiving
IDLE
Pulse
Count++,ResetTimer
BitTimeoutDecode(Count)Store Bit in Byte
Byte Full?enqueue(byte)Yes
ByteTimeOut
Waiting for Bit
IDLE
Count = 1, SetTimer
PulseCount =0
Receiving
ByteTimeOut
OR
IDLE
Waiting For Byte
Validating
IsValid?
Processing
ToAVI
ToON
ToON
ToAVI
YesHerezaByte(ACK)
No
HerezaByte(NAK)
Waiting to Transmit
Waiting to Send Next Byte
West Virginia
University2) Perform Complexity
Analysis
s21
s22
I
init
initI
s11t11
t12
t13
s1
s2
VGx(s11) + VGa(t11) + VGx(s1)+ VGa(t12) + VGe(s2) + VGa(t13) +VGe(s22)
||
1
)()(X
x
ixxi oocpxPSoOCPX
A Transition between Composite States in a component’s Statechart
Operational Complexity of a component using the scenario profile and its complexity per scenario.
West Virginia
University2) Perform Complexity Analysis (cont’d)
A) Quantify Component Complexity Factors using dynamic complexity metrics.
RS CD CG AR VTProgramming ( 0.01) 8.3 67.4 24.3AVI (0.29) 53.2 46.8AAT (0.15) 100AAI (0.20) 100VVI (0.15) 100VVT (0.20) 100% of architecture complexity .083 0.674 0.243 50.428 48.572Normalized to max. complexity 0.002 0.013 0.005 1 0.963
West Virginia
University2) Perform Complexity
Analysis (cont’d)
100|},|),({|
),(
x
jijijix
ji
MT
ooOooooMooEOCx
||
1
)()(X
x
ixxi oOQFSPSoOQFS
||
1
),(),(X
x
jixxji ooEOCPSooEOC
Export Object Coupling
Export Object Coupling(EOC)
EOC with scenario profiles
OQFS with scenario profiles
the export coupling for component Ci with respect to component
Cj, is the percentage of the number of messages sent from Ci to
Cj with respect to the total number of messages exchanged
during the execution of the scenario x
West Virginia
University2) Perform Complexity Analysis (cont’d)
B) Quantify Connector Complexity Factors using dynamic coupling metrics.
RS CD CG AR VT Programmer HeartRS 0.0014 0.0014CD 0.003 0.011CG 0.002 0.0014 0.0014AR 0.25 1VT 0.27 0.873Programmer 0.0014 0.006Heart 0.123 0.307
West Virginia
University3) Perform Severity Analysis
In performing severity analysis, each potential failure mode is ranked according to the consequences of that failure mode.
Steps:
• Identifying Failure Modes Failure modes of individual components.
(Functional faults and state-based faults) Failure modes of individual connectors.
(Interface fault analysis)
West Virginia
University3) Perform Severity Analysis (cont’d)
Steps (cont’d):
• Conducting Effect Analysis Inject the fault. Simulate the faulty model. Monitor output and compare to expected output. Identify the effect of the fault.
• Rank Severity Identify category: Minor, Marginal, Critical, or
Catastrophic. Assign severity index to each component i as (svrtyi),
which takes a value of 0.25, 0.50, 0.75, and 0.95
West Virginia
University
Worst case severity found for the RS, CD, CG, VT, and AR are Minor(0.25), Minor(0.25), Marginal(0.50), Catastrophic(0.95) and Catastrophic (0.95), respectively
Connector Name Failure Mode Cause of Failure Effect of Failure Criticality of effectsRS Failed to enable
communicationError in translatingmagnet command
Unable to program thepacemaker, schedulemaintenance task.
Minor
CD Failed to generategood command
Fault in developingthe command
Unable to program thepacemaker, schedulemaintenance task.
Minor
CG Failed to validatecommand
Fault in thevalidationprocedure
Cannot program thepacemaker, schedulemaintenance task.
Minor
Mis-interpreting aVVT command forVVI
Fault in processingcommand routine
Heart is continuously triggeredbut device is still monitored byphysician, need immediate fixor disable.
Marginal
VT No heart pluses aresensed though heart isworking fine.
Heart sensor ismalfunctioning.
Heart is incorrectly paced,patient could be harmed bycontinuous pulses.
Critical
Refract timer does notgenerate a timeout inan AVI mode
Timer not setcorrectly.
AR and VT are in refactoringstate, no pace is generated forthe heart, patient could die.
Catastrophic
AR Wait timer does notgenerate a timeout inAAI mode
Timer not setcorrectly.
AR stuck at the wait state, nopacing is done to the heart
Catastrophic
FMEA table for the Pacemaker components
West Virginia
UniversityFMEA table for the Pacemaker connectors
Connector Name Failure Mode Cause of Failure Effect of Failure Criticality of effectsRS-CG Failure to enable
communication of theCG
Magnet malfunctioning.RS failed to generatemessage.
Pacemaker is not programmed,schedule maintenance task
Minor
RS-CD Unable to disablecommunication of theCD with theprogrammer
Magnet malfunctioning.RS failed to generatecorrect disable message.
Pacemaker receive bits accidentallyfrom hazards but device is neverprogrammed because CG is disabled,schedule maintenance task.
Minor
CD-Programmer Failed to acknowledgeprogramming
Fault in coding thesending message
Pacemaker is not programmed,schedule maintenance task.
Minor
CD-CG Failed to send bytes ofprogram data to CG
Inappropriate count ofnumber of bits in a byte.
Pacemaker is not programmed,schedule maintenance task.
Minor
CG-AR Send incorrectcommand (ex ToOffinstead of ToIdle)
Incorrect interpretationof program bytes
Incorrect operation mode andincorrect rate of pacing the heart.Device is still monitored by thephysician, immediate maintenance ordisable is required.
Marginal
CG-VT Send incorrectcommand (ex ToOffinstead of ToIdle
Incorrect interpretationof program bytes
Incorrect operation mode andincorrect rate of pacing the heart.Device is still monitored by thephysician, immediate maintenance ordisable is required.
Marginal
AR-Heart Failed to sense heart inAAI mode
Sensor error. Heart is always paced while patientcondition requires only pacing theheart when no pulse is detected
Critical
Failed to pace the heartin AVI mode
Pacing hardware devicemalfunctioning
Heart could be in serious problembecause of no pacing.
Catastrophic
VT-AR VT failed to informAR of finishingrefractoring in AVImode
Timing mismatchesbetween AR and VToperation.
Failure to pace the heart. Catastrophic
West Virginia
University4) Develop Risk Factorshrfi = cpxi x svrtyi
where:
0 <= cpxi <= 1, is the normalized complexity level (dynamic complexity for components or dynamic coupling for connectors), and
0<= svrtyi < 1 , is the severity level for the architecture element.
RS CD CG AR VTDynamicComplexity
0.002 0.013 0.005 1 0.963
Severity 0.25 0.25 0.5 0.95 0.95Risk Factors 0.0005 0.00325 0.0025 0.95 0.91485
Risk Factors for the components in the example
West Virginia
University4) Develop Risk Factors (cont’d)
Comparison between risk factors based on static and dynamic metrics
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
RS CD CG AR VT
Ris
k F
acto
rs
Dynamic
CBO
NAS
Connector Risk Factors RS CD CG AR VT Programmer HeartRS 0.00035 0.00035CD 0.00075 0.00275CG 0.0005 0.0007 0.0007AR 0.2375 0.95VT 0.2565 0.82935Programmer 0.00035 .0015Heart 0.11685 0.29165
Risk Factors for the connectors in the pacemaker example
West Virginia
University5) Constructing the CDG
<Prog., 0,5>
<RS,5x10-4,5>
<CD, 3x10-3,5>
<AR,0.95,40>
<VT,0.9,40>
<Heart,0,5><CG, 2.5x10-2,5>
s
t
t
t
<, 0, .01>
<, 0, .64>
<, 0, .35>
<, 0, .01><, 0, .99>
<, 0, .99>
<, 0, .99>
<, 0, .99>
<, 0, .34><, 0, .36>
<,3.5x10-4, .002>
<,1.5x10-3,.008>
<,2.7x10-3,.008>
<,7.5x10-4,.002>
<,3.5x10-4,.005>
<,3.5x10-3,.005><,7x10-4,.0025>
<,5x10-4,.005>
<,7x10-4,.0025>
<,.12,.35><,.29,.64>
<,.26,.29>
<,.95,.47>
<,.24,.19>
<,.26,.29>
West Virginia
University6) Risk Aggregation Algorithm
The algorithm expands all branches of the CDG starting from the start node.
The breadth expansions of the graph represent logical "OR" paths.
• translated as the summation of aggregated risk factors weighted by the transition probability along each path.
The depth of each path represents the sequential execution of components:
• is given by the aggregate: HRF = 1 - i(1-hrfi)
West Virginia
UniversityRisk Aggregation AlgorithmProcedure AssessRiskParameters
consumes CDG, AEappl,(average execution time for the application)produces Riskappl
Initialization:Rappl = Rtemp = 1 (temporary variables for (1-RiskFactor) )Time = 0Algorithmpush tuple <C1, hrf1, EC1 >, Time, Rtemp
while Stack not EMPTY dopop < Ci, hrfi , ECi >, Time, Rtemp
if Time > AEappl or Ci = t; (terminating node)Rappl += Rtemp ;(an OR path)
else < Cj ,hrfj , ECj > children(Ci)
push (<Cj, hrfj ,ECj>, Time += ECi , Rtemp = Rtemp*(1-hrfi)*(1-hrfij )*PTij ) ( AND path)
endend while
Riskappl = 1- Rappl
end Procedure AssessRisk
West Virginia
UniversityRisk Aggregation Algorithm
The algorithm can be used for
• System-level Risk Assessment The risk of the pacemaker that is found to be ~ 0.9
• Subsystem-level Risk Comparison Complex systems are composed of many subsystems. The algorithm can be used to obtain a risk factor for a
subsystem using risk factors of its individual components. Compare risk factors of individual subsystems.
• Sensitivity Analysis Sensitivity to Uncertainties in Component Risk Factors Sensitivity to Uncertainties in Connector Risk Factors
West Virginia
UniversitySensitivity Analysis
0.0
0.2
0.4
0.6
0.8
1.0
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1
Risk Factor of Individual Components
Ove
rall
Ris
k F
acto
r o
f th
e S
yste
m
R(AR)
R(VT)
R(CG)
R(CD)
R(RS)
0.0
0.2
0.4
0.6
0.8
1.0
0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1
Risk Factor of Individual Connectors
Ove
rall
Sys
tem
Ris
k V
alu
e
R(RS-CD)
R(CG-CD)
R(AR-Heart)
R(VT-AR)
R(VT-Heart)
The Pacemaker risk factor as function of connector risk factors (one at a time)
The Pacemaker risk factor as function of component risk factors (one at a time)
West Virginia
UniversityBenefits
The approach helps in:
• Deciding which components in the architecture require more development resources.
• Deciding which connectors in the architecture are of highest risk. A high risk connector indicates that the interfaces between the corresponding components and the messaging protocol should be carefully designed.
• Studying how uncertainties in component risk factors affect the overall risk value of the system.
• Studying how uncertainties in connector risk factors affect the overall risk value of the system.
West Virginia
UniversityConclusion : Benefits The methodology is applicable early at the
architectural level.
The methodology is based on dynamic metrics. We use dynamic metrics to account for the fact that a fault in a frequently executed component will frequently manifest itself into a failure.
The methodology is based on simulation of architecture models. Simulation helps in:
• Performing FMEA procedures .
• Calculating the CDG parameters such as probability of transitions.
• Obtaining dynamic metrics.
West Virginia
UniversityConclusion : Issues
Using ordinal scale for measuring severity.
Effect of uncertainties in the scenario probabilities and the estimated average execution times.
Scalability issues, applying the methodology to a larger case study.
Methodology is limited to systems with statechart and sequence diagram specifications.
Questions...
West Virginia
UniversityMain Use Case Diagram
Programming Mode
Operational Modes
DoctorsProgramer
Operating_in_AVIOperating_in_ AAT
Operating_in_ VVIOperating_in_ VVT
PatientsHeart
1
1
1 1
1
1
1
1
1
111
1
Operating_in_ AAI
1
1
1
1
Programming
1
1
1
1
«extend»«extend»«extend»«extend»
«extend»«extend»«extend»«extend»
«extend»«extend»