Download - Practical Assurance Case Design
1
Practical Assurance Case Design
IV&V WorkshopS. R. BrownKeyLogic Inc
With my thanks and appreciation Don Ohi – Project Monitor
Travis Dawson – Chief EngineerBill Stanton for tolerating a lot of questions
9/2/2013
Why Assurance Cases• State your case:
• What are we trying to prove• How are we trying to prove it• What evidence supports the proof
Assurance Cases make the pointDesign of AssuranceSupport of Assurance
Related to:
Argumentation
Safety Case
GSN Standard Kelly, T.: 'Arguing Safety: A Systematic Approach to Managing Safety Cases', D.Phil Thesis, University of York (1998). Available for download from http://www-users.cs.york.ac.uk/~tpk 9/2/2013 2
State Your Case
My coffee will provide spousal satisfaction
ID EC_01
Examine each element of coffee satisfaction
Label AC_1
The coffee is adequately hot
ID EC_02
Two creamers have been added to the coffee
ID EC_03
No sugar has been added to the coffee
ID EC_04
The coffee warmers are
operating.
Label EC_03
Two creamer packets are on the
the counter
Label E_C01
No sugar packets are on the counter
Label EC_02
The coffee isserved
immediately
Label EC_04
Argue through coffee temperature discriminators.
Label AC_02
The coffee pot heater is on and operating
ID C_1020
The coffee is serves with no more than three minutes
delay.
ID C_1020
Spousal coffee satisfaction is critical to morning kharma.
Label JC_01
The two key parameters in coffee temperature are
initial temperature and time to serving.
Label JC_02
Spousal Coffee Assurance CaseTop level Claim
Claim Justification
Perspective: Who is the stakeholder?
9/2/2013 3
Make the Argument
My coffee will provide spousal satisfaction
ID EC_01
Examine each element of coffee satisfaction
Label AC_1
The coffee is adequately hot
ID EC_02
Two creamers have been added to the coffee
ID EC_03
No sugar has been added to the coffee
ID EC_04
The coffee warmers are
operating.
Label EC_03
Two creamer packets are on the
the counter
Label E_C01
No sugar packets are on the counter
Label EC_02
The coffee isserved
immediately
Label EC_04
Argue through coffee temperature discriminators.
Label AC_02
The coffee pot heater is on and operating
ID C_1020
The coffee is serves with no more than three minutes
delay.
ID C_1020
Spousal coffee satisfaction is critical to morning kharma.
Label JC_01
The two key parameters in coffee temperature are
initial temperature and time to serving.
Label JC_02
Architectural
decomposition
Behavioral
decomposition
9/2/2013 4
My coffee will provide spousal satisfaction
ID EC_01
Examine each element of coffee satisfaction
Label AC_1
The coffee is adequately hot
ID EC_02
Two creamers have been added to the coffee
ID EC_03
No sugar has been added to the coffee
ID EC_04
The coffee warmers are
operating.
Label EC_03
Two creamer packets are on the
the counter
Label E_C01
No sugar packets are on the counter
Label EC_02
The coffee isserved
immediately
Label EC_04
Argue through coffee temperature discriminators.
Label AC_02
The coffee pot heater is on and operating
ID C_1020
The coffee is serves with no more than three minutes
delay.
ID C_1020
Spousal coffee satisfaction is critical to morning kharma.
Label JC_01
The two key parameters in coffee temperature are
initial temperature and time to serving.
Label JC_02Gather the Evidence
Atomic evidence
9/2/2013 5
Assurance Case Semantic ContentMy coffee will provide
spousal satisfaction
ID EC_01
Examine each element of coffee satisfaction
Label AC_1
The coffee is adequately hot
ID EC_02
Two creamers have been added to the coffee
ID EC_03
No sugar has been added to the coffee
ID EC_04
The coffee warmers are
operating.
Label EC_03
Two creamer packets are on the
the counter
Label E_C01
No sugar packets are on the counter
Label EC_02
The coffee isserved
immediately
Label EC_04
Argue through coffee temperature discriminators.
Label AC_02
The coffee pot heater is on and operating
ID C_1020
The coffee is serves with no more than three minutes
delay.
ID C_1020
Spousal coffee satisfaction is critical to morning kharma.
Label JC_01
The two key parameters in coffee temperature are
initial temperature and time to serving.
Label JC_02
• What I get out of a cup of coffee:• The assurance design is not a description of the system• Assurance network needs only to go as far as necessary to provide stated assurance• Strength of evidence is a constant consideration• Without more information there is little basis for claim prioritization
• Can address risk (uncertainty)• Does not support consequence
9/2/2013 6
Where to Start
The GENERIC Spacecraft will travel to Planet X and
enter orbit, remaining in orbit doing nothing in particular
until the requir
ID G_1
Testable (yes/no)
Is Clearly Stated
Will decompose in a useful
way Lessons Learned:Look for simple and comprehensive claimsClaims must be objective (yes/no)Watch out for claims that must decompose through out-of-scope domains (e.g.:
reset/sideswap hardware timing)Take advantage of self-similarity, patterns and
common arguments.
Initially a major goal: In almost every case it is self-evident if the objective is defined. This is where perspective becomes important. Consider stakeholder needs.
9/2/2013 7
8
Perspectives and Planning• The perspective of the assurance case is important
• Software lifecycle (NPR 7150)• System/Human safety• 09-1 System definition• Project-specific
A place for (almost) everythingDecomposes differently than lifecycle
Represent the stakeholder
9/2/2013
9
Argument: the Art of Assurance Case
Some Useful decomposition classes
Appeal to architecture• Use software architecture to
decompose to subsystems• No uncertainty
Appeal to Usage• Use scenario to decompose
to subsystems by required actions
• Some uncertainty
Note: 3Q is a special case of appeal to usage
Appeal to Specificity• Provide detail to support
a more general claim• Little uncertainty
Arguments describe how the child claims satisfy the parent claim but do not carry the burden of proof
Some approaches require a justification for each argument but this was often redundant.
Assumptions are often necessary in order that the argument is clear to a reviewer
Arguments are in the context of the system
Use cases can be a good basis for an Appeal to Usage. Note semantic equivalence.9/2/2013
10
The spacecraft will calculate estimated attitude based on
IMU, STAR, and SUN sensors
ID G_56
Iterate through AD functional behaviors
Label S_700AD performs sensor integration
via a 12 state Kalman Filter. Steps include forward
propagation, noise covariance update bias update, and check
for divergence.
Label J_701
The AD system will calculate state propagation
ID G_701
The AD system will calculate noise covariance and
accelerometer bias
ID G_702
The AD system will detect Kalman filter divergence
ID G_703
Kalman filter algorithms are not validated (scope). There
is evidence to support this assumption but the decomp is
out of scope.
Label J_702
Ibid J_702
Label J_703
Ibid J_702
Label J_704
If the covariance exceeds a configurable threshold the
Kalman filter will re-initialize
ID G_704
If, after Kalman filter reset, another divergence is
detected within a configurable period, the high
level fault protection will force a side swap.
ID G_705
L5_FSWGNCxxxx
Label E_701
L5_FSWGNCxxxx
Label E_705
UT_xxxx
Label E_702
FVT_AD.301
Label E_703
UT_xxxx
Label E_706
When to Stop• Completeness
• Are all claims fully decomposed?• Are all assumptions described?
• Depth• Is there evidence that is directly relatable?
• Abstraction• Are sub-claims at (more or less) the same level of abstraction?• Is evidence ascribed at the appropriate level of abstraction?
• Generally the lowest level claim can be supported by one or more low level requirements.
• Software architecture level• System-level tests and requirements are evidence too, but lower-level
evidence is earlier in the project and often less ambiguous.
9/2/2013
11
Top Down or Bottom Up?• GSN Standard describes both methods, then tosses in a third – use
both.
• Practical use dictates that both are necessary for IV&V• Top level claims provide the entire motivation for the assurance case• Evidence is determined by a combination of artifacts and IV&V analysis.
9/2/2013
12
GENERIC PBRA Summary5
7 16 20 23 25
4
6 13 18 22 24
3
4 11 15 19 21
2
2 8 10 14 17
1
1 3 5 9 12
1 2 3 4 5
Probability
Impact
5
12 3 4
7
6
Capability 1 Launch 2 Commission 3 Cruise 4 POI 5 Do Nothing
(science) 6 Health and
Maintenance 7 Disposal
GENERIC PBRA Summary
GNC Driver
9/2/2013
13
Using the Assurance Cases with PBRA
L_54CFL = Critical failure list
GENERIC 02-003
C_61The GENERIC POI
maneuver is defined in GMP Section 7 and is comprised
of a series of sequences running in a VM
G_64The FSW will persist in POI in the presence of any failure not
on the CFL
J_61This branch
covers the non-failure nominal
behavior
J_62This branch addresses
failure cases (Q3)
J_64This branch
addresses bad behavior (Q2)
C_62All GNC error
budgets are defined in the PPPC
E_600GNCUTxxxx
The GNC system will autonomously complete the
POI maneuver in the presence of any single failure not on the CFL
ID G_53
Use 3Q decomposition
Label S_1
Behavior decomp could be done using either a step-by-step scenario decomp or the 3Q decomp. In this case the 3Q
decomp prompts a look at the failure cases fairly early, driving out some of the bigger behaviors like reset/reboot
Label J_61
The FSW will execute the maneuvers required for POI
ID G_61
There are two types of maneuvers required, the TCM and the POI burn. Treate as separate cases since they
are distinct commands and have different fault responses.
Label S_S_67
The FSW will not generate or permit commands which
would prevent POI
ID G_62
GENERIC FSW will perform the TCM sequences
correctly
ID G_65
Here we look at each command required for the TCM sequence
Label S_1
The FSW will turn to point at an arbitrary J-2000
reference using thruster and reaction wheel commands.
ID G_67
The FSW will perform translation dv maneuvers
using RCS jets in an arbitrary J-2000 reference
direction.
ID G_68
The FSW will point the spacecraft toward the sun
upon command using reaction wheels and
thrusters if necessary.
ID G_69
SVT_SunPoint
Label E601
The Flight System will autonomously perform the
POI maneuver.
ID G_610
The GENERIC FSW will perform the pre-POI
configuration sequence commands
ID G_66
The sequences are constructed properly
Label A_611
Iterate through POI scenario steps
Label S_65
The flight system will configure COMM for POI
upon command
ID G_611
The Flight System will set POI_Timeout and
POI_DVTarget per POI command parameters
ID G_612The Flight System will set
propulsion for main engine burn using prop/pyros to enable main engine flow
ID G_613
The Flight System will start the main engine burn at the
commanded SC_Time
ID G_614
The Flight System will stop the main engine burn at
POI_DVTarget or POIDVTimeout, whichever is
achieved first.
ID G_615
The scenario steps are based on Mission Plan
(page). (Appeal to behavioral decomposition)
Label J_67
Req L-5FSWGNCx
xxx
Label E603
COMM is out of scope for now
Label J_610
UT_INxxxx
Label E_604
FT_GNCxxxx
Label E_605
Req L5-FSWGNCxxx
x
Label E_606
UT_xxxx
Label E_607
Req L5-FSWGNCxxxx
Label E_608
UT-xxxx
Label E_609
Decompose by scenario steps
(commands)
Label J_600
We can map scenario steps to the assurance case because we used them for decompositionSemantically, it seems that scenarios map to arguments (decomp by behavior), and scenario steps map to claims.
9/2/2013
14
Odd topics• Where does static code analysis fit into the picture?
• Certainly provides assurance in some sense• Applies to the product in general but traceable to architectural components
Claim: “Module XYZ is free of implementation flaws”
• How does uncertainty fit into the picture?• IEEE15026 assumes there is a way to combine the independent (and ill-defined)
components of uncertainty, or at least specify uncertainty. • Even if we could calculate uncertainty, would it be a Good Thing? Would broad
categories be sufficient?
9/2/2013
15
And now for something you will really like• Hot spots for development
• Algorithmic blind spots• Scoring or ranking system• Real project testbed
• Is a practical S/C assurance case possible? • Is a practical S/C assurance case useful?• Is it difficult to write top-level claims?• Is it difficult to develop arguments for decomposition?• Can this be a uniform and systematic process for each project?
Yes
Yes – if well-designed
Not at all
Only in a few cases
Structured GSN
Re-usability of assurance design9/2/2013