1

Practical Assurance Case Design

IV&V WorkshopS. R. BrownKeyLogic Inc

With my thanks and appreciation Don Ohi – Project Monitor

Travis Dawson – Chief EngineerBill Stanton for tolerating a lot of questions

9/2/2013

Why Assurance Cases• State your case:

• What are we trying to prove• How are we trying to prove it• What evidence supports the proof

Assurance Cases make the pointDesign of AssuranceSupport of Assurance

Related to:

Argumentation

Safety Case

GSN Standard Kelly, T.: 'Arguing Safety: A Systematic Approach to Managing Safety Cases', D.Phil Thesis, University of York (1998). Available for download from http://www-users.cs.york.ac.uk/~tpk 9/2/2013 2

State Your Case

My coffee will provide spousal satisfaction

ID EC_01

Examine each element of coffee satisfaction

Label AC_1

The coffee is adequately hot

ID EC_02

Two creamers have been added to the coffee

ID EC_03

No sugar has been added to the coffee

ID EC_04

The coffee warmers are

operating.

Label EC_03

Two creamer packets are on the

the counter

Label E_C01

No sugar packets are on the counter

Label EC_02

The coffee isserved

immediately

Label EC_04

Argue through coffee temperature discriminators.

Label AC_02

The coffee pot heater is on and operating

ID C_1020

The coffee is serves with no more than three minutes

delay.

ID C_1020

Spousal coffee satisfaction is critical to morning kharma.

Label JC_01

The two key parameters in coffee temperature are

initial temperature and time to serving.

Label JC_02

Spousal Coffee Assurance CaseTop level Claim

Claim Justification

Perspective: Who is the stakeholder?

9/2/2013 3

Make the Argument

My coffee will provide spousal satisfaction

ID EC_01

Examine each element of coffee satisfaction

Label AC_1

The coffee is adequately hot

ID EC_02

Two creamers have been added to the coffee

ID EC_03

No sugar has been added to the coffee

ID EC_04

The coffee warmers are

operating.

Label EC_03

Two creamer packets are on the

the counter

Label E_C01

No sugar packets are on the counter

Label EC_02

The coffee isserved

immediately

Label EC_04

Argue through coffee temperature discriminators.

Label AC_02

The coffee pot heater is on and operating

ID C_1020

The coffee is serves with no more than three minutes

delay.

ID C_1020

Spousal coffee satisfaction is critical to morning kharma.

Label JC_01

The two key parameters in coffee temperature are

initial temperature and time to serving.

Label JC_02

Architectural

decomposition

Behavioral

decomposition

9/2/2013 4

My coffee will provide spousal satisfaction

ID EC_01

Examine each element of coffee satisfaction

Label AC_1

The coffee is adequately hot

ID EC_02

Two creamers have been added to the coffee

ID EC_03

No sugar has been added to the coffee

ID EC_04

The coffee warmers are

operating.

Label EC_03

Two creamer packets are on the

the counter

Label E_C01

No sugar packets are on the counter

Label EC_02

The coffee isserved

immediately

Label EC_04

Argue through coffee temperature discriminators.

Label AC_02

The coffee pot heater is on and operating

ID C_1020

The coffee is serves with no more than three minutes

delay.

ID C_1020

Spousal coffee satisfaction is critical to morning kharma.

Label JC_01

The two key parameters in coffee temperature are

initial temperature and time to serving.

Label JC_02Gather the Evidence

Atomic evidence

9/2/2013 5

Assurance Case Semantic ContentMy coffee will provide

spousal satisfaction

ID EC_01

Examine each element of coffee satisfaction

Label AC_1

The coffee is adequately hot

ID EC_02

Two creamers have been added to the coffee

ID EC_03

No sugar has been added to the coffee

ID EC_04

The coffee warmers are

operating.

Label EC_03

Two creamer packets are on the

the counter

Label E_C01

No sugar packets are on the counter

Label EC_02

The coffee isserved

immediately

Label EC_04

Argue through coffee temperature discriminators.

Label AC_02

The coffee pot heater is on and operating

ID C_1020

The coffee is serves with no more than three minutes

delay.

ID C_1020

Spousal coffee satisfaction is critical to morning kharma.

Label JC_01

The two key parameters in coffee temperature are

initial temperature and time to serving.

Label JC_02

• What I get out of a cup of coffee:• The assurance design is not a description of the system• Assurance network needs only to go as far as necessary to provide stated assurance• Strength of evidence is a constant consideration• Without more information there is little basis for claim prioritization

• Can address risk (uncertainty)• Does not support consequence

9/2/2013 6

Where to Start

The GENERIC Spacecraft will travel to Planet X and

enter orbit, remaining in orbit doing nothing in particular

until the requir

ID G_1

Testable (yes/no)

Is Clearly Stated

Will decompose in a useful

way Lessons Learned:Look for simple and comprehensive claimsClaims must be objective (yes/no)Watch out for claims that must decompose through out-of-scope domains (e.g.:

reset/sideswap hardware timing)Take advantage of self-similarity, patterns and

common arguments.

Initially a major goal: In almost every case it is self-evident if the objective is defined. This is where perspective becomes important. Consider stakeholder needs.

9/2/2013 7

8

Perspectives and Planning• The perspective of the assurance case is important

• Software lifecycle (NPR 7150)• System/Human safety• 09-1 System definition• Project-specific

A place for (almost) everythingDecomposes differently than lifecycle

Represent the stakeholder

9/2/2013

9

Argument: the Art of Assurance Case

Some Useful decomposition classes

Appeal to architecture• Use software architecture to

decompose to subsystems• No uncertainty

Appeal to Usage• Use scenario to decompose

to subsystems by required actions

• Some uncertainty

Note: 3Q is a special case of appeal to usage

Appeal to Specificity• Provide detail to support

a more general claim• Little uncertainty

Arguments describe how the child claims satisfy the parent claim but do not carry the burden of proof

Some approaches require a justification for each argument but this was often redundant.

Assumptions are often necessary in order that the argument is clear to a reviewer

Arguments are in the context of the system

Use cases can be a good basis for an Appeal to Usage. Note semantic equivalence.9/2/2013

10

The spacecraft will calculate estimated attitude based on

IMU, STAR, and SUN sensors

ID G_56

Iterate through AD functional behaviors

Label S_700AD performs sensor integration

via a 12 state Kalman Filter. Steps include forward

propagation, noise covariance update bias update, and check

for divergence.

Label J_701

The AD system will calculate state propagation

ID G_701

The AD system will calculate noise covariance and

accelerometer bias

ID G_702

The AD system will detect Kalman filter divergence

ID G_703

Kalman filter algorithms are not validated (scope). There

is evidence to support this assumption but the decomp is

out of scope.

Label J_702

Ibid J_702

Label J_703

Ibid J_702

Label J_704

If the covariance exceeds a configurable threshold the

Kalman filter will re-initialize

ID G_704

If, after Kalman filter reset, another divergence is

detected within a configurable period, the high

level fault protection will force a side swap.

ID G_705

L5_FSWGNCxxxx

Label E_701

L5_FSWGNCxxxx

Label E_705

UT_xxxx

Label E_702

FVT_AD.301

Label E_703

UT_xxxx

Label E_706

When to Stop• Completeness

• Are all claims fully decomposed?• Are all assumptions described?

• Depth• Is there evidence that is directly relatable?

• Abstraction• Are sub-claims at (more or less) the same level of abstraction?• Is evidence ascribed at the appropriate level of abstraction?

• Generally the lowest level claim can be supported by one or more low level requirements.

• Software architecture level• System-level tests and requirements are evidence too, but lower-level

evidence is earlier in the project and often less ambiguous.

9/2/2013

11

Top Down or Bottom Up?• GSN Standard describes both methods, then tosses in a third – use

both.

• Practical use dictates that both are necessary for IV&V• Top level claims provide the entire motivation for the assurance case• Evidence is determined by a combination of artifacts and IV&V analysis.

9/2/2013

12

GENERIC PBRA Summary5

7 16 20 23 25

4

6 13 18 22 24

3

4 11 15 19 21

2

2 8 10 14 17

1

1 3 5 9 12

1 2 3 4 5

Probability

Impact

5

12 3 4

7

6

Capability 1 Launch 2 Commission 3 Cruise 4 POI 5 Do Nothing

(science) 6 Health and

Maintenance 7 Disposal

GENERIC PBRA Summary

GNC Driver

9/2/2013

13

Using the Assurance Cases with PBRA

L_54CFL = Critical failure list

GENERIC 02-003

C_61The GENERIC POI

maneuver is defined in GMP Section 7 and is comprised

of a series of sequences running in a VM

G_64The FSW will persist in POI in the presence of any failure not

on the CFL

J_61This branch

covers the non-failure nominal

behavior

J_62This branch addresses

failure cases (Q3)

J_64This branch

addresses bad behavior (Q2)

C_62All GNC error

budgets are defined in the PPPC

E_600GNCUTxxxx

The GNC system will autonomously complete the

POI maneuver in the presence of any single failure not on the CFL

ID G_53

Use 3Q decomposition

Label S_1

Behavior decomp could be done using either a step-by-step scenario decomp or the 3Q decomp. In this case the 3Q

decomp prompts a look at the failure cases fairly early, driving out some of the bigger behaviors like reset/reboot

Label J_61

The FSW will execute the maneuvers required for POI

ID G_61

There are two types of maneuvers required, the TCM and the POI burn. Treate as separate cases since they

are distinct commands and have different fault responses.

Label S_S_67

The FSW will not generate or permit commands which

would prevent POI

ID G_62

GENERIC FSW will perform the TCM sequences

correctly

ID G_65

Here we look at each command required for the TCM sequence

Label S_1

The FSW will turn to point at an arbitrary J-2000

reference using thruster and reaction wheel commands.

ID G_67

The FSW will perform translation dv maneuvers

using RCS jets in an arbitrary J-2000 reference

direction.

ID G_68

The FSW will point the spacecraft toward the sun

upon command using reaction wheels and

thrusters if necessary.

ID G_69

SVT_SunPoint

Label E601

The Flight System will autonomously perform the

POI maneuver.

ID G_610

The GENERIC FSW will perform the pre-POI

configuration sequence commands

ID G_66

The sequences are constructed properly

Label A_611

Iterate through POI scenario steps

Label S_65

The flight system will configure COMM for POI

upon command

ID G_611

The Flight System will set POI_Timeout and

POI_DVTarget per POI command parameters

ID G_612The Flight System will set

propulsion for main engine burn using prop/pyros to enable main engine flow

ID G_613

The Flight System will start the main engine burn at the

commanded SC_Time

ID G_614

The Flight System will stop the main engine burn at

POI_DVTarget or POIDVTimeout, whichever is

achieved first.

ID G_615

The scenario steps are based on Mission Plan

(page). (Appeal to behavioral decomposition)

Label J_67

Req L-5FSWGNCx

xxx

Label E603

COMM is out of scope for now

Label J_610

UT_INxxxx

Label E_604

FT_GNCxxxx

Label E_605

Req L5-FSWGNCxxx

x

Label E_606

UT_xxxx

Label E_607

Req L5-FSWGNCxxxx

Label E_608

UT-xxxx

Label E_609

Decompose by scenario steps

(commands)

Label J_600

We can map scenario steps to the assurance case because we used them for decompositionSemantically, it seems that scenarios map to arguments (decomp by behavior), and scenario steps map to claims.

9/2/2013

14

Odd topics• Where does static code analysis fit into the picture?

• Certainly provides assurance in some sense• Applies to the product in general but traceable to architectural components

Claim: “Module XYZ is free of implementation flaws”

• How does uncertainty fit into the picture?• IEEE15026 assumes there is a way to combine the independent (and ill-defined)

components of uncertainty, or at least specify uncertainty. • Even if we could calculate uncertainty, would it be a Good Thing? Would broad

categories be sufficient?

9/2/2013

15

And now for something you will really like• Hot spots for development

• Algorithmic blind spots• Scoring or ranking system• Real project testbed

• Is a practical S/C assurance case possible? • Is a practical S/C assurance case useful?• Is it difficult to write top-level claims?• Is it difficult to develop arguments for decomposition?• Can this be a uniform and systematic process for each project?

Yes

Yes – if well-designed

Not at all

Only in a few cases

Structured GSN

Re-usability of assurance design9/2/2013