iv&v facility assertion–based requirements validation patrick theeke (pmp, csqe) john ryan...

IV&V FacilityAssertion–BasedRequirements Validation

Patrick Theeke (PMP, CSQE)John [email protected]

September 8, 2008

IV&V Facility

05/03/23 2

Assertion–Based Requirements ValidationObjectives• Describe the process of using executable

models for requirements validation • Discuss experiences and characterize

outcomes from a pilot project employing the technique

• Identify lessons learned to date and possible next steps

IV&V Facility

05/03/23 3

Assertion–Based Requirements ValidationAgenda• Briefly describe Model-Based Requirements

Validation• Define the process for Assertion-Based

Requirements Validation• Present experiences of pilot project• Present observations• Discuss additional details via panel

IV&V Facility

05/03/23 4

Requirements ValidationGoals of Requirements Validation• Assure that the documented requirements fully specify the capabilities and

characteristics necessary for the system being defined to meet its goals• In particular, NASA IVV is interested in critical capabilities• Requirements defects include

– Missing – Superfluous– Ambiguous– Incomplete– Inconsistent– Unverifiable– Incorrect

IV&V Facility

05/03/23 5

Requirements ValidationValidation Approach• Build a standard of reference for comparison

against the documented requirements of the system of interest

• This standard is the System Reference Model (SRM), which represents IVV’s understanding of the system of interest

Verifiable RequirementsRequirements

Software RequirementsSRM Behaviors

SRM Correct?Requirements Complete?Valid?

Validated Requirements

SRM Correct?Requirements in Scope?Valid?

Correct, Complete,

Consistent, Unambiguous, and

Verifiable Requirements

IV&V Facility

05/03/23 6

Requirements ValidationSRM components• Behaviors

– Use cases and UML diagrams (e.g. activity diagrams)

• Structure– Placeholder for named physical and logical components of the system– Identify component roles and events important to understanding the

system

• Independent Modeling Assertions (IMA)– Formalized IMA extracted from the other SRM components– Collection of unit tests to validate the IMAs

• System Scenarios– Scenarios that describe various sequences of events

• Nominal sequences (what the system is supposed to do)• Unwanted behaviors (what the system is not supposed to do)• Detection and response to adverse conditions (how the systems responds to

adverse conditions)

IV&V Facility

05/03/23 7

Assertion–Based Requirements Validation Pilot Assertion Statechart Definition

• Each Statechart Assertion is a formal specification of a single requirement.

• A statechart assertion is fundamentally a monitoring device that observes system behavior and determines whether that behavior is valid

• Observed behavior is valid when it matches the behavior specification coded into the assertion, and invalid when it violates the specification

• An assertion is run against observable behavior, typically supplied by some executable artifact running under a test scenario

IV&V Facility

05/03/23 8

Requirements ValidationCorrelation-Based Requirements Validation• Capture independent understanding of system

behavior or characteristic in a SRM• Correlate project requirements with SRM behaviors• Analyze results to identify

– Ambiguous, incorrect, incomplete, inconsistent, or non-verifiable requirements that correspond to SRM behaviors

– Requirements without matching SRM behavior– SRM behavior without matching requirements

SRM

Correlationswith SRM

Project Requirements

Analyze Results

Findings

IV&V Facility

05/03/23 9

Assertion–Based Requirements ValidationMotivation• Formalization of natural language requirements reveal errors in

the specification• Leveraging formal assertions for requirements validation

increases the utility of formalized requirements– Formalization of requirements is a necessary step in leveraging the

SRM for verification

• Utilizing assertions more closely mimics the highly parallel threads of execution of real time, reactive systems, including timing and synchronization, allowing more in-depth examination of complex behavioral interactions

• Discovering requirements and representing them as assertions is more readily done early in the modeling process while the SRM is under development – so we might as well use them!

IV&V Facility

05/03/23 10

Assertion–Based Requirements ValidationDependencies• Assertion Based Validation Depends on:

– SRM was built to the appropriate level of depth and details

– The requirements were traced to the SRM– The set of system scenarios provided an appropriate

coverage of the system behaviors– The naming convention for the assertion events

match the events of the formalized unit tests, system scenarios, and assertion state charts

IV&V Facility

05/03/23 11

Assertion–Based Requirements Validation ProcessAssertion-Based Requirements Validation• Formally capture developer’s documented

requirements as Project Based Assertions (PBA)• Exercise PBAs and IMAs using validated System

Scenarios• Analyze results

SRM



Analyze Results

FindingsDevelop & Validate PBAs

Execute Scenarios in Context of PBAs

Develop & Validate IMAs

Develop System Scenarios

IV&V Facility

05/03/23 12

Assertion–Based Requirements Validation Process Develop & Validate Project–Based Assertions • Translate the documented requirements from the development

project into a formal representation that is conducive to comparison with the SRM

• Requires detailed understanding of the requirements • Exposes errors in the requirements specifications

SRM



Analyze Results Findings

Develop & Validate PBAs


Identify Relevant Documented requirements

Interpret Requirements

Formalize Requirements as PBAs

Validate PBAs

IV&V Facility

05/03/23 13

Assertion–Based Requirements Validation Process Execute scenarios in Context of PBAs• We are interested in how the PBAs react to the

system scenarios– Which system scenarios ended as expected? – Which system scenarios ended differently than expected?– Which PBAs were not exercised?

SRM






Include PBAs in executable environment

Execute system scenarios

Collect results for further analysis

IV&V Facility

05/03/23 14

Assertion–Based Requirements Validation Process Analyze Results• Examine each assertion failure for:

– Errors in constructing the SRM (system scenarios, IMAs, etc.)– Errors in constructing the PBA(s)– Errors in a requirement

• Fix the errors in the SRM• Rerun test until no SRM or PBA construction errors are

discovered. Therefore all unexpected results indicate findings

SRM



Analyze Results

FindingsDevelop & Validate PBAs


Ensure system scenario set is correct and complete

Ensure PBA set is complete

Ensure that PBA is in scope

Ensure PBA is correct

IV&V Facility

05/03/23 15

Assertion–Based Requirements Validation PilotDevelop and Validate IMAs• Examine behavior in UML activity diagrams

– Begin with preconditions, triggers, constraints– Assert important sequences that must occur in order (under all

conditions or adverse conditions)– Constraints on looping (will continue to try until …)– Important executions if a critical behavior is unavailable– Any possible/impossible executions that will occur during off-

nominal events– Look at the whole picture and observe modeling issues as well

SRM








Identify Candidate IMAs

Analyze Candidate IMAs

Represent Candidate IMAs in Natural Language

Formalize IMAs as assertion state charts

Validate IMAs

IV&V Facility

05/03/23 16

Assertion–Based Requirements Validation Pilot Identify candidate IMAs

-Actively Damp Nutation[After burn maneuver, need to stabilize damping before selecting Idle Mode to downlink data]

Select Damping Mode Determine Attitude Configure Damping Filter Estimate Nutation Angle IF Angle > target: Then compute damping torque prior to controlling fired thruster burn IF Angle < target: continue process

Estimate target AngleCheck if < targetTrack time passing

Once stablized for minimal time, GNC subsytem can Select Idle Mode

IV&V Facility

05/03/23 17

Assertion–Based Requirements Validation PilotAnalyze candidate IMAs• Analysis

No te: The nutation angle must be less than or equal to target nutation angle for the minimum amount of stable time to maintain an acceptable and stable nutation angle. [P <= R and T1 >= T2 (Tmin)]

Q1. The nutation angle must (/can?) be less than or equal to target nutation angle for the minimum amount of stable time to select Idle Mode.

Domain Answer – Actually both Ground and Fault Protection Mode can select Idle Mode at any time even if the above is not true ([P <= R and T 1 >= T2 (Tmin)]).

Assumption: If there was a satellite timeout occurring, and if the fault protection mode has not been entered (due to # attempts or timeout), then the above is true.

Q1b. The Reference Orientation (GN&C sub system) must maintain a nutat ion angle that is less than or equal to the target nutation angle for a minimal amount of stable time to select Idle Mode. (See Figure 16 and 20)

Q2. What occurs if the goal is met (nutation angle stabilized and maintainable) and something causes the angle to exceed the nutation angle again?

Domain Answer – The issue is monitored and the active damp nutation behavior is called again.

IV&V Facility

05/03/23 18

Assertion–Based Requirements Validation PilotAnalyze candidate IMAs• Analyze each candidate assertion and create good and bad

scenarios to represent the requirements• Use the diagrams in the SRM to derive scenarios or unit tests that

will cause the requirement to succeed and fail• Use the scenarios to challenge the assertion against

– redundancy of events– improper sequencing– Unknowns– misuse of system boundaries– Dependencies– and constraints (i.e time)

• Determine whether or not the sequencing, repetition, looping, constraints, and concurrence always (“must”) occur this way or conditionally throughout the entire SRM

IV&V Facility

05/03/23 19

Assertion–Based Requirements Validation PilotAnalyze candidate IMAs

Good and Bad Scenarios for Q1B: Ex

ceed

Tar

get

Nuta

tion

Angl

e

Reac

h Ta

rget

Nut

atio

nAn

gle

Tim

e pa

ssed

= 2

Tim

e pa

ssed

= 3

Sele

ct Id

le M

ode

Exce

ed T

arge

t Nu

tatio

nAn

gle

Reac

h Ta

rget

Nut

atio

nAn

gle

Tim

e pa

ssed

= 4

Sele

ct Id

le M

ode

Good Scenario Bad Scenario

Timer Tmin is the minimal amount of stable time required to reach a mainta inable angle state and allow ability to select Idle Mode. Tmin = 5.

Exce

ed T

arge

t Nu

tatio

nAn

gle

Reac

h Ta

rget

Nut

atio

nAn

gle

Tim

e pa

ssed

= 4

The Reference Orientation must maintain a nutation angle that is less than or equal to the target nutation angle for a minimal amount of stable time to select Idle Mode.

IV&V Facility

05/03/23 20

Assertion–Based Requirements Validation PilotRepresent the candidate IMAs as NLR • Natural Language Requirement for Q1b

– The Reference Orientation (GN&C sub system) shall achieve a stable nutation angle that is less than or equal to an acceptable target nutation angle for a minimal amount of time prior to selecting Idle Mode.

IV&V Facility

05/03/23 21

Assertion–Based Requirements Validation Pilot Formalize each Natural Language Requirement

IV&V Facility

05/03/23 22

Assertion–Based Requirements Validation Pilot Validate each formalized IMA• Validate the IMA is unambiguous, correct, and consistent

with the SRM behavior and scenarios.– Validate each model-based formalized requirement has been modeled

correctly to represent the intended requirement

• Create formalized unit tests for the scenarios

• Validate the IMA against the formalized unit tests

public void testDampNutation_IMA001_sc001(){// this is testing a good scenario for RO selecting Idle ModeprintState(IMA001);IMA001.reachTargetNutationAngle();printState(IMA001);IMA001.incrSimTime(1);printState(IMA001);IMA001.exceedTargetNutationAngle();printState(IMA001);IMA001.reachTargetNutationAngle();printState(IMA001);IMA001.incrSimTime(5);printState(IMA001);IMA001.incrSimTime(1);printState(IMA001);IMA001.ROselectIdleMode();printState(IMA001);assertTrue(IMA001.isSuccess());System.out.println("Completed testDampNutation_IMA001_sc001");System.out.println();System.out.println("-------------------------");System.out.println();}

public void testDampNutation_IMA001_sc002(){// this is testing a b scenario for RO selecting Idle ModeprintState(IMA001);IMA001.reachTargetNutationAngle();printState(IMA001);IMA001.incrSimTime(1);printState(IMA001);IMA001.exceedTargetNutationAngle();printState(IMA001);IMA001.reachTargetNutationAngle();printState(IMA001);IMA001.incrSimTime(4);printState(IMA001);IMA001.ROselectIdleMode();printState(IMA001);assertFalse(IMA001.isSuccess());System.out.println("CompletedtestDampNutation_IMA001_sc002");}

IV&V Facility

05/03/23 23

Assertion–Based Requirements Validation PilotDevelop & Validate Project–Based Assertions • Translate the documented requirements from the development

project into a formal representation that is conducive to comparison with the SRM

• Requires detailed understanding of the requirements • Exposes errors in the requirements specifications

SRM








Identify Relevant Documented requirements

Interpret Requirements

Formalize Requirements as PBAs

Validate PBAs

IV&V Facility

05/03/23 24

Assertion–Based Requirements Validation PilotDevelop & Validate Project–Based Assertions

IV&V Facility

05/03/23 25

public void testTorqueVector_IMA001(){printState(pba001);pba001.beginJOIPhase();printState(pba001);pba001.SRUUpdate();printState(pba001);pba001.incrSimTime(4);pba001.beginFireThrusters();printState(pba001);pba001.endFireThrusters();printState(pba001);

assertTrue(pba001.isSuccess());System.out.println("Completed testTorqueVector_pba001");System.out.println();System.out.println("-------------------------");System.out.println();

}

Assertion–Based Requirements Validation PilotValidate each formalized PBAValidate the PBA is unambiguous, correct, and consistent.

Validate each project-based formalized requirement has been modeled correctly to represent the intended requirement

Create formalized unit tests for the scenarios

IV&V Facility

05/03/23 26

Assertion–Based Requirements Validation PilotAdd system scenarios to SRM• Derive system scenarios

– Nominal, expected behavioral sequences– Behavioral sequences that should not be permitted– Detection and responses to adverse conditions

SRM








Identify events

Define system scenarios

Implement system scenarios

Validate system scenarios

IV&V Facility

05/03/23 27

Assertion–Based Requirements Validation PilotIdentify Events• Identify events in the activity diagrams

– Activities model the control of execution of one or more actions. Activities do not model sequences of event occurrences.

– Activities do not often model timing constraints– Action names and control flow guards are defined in

natural language and thus prone to ambiguity

IV&V Facility

05/03/23 28

Assertion–Based Requirements Validation PilotDefine System Scenarios• “Good” and “Bad” Scenarios

– Used the semantics of UML sequence diagrams to define a scenario, by modeling sequence of event occurrences (i.e. action execution start/finish events)

– Used guard conditions in activity diagrams that should not hold true in the context of the goal as means of creating “bad scenarios”

IV&V Facility

05/03/23 29

Assertion–Based Requirements Validation PilotImplement System Scenarios• Scenarios are implemented in J-Unit and

reference events in assertions

IV&V Facility

05/03/23 30

Assertion–Based Requirements Validation PilotValidate System Scenarios• Use the IMAs to ensure that the scenarios

return the expected results– Q1 scenarios complete without errors, as expected– Q2 scenarios do not cause any assertions to “pop”– Q3 scenarios result in detection of an adverse

condition and respond in a correct way, as expected

• Scenarios represent a test oracle

IV&V Facility

05/03/23 31

Assertion–Based Requirements Validation Pilot Execute scenarios in Context of PBAs• Interested in how the PBAs react to the system

scenarios– Which system scenarios ended as expected? – Which system scenarios ended differently than expected?– Which PBAs were not exercised?

SRM








Include PBAs in executable environment

Execute system scenarios

Collect results for further analysis

IV&V Facility

05/03/23 32

Assertion–Based Requirements Validation Process Analyze Results• Examine each assertion failure for:

– Errors in constructing the SRM (system scenarios, IMAs, etc.)– Errors in constructing the PBA(s)– Errors in a requirement

• Fix the errors in the SRM• Rerun test until no SRM or PBA construction errors are

discovered. Therefore all unexpected results indicate findings

SRM



AnalyzeResults Findings





Ensure system scenario set is correct and complete

Ensure PBA set is complete

Ensure that PBA is in scope

Ensure PBA is correct

IV&V Facility

05/03/23 33

Assertion–Based Requirements Validation PilotObservations• Developing IMAs helps identify defects and

deficiencies in the SRM. Doing this early in the SRM development process yields early benefits

• The process of examining the documented requirements helps to identify omissions in the correlation process

• The depth of understanding required to create IMAs and PBAs results in more rapid understanding of the system of interest

• Assertions best represent requirements of specific behaviors

• When creating PBAs for high level requirements such as “The system shall have the capability to do X”, these behavioral details need to be supplied from domain knowledge and analysis

IV&V Facility

05/03/23 34

Assertion–Based Requirements Validation PilotObservations• It was easier than expected to:

– Discover early questions and concerns that could lead to good assertions

– Develop a high level understanding of what the system is supposed to do

– Create higher level system scenarios that describe what the system is supposed to do

– Develop Formal Assertion State Charts from Natural Language– Observe the lack of model depth, details– Discover possible missing behavioral IMAs– Write unit tests for IMAs and PBAs (although easier for IMAs if the

SRM provides more information)– Write formalized Junit Tests– Test assertions against the System Scenarios

IV&V Facility

05/03/23 35

Assertion–Based Requirements Validation PilotObservations• It was harder than expected to:

– Vet good/worthwhile NL (informal assertions) from the list of answered questions

– Create System Scenarios with the lack of model depth and details– Create System Scenarios that describe what the system is not supposed

to do, and what the system is supposed to do under adverse conditions– Correlate system scenario events, IMA events, and PBA events without a

data dictionary or complete Domain Model– Discover worthwhile PBAs from documented requirements that define and

monitor a single event, and/or provide limited details and constraints– Discover matching IMAs and PBAs when there are limited requirement

details– Discover matching IMAs and PBAs when the model is not developed to

needed level of coverage/detail– Validate Requirements when the model is not developed to the needed

level of coverage/detail– Set up the test rig for the first time

IV&V Facility

05/03/23 36

Assertion–Based Requirements ValidationRecommendations• Develop IMAs early in the SRM development process• Consider developing IMAs as part of the process for

reviewing activity diagrams• Continue refining techniques for using assertion-

based methods for requirements validation

IV&V Facility

05/03/23 37

Assertion–Based Requirements ValidationSummary• Model–Based Requirements Validation is part of the SRM

Validation process– The purposes for using assertions include:

• Have ability to test conflicts and validate complex requirements• Provides an advantage of using formal requirements over natural language• Sets groundwork for automated testing

– Use assertions to validate requirements that correspond to SRM behaviors and IMAs as unambiguous, correct, complete, consistent, and verifiable.

– Use assertions to ensure that the right behaviors have been defined and the behaviors are of high quality. The right behaviors are those that adequately describe:

• What the system is supposed to do• What the system in not supposed to do• What the system is supposed to do under adverse conditions

Assertion–based validation leverages our independent Assertion–based validation leverages our independent modeling to find issues earlier in the life–cyclemodeling to find issues earlier in the life–cycle

IV&V Facility

05/03/23 38

Assertion–Based Requirements ValidationPanel Discussion• Michael Facemire• John Ryan• Bill Stanton• Patrick Theeke

IV&V Facility

05/03/23 39

References• D. Drusinsky, “From AD to Assertions”; informal presentation,

2008. • D. Drusinsky, J. B. Michael, and M. Shing, “A Framework for

Computer-aided Validation”; Naval Postgraduate School, NPS-CS-07-008, 08/2007.

• K. Hurley, and D. Kranz, “Executable Reference Model”; informal presentation, 2008.

• Juno IVV Team, “Juno Level-4 Flight Software Requirements Validation Report”; NASA IV&V, 2008.

• D. Kranz, and J. Ryan, “Juno and Assertion Based Validation”; NASA IV&V, white paper, 2008.

• NASA IV&V Facility, “System Level Procedure 09-1”; 2007.• S. Raque, “SRM Example Overview”; informal presentation ,

2008.• P. Theeke, “A process architecture for reference model based

validation”; NASA IV&V, white paper, 2007.• P. Theeke, “System Reference Models and Executable

Assertions for Requirements Validation”; NASA IV&V, white paper, 2007.

iv&v facility assertion–based requirements validation patrick theeke (pmp, csqe) john ryan...

Documents