ESA D/TEC-QQS Software Product Assurance 1

SAS, September 2007

An Overview of ESA SoftwareProduct Assurance

Services

M. Garcia, R. PradesESA D/TEC-QQS

SAS 2007Morgantown, West Virginia, USA

September 25-27, 2007

ESA D/TEC-QQS Software Product Assurance 2

SAS, September 2007

Presentation Overview

• Software product assurance context in ESA• Software product assurance services provided to European

space projects and industry– Software Product Evaluation for Conformity (SPEC)– Software Safety and Dependability Evaluation (RAMS)– Software Process Assessment (S4S)

ESA D/TEC-QQS Software Product Assurance 3

SAS, September 2007

SW PA Context in ESA

• ESA Directorate Organization• Program Directorates: D/SCI, D/EOP, D/HME, …• Technical Directorate: D/TEC

• Systems design• Electrical Engineering• Mechanical Engineering• Quality

• Materials• Component• Software, Dependability and Safety

• Project teams/technical support

ESA D/TEC-QQS Software Product Assurance 4

SAS, September 2007

SW PA Context in ESA

• D/TEC-QQS• SW PA support to projects

• Meeting requirements specified in applicable documentation →development of methods, approaches, tools

• R&D in the SW PA domain with focus on present and future needs ofprojects and industrial partners

• Handbooks containing guidance on how to apply SW PA in projects• Services

• Services• Software Product Evaluation for Conformity• Software Safety and Dependability Evaluation• Software Process Assessment

ESA D/TEC-QQS Software Product Assurance 5

SAS, September 2007

Software Process Assessment (S4S)

ESA D/TEC-QQS Software Product Assurance 6

SAS, September 2007

Software Process Assessment (S4S)

• What is S4S?– A method of evaluating space software processes– ISO 15504 (Process Assessment) conformant– Can be tailored to different classes of safety-critical software

• Scope– Processes include acquisition, supply, operation, engineering,

supporting, management, process improvement, resource andinfrastructure and reuse

– Can select which processes to assess– Each process is assessed on a scale of capability

ESA D/TEC-QQS Software Product Assurance 7

SAS, September 2007

Software Process Assessment (S4S)• 25 assessment performed

– Prime contractors– Small and medium enterprises

• Performed on a voluntary basis– few in response to specific project requirements

– Input to improvement programs

– Confirmation of SW development capabilities towards customers

ESA D/TEC-QQS Software Product Assurance 8

SAS, September 2007

Software Process Assessment (S4S)• Assessment organization

– Pre-assessment visit• Selection of processes• Determination of the capability levels to assess• Schedule and participants

– Assessment• Improvement Workshop

– Identification of improvement opportunities from assessment report– Discussion and setup of improvement projects

ESA D/TEC-QQS Software Product Assurance 9

SAS, September 2007

Risk Reduction with S4S• What is R4S?

– An extension of S4S with a risk dimension integrated in the method– Basic idea: use risk analysis post-assessment to identify most critical

processes for improvement– Relies on correlations between commonly known risks & S4S

processes• Objectives

– Identification, assessment and management of process related risks– Link between process and risk management– Supports identification and prioritization of improvement actions to

• improve processes towards target capability levels• reduce process-related risks to an acceptable level

ESA D/TEC-QQS Software Product Assurance 10

SAS, September 2007

Risk Reduction with S4S

ProcessImprovement

Business Goals,Customer

ProcessAssessment

RiskManagementR4S

processrelated risks

risks to bemitigated

target capability profile

improvementactions

actual profile

acceptablerisks

mitigatedrisks

ESA D/TEC-QQS Software Product Assurance 11

SAS, September 2007

Software Product Evaluation forConformity (SPEC)

ESA D/TEC-QQS Software Product Assurance 12

SAS, September 2007

Software Product Evaluation for Conformity (SPEC)

• ESA’s Initiative on Software Product Evaluation– Provide a framework to evaluate, and possibly confirm

conformity of a space software product.– Provide a good bases for specifying the non-functional

requirements through a quality model– Reduce significantly the cost associated with the product

development by adopting a quality model from the beginningof a project.

– Increase the ability to produce quality software and to specifyand assess this quality

ESA D/TEC-QQS Software Product Assurance 13

SAS, September 2007

Software Product Evaluation for Conformity (SPEC)The context

• ECSS-Q80B “Software product assurance,” definesrequirements on Software product quality assurance– Definition or adoption of a quality model– Definition of a metrication program– Definition of metrics– Verification of the achievements– Metrics trend– Improvement actions

• HB-Q80-04 “Software metrication programmedefinition and implementation”

ESA D/TEC-QQS Software Product Assurance 14

SAS, September 2007

Software Product Evaluation for Conformity (SPEC)SPEC method

• SPEC is a method to evaluate aSpace Software Product bymeasuring its quality, based ona tailored quality model.

• SPEC defines:- Quality Model- Evaluation framework

ESA D/TEC-QQS Software Product Assurance 15

SAS, September 2007

Software Product Evaluation for Conformity (SPEC)SPEC on on-board OS

• Goals of the evaluation:– Check the usability of the SPEC scheme for Open Source

Software– Check whether the OS satisfies the SPEC criteria for mission

critical software.– Obtain substantial information about its features and

determine potential areas for (future) in depth analysis– Identify weaknesses and potential areas of improvements for

the SPEC scheme

ESA D/TEC-QQS Software Product Assurance 16

SAS, September 2007

Software Product Evaluation for Conformity (SPEC) SPEC on on-board OS

• Constraints of the evaluation– Product was an Open Source Software with a limited set of

documentation– The provider did not issue any documentation about the

verification activities (methodology and results)• Evaluation results

– Methodological SPEC related results (SPEC to OSS)– Software product (On-board OS) related results

ESA D/TEC-QQS Software Product Assurance 17

SAS, September 2007

Software Product Evaluation for Conformity (SPEC)SPEC on GSI

• Goals of the evaluation– Evaluation of GSI in a generic scenario (i.e. typical scientific mission)– Application of SPEC to real project– SPEC as a mean to improve software products– Use of S4S* for process related goal properties

(*) Spice for Space : Process assessment method compliant with ISO15504

ESA D/TEC-QQS Software Product Assurance 18

SAS, September 2007

Phase I: Baseline EvaluationPhase I: Baseline Evaluation• Tailoring of Quality Model (based on criticality classes)• Evaluation Plan & Data Collection• Evaluation Report (with recommendations)

Phase III: Delta EvaluationPhase III: Delta Evaluation• Re-measurement phase (including delta S4S)• Re-issue of Evaluation Report (21 detailed recommendations )

Software Product Evaluation for Conformity (SPEC)SPEC on SGI -- Phases

Phase II: ImprovementPhase II: Improvement• Implementation of recommendations (including specific

process improvement actions)

ESA D/TEC-QQS Software Product Assurance 19

SAS, September 2007

– Identification of recommendations allows the implementation ofimprovements in some processes of the software developmentand in further versions of the software products.

– Although SPEC was not meant to be used “after the fact” onexisting products, it was demonstrated that SPEC can beapplied successfully on finalized SW products.

– SPEC needs to be tailored before it can be used to evaluateOSS.

– SPEC is being updated based on the experience gained duringits application on real projects.

Software Product Evaluation for Conformity (SPEC) Conclusions

ESA D/TEC-QQS Software Product Assurance 20

SAS, September 2007

Software Safety andDependability Evaluation

ESA D/TEC-QQS Software Product Assurance 21

SAS, September 2007

Software Safety and Dependability Evaluation

• ESA’s Initiative on Software Dependability and SafetyEvaluations– Evaluate dependability and safety of software to reduce

the risk of mission failures– Help assure correct implementation of system PA

requirements– Evaluate the applicability and usefulness of selected

techniques– Identify improvements of the techniques

ESA D/TEC-QQS Software Product Assurance 22

SAS, September 2007

Software Safety and Dependability EvaluationThe context

• ECSS-Q80B “Software product assurance,” definesrequirements on safety and mission critical software– Identification of the criticality of the software function in the

system context– Software dependability and safety analysis to be performed– Design of the software to minimize number of critical

components– Use of criticality for tailoring of the “development rigour”– Special rules for handling of critical software components– Organizational constraints

• HB-Q80-03 “ Methods and techniques to support theassessment of software dependability and safety”

ESA D/TEC-QQS Software Product Assurance 23

SAS, September 2007

Software Safety and Dependability EvaluationSFMECA on GSI

– Software Failure Modes, Effects and Criticality Analysisperformed on the Spacecraft Control and OperationsSystem used for spacecraft operation– A typical scientific mission was used as a reference

– Performed starting from top-level requirements, then analyzingthe failure modes of the SW components

– Main result was the criticality level assigned to SWcomponents based on the severity of the consequences offailures

ESA D/TEC-QQS Software Product Assurance 24

SAS, September 2007

Software Safety and Dependability EvaluationCode Analysis on GSI

– Applied to components having highest criticality andhighest density of severe failure modes associated

– Based on checklist derived from applicable codingstandards

– Main output was the report on coding standard rulesviolated and metrics values.

ESA D/TEC-QQS Software Product Assurance 25

SAS, September 2007

Software Safety and Dependability EvaluationRobustness and stress testing of on-board OS

– This off-the-shelf open-source real time operating system isbroadly used in space applications

– Tested against the available “specifications”

– Robustness-tested with singular and boundary input values,exercising error handling mechanisms

– Stress-tested in conditions of extreme workload

– Faults mainly from robustness tests:– Incorrect control flow– Memory alignment/Illegal instruction exceptions– Unexpected error codes returned

ESA D/TEC-QQS Software Product Assurance 26

SAS, September 2007

Software Safety and Dependability EvaluationHSIA on scientific instrument

– Hardware-Software Interaction Analysis performed on ascience payload of the space observatory

– HW failure modes extracted from FMECA (some newlyidentified from system analysis)

– Several issues identified:– Lack of failure detection and recovery mechanisms– Deficient failure reporting– System FMECA Report updates necessary

ESA D/TEC-QQS Software Product Assurance 27

SAS, September 2007

Software Safety and Dependability EvaluationHazard Analysis on a Flight Application Software

– Methodology used from ESA standard HB-Q80-03

– Based on list of feared events and severity levels derivedfrom system-level analyses

– Software Fault Tree Analysis starting from selected fearedevent, to identify basic events; then SW FMECA on the SWfunctions/components that can be traced to the basicevents

– As a result, the combination of techniques allowed theidentification of potential hazard scenarios andinvestigation of hazard reduction/control mechanisms

ESA D/TEC-QQS Software Product Assurance 28

SAS, September 2007

– Systematic identification of SW failure modes andcriticality classification of SW components

– To drive the verification activities and the “rigour” of thedevelopment, based on the criticality classification ofthe SW components

– Verification that for all potential HW failures the SWreaction is correctly specified

– Improvement of system-level Failure Detection,Isolation and Recovery (FDIR) through Integration ofSW-level analyses results into system-level analyses

Software Safety and Dependability Evaluation Conclusions