qualification of complex avionics systems - mys5.org 3/qualification and reliability...federated...
TRANSCRIPT
1
Qualification and Reliability of Complex Electronic Rotorcraft Systems
by Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
UH-60M Blackhawk Upgrade Picture by US Army
CH47F Chinook CAAS Glass CockpitPicture by US Army
DISCLAIMER Presented at theSafe and Secure Systems and Software Symposium (S5) on 15-17 June 2010 This material is declared a work of theUS Government and is not subject of copyright materials Approved for public release distribution unlimited Review completed by the AMRDEC PublicAffairs Office 21 Sep 2009 FN4208 Reference herein to any specific commercial private or public products process or service by trade name trademarkmanufacturer or otherwise does not constitute or imply its endorsement recommendation or favoring by the United States Government The views andopinions expressed herein are strictly those of the authors and do not represent or reflect those of the United States Government
2
Agenda
bull Objectivebull Defense Acquisition Approach to Systems Development and Testbull US Army Airworthinessbull AED and Qualificationbull Evolution of Helicopter Systemsbull Present Approach to Testingbull Development Challengesbull Complexity Issuesbull Complexity vs Reliability Cost and Schedulebull Complex System Examples and Failuresbull Lessons Learned from Failuresbull Current Guidelines and Certification Assessment Considerationsbull Definition of Complexity and Reliability Neededbull Analytical Modeling and Analysisbull Call for System Reliability Standard Establishment
3
Objective
ldquoDevelop an industry standard method for qualifying complex integrated systems to a specified reliabilityrdquo
RA-66 Comanche Picture by US Army
4
Defense Acquisition Approach to Systems Development and Test
Requirements Establishment
Analysis
High Level Design
Detailed Specifications
Implementation Coding
Operational Testing amp Validation
Verification
Development Testing
Deployed System
5
US Army Airworthiness
Airworthiness Qualification meansThe system is safe and reliable to operate and willperform the mission when deliveredIt will continue to safely perform the mission ifmaintainedoperated per the manualParts and overhaul work must be high quality to maintain airworthinessFlight control systems have high reliability requirements - 10-9 for civil airspace critical IFR functions [35]- 10-6 for tactical airspace [36]
6
AED and Qualification
bull AEDrsquos mission is to ensure airworthiness qualification for aircraft and subsystems used in the US Army fleet
bull Airworthiness Qualification isndash Demonstration of an aircraft or aircraft subsystem or
component including modifications to function safely meeting performance specifications when used and maintained within prescribed limits (AR 70-62)
bull Traditionally qualified systems by ndash Similarity ndash Analysisndash Testndash Demonstrationndash Examination
7
Evolution of Helicopter Systems
bull Past systems historically federated ndash Distributed Functionality ndash Hardware basedndash Simple easier to test
bull Now systems are becoming integratedndash Combined Functionality ndash More Software Intensivendash Complex more difficult to test
Chief Warrant Office Jim Beaty (back row far left) and crew of the of the Vietnam UH-1 Flying Bikinis (friend of Alex Boydston)
UH-1 Cockpit (US Army)
Chinook CAAS Cockpit (US Army)
CH-47 Chinook (US Army)
8
Present Approach to Testing
bull Several disciplines weigh in such as software avionics GNampCenvironmental E3 electrical human factors structures aerodynamics etc
bull Current test methodology per older federated systemsndash Hardware Mil-Std 810 Mil-Std 461ndash Requirements Analysis (Traceability)
bull Test at different levelsndash Individual software module testingndash Black box testingndash System Integration testing
bull Hot benchbull System Integration Lab (SIL)
ndash Aircraft level Testing bull Groundbull Flight Aviation Flight Test Directorate (AFTD) Testing (US Army Photo)
Aviation System Integration Facility (ASIF) (US Army Photo)
9
Development Challenges
bull Legacy Aircraft often upgraded in a piecemeal fashionndash Makes certification difficultndash Desire to increase to modern requirements based on size of upgrade and
what it includes ndash hard to scope
bull New system requirements must be clear complete and testable ndash Certification requirements must be obvious
bull Orchestrating agreement between stakeholders is necessary to mitigatendash Juggling of multiple software buildsndash End system that is difficult to test certify and deployndash Escalating Costsndash System Safety from being poorly understoodndash Design iterations
10
Complexity Issues
bull System Development costs and schedule increase with complexityndash Existing lack of schedule and funding resources
bull Keeps systems from achieving full compliance with specifications and requirements
bull Garbage in -gt Garbage OuthellipPoor requirements -gt Poor Systemndash Finding problems in new designs at PDR is too latendash Difficult to correct existing poorly designed fielded complex systems
bull Complexity amp reliability of complex systems is not fully understoodndash How do we accurately assess operating risk performance reliability of
complex systems based on limited testing and analysisndash How do we know when system design is good enoughndash Latent defects occur in supposedly well-tested mature systems
bull Avionics parts and software change constantlyndash Spiral development -gt new softwarehardware qualification required frequently ndash How do we streamline the process (partition the system) so the need for
complete re-qualification after changes is lessened
11
Complexity Issues (continued)
bull Functional Hazard Assessments and related documentation are crucial
ndash Understanding risks ndash Performing the correct tests at the right level Lab test vs Flight Test
bull Saves flight time and money
bull Systems Integration for complex systems is a schedule driver
bull Need experienced personnel to work complex systems
bull Need a centralized database - just doesnrsquot existndash Determine data needed for quantifying reliability of complex systemsndash Capture the pertinent data on previous or existing complex systemsndash Understand successes and failures of previous and present complex systemsndash Establish baseline reliability figures for known architectures
bull Complex System of Systems exacerbates problem
12
Reliability vs Complexity ampCost vs Complexity
Notional Graphs
bull Reliability vs Complexity bull Cost amp Schedule vs Complexity
Rel
iabi
lity
Complexity Complexity
Cos
t amp S
ched
uleOptimum
Aggregation of part reliability
feeds into overall system reliability
Desired
13
A Few Examples of Complex Systems
bull This is not a new problem Other have struggled with the challenges of establishing confidence in complex systems
ndash NASAbull Apollo Guidance Computerbull Dryden F8 Crusaderbull Space Shuttlebull International Space Station
ndash Commercial Airlinersbull Airbus A320 and higherbull Boeing B777 B787
ndash Militarybull Ships and Submarinesbull Jets (F14F15 F16 F18 F22 F35 etc)bull Cargo Planes (C130J Hercules C17 Globemaster etc)bull Helicopters (Chinook Blackhawk Sea Stallion etc)bull Rocketsbull Unmanned Aerial Systemsbull Unmanned Ground Systemsbull Unmanned Submarine Systems Photos by US Army NASA US Navy and US Air Force
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
2
Agenda
bull Objectivebull Defense Acquisition Approach to Systems Development and Testbull US Army Airworthinessbull AED and Qualificationbull Evolution of Helicopter Systemsbull Present Approach to Testingbull Development Challengesbull Complexity Issuesbull Complexity vs Reliability Cost and Schedulebull Complex System Examples and Failuresbull Lessons Learned from Failuresbull Current Guidelines and Certification Assessment Considerationsbull Definition of Complexity and Reliability Neededbull Analytical Modeling and Analysisbull Call for System Reliability Standard Establishment
3
Objective
ldquoDevelop an industry standard method for qualifying complex integrated systems to a specified reliabilityrdquo
RA-66 Comanche Picture by US Army
4
Defense Acquisition Approach to Systems Development and Test
Requirements Establishment
Analysis
High Level Design
Detailed Specifications
Implementation Coding
Operational Testing amp Validation
Verification
Development Testing
Deployed System
5
US Army Airworthiness
Airworthiness Qualification meansThe system is safe and reliable to operate and willperform the mission when deliveredIt will continue to safely perform the mission ifmaintainedoperated per the manualParts and overhaul work must be high quality to maintain airworthinessFlight control systems have high reliability requirements - 10-9 for civil airspace critical IFR functions [35]- 10-6 for tactical airspace [36]
6
AED and Qualification
bull AEDrsquos mission is to ensure airworthiness qualification for aircraft and subsystems used in the US Army fleet
bull Airworthiness Qualification isndash Demonstration of an aircraft or aircraft subsystem or
component including modifications to function safely meeting performance specifications when used and maintained within prescribed limits (AR 70-62)
bull Traditionally qualified systems by ndash Similarity ndash Analysisndash Testndash Demonstrationndash Examination
7
Evolution of Helicopter Systems
bull Past systems historically federated ndash Distributed Functionality ndash Hardware basedndash Simple easier to test
bull Now systems are becoming integratedndash Combined Functionality ndash More Software Intensivendash Complex more difficult to test
Chief Warrant Office Jim Beaty (back row far left) and crew of the of the Vietnam UH-1 Flying Bikinis (friend of Alex Boydston)
UH-1 Cockpit (US Army)
Chinook CAAS Cockpit (US Army)
CH-47 Chinook (US Army)
8
Present Approach to Testing
bull Several disciplines weigh in such as software avionics GNampCenvironmental E3 electrical human factors structures aerodynamics etc
bull Current test methodology per older federated systemsndash Hardware Mil-Std 810 Mil-Std 461ndash Requirements Analysis (Traceability)
bull Test at different levelsndash Individual software module testingndash Black box testingndash System Integration testing
bull Hot benchbull System Integration Lab (SIL)
ndash Aircraft level Testing bull Groundbull Flight Aviation Flight Test Directorate (AFTD) Testing (US Army Photo)
Aviation System Integration Facility (ASIF) (US Army Photo)
9
Development Challenges
bull Legacy Aircraft often upgraded in a piecemeal fashionndash Makes certification difficultndash Desire to increase to modern requirements based on size of upgrade and
what it includes ndash hard to scope
bull New system requirements must be clear complete and testable ndash Certification requirements must be obvious
bull Orchestrating agreement between stakeholders is necessary to mitigatendash Juggling of multiple software buildsndash End system that is difficult to test certify and deployndash Escalating Costsndash System Safety from being poorly understoodndash Design iterations
10
Complexity Issues
bull System Development costs and schedule increase with complexityndash Existing lack of schedule and funding resources
bull Keeps systems from achieving full compliance with specifications and requirements
bull Garbage in -gt Garbage OuthellipPoor requirements -gt Poor Systemndash Finding problems in new designs at PDR is too latendash Difficult to correct existing poorly designed fielded complex systems
bull Complexity amp reliability of complex systems is not fully understoodndash How do we accurately assess operating risk performance reliability of
complex systems based on limited testing and analysisndash How do we know when system design is good enoughndash Latent defects occur in supposedly well-tested mature systems
bull Avionics parts and software change constantlyndash Spiral development -gt new softwarehardware qualification required frequently ndash How do we streamline the process (partition the system) so the need for
complete re-qualification after changes is lessened
11
Complexity Issues (continued)
bull Functional Hazard Assessments and related documentation are crucial
ndash Understanding risks ndash Performing the correct tests at the right level Lab test vs Flight Test
bull Saves flight time and money
bull Systems Integration for complex systems is a schedule driver
bull Need experienced personnel to work complex systems
bull Need a centralized database - just doesnrsquot existndash Determine data needed for quantifying reliability of complex systemsndash Capture the pertinent data on previous or existing complex systemsndash Understand successes and failures of previous and present complex systemsndash Establish baseline reliability figures for known architectures
bull Complex System of Systems exacerbates problem
12
Reliability vs Complexity ampCost vs Complexity
Notional Graphs
bull Reliability vs Complexity bull Cost amp Schedule vs Complexity
Rel
iabi
lity
Complexity Complexity
Cos
t amp S
ched
uleOptimum
Aggregation of part reliability
feeds into overall system reliability
Desired
13
A Few Examples of Complex Systems
bull This is not a new problem Other have struggled with the challenges of establishing confidence in complex systems
ndash NASAbull Apollo Guidance Computerbull Dryden F8 Crusaderbull Space Shuttlebull International Space Station
ndash Commercial Airlinersbull Airbus A320 and higherbull Boeing B777 B787
ndash Militarybull Ships and Submarinesbull Jets (F14F15 F16 F18 F22 F35 etc)bull Cargo Planes (C130J Hercules C17 Globemaster etc)bull Helicopters (Chinook Blackhawk Sea Stallion etc)bull Rocketsbull Unmanned Aerial Systemsbull Unmanned Ground Systemsbull Unmanned Submarine Systems Photos by US Army NASA US Navy and US Air Force
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
3
Objective
ldquoDevelop an industry standard method for qualifying complex integrated systems to a specified reliabilityrdquo
RA-66 Comanche Picture by US Army
4
Defense Acquisition Approach to Systems Development and Test
Requirements Establishment
Analysis
High Level Design
Detailed Specifications
Implementation Coding
Operational Testing amp Validation
Verification
Development Testing
Deployed System
5
US Army Airworthiness
Airworthiness Qualification meansThe system is safe and reliable to operate and willperform the mission when deliveredIt will continue to safely perform the mission ifmaintainedoperated per the manualParts and overhaul work must be high quality to maintain airworthinessFlight control systems have high reliability requirements - 10-9 for civil airspace critical IFR functions [35]- 10-6 for tactical airspace [36]
6
AED and Qualification
bull AEDrsquos mission is to ensure airworthiness qualification for aircraft and subsystems used in the US Army fleet
bull Airworthiness Qualification isndash Demonstration of an aircraft or aircraft subsystem or
component including modifications to function safely meeting performance specifications when used and maintained within prescribed limits (AR 70-62)
bull Traditionally qualified systems by ndash Similarity ndash Analysisndash Testndash Demonstrationndash Examination
7
Evolution of Helicopter Systems
bull Past systems historically federated ndash Distributed Functionality ndash Hardware basedndash Simple easier to test
bull Now systems are becoming integratedndash Combined Functionality ndash More Software Intensivendash Complex more difficult to test
Chief Warrant Office Jim Beaty (back row far left) and crew of the of the Vietnam UH-1 Flying Bikinis (friend of Alex Boydston)
UH-1 Cockpit (US Army)
Chinook CAAS Cockpit (US Army)
CH-47 Chinook (US Army)
8
Present Approach to Testing
bull Several disciplines weigh in such as software avionics GNampCenvironmental E3 electrical human factors structures aerodynamics etc
bull Current test methodology per older federated systemsndash Hardware Mil-Std 810 Mil-Std 461ndash Requirements Analysis (Traceability)
bull Test at different levelsndash Individual software module testingndash Black box testingndash System Integration testing
bull Hot benchbull System Integration Lab (SIL)
ndash Aircraft level Testing bull Groundbull Flight Aviation Flight Test Directorate (AFTD) Testing (US Army Photo)
Aviation System Integration Facility (ASIF) (US Army Photo)
9
Development Challenges
bull Legacy Aircraft often upgraded in a piecemeal fashionndash Makes certification difficultndash Desire to increase to modern requirements based on size of upgrade and
what it includes ndash hard to scope
bull New system requirements must be clear complete and testable ndash Certification requirements must be obvious
bull Orchestrating agreement between stakeholders is necessary to mitigatendash Juggling of multiple software buildsndash End system that is difficult to test certify and deployndash Escalating Costsndash System Safety from being poorly understoodndash Design iterations
10
Complexity Issues
bull System Development costs and schedule increase with complexityndash Existing lack of schedule and funding resources
bull Keeps systems from achieving full compliance with specifications and requirements
bull Garbage in -gt Garbage OuthellipPoor requirements -gt Poor Systemndash Finding problems in new designs at PDR is too latendash Difficult to correct existing poorly designed fielded complex systems
bull Complexity amp reliability of complex systems is not fully understoodndash How do we accurately assess operating risk performance reliability of
complex systems based on limited testing and analysisndash How do we know when system design is good enoughndash Latent defects occur in supposedly well-tested mature systems
bull Avionics parts and software change constantlyndash Spiral development -gt new softwarehardware qualification required frequently ndash How do we streamline the process (partition the system) so the need for
complete re-qualification after changes is lessened
11
Complexity Issues (continued)
bull Functional Hazard Assessments and related documentation are crucial
ndash Understanding risks ndash Performing the correct tests at the right level Lab test vs Flight Test
bull Saves flight time and money
bull Systems Integration for complex systems is a schedule driver
bull Need experienced personnel to work complex systems
bull Need a centralized database - just doesnrsquot existndash Determine data needed for quantifying reliability of complex systemsndash Capture the pertinent data on previous or existing complex systemsndash Understand successes and failures of previous and present complex systemsndash Establish baseline reliability figures for known architectures
bull Complex System of Systems exacerbates problem
12
Reliability vs Complexity ampCost vs Complexity
Notional Graphs
bull Reliability vs Complexity bull Cost amp Schedule vs Complexity
Rel
iabi
lity
Complexity Complexity
Cos
t amp S
ched
uleOptimum
Aggregation of part reliability
feeds into overall system reliability
Desired
13
A Few Examples of Complex Systems
bull This is not a new problem Other have struggled with the challenges of establishing confidence in complex systems
ndash NASAbull Apollo Guidance Computerbull Dryden F8 Crusaderbull Space Shuttlebull International Space Station
ndash Commercial Airlinersbull Airbus A320 and higherbull Boeing B777 B787
ndash Militarybull Ships and Submarinesbull Jets (F14F15 F16 F18 F22 F35 etc)bull Cargo Planes (C130J Hercules C17 Globemaster etc)bull Helicopters (Chinook Blackhawk Sea Stallion etc)bull Rocketsbull Unmanned Aerial Systemsbull Unmanned Ground Systemsbull Unmanned Submarine Systems Photos by US Army NASA US Navy and US Air Force
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
4
Defense Acquisition Approach to Systems Development and Test
Requirements Establishment
Analysis
High Level Design
Detailed Specifications
Implementation Coding
Operational Testing amp Validation
Verification
Development Testing
Deployed System
5
US Army Airworthiness
Airworthiness Qualification meansThe system is safe and reliable to operate and willperform the mission when deliveredIt will continue to safely perform the mission ifmaintainedoperated per the manualParts and overhaul work must be high quality to maintain airworthinessFlight control systems have high reliability requirements - 10-9 for civil airspace critical IFR functions [35]- 10-6 for tactical airspace [36]
6
AED and Qualification
bull AEDrsquos mission is to ensure airworthiness qualification for aircraft and subsystems used in the US Army fleet
bull Airworthiness Qualification isndash Demonstration of an aircraft or aircraft subsystem or
component including modifications to function safely meeting performance specifications when used and maintained within prescribed limits (AR 70-62)
bull Traditionally qualified systems by ndash Similarity ndash Analysisndash Testndash Demonstrationndash Examination
7
Evolution of Helicopter Systems
bull Past systems historically federated ndash Distributed Functionality ndash Hardware basedndash Simple easier to test
bull Now systems are becoming integratedndash Combined Functionality ndash More Software Intensivendash Complex more difficult to test
Chief Warrant Office Jim Beaty (back row far left) and crew of the of the Vietnam UH-1 Flying Bikinis (friend of Alex Boydston)
UH-1 Cockpit (US Army)
Chinook CAAS Cockpit (US Army)
CH-47 Chinook (US Army)
8
Present Approach to Testing
bull Several disciplines weigh in such as software avionics GNampCenvironmental E3 electrical human factors structures aerodynamics etc
bull Current test methodology per older federated systemsndash Hardware Mil-Std 810 Mil-Std 461ndash Requirements Analysis (Traceability)
bull Test at different levelsndash Individual software module testingndash Black box testingndash System Integration testing
bull Hot benchbull System Integration Lab (SIL)
ndash Aircraft level Testing bull Groundbull Flight Aviation Flight Test Directorate (AFTD) Testing (US Army Photo)
Aviation System Integration Facility (ASIF) (US Army Photo)
9
Development Challenges
bull Legacy Aircraft often upgraded in a piecemeal fashionndash Makes certification difficultndash Desire to increase to modern requirements based on size of upgrade and
what it includes ndash hard to scope
bull New system requirements must be clear complete and testable ndash Certification requirements must be obvious
bull Orchestrating agreement between stakeholders is necessary to mitigatendash Juggling of multiple software buildsndash End system that is difficult to test certify and deployndash Escalating Costsndash System Safety from being poorly understoodndash Design iterations
10
Complexity Issues
bull System Development costs and schedule increase with complexityndash Existing lack of schedule and funding resources
bull Keeps systems from achieving full compliance with specifications and requirements
bull Garbage in -gt Garbage OuthellipPoor requirements -gt Poor Systemndash Finding problems in new designs at PDR is too latendash Difficult to correct existing poorly designed fielded complex systems
bull Complexity amp reliability of complex systems is not fully understoodndash How do we accurately assess operating risk performance reliability of
complex systems based on limited testing and analysisndash How do we know when system design is good enoughndash Latent defects occur in supposedly well-tested mature systems
bull Avionics parts and software change constantlyndash Spiral development -gt new softwarehardware qualification required frequently ndash How do we streamline the process (partition the system) so the need for
complete re-qualification after changes is lessened
11
Complexity Issues (continued)
bull Functional Hazard Assessments and related documentation are crucial
ndash Understanding risks ndash Performing the correct tests at the right level Lab test vs Flight Test
bull Saves flight time and money
bull Systems Integration for complex systems is a schedule driver
bull Need experienced personnel to work complex systems
bull Need a centralized database - just doesnrsquot existndash Determine data needed for quantifying reliability of complex systemsndash Capture the pertinent data on previous or existing complex systemsndash Understand successes and failures of previous and present complex systemsndash Establish baseline reliability figures for known architectures
bull Complex System of Systems exacerbates problem
12
Reliability vs Complexity ampCost vs Complexity
Notional Graphs
bull Reliability vs Complexity bull Cost amp Schedule vs Complexity
Rel
iabi
lity
Complexity Complexity
Cos
t amp S
ched
uleOptimum
Aggregation of part reliability
feeds into overall system reliability
Desired
13
A Few Examples of Complex Systems
bull This is not a new problem Other have struggled with the challenges of establishing confidence in complex systems
ndash NASAbull Apollo Guidance Computerbull Dryden F8 Crusaderbull Space Shuttlebull International Space Station
ndash Commercial Airlinersbull Airbus A320 and higherbull Boeing B777 B787
ndash Militarybull Ships and Submarinesbull Jets (F14F15 F16 F18 F22 F35 etc)bull Cargo Planes (C130J Hercules C17 Globemaster etc)bull Helicopters (Chinook Blackhawk Sea Stallion etc)bull Rocketsbull Unmanned Aerial Systemsbull Unmanned Ground Systemsbull Unmanned Submarine Systems Photos by US Army NASA US Navy and US Air Force
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
5
US Army Airworthiness
Airworthiness Qualification meansThe system is safe and reliable to operate and willperform the mission when deliveredIt will continue to safely perform the mission ifmaintainedoperated per the manualParts and overhaul work must be high quality to maintain airworthinessFlight control systems have high reliability requirements - 10-9 for civil airspace critical IFR functions [35]- 10-6 for tactical airspace [36]
6
AED and Qualification
bull AEDrsquos mission is to ensure airworthiness qualification for aircraft and subsystems used in the US Army fleet
bull Airworthiness Qualification isndash Demonstration of an aircraft or aircraft subsystem or
component including modifications to function safely meeting performance specifications when used and maintained within prescribed limits (AR 70-62)
bull Traditionally qualified systems by ndash Similarity ndash Analysisndash Testndash Demonstrationndash Examination
7
Evolution of Helicopter Systems
bull Past systems historically federated ndash Distributed Functionality ndash Hardware basedndash Simple easier to test
bull Now systems are becoming integratedndash Combined Functionality ndash More Software Intensivendash Complex more difficult to test
Chief Warrant Office Jim Beaty (back row far left) and crew of the of the Vietnam UH-1 Flying Bikinis (friend of Alex Boydston)
UH-1 Cockpit (US Army)
Chinook CAAS Cockpit (US Army)
CH-47 Chinook (US Army)
8
Present Approach to Testing
bull Several disciplines weigh in such as software avionics GNampCenvironmental E3 electrical human factors structures aerodynamics etc
bull Current test methodology per older federated systemsndash Hardware Mil-Std 810 Mil-Std 461ndash Requirements Analysis (Traceability)
bull Test at different levelsndash Individual software module testingndash Black box testingndash System Integration testing
bull Hot benchbull System Integration Lab (SIL)
ndash Aircraft level Testing bull Groundbull Flight Aviation Flight Test Directorate (AFTD) Testing (US Army Photo)
Aviation System Integration Facility (ASIF) (US Army Photo)
9
Development Challenges
bull Legacy Aircraft often upgraded in a piecemeal fashionndash Makes certification difficultndash Desire to increase to modern requirements based on size of upgrade and
what it includes ndash hard to scope
bull New system requirements must be clear complete and testable ndash Certification requirements must be obvious
bull Orchestrating agreement between stakeholders is necessary to mitigatendash Juggling of multiple software buildsndash End system that is difficult to test certify and deployndash Escalating Costsndash System Safety from being poorly understoodndash Design iterations
10
Complexity Issues
bull System Development costs and schedule increase with complexityndash Existing lack of schedule and funding resources
bull Keeps systems from achieving full compliance with specifications and requirements
bull Garbage in -gt Garbage OuthellipPoor requirements -gt Poor Systemndash Finding problems in new designs at PDR is too latendash Difficult to correct existing poorly designed fielded complex systems
bull Complexity amp reliability of complex systems is not fully understoodndash How do we accurately assess operating risk performance reliability of
complex systems based on limited testing and analysisndash How do we know when system design is good enoughndash Latent defects occur in supposedly well-tested mature systems
bull Avionics parts and software change constantlyndash Spiral development -gt new softwarehardware qualification required frequently ndash How do we streamline the process (partition the system) so the need for
complete re-qualification after changes is lessened
11
Complexity Issues (continued)
bull Functional Hazard Assessments and related documentation are crucial
ndash Understanding risks ndash Performing the correct tests at the right level Lab test vs Flight Test
bull Saves flight time and money
bull Systems Integration for complex systems is a schedule driver
bull Need experienced personnel to work complex systems
bull Need a centralized database - just doesnrsquot existndash Determine data needed for quantifying reliability of complex systemsndash Capture the pertinent data on previous or existing complex systemsndash Understand successes and failures of previous and present complex systemsndash Establish baseline reliability figures for known architectures
bull Complex System of Systems exacerbates problem
12
Reliability vs Complexity ampCost vs Complexity
Notional Graphs
bull Reliability vs Complexity bull Cost amp Schedule vs Complexity
Rel
iabi
lity
Complexity Complexity
Cos
t amp S
ched
uleOptimum
Aggregation of part reliability
feeds into overall system reliability
Desired
13
A Few Examples of Complex Systems
bull This is not a new problem Other have struggled with the challenges of establishing confidence in complex systems
ndash NASAbull Apollo Guidance Computerbull Dryden F8 Crusaderbull Space Shuttlebull International Space Station
ndash Commercial Airlinersbull Airbus A320 and higherbull Boeing B777 B787
ndash Militarybull Ships and Submarinesbull Jets (F14F15 F16 F18 F22 F35 etc)bull Cargo Planes (C130J Hercules C17 Globemaster etc)bull Helicopters (Chinook Blackhawk Sea Stallion etc)bull Rocketsbull Unmanned Aerial Systemsbull Unmanned Ground Systemsbull Unmanned Submarine Systems Photos by US Army NASA US Navy and US Air Force
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
6
AED and Qualification
bull AEDrsquos mission is to ensure airworthiness qualification for aircraft and subsystems used in the US Army fleet
bull Airworthiness Qualification isndash Demonstration of an aircraft or aircraft subsystem or
component including modifications to function safely meeting performance specifications when used and maintained within prescribed limits (AR 70-62)
bull Traditionally qualified systems by ndash Similarity ndash Analysisndash Testndash Demonstrationndash Examination
7
Evolution of Helicopter Systems
bull Past systems historically federated ndash Distributed Functionality ndash Hardware basedndash Simple easier to test
bull Now systems are becoming integratedndash Combined Functionality ndash More Software Intensivendash Complex more difficult to test
Chief Warrant Office Jim Beaty (back row far left) and crew of the of the Vietnam UH-1 Flying Bikinis (friend of Alex Boydston)
UH-1 Cockpit (US Army)
Chinook CAAS Cockpit (US Army)
CH-47 Chinook (US Army)
8
Present Approach to Testing
bull Several disciplines weigh in such as software avionics GNampCenvironmental E3 electrical human factors structures aerodynamics etc
bull Current test methodology per older federated systemsndash Hardware Mil-Std 810 Mil-Std 461ndash Requirements Analysis (Traceability)
bull Test at different levelsndash Individual software module testingndash Black box testingndash System Integration testing
bull Hot benchbull System Integration Lab (SIL)
ndash Aircraft level Testing bull Groundbull Flight Aviation Flight Test Directorate (AFTD) Testing (US Army Photo)
Aviation System Integration Facility (ASIF) (US Army Photo)
9
Development Challenges
bull Legacy Aircraft often upgraded in a piecemeal fashionndash Makes certification difficultndash Desire to increase to modern requirements based on size of upgrade and
what it includes ndash hard to scope
bull New system requirements must be clear complete and testable ndash Certification requirements must be obvious
bull Orchestrating agreement between stakeholders is necessary to mitigatendash Juggling of multiple software buildsndash End system that is difficult to test certify and deployndash Escalating Costsndash System Safety from being poorly understoodndash Design iterations
10
Complexity Issues
bull System Development costs and schedule increase with complexityndash Existing lack of schedule and funding resources
bull Keeps systems from achieving full compliance with specifications and requirements
bull Garbage in -gt Garbage OuthellipPoor requirements -gt Poor Systemndash Finding problems in new designs at PDR is too latendash Difficult to correct existing poorly designed fielded complex systems
bull Complexity amp reliability of complex systems is not fully understoodndash How do we accurately assess operating risk performance reliability of
complex systems based on limited testing and analysisndash How do we know when system design is good enoughndash Latent defects occur in supposedly well-tested mature systems
bull Avionics parts and software change constantlyndash Spiral development -gt new softwarehardware qualification required frequently ndash How do we streamline the process (partition the system) so the need for
complete re-qualification after changes is lessened
11
Complexity Issues (continued)
bull Functional Hazard Assessments and related documentation are crucial
ndash Understanding risks ndash Performing the correct tests at the right level Lab test vs Flight Test
bull Saves flight time and money
bull Systems Integration for complex systems is a schedule driver
bull Need experienced personnel to work complex systems
bull Need a centralized database - just doesnrsquot existndash Determine data needed for quantifying reliability of complex systemsndash Capture the pertinent data on previous or existing complex systemsndash Understand successes and failures of previous and present complex systemsndash Establish baseline reliability figures for known architectures
bull Complex System of Systems exacerbates problem
12
Reliability vs Complexity ampCost vs Complexity
Notional Graphs
bull Reliability vs Complexity bull Cost amp Schedule vs Complexity
Rel
iabi
lity
Complexity Complexity
Cos
t amp S
ched
uleOptimum
Aggregation of part reliability
feeds into overall system reliability
Desired
13
A Few Examples of Complex Systems
bull This is not a new problem Other have struggled with the challenges of establishing confidence in complex systems
ndash NASAbull Apollo Guidance Computerbull Dryden F8 Crusaderbull Space Shuttlebull International Space Station
ndash Commercial Airlinersbull Airbus A320 and higherbull Boeing B777 B787
ndash Militarybull Ships and Submarinesbull Jets (F14F15 F16 F18 F22 F35 etc)bull Cargo Planes (C130J Hercules C17 Globemaster etc)bull Helicopters (Chinook Blackhawk Sea Stallion etc)bull Rocketsbull Unmanned Aerial Systemsbull Unmanned Ground Systemsbull Unmanned Submarine Systems Photos by US Army NASA US Navy and US Air Force
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
7
Evolution of Helicopter Systems
bull Past systems historically federated ndash Distributed Functionality ndash Hardware basedndash Simple easier to test
bull Now systems are becoming integratedndash Combined Functionality ndash More Software Intensivendash Complex more difficult to test
Chief Warrant Office Jim Beaty (back row far left) and crew of the of the Vietnam UH-1 Flying Bikinis (friend of Alex Boydston)
UH-1 Cockpit (US Army)
Chinook CAAS Cockpit (US Army)
CH-47 Chinook (US Army)
8
Present Approach to Testing
bull Several disciplines weigh in such as software avionics GNampCenvironmental E3 electrical human factors structures aerodynamics etc
bull Current test methodology per older federated systemsndash Hardware Mil-Std 810 Mil-Std 461ndash Requirements Analysis (Traceability)
bull Test at different levelsndash Individual software module testingndash Black box testingndash System Integration testing
bull Hot benchbull System Integration Lab (SIL)
ndash Aircraft level Testing bull Groundbull Flight Aviation Flight Test Directorate (AFTD) Testing (US Army Photo)
Aviation System Integration Facility (ASIF) (US Army Photo)
9
Development Challenges
bull Legacy Aircraft often upgraded in a piecemeal fashionndash Makes certification difficultndash Desire to increase to modern requirements based on size of upgrade and
what it includes ndash hard to scope
bull New system requirements must be clear complete and testable ndash Certification requirements must be obvious
bull Orchestrating agreement between stakeholders is necessary to mitigatendash Juggling of multiple software buildsndash End system that is difficult to test certify and deployndash Escalating Costsndash System Safety from being poorly understoodndash Design iterations
10
Complexity Issues
bull System Development costs and schedule increase with complexityndash Existing lack of schedule and funding resources
bull Keeps systems from achieving full compliance with specifications and requirements
bull Garbage in -gt Garbage OuthellipPoor requirements -gt Poor Systemndash Finding problems in new designs at PDR is too latendash Difficult to correct existing poorly designed fielded complex systems
bull Complexity amp reliability of complex systems is not fully understoodndash How do we accurately assess operating risk performance reliability of
complex systems based on limited testing and analysisndash How do we know when system design is good enoughndash Latent defects occur in supposedly well-tested mature systems
bull Avionics parts and software change constantlyndash Spiral development -gt new softwarehardware qualification required frequently ndash How do we streamline the process (partition the system) so the need for
complete re-qualification after changes is lessened
11
Complexity Issues (continued)
bull Functional Hazard Assessments and related documentation are crucial
ndash Understanding risks ndash Performing the correct tests at the right level Lab test vs Flight Test
bull Saves flight time and money
bull Systems Integration for complex systems is a schedule driver
bull Need experienced personnel to work complex systems
bull Need a centralized database - just doesnrsquot existndash Determine data needed for quantifying reliability of complex systemsndash Capture the pertinent data on previous or existing complex systemsndash Understand successes and failures of previous and present complex systemsndash Establish baseline reliability figures for known architectures
bull Complex System of Systems exacerbates problem
12
Reliability vs Complexity ampCost vs Complexity
Notional Graphs
bull Reliability vs Complexity bull Cost amp Schedule vs Complexity
Rel
iabi
lity
Complexity Complexity
Cos
t amp S
ched
uleOptimum
Aggregation of part reliability
feeds into overall system reliability
Desired
13
A Few Examples of Complex Systems
bull This is not a new problem Other have struggled with the challenges of establishing confidence in complex systems
ndash NASAbull Apollo Guidance Computerbull Dryden F8 Crusaderbull Space Shuttlebull International Space Station
ndash Commercial Airlinersbull Airbus A320 and higherbull Boeing B777 B787
ndash Militarybull Ships and Submarinesbull Jets (F14F15 F16 F18 F22 F35 etc)bull Cargo Planes (C130J Hercules C17 Globemaster etc)bull Helicopters (Chinook Blackhawk Sea Stallion etc)bull Rocketsbull Unmanned Aerial Systemsbull Unmanned Ground Systemsbull Unmanned Submarine Systems Photos by US Army NASA US Navy and US Air Force
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
8
Present Approach to Testing
bull Several disciplines weigh in such as software avionics GNampCenvironmental E3 electrical human factors structures aerodynamics etc
bull Current test methodology per older federated systemsndash Hardware Mil-Std 810 Mil-Std 461ndash Requirements Analysis (Traceability)
bull Test at different levelsndash Individual software module testingndash Black box testingndash System Integration testing
bull Hot benchbull System Integration Lab (SIL)
ndash Aircraft level Testing bull Groundbull Flight Aviation Flight Test Directorate (AFTD) Testing (US Army Photo)
Aviation System Integration Facility (ASIF) (US Army Photo)
9
Development Challenges
bull Legacy Aircraft often upgraded in a piecemeal fashionndash Makes certification difficultndash Desire to increase to modern requirements based on size of upgrade and
what it includes ndash hard to scope
bull New system requirements must be clear complete and testable ndash Certification requirements must be obvious
bull Orchestrating agreement between stakeholders is necessary to mitigatendash Juggling of multiple software buildsndash End system that is difficult to test certify and deployndash Escalating Costsndash System Safety from being poorly understoodndash Design iterations
10
Complexity Issues
bull System Development costs and schedule increase with complexityndash Existing lack of schedule and funding resources
bull Keeps systems from achieving full compliance with specifications and requirements
bull Garbage in -gt Garbage OuthellipPoor requirements -gt Poor Systemndash Finding problems in new designs at PDR is too latendash Difficult to correct existing poorly designed fielded complex systems
bull Complexity amp reliability of complex systems is not fully understoodndash How do we accurately assess operating risk performance reliability of
complex systems based on limited testing and analysisndash How do we know when system design is good enoughndash Latent defects occur in supposedly well-tested mature systems
bull Avionics parts and software change constantlyndash Spiral development -gt new softwarehardware qualification required frequently ndash How do we streamline the process (partition the system) so the need for
complete re-qualification after changes is lessened
11
Complexity Issues (continued)
bull Functional Hazard Assessments and related documentation are crucial
ndash Understanding risks ndash Performing the correct tests at the right level Lab test vs Flight Test
bull Saves flight time and money
bull Systems Integration for complex systems is a schedule driver
bull Need experienced personnel to work complex systems
bull Need a centralized database - just doesnrsquot existndash Determine data needed for quantifying reliability of complex systemsndash Capture the pertinent data on previous or existing complex systemsndash Understand successes and failures of previous and present complex systemsndash Establish baseline reliability figures for known architectures
bull Complex System of Systems exacerbates problem
12
Reliability vs Complexity ampCost vs Complexity
Notional Graphs
bull Reliability vs Complexity bull Cost amp Schedule vs Complexity
Rel
iabi
lity
Complexity Complexity
Cos
t amp S
ched
uleOptimum
Aggregation of part reliability
feeds into overall system reliability
Desired
13
A Few Examples of Complex Systems
bull This is not a new problem Other have struggled with the challenges of establishing confidence in complex systems
ndash NASAbull Apollo Guidance Computerbull Dryden F8 Crusaderbull Space Shuttlebull International Space Station
ndash Commercial Airlinersbull Airbus A320 and higherbull Boeing B777 B787
ndash Militarybull Ships and Submarinesbull Jets (F14F15 F16 F18 F22 F35 etc)bull Cargo Planes (C130J Hercules C17 Globemaster etc)bull Helicopters (Chinook Blackhawk Sea Stallion etc)bull Rocketsbull Unmanned Aerial Systemsbull Unmanned Ground Systemsbull Unmanned Submarine Systems Photos by US Army NASA US Navy and US Air Force
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
9
Development Challenges
bull Legacy Aircraft often upgraded in a piecemeal fashionndash Makes certification difficultndash Desire to increase to modern requirements based on size of upgrade and
what it includes ndash hard to scope
bull New system requirements must be clear complete and testable ndash Certification requirements must be obvious
bull Orchestrating agreement between stakeholders is necessary to mitigatendash Juggling of multiple software buildsndash End system that is difficult to test certify and deployndash Escalating Costsndash System Safety from being poorly understoodndash Design iterations
10
Complexity Issues
bull System Development costs and schedule increase with complexityndash Existing lack of schedule and funding resources
bull Keeps systems from achieving full compliance with specifications and requirements
bull Garbage in -gt Garbage OuthellipPoor requirements -gt Poor Systemndash Finding problems in new designs at PDR is too latendash Difficult to correct existing poorly designed fielded complex systems
bull Complexity amp reliability of complex systems is not fully understoodndash How do we accurately assess operating risk performance reliability of
complex systems based on limited testing and analysisndash How do we know when system design is good enoughndash Latent defects occur in supposedly well-tested mature systems
bull Avionics parts and software change constantlyndash Spiral development -gt new softwarehardware qualification required frequently ndash How do we streamline the process (partition the system) so the need for
complete re-qualification after changes is lessened
11
Complexity Issues (continued)
bull Functional Hazard Assessments and related documentation are crucial
ndash Understanding risks ndash Performing the correct tests at the right level Lab test vs Flight Test
bull Saves flight time and money
bull Systems Integration for complex systems is a schedule driver
bull Need experienced personnel to work complex systems
bull Need a centralized database - just doesnrsquot existndash Determine data needed for quantifying reliability of complex systemsndash Capture the pertinent data on previous or existing complex systemsndash Understand successes and failures of previous and present complex systemsndash Establish baseline reliability figures for known architectures
bull Complex System of Systems exacerbates problem
12
Reliability vs Complexity ampCost vs Complexity
Notional Graphs
bull Reliability vs Complexity bull Cost amp Schedule vs Complexity
Rel
iabi
lity
Complexity Complexity
Cos
t amp S
ched
uleOptimum
Aggregation of part reliability
feeds into overall system reliability
Desired
13
A Few Examples of Complex Systems
bull This is not a new problem Other have struggled with the challenges of establishing confidence in complex systems
ndash NASAbull Apollo Guidance Computerbull Dryden F8 Crusaderbull Space Shuttlebull International Space Station
ndash Commercial Airlinersbull Airbus A320 and higherbull Boeing B777 B787
ndash Militarybull Ships and Submarinesbull Jets (F14F15 F16 F18 F22 F35 etc)bull Cargo Planes (C130J Hercules C17 Globemaster etc)bull Helicopters (Chinook Blackhawk Sea Stallion etc)bull Rocketsbull Unmanned Aerial Systemsbull Unmanned Ground Systemsbull Unmanned Submarine Systems Photos by US Army NASA US Navy and US Air Force
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
10
Complexity Issues
bull System Development costs and schedule increase with complexityndash Existing lack of schedule and funding resources
bull Keeps systems from achieving full compliance with specifications and requirements
bull Garbage in -gt Garbage OuthellipPoor requirements -gt Poor Systemndash Finding problems in new designs at PDR is too latendash Difficult to correct existing poorly designed fielded complex systems
bull Complexity amp reliability of complex systems is not fully understoodndash How do we accurately assess operating risk performance reliability of
complex systems based on limited testing and analysisndash How do we know when system design is good enoughndash Latent defects occur in supposedly well-tested mature systems
bull Avionics parts and software change constantlyndash Spiral development -gt new softwarehardware qualification required frequently ndash How do we streamline the process (partition the system) so the need for
complete re-qualification after changes is lessened
11
Complexity Issues (continued)
bull Functional Hazard Assessments and related documentation are crucial
ndash Understanding risks ndash Performing the correct tests at the right level Lab test vs Flight Test
bull Saves flight time and money
bull Systems Integration for complex systems is a schedule driver
bull Need experienced personnel to work complex systems
bull Need a centralized database - just doesnrsquot existndash Determine data needed for quantifying reliability of complex systemsndash Capture the pertinent data on previous or existing complex systemsndash Understand successes and failures of previous and present complex systemsndash Establish baseline reliability figures for known architectures
bull Complex System of Systems exacerbates problem
12
Reliability vs Complexity ampCost vs Complexity
Notional Graphs
bull Reliability vs Complexity bull Cost amp Schedule vs Complexity
Rel
iabi
lity
Complexity Complexity
Cos
t amp S
ched
uleOptimum
Aggregation of part reliability
feeds into overall system reliability
Desired
13
A Few Examples of Complex Systems
bull This is not a new problem Other have struggled with the challenges of establishing confidence in complex systems
ndash NASAbull Apollo Guidance Computerbull Dryden F8 Crusaderbull Space Shuttlebull International Space Station
ndash Commercial Airlinersbull Airbus A320 and higherbull Boeing B777 B787
ndash Militarybull Ships and Submarinesbull Jets (F14F15 F16 F18 F22 F35 etc)bull Cargo Planes (C130J Hercules C17 Globemaster etc)bull Helicopters (Chinook Blackhawk Sea Stallion etc)bull Rocketsbull Unmanned Aerial Systemsbull Unmanned Ground Systemsbull Unmanned Submarine Systems Photos by US Army NASA US Navy and US Air Force
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
11
Complexity Issues (continued)
bull Functional Hazard Assessments and related documentation are crucial
ndash Understanding risks ndash Performing the correct tests at the right level Lab test vs Flight Test
bull Saves flight time and money
bull Systems Integration for complex systems is a schedule driver
bull Need experienced personnel to work complex systems
bull Need a centralized database - just doesnrsquot existndash Determine data needed for quantifying reliability of complex systemsndash Capture the pertinent data on previous or existing complex systemsndash Understand successes and failures of previous and present complex systemsndash Establish baseline reliability figures for known architectures
bull Complex System of Systems exacerbates problem
12
Reliability vs Complexity ampCost vs Complexity
Notional Graphs
bull Reliability vs Complexity bull Cost amp Schedule vs Complexity
Rel
iabi
lity
Complexity Complexity
Cos
t amp S
ched
uleOptimum
Aggregation of part reliability
feeds into overall system reliability
Desired
13
A Few Examples of Complex Systems
bull This is not a new problem Other have struggled with the challenges of establishing confidence in complex systems
ndash NASAbull Apollo Guidance Computerbull Dryden F8 Crusaderbull Space Shuttlebull International Space Station
ndash Commercial Airlinersbull Airbus A320 and higherbull Boeing B777 B787
ndash Militarybull Ships and Submarinesbull Jets (F14F15 F16 F18 F22 F35 etc)bull Cargo Planes (C130J Hercules C17 Globemaster etc)bull Helicopters (Chinook Blackhawk Sea Stallion etc)bull Rocketsbull Unmanned Aerial Systemsbull Unmanned Ground Systemsbull Unmanned Submarine Systems Photos by US Army NASA US Navy and US Air Force
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
12
Reliability vs Complexity ampCost vs Complexity
Notional Graphs
bull Reliability vs Complexity bull Cost amp Schedule vs Complexity
Rel
iabi
lity
Complexity Complexity
Cos
t amp S
ched
uleOptimum
Aggregation of part reliability
feeds into overall system reliability
Desired
13
A Few Examples of Complex Systems
bull This is not a new problem Other have struggled with the challenges of establishing confidence in complex systems
ndash NASAbull Apollo Guidance Computerbull Dryden F8 Crusaderbull Space Shuttlebull International Space Station
ndash Commercial Airlinersbull Airbus A320 and higherbull Boeing B777 B787
ndash Militarybull Ships and Submarinesbull Jets (F14F15 F16 F18 F22 F35 etc)bull Cargo Planes (C130J Hercules C17 Globemaster etc)bull Helicopters (Chinook Blackhawk Sea Stallion etc)bull Rocketsbull Unmanned Aerial Systemsbull Unmanned Ground Systemsbull Unmanned Submarine Systems Photos by US Army NASA US Navy and US Air Force
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
13
A Few Examples of Complex Systems
bull This is not a new problem Other have struggled with the challenges of establishing confidence in complex systems
ndash NASAbull Apollo Guidance Computerbull Dryden F8 Crusaderbull Space Shuttlebull International Space Station
ndash Commercial Airlinersbull Airbus A320 and higherbull Boeing B777 B787
ndash Militarybull Ships and Submarinesbull Jets (F14F15 F16 F18 F22 F35 etc)bull Cargo Planes (C130J Hercules C17 Globemaster etc)bull Helicopters (Chinook Blackhawk Sea Stallion etc)bull Rocketsbull Unmanned Aerial Systemsbull Unmanned Ground Systemsbull Unmanned Submarine Systems Photos by US Army NASA US Navy and US Air Force
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
14
Some Complex System Failures
bull V-22 Osprey crashesbull Mars Climate Orbiter crashbull Mars Pathfinder software resetbull USS Vincennes downing an Airbus 320bull Therac-25 software radiation treatment
failurebull 1989 Airbus A320 air show crashbull China Airlines Airbus Industries A300
crashbull Ariane 5 satellite launcher malfunctionbull Failure of the primary flight system to
sync with the backup during prelaunch of STS-1
bull Mexicana Airlines Boeing 727 airliner crashed into a mountain due to the software not correctly determining the mountain position
bull Loss of the first American probe to Venus
bull Korean Airlines KAL 901 accidentbull Soviet Phobos I Mars probe lost
bull Three Mile Islandbull F-18 fighter plane crash due to bad
exceptionbull F-14 fighter plane lost to
uncontrollable spinbull Swedish Gripen prototype crashedbull Swedish Gripen air-show crashbull F-22 failure crossing the IDLbull 2006 German-Spanish Barracuda UAVbull 2004 FA-22 Raptor stealth fighter jet
crash bull FA-22 Raptor navigation system
software error at Nellis AFBbull 50 cockpit blackouts on A320bull A320 multiple avionics and electrical
failures at Newark NJbull Boeing 777 Malaysian Airlines jetlinerrsquos
nightmarish autopilot rollercoaster ridebull 3000 feet US Army and Air Force UAV
Crashes
bull hellip And Many Morehellip
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
15
Lessons Learned from Failures
bull From Nancy Levesonrsquos paper ldquoThe Role of Software in Spacecraft Accidentsrdquondash ldquoFlaws in the safety culture diffusion of responsibility and
authorityndash Limited communication channels and poor information flowndash Inadequate system and software engineering ndash Poor or missing specifications ndash Unnecessary complexity and software functionality ndash Software reuse or changes without appropriate safety analysisndash [Shortcomings] in safety engineering practices ndash Flaws in test and simulation environments ndash Inadequate human factors design for softwarerdquo
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
16
Some Current Guidelines
bull DO-178B - Software Considerations in Airborne Systems and Equipment Certification bull DO-248B ndash Final Report for the Clarification of DO-178Bbull DO-278 - Guidelines for Communications Navigation Surveillance and Air Traffic Management
(CNSATM) Systems Software Integrity Assurancebull DO-254 - Design Assurance Guidance for Airborne Electronic Hardware bull DO-297 ndash Integrated Modular Avionics (IMA) Development Guidance and Certification
Considerationsbull SAE-ARP4754 ndash Certification Consideration for Highly Integrated or Complex Aircraft Systemsbull SAE-ARP4671- Guidelines and Methods for Conducting the Safety Assessment Process on
Airborne Systems and Equipmentbull FAA Advisory Circular AC27-1B - Certification of Normal Category Rotorcraftbull FAA Advisory Circular AC29-2C - Certification of Transport Category Rotorcraftbull ISOIEC 12207 - Software Life Cycle Processesbull ARINC 653 - Specification Standard for Time and System Partitionbull MIL-STD-882D - DoD System Safetybull ADS-51-HDBK - Rotorcraft and Aircraft Qualification Handbookbull AR-70-62 - Airworthiness Release Standardbull SED-SES-PMHFSA 001 - Software Engineering Directorate (SED) Software Engineering
Evaluation System (SEES) Program Manager Handbook for Flight Software Airworthinessbull SED-SES-PMHSS 001 - SED SEES Program Manager Handbook for Software Safety
WHATrsquoS MISSING - Reliability Standard for Complex Systems
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
17
Certification Assessment Considerations
bull Sufficient data and time must be available for air worthiness evaluation
bull Certification processndash Currently lengthy ndash Depends much on human interpretation trade offs and risk mitigation ndash Overwhelming for complex integrated systems (FHAs FTAs FMECAs
risk mitigation etc)
bull Consistent industry-wide method to assess a system at any stage of the life-cycle to allow a tradeoff of design alternatives
bull Certification Tasks outlined in DO-297 should be consideredndash Task 1 Module Acceptancendash Task 2 Application softwarehardware acceptancendash Task 3 IMA system acceptancendash Task 4 Aircraft integration of IMA system ndash including VampVndash Task 5 Change of modules or applicationsndash Task 6 Reused of modules or applications
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
18
TRL 3 or 4 TRL 6 or 7 TRL 8 or 9
Component 4
ComplexityFundamentals
Reliability Parametrics
Component 3
ComplexityFundamentals
Reliability Parametrics
Definition of Complexity and Reliability is Needed
Subsystem 1
SystemIntegration ofComponents
ReliabilityDependencies
Component 2
ComplexityFundamentals
Reliability Parametrics
Component 1
ComplexityFundamentals
Reliability Parametrics
Subsystem 2
SystemIntegration ofComponents
ReliabilityDependencies
Integration System
Realized System
ReliabilitySensitivities
High Reliable Complex System
Certificate(eg AWR)
Integration
Integration
Integration
Integration
Integration
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
19
Analytical Models and Reliability
bull Analytical models of hardware reliability are well understood
bull Architecture modeling and software reliability modeling is not a novel idea but is highly debated
ndash There are many approaches and little consensus as to best wayndash Many models (Jelinski-Moranda Littlewood-Verrall Musa-Okumoto etc) [1]ndash Many tools (over 200+ tools since 1970s have been built) [2]
bull Predictability of software reliability is of great concern because it is a major contributor to unreliability[2]
bull Software Reliability is the probability of error-free operation of a computer program in a specified environment for a specified time [1]
bull Need basis for setting reliability figures based on previous systems and iteratively refine those figures in the future
bull NOT A REPLACEMENT FOR TESTING AND VERIFICATION
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
20
Tools for Modeling and Analysis
bull Universal Systems Language (USL)bull Unified Modeling Language (UML)bull Systems Modeling Language (SysML)bull MATLABSimulinkbull Telelogic Rhapsodybull MathCadbull Colored Petri Netsbull Rate Monotonic Analysis (RMA)bull STATEMATE (Used by Airbus)bull SCADEbull OPNETbull Embedded System Modeling Language (ESML)bull Component Synthesis using Model-Integrated Computing (CoSMIC)bull Architectural Analysis and Design Language (AADL)bull At least 200+ more packages since the 70rsquosbull Certified tools needs to converge to an accepted standard
modelinganalysis method for complex system reliability
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
21
Modification to Acquisition Model
Requirements Establishment
High Level Design
Detailed Specifications
Implementation Coding
Verification
Development TestingA
rchi
tect
ural
Mod
el amp
A
naly
sis
Propose standard modeling methodology to be applied at different phases of development to enhance requirements development reliability allocation reliability
measurement and testing (DISCLAIMER DOES NOT REPLACE TESTING)
Reliabilityallocated Reliability
measured
Operational Testing amp Validation
Deployed System
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
22
Systems Reliability Standard Establishment
bull Establish a working group to define this standardndash Need a technical society to lead the charge on this
bull Collaborate with industry academia military and societiesndash Focus on development of a reliability standard with AWR safety in mindndash Draw upon the experiences to feed into this standard
bull Study existing and previous complex systemsndash Shuttle Space Station missile systems nuclear submarine and ship
systems nuclear control systems military and commercial jet systems ndash Obtain software reliability information from given existing and previous
systemsndash Build database which would serve as basis for future reliability
bull Research prior efforts in complex systems analysis
bull Establish consensus based modeling and analysis method
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
23
BACKUP SLIDES
BACKUP SLIDES
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
24
PEO Aviation System Safety Management Decision Authority Matrix
Severity(Most Credible)
FrequentP gt 1E-3
A
Probable 1D-4 lt P lt= 1E-3
B
Occasional1E-5 lt P lt= 1E-4
C
Remote1E-6 lt P lt= 1E-5
D
Improbable1E-7 lt P lt= 1E-6
E
Catastrophic1
Critical 2
Marginal 3
Negligible4
Army Acquisition
PEO Aviation
ProgramManagement
HazardCategory
Description
1 Catastrophic Death or permanent total disability system loss
2 Critical Severe injury or minor occupational illness (no permanent effect) minor system or environmental damage
3 Marginal Minor injury or minor occupational illness (no permanent effect) minor system or environmental damage
4 Negligible Less than minor injury or occupational illness (no lost workdays) or less than minor environmental damage
RiskLevel
Description Probability (Frequency) (per 100000 flight hours)
A Frequent gt 100 (P gt 1E-3)
B Probable lt=100 and gt10 (1E-4 lt P lt= 1E-3)
C Occasional lt= 10 and gt1 (1E-5 lt P lt= 1E-4)
D Remote lt=1 and gt01 (1E-6 lt P lt= 1E-5)
E Improbable lt=01 and gt001 (1E-7 lt P lt= 1E-6)
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
25
Reliability Defined
bull Software Reliability - the probability that a given piece of software will execute without failure in a given environment for a given time [40] ndash Often debated as to how to measure
bull Hardware Reliability - the probability that a hardware component fails over time ndash Well defined and established
bull System Reliability - the probability of success or the probability that the system will perform its intended function under specified design limits [over a given amount of time] [39] ndash A combination of software and hardware reliability
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
26
Hardware vs Software Reliability
Hardware Reliability Software ReliabilityFailure rate has a bathtub curve The burn-in state is similar to the software debugging state
Without considering program evolution failure rate is statistically non-increasing
Material deterioration can cause failures even though the system is not used
Failures never occur if the software is not used
Failure data are fitted to some distributions The selection of the underlying distribution is based on the analysis of failure data and experiences Emphasis is placed on analyzing failure data
Most models are analytically derived from assumptions Emphasis is on developing the model the interpretation of the model assumptions and the physical meaning of the parameters
Failures are caused by material deterioration design errors misuse and environment
Failures are caused by incorrect logic incorrect statements or incorrect input data
Can be improved by better design better material applying redundancy and accelerated life cycle testing
Can be improved by increasing testing effort and correcting discovered faults Reliability tends to change continuously during testing due to the addition of problems in new code or the removal of problems by debugging errors
Hardware repairs restore the original condition Software repairs establish a new piece of software
Hardware failures are usually preceded by warnings Software failures are rarely preceded by warnings
Hardware components can be standardized Software components have rarely been standardized
Hardware can usually be tested exhaustively Software essentially requires infinite testing for completeness
Reference [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
27
Acronym ListACRONYM DEFINITIONAADL Architectural Analysis and Design LanguageAC Advisory Circular (FAA)ACM Association of Computing MachineryAED Aviation Engineering Directorate (AMRDEC)AFTD Aviation Flight Test Directorate (US Army)AGC Apollo Guidance ComputerAHS American Helicopter SocietyAIAA American Institute of Aeronautics and Astronautics (Inc)AMCOM Aviation and Missile Command (US Army)AMRDEC Aviation and Missiles Research Development and Engineering Center (US Army)AR Army RegulationARINC Aeronautical Radio Inc ARP Aerospace Recommended PracticeASIF Avionics Software Integration FacilityATAM Architecture Tradeoff Analysis MethodATM Air Traffic ManagementAWR Airworthiness ReleaseCAAS Common Avionics Architecture SystemCH-47 Cargo Helicopter ChinookCMM Capability Maturity ModelCMMI Capability Maturity Model IndexCMU Carnegie Mellon UniversityCNS Communications Navigation SurveillanceCoSMIC Component Synthesis using Model-Integrated ComputingCPS Cyber-Physical SystemCRC Chemical Rubber Company (ie CRC Press)DFBW Digital Fly-By-WireDoD Department of DefenseE3 Electrical and Electromagnetic EffectsESML Embedded System Modeling LanguageFAA Federal Aviation AdministrationFCS Future Combat SystemsFHA Functional Hazard AssessmentFMEA Failure Modes Effects Analysis
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
28
Acronym List (concluded)
ACRONYM DEFINITIONGPWS Ground Proximity Warning SystemIBM International Business MachinesIEC International Engineering ConsortiumIL Instrumentation Lab (now Draper Laboratory)IMA Integrated Modular AvionicsINCOSE International Council On Systems EngineeringISO International Organization for StandardizationISS International Space StationKAL Korean AirlinesMISRA Motor Industry Standard Software Reliability AssociationMIT Massachusetts Institute of TechnologyNASA National Aeronautics and Space Administration (USA)PDR Preliminary Design ReviewPEO Program Element OfficePNAS Proceedings of the National Academy of SciencesRAQ Rotorcraft and Aircraft QualificationRMA Rate Monotonic AnalysisRTC Redstone Test Center (US Army) RTTC Redstone Technical Test Center (US Army)RTCA Radio Technical Commission for AeronauticsSAE Society of Automotive EngineersSED Software Engineering Directorate (AMRDEC)SEES Software Engineering Evaluation SystemSEI Software Engineering Institute (CMU)SIL System Integration LaboratorySSA System Safety AssessmentSTS Space Transportation SystemSysML Systems Modeling LanguageTMR Triple Modular RedundantTRL Technical Readiness LevelUAS Unmanned Aircraft SystemUH-60 Utility Helicopter BlackhawkUML Unified Modeling LanguageUS United StatesUSL Universal Systems Language
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
29
References
bull [1] Israel Koren and Mani Krishna ldquoFault-Tolerant Systemsrdquo Morgan Kaufmann 2007bull [2] Jianto Pan ldquoSoftware Reliabilityrdquo Carnegie Mellon University Spring 1999bull [3] Nachum Dershowitz httpwwwcstauacil~nachumdhorrorhtmlbull [4] httpwwwair-attackcombull [5] David A Mindell ldquoDigital Apollo Human and Machine in Spaceflightrdquo The MIT Press 2008bull [6] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Flight Software Airworthiness SED-SES-PMHFSA 001 December 2003bull [7] Software Engineering Directorate Software Engineering Evaluation System (SEES) ldquoProgram Manager Handbook for
Software Safety SED-SES-PMHSSA 001 February 2006bull [8] AMCOM Regulation 385-17 Software System Safety Policy 15 March 2008bull [9] NASA Software Safety Guidebook NASA-GB-871913 31 March 2004bull [10] Margaret Hamilton amp William Hackler ldquoUniversal Systems Language Lessons Learned from Apollordquo IEEE Computer
Society 2008bull [11] Margaret Hamilton ldquoFull Life Cycle Systems Engineering and Software Development Environment Development Before
The Fact In Actionrdquo httpwwwhtiuscomArticlesFull_Life_Cyclehtmbull [12] Peter Feiler David Gluch John Hudak ldquo The Architecture Analysis amp Design Language (AADL) An Introductionrdquo
CMUSEI-2006-TN-011 February 2006bull [13] Peter Feiler John Hudak ldquoDeveloping AADL Models for Control Systems A Practitionerrsquos Guiderdquo CMUSEI-2007-TR-
014 July 2007bull [14] Bruce Lewis ldquoUsing the Architecture Analysis and Design Language for System Verification and Validationrdquo SEI
Presentation 2006bull [15] Feiler Gluch Hudak Lewis ldquoEmbedded System Architecture Analysis Using SAE AADLrdquo CMUSEI-2004-TN-004 June
2004bull [16] Charles Pecheur Stacy Nelson ldquoVampV of Advanced Systems at NASArdquo NASACR-2002-211402 April 2002bull [17] Systems Integration Requirements Task Group ldquoARP 4754 Certification Considerations for Highly-Integrated or
Complex Aircraft Systemsrdquo SAE Aerospace 10 April 1996bull [18] SAE ldquoARP 4761 Guidelines and Methods for Conducting the Safety Assessment Process on Civil Airborne System and
Equipmentrdquo December 1996bull [19] Aeronautical Radio Inc (ARINC) ldquoARINC Specification 653P1-2 Avionics Application Software Standard Interface Part 1
ndash Required Servicesrdquo 7 March 2006
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
30
References (Continued)
bull [20] Department of Defense ldquoMIL-STD-882D Standard Practice for System Safetyrdquo 19 January 1993bull [21] RTCA Incorporated ldquoDO-178 Software Considerations in Airborne Systems and Equipment Certificationrdquo 1
December 1992bull [22] RTCA Incorporated ldquoDO-254 Design Assurance Guidance for Airborne Electronic Hardwarerdquo 19 April 2000bull [23] US Army ldquoAeronautical Design Standard Handbook Rotorcraft and Aircraft Qualification (RAQ) Handbookrdquo 21
October 1996bull [24] Cary R Spitzer (Editor) ldquoAvionics Elements Software and Functionsrdquo CRC Press 2007bull [25] US Army ldquoArmy Regulation 70-62 Airworthiness Qualification of Aircraft Systemsrdquo 21 May 2007bull [26] US Army ldquoArmy Regulation 95-1 Aviation Flight Regulationsrdquo 3 February 2006bull [27] ldquoUsing the Architecture Tradeoff Analysis Method (ATAM) to Evaluate the Software Architecture for a Product Line
of Avionics Systems A Case Studyrdquo Barbacci Clements Lattanze Northrop Wood July 2003 CMUSEI-2003-TN-012bull [28] ldquoAll in the Family CAAS amp AADLrdquo Peter Feiler August 2008 CMUSEI-2008-SR-021bull [29] ldquoCMMI Guidelines for Process Integration and Product Improvementrdquo Chrissis Konrad Shrum Pearson Education
2007bull [30] ldquoModel Driven Performance Analysis for Avionics Systemsrdquo Brendan OrsquoConnell Draper Laboratory January 2006bull [31] John F Hanaway Robert W Moorehead ldquoSpace Shuttle Avionics Systemsrdquo NASA SP-504 1989bull [32] Lui Sha ldquoThe Complexity Challenge in Modern Avionics Softwarerdquo August 14 2006bull [33] ldquoIncidents Prompt New Scrutiny of Airplane Software Glitchesrdquo 30 May 2006 Wall Street Journalbull [34] Eyal Ophir Clifford Nass and Anthony Wagner ldquo Cognitive Control in Media Multitaskersrdquo PNAS 20 July 2009bull [35] ldquoAdvisory Circular AC 251309-1A System Design and Analysisrdquo Federal Aviation Administration 21 June 1988bull [36] Program Element Office Policy Memorandum 08-03bull [38] httpwwwnsfgovpubs2008nsf08611nsf08611htm National Science Foundation webpage on Cyber-Physical
Systemsbull [39] Hoang Pham ldquoSoftware Reliabilityrdquo Springer 2000bull [40] Paul Rook editor ldquoSoftware Reliability Handbookrdquo Elsevier Science Publishers LTD 1990
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-
31
References (Concluded)
bull [41]httpwwwnavairnavymilv22indexcfmfuseaction=newsdetailampid=128bull [42]httpmars8jplnasagovmsp98newsmco990930htmlbull [43]John Garmen ldquoThe Bug Heard Around the Worldrdquo ACM SIGSOFT October 1981bull [44]httpmarsprogramjplnasagovMPFnewspiompfstatuspf970715html ldquoMars Pathfinder Mission Statusrdquo July 15
1997bull [45] Nancy Leveson ldquoSafeware System Safety and Computersrdquo Addison-Wesley Publishing Company 1995bull [46] httpwwwelectronicaviationcomaircraftJAS-39_Gripen810bull [47] Brandon Hillhttpwwwfreerepubliccomfocusf-news1791574posts Lockheeds F-22 Raptor Gets Zapped by
International Date Line DailyTech LLC February 26 2007 bull [48] httpwwwmilitarycomnewsarticlehuman-error-cited-in-most-uav-crasheshtmlbull [49] Daniel Michaels and Andy Pasztor ldquoIncidents Prompt New Scrutiny Of Airplane Software Glitches As Programs
Grow Complex Bugs Are Hard to Detect A Jets Roller-Coaster Ride Teaching Pilots to Get Controlrdquo Wall-Street Journal May 30 2006
- Qualification and Reliability of Complex Electronic Rotorcraft Systemsby Alex Boydston amp Dr William Lewis AMRDECfor AFRL Safe and Secure Symposium 15-17 June 2010
- Agenda
- Objective
- Defense Acquisition Approach to Systems Development and Test
- US Army Airworthiness
- AED and Qualification
- Evolution of Helicopter Systems
- Present Approach to Testing
- Development Challenges
- Complexity Issues
- Complexity Issues (continued)
- Reliability vs Complexity amp Cost vs Complexity
- A Few Examples of Complex Systems
- Some Complex System Failures
- Lessons Learned from Failures
- Some Current Guidelines
- Certification Assessment Considerations
- Definition of Complexity and Reliability is Needed
- Analytical Models and Reliability
- Tools for Modeling and Analysis
- Modification to Acquisition Model
- Systems Reliability Standard Establishment
- BACKUP SLIDES
- PEO Aviation System Safety Management Decision Authority Matrix
- Reliability Defined
- Hardware vs Software Reliability
- Acronym List
- Acronym List (concluded)
- References
- References (Continued)
- References (Concluded)
-