from virtual system integration to incremental lifecycle ... · distribution statement ... power,...
Post on 08-Aug-2018
223 Views
Preview:
TRANSCRIPT
© 2015 Carnegie Mellon University
Software Solutions Conference 2015November 16–18, 2015
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
From Virtual System Integration to Incremental Lifecycle AssurancePeter H. Feiler
2From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Copyright 2015 Carnegie Mellon University
This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.
[Distribution Statement A] This material has been approved for public release and unlimited distribution. Please see Copyright notice for non-US Government use and distribution.
This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu.
Carnegie Mellon® is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
DM-0003034
3From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Challenges in Software Reliant Systems
Four Pillar Improvement Strategy
Virtual System Integration
Incremental Lifecycle Assurance
Agenda
4From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
We Rely on Software for Safe Aircraft Operation
Embedded software systems introduce a new
class of problems not addressed by traditional system safety analysis
5From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Software Problems not just in Aircraft
How do you upgrade washing machine software?
6From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
High Fault Leakage Drives Major Increase in System Cost
5x
SoftwareArchitectural
Design
SystemDesign
ComponentSoftwareDesign
CodeDevelopment
UnitTest
SystemTest
Integration Test
Acceptance Test
RequirementsEngineering
300‐1000x
Where faults are introducedWhere faults are foundThe estimated nominal cost for fault removal
20.5%
1x
20%, 16%
10%, 50.5%
0%, 9% 80x
70%, 3.5% 20x
Sources:
NIST Planning report 02-3, The Economic Impacts of Inadequate Infrastructure for Software Testing, May 2002.
D. Galin, Software Quality Assurance: From Theory to Implementation, Pearson/Addison-Wesley (2004)
B.W. Boehm, Software Engineering Economics, Prentice Hall (1981)
70% Requirements & system interaction errors 80% late error
discovery at high repair cost
80% late error discovery at high
repair cost
80% late error discovery at high
rework cost
Aircraft industry has reached limits of affordability due to exponential growth in SW size and complexity.
Major cost savings through rework avoidance by early discovery and correction
A $10k architecture phase correction saves $3M
Software as % of total system cost1997: 45% 2010: 66% 2024: 88%
Post-unit test software rework cost 50% of total system cost and growing
7From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Mismatched Assumptions in System Interactions
System Engineer Control Engineer
SystemUnder Control
ControlSystem
Physical Plant CharacteristicsLag, proximity
Operator ErrorAutomation & human actions
Syst
em U
ser/E
nviro
nmen
t
HazardsImpact of
system failures Application D
eveloper
ComputePlatform
RuntimeArchitecture
ApplicationSoftware
Embedded SW System Engineer
Data Stream Characteristics
Latency jitter affects control behavior
Potential event loss
Measurement Units, value range Boolean/Integer abstraction
Air Canada, Ariane, 7500 Boolean variable architecture
Concurrency Communication
ITunes crashes on dual-cores
Distribution & RedundancyVirtualization, load balancing,
mode confusion
HardwareEngineer
Embedded software system as major source of hazards
Why do system level failures still occur despite fault tolerance techniques being deployed in systems?
8From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Model-based Engineering Pitfalls
The system
System models
System implementation
Inconsistency between independently developed
analytical models
Confidence that model reflects implementation
This aircraft industry experience has led to the System Architecture Virtual Integration (SAVI) initiative
9From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Awareness of Requirement QualityTextual requirement quality statistics
• Current requirement engineering practice relies on stakeholders traceability and document reviews resulting in high rate of requirement change
Managed awareness of requirement uncertainty reduces requirement changes by 50%
• 80% of requirement changes from development team
• Expert assessment of change uncertainty• Focus on high uncertainty and high importance
areas• Engineer for inherent variability Rolls Royce Study
NIST Study
10From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Challenges in Software Reliant Systems
Four Pillar Improvement Strategy
Virtual System Integration
Incremental Lifecycle Assurance
Agenda
11From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Assurance & Qualification Improvement Strategy
2010 SEI Study for AMRDEC Aviation Engineering Directorate
Assurance: Sufficient evidence that a system implementation meets system requirements
Architecture-centric Virtual System
Integration
Model RepositoryArchitecture
Model
Component Models
System Implementation
Resource, Timing &
Performance Analysis
Reliability, Safety,
Security Analysis
Operational & failure modes
Static Analysis & Compositional
Verification
System configuration
Early Problem Discovery through Virtual System Integration & Analysis
Incremental Assurance Plans & Cases
throughout Life Cycle
Mission Requirements
FunctionBehavior
Performance
Survivability Requirements
ReliabilitySafety
Security
Architecture-led Requirement Specification
Improved Assurance through Better Requirements & Automated Verification
12From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Assure the System
Improved Cost, Time and Quality
80% Post Unit Test Discovery
70% Defect Introduction Reduced Cost and Time through Early Discovery
Improved Quality through Better Requirements & Evidence
13From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Challenges in Software Reliant Systems
Four Pillar Improvement Strategy
Virtual System Integration
Incremental Lifecycle Assurance
Agenda
14From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
SAE Architecture Analysis & Design Language (AADL) to the Rescue
Distributed Computer Platform
Physical system
Command & Control
Deployed on
Physical interface
AADL Addresses Increasing Interaction Complexity and Mismatched Assumptions
Task & Communication Architecture
SW Design Architecture
15From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Analysis of Virtually Integrated Software Systems
Security• Intrusion• Integrity• Confidentiality
Safety & Reliability• MTBF• FMEA• Hazard analysis
Real-timePerformance• Execution time/Deadline • Deadlock/ starvation• Latency
ResourceConsumption• Bandwidth• CPU time• Power consumption
• Data precision/accuracy• Temporal correctness• Confidence
Data Quality
Architecture Model
Single Annotated Architecture Model Addresses Impact Across Operational Quality Attributes
Auto-generated analytical models
Change of Encryption from 128 bit to 256 bit
Higher CPU demandIncreased latency
Affects temporalcorrectness
Potential new hazard
16From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
SAE AADL Standard & AADL Workbench: Research Transition Platform
2004 2016
Army and other Government Shadow Projects
CommonAvionics
ArchitectureSystem
ApacheBlock IIIATAM CH47F
Health Monitor
JPLMission Data
System
DARPAMetaHACME
AADLError Model
US & European Research Initiatives
EuropeanCommissionSLIM/FIACRE
DARPAMETA
DARPAHACMSSecurity
Other Standards and Regulatory Guidance
OMG MARTE
EmbeddedSystems
ARINC653Partitions
Regulatory GuidanceNRC, FDA, UL
Avionics Network Standards
System Safety Practice Standards
System Architecture Virtual Integration (SAVI) Software & Systems Engineering
AADLSoftware & System
Co-engineeringRequirements
AssuranceMulti-team
Safety
JMR TD: ACVIP Shadow Projects
Future Vertical Lift
Virtual System Integration System Assurance
Architecture-centric Acquisition
Towards an Architecture-Centric Virtual Integration Practice (ACVIP)
17Virtual Integration and Assurance: Impact on FVL July 14, 2015© 2015 Carnegie Mellon University
Finding Problems Early
Issue: Contractor could not assess integration risk early enough.
Action: 6 Week Virtual Integration identified 20 major issues.
Result: Adjusted CDR Schedule to remediate.
• Prevented 12 month delay in a 2 year project.
The current method would not have identified the issues until 3 months before delivery
Image: http://www army mil/
System Architecture Virtual Integration (SAVI) 2008-Proof of concept with AADL led to ten year commitment
SAVI ROI Study (2009/10) $2B savings on $10B aircraft through 33% early detection
International Commercial Aircraft Industry Consortium
2014/15 Virtual Integration Shadow led to early discovery of 85+ potential integration issues
Led to acceleration of adoption by JMR contractors andinclusion in RFP for FY16/17 projects
Architecture-centric Virtual Integration Practice (ACVIP)
18From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Multi-tier system & software architecture (in AADL) Incremental end-to-end verification of system properties
LRU/IMA System: (Tier 2)Hardware platform, software partitionsPower, MIPS, RAM capacity & budgetsEnd-to-end flow latency
Subcontracted software subsystem: (Tier 3)Tasks, periods, execution timeSoftware allocation, schedulabilityGenerated executables
OEM & Subcontractor:Subsystem proposal validationFunctional integration consistencyData bus protocol mappings
Incremental Multi-Tier Assurance in SAVI
Proof of Concept Demonstration and Transition by Aerospace industry initiative• Architecture-centric model-based software and system engineering• Architecture-centric model-based acquisition and development process• Multi notation, multi team model repository & standardized model interchange
Aircraft: (Tier 0)
Repeated Virtual Integration Analyses:Power/weightMIPS/RAM, SchedulingEnd-to-end latencyNetwork bandwidth
System & SW Engineering:Mechatronics: Actuator & WingsSafety Analysis (FHA, FMEA)Reliability Analysis (MTTF)
Aircraft system: (Tier 1)Engine, Landing Gear, Cockpit, …Weight, Electrical, Fuel, Hydraulics,…
19From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Automated FMEA ExperienceFailure Modes and Effects Analyses are rigorous and comprehensive reliability and safety design evaluations
• Required by industry standards and Government policies• When performed manually are usually done once due to cost and schedule• If automated allows for
• multiple iterations from conceptual to detailed design• Tradeoff studies and evaluation of alternatives• Early identification of potential problems
Largest analysis of satellite to date consists of 26,000 failure modes• Includes detailed model of satellite bus• 20 states perform failure mode• Longest failure mode sequences have 25 transitions (i.e., 25 effects)
19
Myron Hecht, Aerospace Corp.Safety Analysis for JPL, member of DO-178C committee
20From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Challenges in Software Reliant Systems
Four Pillar Improvement Strategy
Virtual System Integration
Incremental Lifecycle Assurance
Agenda
21From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Incremental Lifecycle Assurance Objectives
Measurably improve critical system assurance through
• Better coverage and managed uncertainty• Incremental analytical verification throughout lifecycle• Focus on high payoff areas
22From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Requirements & Architecture Design Constraints
We have effectively specified a system partial architecture
Textual Requirements for a Patient Therapy System
Importance of understanding system boundary
U Minnesota Study
Same Requirements Mapped to an Architecture Model
23From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
VAVAVA
Compositional Verification
RS RS RS
Design & ReqRefinement
VAVAVA
Compositional Verification
Compositional verification and partitions to limit assurance impact
RS
RS RS RS
Design & ReqRefinement
RequirementCoverage
Incremental assurance through virtual system integration for early discovery
Early Discovery leads to Rework Reduction
Priority focused architecture design exploration for high payoff
Timing (H)Performance (M)Safety (H)Security (L)Reliability (L)Modifiability (L)Portability (M)Configurability (M)
C
C
C
Three Dimensions of Incremental Assurance
24From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Three Dimensions of Requirement Coverage
GuaranteesAssumptions
Implementation constraints
Invariants
Exceptionalconditions
System interactions, state, behavior Design & operational quality attributes
System Under Control
Behavior
Actuator Sensor
State
Control System
Behavior
Output InputStateValue errors
Timing errors
Rate errors Concurrency errors
Replication errors
Sequence errors
Omission errors
Commission errors
Authentication errors
Authorization errors
Fault Propagation Ontology
Fault impact & contributors
25From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Stakeholder Goals
Tier 0
Tier 1
Tier 2 Model+2’
Ver Plan
Ver Plan
Req+2Ver Plan
Req+1
Req
Model+1
Model
Automated Incremental Assurance Workbench
Abstraction
Level
Low LevelClose to Implementation
High Abstraction
Model+2
Assurance Case
Identify Assurance Hotspots throughout Lifecycle
26From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Secure Mathematically-Assured Composition of Control Models
Technical Approach• Develop a complete, formal architecture model for UAVs
that provides robustness against cyber attack
• Develop compositional verification tools driven from the architecture model for combining formal evidence from multiple sources, components, and subsystems
• Develop synthesis tools to generate flight software for UAVs directly from the architecture model, verified components, and verified operation system
Accomplishments• Created AADL model of vehicle hardware & software
architecture• Identified system-level requirements to be verified
based on input from Red Team evaluations• Developed Resolute analysis tool for capturing and
evaluating assurance case arguments linked to AADL model
• Developed example assurance cases for two security requirements
• Developed synthesis tool for auto-generation of configuration data and glue code for OS and platform hardware
ARCHITECTURE-CENTRIC PROOF
TA4 – Research Integration and Formal Methods Workbench
Rockwell Collins and University of Minnesota
Key ProblemMany vulnerabilities occur at component interfaces. How can we use formal methods to detect these vulnerabilities and build provably secure systems?
Contract-based Compositional Verification
16 months into the project
Draper Labs could not hack into the system in 6 weeks
Had access to source code
27From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Challenges in Software Reliant Systems
Four Pillar Improvement Strategy
Virtual System Integration
Incremental Lifecycle Assurance
Agenda
28From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Benefits of Virtual System Integration & Incremental Lifecycle AssuranceReduce risks
• Understand system wide impact early• Verify assumptions across system
Increase confidence• Verified models to complement integration tests• System design evolved from verified models
Reduce cost• Fewer system integration problems• Less assurance related rework
29From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
ReferencesAADL Website www.aadl.info and AADL Wiki www.aadl.info/wikiBlog entries and podcasts on AADL at www.sei.cmu.eduAADL Book in SEI Series of Addison-Wesley http://www.informit.com/store/product.aspx?isbn=0321888944On AADL and Model-based Engineeringhttp://www.sei.cmu.edu/library/assets/ResearchandTechnology_AADLandMBE.pdfOn an architecture-centric virtual integration practice and SAVIhttp://www.sei.cmu.edu/architecture/research/model-based-engineering/virtual_system_integration.cfmOn an a four pillar improvement strategy for software system verification and qualificationhttp://blog.sei.cmu.edu/post.cfm/improving-safety-critical-systems-with-a-reliability-validation-improvement-frameworkWebinars on system verification https://www.csiac.org/event/architecture-centric-virtual-integration-strategy-safety-critical-system-verification and on architecture trade studies with AADL https://www.webcaster4.com/Webcast/Page/139/5357
30From Virtual System Integration to Incremental Lifecycle AssuranceNov 18, 2015© 2015 Carnegie Mellon University
Distribution Statement A: Approved for Public Release; Distribution is Unlimited
Contact Information
Peter H. FeilerPrincipal ResearcherRTSSTelephone: +1 412-268-7790Email: phf@sei.cmu.edu
U.S. MailSoftware Engineering InstituteCustomer Relations4500 Fifth AvenuePittsburgh, PA 15213-2612USA
WebWiki.sei.cmu.edu/aadlwww.aadl.info
Customer RelationsEmail: info@sei.cmu.eduSEI Phone: +1 412-268-5800SEI Fax: +1 412-268-6257
top related