ilities tradespace workshop summary barry boehm, supannika koolmanojwong usc-csse arr 20 march 14,...
TRANSCRIPT
ilities Tradespace Workshop Summary
Barry Boehm, Supannika Koolmanojwong USC-CSSE ARR 20 March 14, 2013
03-12-2013 1
Importance of ility TradeoffsMajor source of DoD system overruns
• System ilities have systemwide impact– System elements generally just have local impact
• ilities often exhibit asymptotic behavior– Watch out for the knee of the curve
• Best architecture is a discontinuous function of ility level– “Build it quickly, tune or fix it later” highly risky– Large system example below
03-12-2013 2
Importance of Cyber-Physical Systems Major gap in tradespace analysis capabilities
• Current ERS, DARPA tradespace research focused on physical system tradeoffs– Range, payload, size, weight, lethality, power and fuel
consumption, communications bandwidth, etc.– Some focus on physical modularity, composability
• Current cyber tradespace research focused on software, computing, human factors tradeoffs– security, safety, interoperability, usability, flexibility,
adaptability, dependability, response time, throughput, etc.
• Gaps in capabilities for co-design of hardware and software, integration of tradespace analyses
03-12-2013 3
iTAP Startup Results• Top-level ility hierarchy view
– Relation to JCIDS combat command user view– Draft survey to compare with acquirer, developer, supporter views
• Affordability means-ends framework view– Quantification via COCOMO, COSYSMO, CORADMO models
• Architecture-strategy synergies and conflicts views– Conflicts with other capabilities
• Change-Oriented Views: Incremental Commitment– The Cones of Uncertainty– MIT Epoch-Era Approach– Agile SE Schedule Acceleration Model
• Domain-Oriented Views– Ground: Wayne State, Georgia Tech– Sea: NPS; Air: AFIT; Space: MIT, Stevens, USC, U. Virginia
03-12-2013 4
SERC Value-Based ilities HierarchyBased on ISO/IEC 9126, 25030; JCIDS; previous SERC research
• Individual ilities– Quality of Service: Performance, Accuracy, Usability, Scalability, Versatility– Resource Utilization: Cost, Duration, Personnel, Scarce Quantities (size,
weight, energy, …)– Protection: Safety, Security, Privacy– Robustness: Reliability, Availablilty, Maintainability– Flexibility: Modifiability, Tailorability/Extendability, Adaptability– Composability: Interoperability/Portability, Openness/Standards Compliance,
Service-Orientation
• Composite ilities– Comprehensiveness/Suitability: all of the above– Dependability: Quality of Service, Protection, Robustness– Resilience: Protection, Robustness, Flexibility– Affordability: Quality of Service, Resource Utilization
03-12-2013 5
Prioritized JCIDS ilitiesUser View by Combatant Commands: Top priority first
• Intelligence, Surveillance, and Reconnaissance– Comprehensive Persistent Survivable Integrated Timely Credible Adaptable
Innovative
• Command and Control (note emphasis on Usability aspects)– Interoperability Understanding Timeliness Accessibility Simplicity Completeness
Agility Accuracy Relevance Robustness Operational Trust
• Logistics: Supply– Responsiveness Sustainability Flexibility Survivability Attainability Economy Simplicity
• Logistics: Maintenance– Sustainability Responsiveness Attainability Flexibility Economy Survivability Simplicity
• Net-Centric: Information Transport– Accessible Capacity Accurate Timely Throughput Expeditionary Latency
03-12-2013 6
703-12-2013
Legacy System Repurposing
Eliminate Tasks
Eliminate Scrap, Rework
Staffing, Incentivizing, Teambuilding
Kaizen (continuous improvement)
Work and Oversight StreamliningCollaboration Technology
Early Risk and Defect Elimination
Modularity Around Sources of ChangeIncremental, Evolutionary Development
Risk-Based Prototyping
Satisficing vs. Optimizing Performance
Value-Based Capability Prioritization
Composable Components,Services, COTS
Affordability Improvements and Tradeoffs
Get the Best from People
Make Tasks More Efficient
Simplify Products (KISS)
Reuse Components
Facilities, Support Services
Tools and Automation
Lean and Agile Methods
Evidence-Based Decision Gates
Domain Engineering and Architecture
Task AutomationModel-Based Product Generation
Value-Based, Agile Process Maturity
Means-Ends Framework: Affordability
Reduce Operations, Support Costs
Streamline Supply ChainDesign for Maintainability, EvolvabilityAutomate Operations Elements
Anticipate, Prepare for ChangeValue- and Architecture-Based Tradeoffs and Balancing
03-12-2013 8
USC: COCOMO II-Based Tradeoff AnalysisBetter, Cheaper, Faster: Pick Any Two?
Slider-based equalizer version being developed
0
1
2
3
4
5
6
7
8
9
0 10 20 30 40 50
Development Time (Months)
Co
st
($M
)
(VL, 1)
(L, 10)
(N, 300)
(H, 10K)
(VH, 300K)
-- Cost/Schedule/RELY:
“pick any two” points
(RELY, MTBF (hours))
•For 100-KSLOC set of features•Can “pick all three” with 77-KSLOC set of features
iTAP Startup Results• Top-level ility hierarchy view– Relation to JCIDS combat command user view– Draft survey to compare with acquirer, developer, supporter views
• Affordability means-ends framework view– Quantification via COCOMO, COSYSMO, CORADMO models
• Architecture-strategy synergies and conflicts views– Conflicts with other capabilities
• Change-Oriented Views: Incremental Commitment– The Cones of Uncertainty– MIT Epoch-Era Approach– Agile SE Schedule Acceleration Model
• Domain-Oriented Views– Ground: Wayne State, Georgia Tech– Sea: NPS; Air: AFIT; Space: MIT, Stevens, USC, U. Virginia
03-12-2013 9
Architecture-Based Attribute Trades: Flexibility Example (RT-18a)
Flexibility Arch. Strategy Synergies Conflicts
High module cohesion; Low module coupling
InteroperabilityReliability
High Performance via Tight coupling
Service-oriented architecture Composability, Usability, Testability High Performance via Tight coupling
Autonomous adaptive systems Affordability via task automation; Response time
Excess autonomy reduces human Controllability
Modularization around sources of change
Interoperability, Usability, Reliability, Availability
Extra time on critical path of Rapid Fielding
Multi-layered architecture Reliability, Availability Lower Performance due to layer traversal overhead
Many built-in options, entry points Functionality, Accessibility Reduced Usability via options proliferation; harder to Secure
User programmability Usability, Mission Effectiveness Full programmability causes Reliability, Safety, Security risks
Spare/expandable capacity Performance, Reliability Added cost
Product line architecture, reusable components
Cost, Schedule, Reliability Some loss of performance vs. optimized stovepipes
03-12-2013 10
11
MIT: ilities in Tradespace ExplorationBased on Lean Aerospace, DARPA research
For this plot, Ĉ=C∞
More changeable(ie including flexible, adaptable, scalable
and modifiable)
Colored by outdegree
Enabling Construct: Tradespace Networks Changeability
Survivability
Value Robustness
Enabling Construct: Epochs and Eras
Set of Metrics
03-12-2013
WSU: Versatility Factors and Physical OrganizationComponents that Can be in Different Positions or Orientations
Isolated or Separated Compartments
Running Gear
Chassis
Turret
Sight Weapon
suspension
drive
drivedriveMass & Structure Properties•Mass •Angular moments•Imbalances*•Load bearing wall strength•Deck surface area•Interior volumes**•Interior surface areas**
*Angular moments of the CG about axes of rotation** By crew station and compartment
03-12-2013 12
Workshop objectives and approach • Workshop objectives
– Identify interested collaborators and data– Identify user needs for better reasoning about ility tradeoffs
and affordability– Identify improved approaches for cyber-physical system co-
design
• Workshop approach– Ask participants about their current and likely future
challenges and research needs• For ilities and their tradeoffs• For cyber-physical co-design
– Prioritize research with respect to strength of need, difficulty
03-12-2013 13
Workshop approach
• Ask participants about their current and likely future challenges and research needs– For -ilities and their tradeoffs– For cyber-physical co-design
• Prioritize research with respect to strength of need, difficulty
Participants• Shawn Rahmani• Gary Hafen• Winsor Brown• J.D. Baker• Ed Colbert• Thammanoon K.• Peter Suk• Rachchabhorn W.• Sue K.
• Qi Li• Lori Vaughan• Qing Wang• Jing Du• Liming Zhu• Da Yang• Lee Osterweil• Barry Boehm
Current and likely future challenges and research needs for -ilities and their tradeoffs• Simulation model to address performance, usability model, comparing -ilities• Define hierarchy, top factors• Quantification of value of each –ilities, matrices • Contribution of each –ilities (H/M/L. option1/2) • Prototype for a short life vs requirements in terms of ilitlities• Scale the tradeoff, scale up? • Affordability (prioritizing issues, not only low cost)• Inter-dependency of -ilities • Pair-wise comparison tool (lockheed)• Issue of subjective tradeoff study • Modeling and trade study, matrices of architecture and –ilities• Healthcare group (right distribution of data vs performance, privacy control/security)• Usability vs design• Parametric diagram • Agile architecture development – tradeoff, architecture your asset & product• How to specify requirements in a measurable way• Security, accessibility, QoS • Scalability – esp. diseconomy of scale, e.g. mobile network• Reliability - 5 9s system in banking sector• Macro view & micro view; complimentary process modeling (agent, task, resources) optimize the resources & others to optimize the choices,
a tool that guide resource allocation (TWINS)• Statistical mechanic to understand the process & relationship to COCOMO• How to deploy, data /design selection; tradeoff guideline / tool for system implementation• Architectural style tradeoff; compare & normalize & confidence level; validation tool &Sensitivity analysis tool to improve confidence level• Fault tolerance; process view supporting tradeoff analysis• How many (functional ) requirements is enough; size, # per release , customized development vs maintenance/enhancement • unstated non-functional requirements• Estimation – over/under-estimation• Adaptability & flexibility – adapt to emergent behavior/requirements – not enough representative users • How to help decision makers – can we perform as stated in the proposal• Cyber vs physical; dual cone of uncertainty; • Guidance /model ; how to do the trade/ how to weigh, checklist ; human process part • Process for continuous delivery, how to prioritize requirements and assign them into each release based on revenue (considering dependency
& process adoption) • Using checklist – definition of Done(value, cost, quality)• Scenario generators• Product vs program characteristics - executability
Prioritization resultTools -ilities
•Models & Simulations (21)• Multilevel
•Architecting (17)•Prioritization (12)•Metrics/ Matrices (8)•Scenario generators (8)•Checklists (7)•Parametric diagrams (2)
•Affordability (14) •Timeliness (12)•Reliability / Fault Tolerance (11)
• Safety•Scalability (10)•Security / Privacy (10)•Performance (5)•Usability (4)•Accessibility (3)•Adaptability (3)•Flexibility (1)
Afternoon Agenda
• Exploring high score -ilities factors– Exploring degree of difficulties
• Exploring high score tools• Exploring cyber-physical co-design methods
Affordability• Definition = Effectiveness & Cost (INCOSE)• Fixed effectiveness & try to reduce cost• Define cost (Total cost of ownership, life cycle cost)• Cost = including operational & support cost? System cost
(e.g. cheap to build to maintain but not cheap to operate)
• What is the biggest gap that is not covered when doing affordability analysis? – Technical debt
• As long as you are aware of your debt, give you leeway• Technical debt identification tool
– HW SW integration
Timeliness• Define timeliness
– Meet deadline subject to definition of effectiveness (timebox – discard low priority items to meet schedule; agile rebaselining)
– Version control issues– Conflict in reusing items
Reliability / Fault Tolerance /Safety• Standard assurance issue• Define “safe”• Reliability = impact of defect (loss of life/properties/ $)• Dependent on other systems
– Cloud – no full control
• How can your system tolerate other systems?• Not equal (reliable <> fault tolerance <> safety)• Acceptable levels• A tool to provide analysis of balancing these factors • KPP – key performance parameter – no single number• Estimating certification cost & schedule• Level of testing
Scalability• Scalability of Product• Architecture evaluation tool
– Something like static source code analysis tool• Modeling tool• Network traffic & overhead• Communication mechanism• Co-dependent systems (with internet connection? )• Diversity
– Versions or platforms• Horizontal vs Vertical (scale out vs scale up)• Consistency among diversity/nodes• Timing • How to upgrade
Security / Privacy• Acceptable levels / level of assurance• Security vs complexity tradeoff (also with reliability, availability)• multiple independent level of security• Certification cost & schedule• Scope of certification (privacy)• Lifetime of data (dynamic data)• Anti-tamper • communication security• Scaling• How to show feasibility evidence• How to justify being assured (how the entire thing is developed)• Proof of correctness • Domain dependent
Prioritization resultTools -ilities
•Models & Simulations (21)• Multilevel
•Architecting (17)•Prioritization (12)•Metrics/ Matrices (8)•Scenario generators (8)•Checklists (7)•Parametric diagrams (2)
•Affordability (14) •Timeliness (12)•Reliability / Fault Tolerance (11)
• Safety•Scalability (10)•Security / Privacy (10)•Performance (5)•Usability (4)•Accessibility (3)•Adaptability (3)•Flexibility (1)
Models & Simulations (Multilevel)
• Accuracy / fidelity• VV&A (verification, Validation & accreditation)• Scalability• Modeling environment (jungle/desert)• Modeling language • Visualization• Test cases, usage(scope of use), underlying
assumption
Architecting
• Language (dependent)• Visualization (static & dynamic)• Executable architecture• Analyzable• Multiple views (like DODAF), integration of views• Generation of the system from the architecture• Architecture pattern, reuse
Prioritization
• Multiple stakeholder value proposition, criteria analysis
• Tradeoff• Interdependencies• Cost & schedule• Value estimation• Environment (scenario)