test & evaluation - itea
TRANSCRIPT
TEST & EVALUATION
T H E K N O W L E D G E T O M A K E I T H A P P E N
SYSTEMS ENGINEERING
F R O M T H E M I N D ’ S E Y E T O T H E B E H O L D E R ’ S
PROGRAM MANAGEMENT
T H E G U I D A N C E A N D L E A D E R S H I P
INTEGRATED PROCESSES
A Short Course By:
Dr. W. David Bell
Dr. C. David Brown
Dave Bell Principal Multi-Discipline Systems Engineer
MITRE Corp
Things done: • Data Analysis • Flight test direction - DT and OT • Scientific experiments • Systems engineering from beginning to end
– Requirements definition – Tech development, tech maturing – I&T at all levels - assembly, subsystem & system
• Managed S&T programs • VP of operations • Adjunct Professor of systems engineering - SMU
Education • BS Physics, MBA, D. Engineering
1 Feb 2012, 1100 2
Dave Brown Consulting Systems Engineer
MITRE Corp and Institute for Defense Analyses
Things done: • Colonel, US Army Signal Corps
• Developing DT instrumentation and test facilities
• Development of M&S for test support – Army Virtual Proving Ground
• Live Fire T&E
• Army Senior Executive – Director of Test and Technology, Army Developmental Test Command
• Deputy PM/Executive Director for Test, Army Future Combat Systems Program
• Instructor – JHU Masters in Systems Engineering
Education • BS, MS, PhD Electrical Engineering
• Masters in National Security Policy, ICAF
1 Feb 2012, 1100 3
Our Challenging World
• Terrorism
• Global culture clashes
• Information overload
• Disease – Health Care
• Infrastructure
• Technical Complexity
• Business and Finance
Problem Attributes
• Technical
• Size - Scope
• Complexity
• Importance
• Social, political, fiscal
Systems Engineering is increasingly the
field expected to solve the problems. 1 Feb 2012, 1100 4
Form
− Incorporates Many Capabilities
− Comprised of Interacting, Diverse Elements
− Definable Structure and Interconnections
− Bounded; Has Inputs and Outputs
Function
− It Does Something Valuable
− It is Dynamic—Not Inert
Interfaces
− Internal
− External
Attributes View of a Complex System
1 Feb 2012, 1100 5
This slide adapted from a brief by:
Dr. Samuel Seymour, JHU Applied Physics Lab
Each system can be broken down into components which in turn can
be broken down into smaller components, etc.
System
Subsystem
Component Component
Subsystem
Component Component
Subcomponent Subcomponent
Part Part
Air Defense System
Search Radar
T/R Module
Screw
Assembly
Partitioning View of a Complex System
1 Feb 2012, 1100 6 This slide adapted from a brief by:
Dr. Samuel Seymour, JHU Applied Physics Lab
Program
Management
OR THIS WAY!!! NOT THIS WAY!!!
THIS WAY!!!
Analysis Design Integration Test Systems
Engineering
Program
Management
Systems
Engineering
Analysis Design Integration Test
Program
Management
Systems
Engineering
Analysis Design Integration Test
Systems Engineering Role and Program Culture Matters!
Organizational View of a Complex System
1 Feb 2012, 1100 7
Slide adapted from a brief by:
Dr. Samuel Seymour,
JHU Applied Physics Lab
System Management
System Design
& Engineering
Interface
Management
Analysis &
Evaluation
Integration
& Test
Technical Performance,
Cost, Schedule
Data Management
Project Administration
& Support
Configuration
Management
Quality Control
Contract
Management
Production
Management
Systems
Engineering Program/Project
Management
Test &
Evaluation
C
1 Feb 2012, 1100 8
9 9
A Systems Approach View
Data Collection
Mission Performance Analysis
Operational Data Collection
Lessons Learned
Test & Evaluation
Product Development & Production
Prototype Development
Performance Demonstration
Critical Field Experiments
Enabling Science & Technology
Hypothesis, Concept Development Trade-offs & Critical Experiments
Modeling and Simulations
Capabilities Improvement
Needs Definition
Technology Knowledge Transfer
Technology
Operations
This slide adapted from a brief by:
Dr. Samuel Seymour, JHU Applied Physics Lab 1 Feb 2012, 1100
Systems Perspectives View
Systems Thinking Systems Engineering Engineering Systems
Focus on Process Focus On Whole Product Focus on Both Process and Product
Consideration of Issues Solve Complex Technical
Problems Solve Complex Interdisciplinary
Technical, Social and Management
Issues
Evaluation of Multiple
Factors and Influences Develop and Test Tangible
System Solutions Influence Policy, Processes and use
Systems Engineering to Develop
Systems Solutions
Inclusion of Patterns
Relationships, and Common
Understanding
Need to Meet Requirements,
Measure Outcomes and Solve
Problems
Integrate Human and Technical
Domain Dynamics and Approaches
1 Feb 2012, 1100 10
This slide adapted from a brief by:
Dr. Samuel Seymour, JHU Applied Physics Lab
Systems Management
Key Issues
• SE and T&E have traditionally been very separate processes accomplished by widely separate communities
• SE and PM require information to accomplish knowledge-based development and acquisition
• T&E provides information, but it must be the right information, at the right time, fully understood, and correctly used
• The Test setup is a system that must be System Engineered.
• The T&E program must be Program Managed.
C
1 Feb 2012, 1100 11
What is a System?
“A System is a set of interrelated components working together toward some common objective.” - (Kossiakoff, et al) The organization of hardware, software, materials, facilities, personnel, data and services needed to perform a designated function with specified results...(Defense Acquisition University) A bounded process involving specific interactions among a number of separately distinguishable functional elements (Amsler, JHU/APL)
1 Feb 2012, 1100 12
Systems Scope Relationships
Examples
Complexity
Scale
Device – Detector
Component - Sensor
Subsystem - IR Seeker
System – STD Missile
System of Systems - AEGIS BMD
Enterprise Systems – BMD Community
Global Family of Systems -
Theater Defense
1 Feb 2012, 1100 13
This slide adapted from a brief by:
Dr. Samuel Seymour, JHU Applied Physics Lab
A System Includes…
Benjamin S. Blanchard, System
Engineering Management
1 Feb 2012, 1100 14
System Environment
Benjamin S. Blanchard, System
Engineering Management
1 Feb 2012, 1100 15
Definition of Systems
Engineering (SE)
• An iterative and interdisciplinary management and development process that defines and transforms requirements into an operational system.
• Features: Typically, this process involves environmental, economic, political, social, and other non-technological aspects. Activities include conceiving, researching, architecting, utilizing, designing, developing, fabricating, producing, integrating, testing, deploying, operating, sustaining, and retiring system elements.
• Note: The customer for or user of the system usually states the initial version of the requirements. Systems engineering should be used to better define and refine these requirements. Often the requirements change as further decisions are made.
1 Feb 2012, 1100 16
System Engineering Identifying
Qualities
• Top down - viewing system as a whole.
• Life cycle view.
• “Complete” effort to identify system
requirements “up-front”.
• Interdisciplinary team approach.
1 Feb 2012, 1100 17
Scientific Method
Problem
Definition
Select
Hypothesis
Test Hypothesis
Problem
Next Problem
Conclude
Confirm, Deny or
Modify Hypothesis
1 Feb 2012, 1100 18
Systems Engineering Process (Method) (Kossiakoff, et al)
1. Requirements
Analysis
2. Functional
Definition
Need
Solution(s)
4. Design
Validation
Requirements
Functions
Potential Solutions
(System Models)
4 Basic SE Activities: executed
repetitively over life cycle
leading to successfully
developed and validated
system
Problem
Definition
3. Physical
Definition or
Design Synthesis
1 Feb 2012, 1100 19
Process Input
Process Output
Requirements
Analysis
Functional Analysis
& Allocation
Design
Synthesis
System
Analysis & Control
(Balance)
Validation
Requirements
Loop
Design
Loop
DOD Systems Engineering Process (MIL-STD-499A)
Ref; DSMC/DAU
C
1 Feb 2012, 1100 20
Common SE “Steps”
• Problem definition
• Consumer need definition
• Feasibility determination
• System operational requirements definition
• Maintenance and support concepts development
• TPM determination & prioritization
• Functional analysis
• Requirements allocation
• System synthesis
• Integrated design
• T&E
• Production
• Operational use and support
• Retirement & disposal
1 Feb 2012, 1100 21
22
Systems Engineering Approaches
Views
Waterfall
Spiral
The “V” Linear
Life cycle
Systems
Engineering
Lifecycle
Concept
Development
Engineering
Development
Post
Development
Operational
Deficiencies
Technological
Opportunities
System Functional
Specifications
Defined System
Concept
Production
Specifications
Production
System
Operation &
Maintenance
Documentation
Installed Operational
System
Systems
Engineering
Method Requirements
Analysis (Problem Definition)
Functional
Definition (Functional Analysis &
Allocation)
Physical
Definition (Synthesis, Physical Analysis
& Allocation)
Design
Validation (Verification, Evaluation)
Next
Phase
Objectives
Requirements
Functions
System
Model
Validated
System
Model
P&D
E&MD
D&V
CE/D
Need
1
2
3
4
5
Quality
Mgt
Users - Operators
Market Pull
Customer Requirements
Concept
New Product Idea
Technology Push
Preliminary System/Product
Concept Definition
Market Assessment
RFP/|BAA
Discuss ions
Collaboration
Brainstorming
War rooms
“Draw a Cartoon”
Proposal
Statement of Work
Product Defn Statement
Functional/System
Block Diagram
Pricing/Estimating
WIN! Form Project Office
START WORK
Work breakdown
Structure WBS
Risk Assessment
Plan
Budget and Schedules
PERT and GANTT charts
Task Work Orders
Work Authorizations Develop Prototype
Evaluate Prototype
“beta tests”
Specs
Des ign
Linear Responsibility Charts
Critical Path Analysis
Contracting
Production
Q uantities
Sub System Fabrication
PDR CDR
System Integration
and Verification
System Test and Evaluation
Configuration
Mgmt
Field Test and
Evaluation
Operations and
Maintenance
Project Closeout
Follow-On?
Delivery
Install/
Acceptance
Logis tics
Warehousing
Sales
NEEDS ANALYSIS CONCEPT EXPLORATION
CONCEPT/PROGRAM DEFINITION
DESIGN / TECHNOLOGY VALIDATION / ENGINEERING DEVELOPMENT PRODUCTION / MANUFACTURING
T/E AND OPERATIONAL SUPPORT SYSTEM USE
Systems Engineering LIFE CYCLE ACTIVITIES
9/20/01 SJSeymour
“PLANNING”
“DIRECTION, MONITOR,CONTROL”
Organizational Structures
Project Mgr Attributes/Authority
Technical Performance
Budget Schedule
Systems
Engineering
Communications and Financial Management All the Time
Slide adapted from a brief by:
Dr. Samuel Seymour,
JHU Applied Physics Lab
1 Feb 2012, 1100
Life Cycle or Linear Approach to
Systems Engineering Phase
Step
Requirements
Analysis
Functional
Definition
Physical
Definition
Design
Validation
Needs
Analysis
Concept
Exploration
Concept
Definition
Technology
Validation
Engineering
Design Integration
& Evaluation
Analyze
needs
Analyze
operational
reqmts
Analyze
performance
reqmts
Analyze
functional
reqmts
Analyze
design
reqmts
Analyze
reqmts
Define
system
functions
Define
subsystem
functions
Define
component
functions &
interactions
Define
subcomponent
functions
Define
part
functions
Define
functional
tests
Visualize
subsystems,
technology
Visualize
components,
architectures
Select
components,
architectures
Specify
component
construction
Specify
subcomponent
construction
Specify
test
equipment
Validate
needs,
feasibility
Simulate,
validate
performance
reqmts
Simulate,
validate
system
effectiveness
Test
critical
subsystems
Validate
component
design &
construction
Test &
evaluate
production
system
T&E is a key part of every phase
1 Feb 2012, 1100 23
Motivation for Better
Systems Management
Benjamin S. Blanchard, System
Engineering Management
1 Feb 2012, 1100 24
The Role of a Systems Engineer
• The technical “leader” and “conductor”
– Sets the objectives (mission needs, system requirements)
– Establishes the plan
– Oversees its execution
– Monitors and guides progress
– Coordinates all technical activities
– Enforces requirements
– Is the final judge of performance/capabilities demonstrated
– Works with Program Manager to orchestrate resources
– Makes the difficult technical decisions
– Manages resolution of technical problems
– Adjusts the plan as necessary
– Advises the Program Manager
– Ultimately responsible for the technical product
1 Feb 2012, 1100 25
Systems Enterprise
• System-of-Systems (SOS) - Combination of interrelated systems working with direct interaction to achieve a broader range of activities, with the systems generally fixed and invariant.
• Family-of-Systems (FOS) – Loosely coupled or non-interacting combination of systems intended to perform a broad function where the systems may or may not directly interact and may change from one situation to another
• System Architecture or View (SV) - The fundamental organization of a system embodied in its components, their relationships to each other and to the environment and the principles guiding its design and evolution
• Operational and Technical perspective
1 Feb 2012, 1100 26
System of Systems Engineering
• Traditionally, SE is focused on the development and implementation of a well-bounded and firmly established set of requirements through the development and integration of subsystems that meet those requirements.
• SoSE leverages independent, interoperable, and integrate-able systems that serve a meaningful purpose in a stand-alone mode, to implement operationally flexible capabilities by building a collaborative SoS within an evolving concept of operations to effectively respond to an evolving threat.
1 Feb 2012, 1100 27
What makes SoS Engineering
Different?
• Capability focused (e.g., engineering a capability)
• Generally does not have specified requirements
• Focus is more on the architecture than the systems
• Focus is more on interoperability among systems
• Individual systems treated as potentially interchangeable elements
• More difficult to test—especially at full-up level in lab
• However…similar engineering practices apply – e.g., SE method – Integrate and test
1 Feb 2012, 1100 28
The SE Way
• Customer develops requirements
• PM leads and manages process
• SE guides the
development
process
• T&E informs the
process
• Customer gets the stuff
1 Feb 2012, 1100 29
The DOD Way
• DOD Acquisition has two customers – The Warfighter/User
– The US Taxpayer/Congress
• Requirements from the User – T&E to Verify/Validate
• Cost and Schedule accountable to the Taxpayer/Congress – T&E to Understand Risks –
Programmatic and Technology
1 Feb 2012, 1100 30
How the Military Works
1 Feb 2012, 1100 31
Requirements
Deploy
Secretary of Defense
Service Secretaries
Develop
Build
Buy
32 1 Feb 2012, 1100
What Happens to the SE “V”
33 1 Feb 2012, 1100
The PM
What Happens to the SE “V”
More Stakeholders
DOD Acquisition – “The Rules”
1 Feb 2012, 1100 34
Title 10 USC
Armed Forces
Federal
Acquisition
Regulation
(FAR)
Defense
Acquisition
System
DOD 5000
Service Acquisition Regulations, Instructions, Guidance
Contracting Missile
Defense
Army Marines Air Force Navy
National Defense Authorization Acts (NDAA)
• National Defense Authorization Act is the name of a United
States federal law that has been enacted for each of the past 48
fiscal years to specify the budget and expenditures of the
United States Department of Defense.
• Often additional provisions that deal with Defense
development and acquisition
– 1982 – Nunn-McCurdy Amendment, designed to curtail cost growth
in American weapons procurement programs
– 1984 – DOT&E
– 1996 - Clinger-Cohen Act (CCA), designed to improve the way the
federal government acquires, uses and disposes information technology
– 2009 – Weapon Systems Acquisition Reform Act (WSARA)
1 Feb 2012, 1100 35
DOD Acquisition
• The DOD has three principal decision-making support systems associated with military acquisition: – Planning, Programming, Budgeting and Execution
(PPBE) Process - Process for strategic planning, program development, and resource determination.
– Joint Capabilities Integration and Development System (JCIDS) - The systematic method established by the Joint Chiefs of Staff for assessing gaps in military joint warfighting capabilities and recommending solutions to resolve these gaps.
– Defense Acquisition System - The management process used to acquire weapon systems and automated information systems, the operation of which is described in DOD 5000 regulation, instruction, guidance.
1 Feb 2012, 1100 36
• User Articulates the Needs (Sometimes)
• JCIDS Process Develops Requirements (ICD/CDD)
• Acquisition Community Develops Contract SOW with Contractor (SE Executer)
• PPBE Process Provides the Funding
• Acquisition Community Conducts T&E (DT&E) Informed Reviews (Programmatic and Technical) of SE Progression (PDR, CDR, DAES, MS-C)
• Service/Agency OTAs and DOT&E Conduct Final Acceptance T&E (IOT&E) & Report to Congress (Representing the US Taxpayer)
• User Gets the Stuff
1 Feb 2012, 1100 37
The DOD Way
Operation of the Defense Acquisition System
First Acquisition Framework in 1971
FULL-
SCALE
DEVELOPMENT
Full-Scale
Development
Decision
Program
Initiation
Production
Go-ahead
Decision
CONCEPTUAL
EFFORT PRODUCTION/ DEPLOYMENT
• Decision points: 3
• Phases: 3
• Milestone documents: 1 (Decision Coordinating Paper (DCP))
1 Feb 2012, 1100 38
The Defense Acquisition Management System
DODI 5000.02*
Relationship to JCIDS
Initial Capabilities
Document (ICD)
Capability Development
Document (CDD)
Capability Production
Document (CPD)
• The Materiel Development Decision precedes entry into
any phase of the acquisition management system
• Entrance Criteria met before entering phase
• Evolutionary Acquisition or Single Step to Full Capability
IOC: Initial Operational Capability
FOC: Full Operational Capability
PDR: Preliminary Design Review
CDR: Critical Design Review
FRP: Full Rate Production
IOC B A
Technology Opportunities & Resources
Materiel Solution Analysis
FRP Decision Review
FOC
Materiel Development Decision
User Needs
PDR CDR
CDD
CPD
ICD
AoA
Pre-Systems Acquisition Systems Acquisition Sustainment
Post CDR Assessment
PDR
Technology
Development
Production &
Deployment
Operations &
Support
Engineering and
Manufacturing Development
Post PDR Assessment
C
or
1 Feb 2012, 1100 39
* DOD Instruction 5000.02, Operation of
the Defense Acquisition System,
December 8, 2008, USD AT&L
40 1 Feb 2012, 1100
What Happens to the Development & Acquisition Process
Systems Engineering - Ideal
Time
Requirements
Design
Implementation
Integration
& Test
Full System
Test
Validate
Verify
1 Feb 2012, 1100 41
Requirements
Design
Implementation
Integration
& Test
Full System
Test
Systems Engineering – How it Happens
Time
Requirements
Design
Implementation
Integration
& Test
Full System
Test
Validate
Verify
Time
1 Feb 2012, 1100 42
Requirements
Design
Implementation
Integration
& Test
Full System
Test
Systems Engineering – The Result
Time
Requirements
Design
Implementation
Integration
& Test
Full System
Test
Validate
Verify
Time
1 Feb 2012, 1100 43
Requirements
Implementation
Integration
& Test
Full System
Test
Time
Validate
Verify Design
1 Feb 2012, 1100 44
Systems Engineering – How it Should Be
V&V Implications
• DT is more heavily focused around verification
– Among other uses, DT is a main player in deciding to
accept items from contractors
• OT, the focus is both were the requirements met
and are they still the right requirements
– This is a source of friction with PM’s as they believe
they should only be judged by the requirements as
stated not as currently needed.
1 Feb 2012, 1100 45
46
• Over the past 10 years, DoD systems have experienced a 33% cost growth due to “RDT&E mistakes”…
• DoD IOT&E results, FY2001-2006 – 29 systems; mix of ACAT II, 1C, 1D across 3 Services – Approx. 50% were deemed “Not Suitable”, or partially NS – Approx. 33% were deemed “Not Effective”, or partially NE
• Preliminary study conducted by DOT&E (July 2007) determined that lack of Suitability is a significant life cycle sustainment cost driver – Reliability is the main component – Study revealed a strong correlation between reliability growth (mostly
Testing and M&S) program investments and reductions in support costs
From: Dave Castellano, Deputy Director,
Assessments and Support ODUSD(ATL)
1 Feb 2012, 1100
47
…We Don’t Start Them Right • Insufficient requirements analysis and definition at program initiation
– Not tangible, measurable, testable, stable
– User R&M requirements are not underpinned by sound rationale
• Acquisition strategies based on poor technical assumptions, competing budget priorities, and unrealistic expectations
• Budget not properly phased
• Lack of rigorous systems engineering approach
• Schedule realism – success oriented, concurrent, poor estimation and/or planning
• Inadequate test planning – breadth, depth, resources
• Optimistic/realistic reliability growth – not a priority during development
• Inadequate software architectures, design/development discipline, and organizational competencies
• Sustainment/life-cycle costs not fully considered (short-sighted)
From: Dave Castellano, Deputy Director, Assessments and Support ODUSD(ATL)
Red denotes areas in which testers can be especially helpful.
1 Feb 2012, 1100
48
…We Don’t Manage Them Right • Insufficient trade space
– Resources, schedule, performance, requirements
• Insufficient risk management
• Inadequate IMP, IMS, EVMS
• Most programs lack quantifiable entrance/exit criteria
• Maturing “suitability” (e.g., RAM) is not always a priority
• Maturing “effectiveness” is not always a priority
• Concurrent test program; inadequate scope due to schedule and resource insufficiencies, etc.
• Inadequate OTRR process – no strong DT&E gate prior to IOT&E
• Inadequate government staff; Inexperienced and/or limited contractor staffing
• Poorly defined IPT roles, responsibilities and authority
– Overall poor communications across government and industry staff
From: Dave Castellano, Deputy Director, Assessments and Support ODUSD(ATL)
Red denotes areas in which testers can be especially helpful.
1 Feb 2012, 1100
49
PMs Avoid Testers
As Long As Possible…
• The Situation: – Gov’t program managers typically limit T&E involvement until
absolutely forced to do so. • Perceived as adding schedule and fiscal expense in a time-period
already under schedule and fiscal stress. – (Cynically) Do not want to ask the question if they might not be able to
stand the answer.
– We’ll make up lost time and fix the problems later in the last few weeks/months of the program known as T&E.
– By forgoing T&E insights the program manager du jour might: • Escape problems on their watch but the DOD does not escape
discovering problems and ultimately fixing them
• Have their program cancelled or restructured therefore not fielding important warfighter capabilities
1 Feb 2012, 1100
A Word About USC Title X, DOT&E, OT&E, & OTA’s
• Do OTAs & DOT&E Represent the User?
• Do They Conduct and Evaluate “User Testing” - Is OT&E User T&E?
• Involved mostly at end of the process
• Report to:
– User?
– Congress
• What happens when the
User gets the stuff?
1 Feb 2012, 1100 50
The DOD Way Updated • Is the stuff ready for OT?
• Congress Decides they Shouldn’t Wait Until the End of the
Process
• WSARA 2009
– Create DASD(SE)
– Create DASD(DT&E)
– More Emphasis on DT&E (Theoretically throughout the
process)
– However: Better DT&E does not directly translate to better
informed acquisition and development and certainly does not
enhance user involvement in the process
1 Feb 2012, 1100 51
A Word About System-of-Systems (SOS)
• SOS is the way almost all military equipment gets used
• But it is the way almost none of it is developed
• Recent activity on SOS T&E (mainly Army)
– Mandates to conduct SOS T&E
– Need to address SOS requirements
– Need to manage development by Program of Programs
– Can’t just integrate individual programs (Army ASA(ALT)
SOS Integration Directorate challenges)
– Interoperability testing and integrated testing are not SOS
testing
1 Feb 2012, 1100 52
The Future • Can the Pure SE Way Work for DOD?
– Not as Long as the User and Purchaser are Different Customers
• Can a Hybrid Work?
– Does the Current Hybrid Work?
• Congress thinks not
• The GAO thinks not
• The User thinks not
– A Better Hybrid?
• COCOM Driven Rapid Acquisition?
– Working, but……
– COCOM of current ops gets all the attention and resources
– Other COCOMs of possible next conflict get little to nothing
• Budget cutting will dictate:
– Fewer new systems
– More evolution – less revolution
– More discipline and rigor
– Better informed acquisition – T&E
– Better Systems Engineering
1 Feb 2012, 1100 53
TEST & EVALUATION
T H E K N O W L E D G E T O M A K E I T H A P P E N
SYSTEMS ENGINEERING
F R O M T H E M I N D ’ S E Y E T O T H E B E H O L D E R ’ S
PROGRAM MANAGEMENT
T H E G U I D A N C E A N D L E A D E R S H I P
INTEGRATED PROCESSES
A Short Course By:
Dr. W. David Bell
Dr. C. David Brown