Collaboration to Meet Future T&E Needs
ITEA14 September 2010
1
Mr. Mike Crisp
Deputy Director, Air Warfare
Operational Test and Evaluation
Need vs. Speed vs. Cost:A Balancing Act in Acquisition
• Requirements Process stressed in Meeting Today’s Needs• DoD Budgets Under Pressure• Warfighter Demand Cycle inside Acquisition OODA Loop• Concurrent Development, Production, Operations, and Support
– Program Managers doing it all• Configuration Control
– How many one or two of a kind can we manage?• DoD Acquisition Life Cycle Framework
– Complex Systems Integration in Fluid Operational Environments– Adaptive and Flexible enough to support rapid fielding?
• JUONS, Rapid Block upgrades, Milestones, Increments
Innovative and Adaptive Approaches Required
3
Challenges for Building Effective Integrated Test Partnership
• Limited Test Resources: Range Capabilities, Size, Forces (Manpower & Materiel)
• Limited Test Time: Fielding & Production Schedules, Range Availability
• Limited Test Articles: Unit Cost, Production Time
• Requirements Validation: Do requirements satisfy mission requirements? Are they unambiguous, testable and technically realistic?
• Test Planning: Integrated test approach or serial? Active Participation in T&E WIPT? Early Identification of Operational Test Requirements?
• Test Execution: Operationally Representative Test Environments?
Integrated Test Approach and Partnering Can be Improved
DOT&E Initiatives
4
Integrated Testing
DevelopmentalOperational
Live FireImprove
SuitabilityReliability Growth
Testers Engage Early Requirements are
UnambiguousTestable
Relevant to mission accomplishment
Technically Realistic
Field New Capabilities
Rapidly Accelerated
Testing
Rapid Fielding• Number One Priority of SECDEF
• Examples:– Extended Range Multi-Purpose UAS
• Early Fielding Report based on Customer Test
– Project ODIN• ATEC Observations Led to Improved Effectiveness
– MRAP• DoD 5000 Process Adapted for Rapid Development
55
ERMP UAS Counter IED OPS MRAP Live Fire Test
Field New Capabilities
Rapidly Accelerated
Testing
Engage Early
• DOT&E Action Officer as a Team Member– Requirements Validation
• Do requirements satisfy mission requirements?• Are they unambiguous, testable and technically
realistic
– Test Planning• Active Participation in T&E WIPT• Early Identification of Operational Test
Requirements
– Test Execution• Operationally Representative Test Environments 6
Testers Engage Early
Requirements are …
Integrated Testing• Weapon System Acquisition Reform Act
– Established Director of Developmental Test– Formally Implemented Integrated Testing
• Test Process as a Continuum– Developmental Testing– Live-fire Testing– Operational Testing
• Optimize Testing for Program Execution– Design of Experiments– Operational Test Evaluation Utilizes All Data
7
Integrated Testing
DevelopmentalOperational
Live Fire
Improve Suitability• After System Effectiveness . . .
Suitability is the Key Concern
8
Improve Suitability
Reliability Growth
• Systems frequently judged Un-Suitable
• Reliability and Maintainability account for 80% of Not Suitable ratings
Implementation (Continued)
• Integrated Testing– Test Process as a Continuum
• Developmental Testing• Live-fire Testing• Operational Testing
– Use Statistical Analysis• Data Collected over Long Period & Multiple Tests• Design of Experiments• Normalization of Data
9
10
Common Activities
• Replicating the “real world” environment as closely as practical
• Need for a distributive live/virtual/constructive (LVC) representation of the Joint Operational Environment
• Development and use of validated Tactics, Techniques, and Procedures (TTPs)
• Development of COCOM/JS approved Joint Mission Threads
• Data collection, management, archiving, and retrieval processes
The Test and Training Communities are Missing Opportunities for Cost-effective, Relevant Partnerships
11
Common Activities
• Replicating the “real world” environment as closely as practical
• Need for a distributive live/virtual/constructive (LVC) representation of the Joint Operational Environment
• Development and use of validated Tactics, Techniques, and Procedures (TTPs)
• Development of COCOM/JS approved Joint Mission Threads
• Data collection, management, archiving, and retrieval processes
The Test and Training Communities are Missing Opportunities for Cost-effective, Relevant Partnerships
12
Bottom Line on What’s Needed
• Strong partnership between the COCOMS, Test, and Training communities
• Committed leadership for strategic institutional funding, co-investment, and cross-community management of joint test and training resources and infrastructure– Need to overcome preference for OPM (Other People’s Money)– Need to continue to work towards implementation of joint test and
training roadmaps
It Takes Time and Effort – but Potential for Increased Efficiencies and Effectiveness Justifies Both
Design of Experiments (DOE)• Test planning is a science!• DOE is a scientific tool for
developing robust test plans.
• DOE equips us to determine:
– Power– Confidence– Breadth of coverage
• The iterative nature of DOE facilitates Integrated Test design
DOE
Manage
Mother nature
Method MachineMeasurement
Material Manpower
Plan
Test
DesignAnalyze
Design of Experiments• Integrated Test – DOE Approach
– Inform each stage of testing from previous tests.– Span the battlespace through a compilation of tests.– Use screening experiments in DT to ensure a rigorous OT.
• DOE Provides:– Most powerful allocation of test resources for a given number of tests.– A scientific, structured, objective way to plan tests.– An approach to integrated test.– A structured, mathematical analysis for summarizing test results.
Developmental Test
DT/OT Testing
Operational Test
14
Design of Experiments:JASSM-ER
• Binomial Distribution was Primary Method to Establish JASSM-ER Integrated Test Design• Resulted in 21-shot Design
• Assumes ≤ 2 failures, to Demonstrate 0.80 Reliability with 80% Confidence
• 24 Factors Considered; Refined Design by Reducing Factors Previously Evaluated in Baseline JASSM OT
• Primary Factors: Range, Release Mach, Release Altitude, & Target Type• Ensures Key Performance Differences between Baseline JASSM
& JASSM-ER are Evaluated
• 21-event Test Design Created Using ¾–fractional Design to Estimate Main Effects & Interactions with Range• Uses 5 Combined DT/OT Shots
15
Conclusion• Our Goals
– Greater Rigor and Effectiveness in Testing
– Introduce New Statistical Developments
• Results
– Better Confidence, Power & Breadth in Tests
• Our Products
– More Effective and Suitable Systems
• Rigor is Key to Constructing & Executing Efficient Tests
• Only Efficient Testing can be Defended in the Current and Future Budget Environment
16
Questions
17