33nd annual international t&e symposium
TRANSCRIPT
33nd Annual International T&E Symposium
How to Manage the Complexity
of Testing the SoS
Presented By:
Ms. Paola Pringle, NAVAIR
Mr. Todd Standard, NAVAIR
Enhancing our Competitive Edge: T&E to Meet the Pace of Need
October 4-6, 2015 Hyatt Regency~ Reston, VA
Distribution Statement A - approved for public release; distribution is unlimited,
as under NAVAIR Public Release Authorization 2016-775
BLUF
• Integrated and Interoperable (I&I) systems must be defined and
designed as such
• Test cannot create I&I
• I&I requires more than a single platform
• Our Naval Forces operate in a communication rich System of Systems
(SoS) and Family of Systems (FoS) that accomplish mission objectives
using capabilities
• Test focus must shift to testing capabilities within a mission context to
determine
How effective our platform is within each required missions
How confident we are that the platform can execute capabilities to accomplish
the required missions
Distribution Statement A - approved for public release; distribution is unlimited,
as under NAVAIR Public Release Authorization 2016-7752
Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2015-710Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2015-7103
Universal High-Level OV-1
4 Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775
P-8A Platform Centric View
5
Requirements•Multi-static Active Coherent
Acoustic Sensors
•Automatic Identification System
•Rapid Capability Insertion
•High Altitude ASW
•Weapons
•Communications & Sensor
Upgrades
•Net-Ready KPP
•Net-enabled ASuW Weapon
•Enhanced Security Architecture
5
6
Communications
Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775
System of Systems View
7
Capabilities
• Communications
• Acoustics
• Service Oriented Architecture (SOA)
• Track Management
• Targeting
• WeaponsTRITON UAS
TRITON UAS MCS
Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release
Authorization 2016-7757
8
~1.4X
(Convergence Zone)
Acoustics
Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775
9
Weapons and Targeting
Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775
• Capabilities-based perspectives are more likely to identify
needed relationships among what were previously considered
independent systems
−SoS evolve over time making requirements
development an ongoing activity
• SoS systems engineer integrates the end-to-end functionality
and performance across the SoS mission capability
• SoS T&E engineer applies capabilities based test methods to
a mission (CBT&E) is required to verify and validate integration
of the end-to-end functionality and performance
ANALYSIS AND EVALUATION CHANGES FROM A ONE-TIME, UP-FRONT PROCESS
TO A MORE-OR-LESS CONTINUOUS PROCESS
10
CBT&E Complexity
Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775
• CBTE supports the continuum of upfront and early materiel solutions
analysis, completes the Systems Engineering “V”, and addresses testing from
a capabilities viewpoint
− T&E process is both iterative and recursive with each pass providing an increasing
level of complexity and realism to the system under test with a corresponding level
of confidence
CBTE CLOSES THE LOOP ON MATERIEL SOLUTION ANALYSIS THROUGH
VERIFICATION AND VALIDATION OF INTEGRATED WARFIGHTING CAPABILITIES
SYSTEM T&E IN A MISSION CONTEXT
11 Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775
DoD Architecture Framework
12Distribution Statement A - approved for public release;
distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775
System of Systems (SoS) Engineering
Product Integration
Operational Evaluation Views
System/Services
Functionality
System/Services Evaluation Views
Operational
Concept
Operational
Activity
Component &Qualification Testbeds
LVC Distributed Events
Mission
and
Capabilities
Based
Architectures
Live, Virtual,
Constructive
(LVC) Test
Environments
Desired Capability
PerformanceGap: Mission
Level Assessment
& Evaluation
Across SoS
Solution:
Capabilities
Based T&E
(CBTE)
Mission
Based
Verified &
Validated
Capability
BaselinesCapability Actual
Performance
13
CBTE is a Catalyst for Architecture-driven
Distributed Capabilities Verification and Validation
Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775
CBTE Intent
14
Cross-competency expanded processes
required to rapidly assess SoS capabilities
Development
SoS / SUT Performance
Requirements:
Unambiguous,
Measurable, Testable
Tasks/Conditions:
UNTL, UJTL
Mission
Desired Effects Fleet Exercise
Operational
Test
Development
Test
CBTE Precursor Products
• Mission Effectiveness Baseline (MEB)
• Mission Confidence Baseline (MCB)
Mission Context (MTB / ICTB)
Mission Evaluation Structure (Mission Study Views / MVM)
Operational
Capabilities
Operational Capability
Needs
ICTB – Integrated Capability Technical
Baseline
MTB – Mission Technical Baseline
MVM – Mission Verification Matrix
SoS – System of Systems
SUT – System Under Test
UJTL – Universal Joint Task List
UNTL – Universal Naval Task List
Mission Based Test & Evaluation “V”
CBTE Products:
• Mission Test Strategy (MTS)
• SoS Test Environment Investment
Strategy (STEIS)
• Mission Test Matrix (MTM)
with Mission Test Architecture
baseline (MTA)
• Provides
traceability from
mission capability
requirements to
mission testing
• Provide timely,
credible, and
continuous
feedback to
decision makers
Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775
MTB / ICTB Product One Liners
MTB
• Documents a mission based need using scenarios
ICTB Level 1
• Naval Force apportionment for mission area
ICTB Level 2
• Describes how systems are organized and allocates performance requirements
ICTB Level 3
• Used to manage requirements across multiple programs
15Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775
Mission Array Inputs to Mission Verification Matrix (MVM) DoDAF Viewpoint Model
16
Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775
Mission Array Inputs to Mission Verification Matrix (MVM)
17
W1
Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775
Outcomes for SoS using CBT&E
• CBTE has the potential to increase acquisition efficiency for the SoS
SoS viewed within a data driven high accuracy mission context
Improved I&I of the SoS throughout the acquisition lifecycle
SoS V&V across mission, task, and system requirements
Better coordinated, more efficient acquisition workflow
• System Under Test has high confidence that mission-level
requirements are met when fielded
• CBTE facilitates IT: Complements MBTD, COMOPTEVFOR’s
planning method for OT promoting operational view of SUT in DT
• CBTE promotes end-to-end testing of the SUT using LVC test
environments in an operationally relevant mission context
• “Closes the loop” on materiel solution analysis
• Focuses on continuous improvement of mission / SoS capabilities
18 Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775
Questions
19 Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2015-710
Contact Information
Ms. Paola Pringle
Integrated Warfare Test and Evaluation Division, 5.1L
Cyberspace T&E Branch 51L300E
P-8A Increment 3 Interoperability LTE
(805) 816-3038
Mr. Todd Standard
Integrated Systems Evaluation, Experimentation and Test, AIR 5.1G
Capabilities-Based T&E Chief Test Engineer
(301) 757-7070
Hank 'Wizard' Steinfeld, APM T&E P-8A Inc 3, P-3C
APM T&E PMA-264
ASW IWC APM T&E
PMA-290 Cyber Test Strategy Lead
301-342-3041
20 Distribution Statement A - approved for public release; distribution is unlimited, as under NAVAIR Public Release Authorization 2016-775