H e l p i n g Wa r f i g h t e r s a d a p t
UNCLASSIFIED
UNCLASSIFIED
(U) Rapid Test and Evaluation
18 January 2018Thomas C. Clutz, PhD
2
UNCLASSIFIED
UNCLASSIFIED
(U) Background
(U) JIDO enables DoD actions to counter improvised-threats
with tactical responsiveness and, through rapid solution
development and delivery, to support the Combatant
Commands‘ efforts to prepare for and adapt to battlefield
surprise in support of counter-terrorism, counter-insurgencies,
and other related mission areas, including countering
improvised explosive devices.
(U) “Rapid solution development and delivery” requires a rapid test capability
3
UNCLASSIFIED
UNCLASSIFIED
(U) Enabling Rapid Fielding
Vet Deploy Sustain
CONUS Based Testing
Theater Support Testing
In-Theater Testing
Forward Operations
Support
JIDO Risk Mitigation
4
UNCLASSIFIED
UNCLASSIFIED
(U) Vocabulary
• (U) Test – a formal activity designed to provide an official report on/certification for a system’s capabilities- Government sponsored
• (U) Experiment – an activity intended to provide understanding about some (perhaps all) of a system’s capabilities– Government/vendor sponsored
• (U) Demonstration – an activity designed to showcase a system’s capabilities– Vendor sponsored
(U) A specific purpose provides a specific outcome
5
UNCLASSIFIED
UNCLASSIFIED
• Needs for rapid test execution:• (U) Pre-defined assessment processes
– (U) Establish an understanding (if possible, a specific plan) of how a particular technology needs to be tested
– (U) Includes an understanding of classification• (U) Pre-positioned equipment (systems and devices)
– (U) Have all the relevant equipment pre-positioned at the test range
– (U) Have a common representation of a threat device (ideally across all test centers involved in testing a particular capability)
• (U) Pre-positioned funding– (U) Front-load test activity coffers; retain expenditure authority
(U) Rapid Test & Evaluation Principles
(U) The more time spent in test preparation,the less time required for test execution
6
UNCLASSIFIED
UNCLASSIFIED
• Common standards and methodologies
• Statistically relevant testing
• Operational realism
• Consistent T&E across DoD
• Comparability of test results
Consistent, Comparable Results
(U) Test Protocols
7
UNCLASSIFIED
UNCLASSIFIED
(U) Test in Integrated Environments
(U) Realistic testing in integrated, realistic environments increases the chances of utility in a theater environment
8
UNCLASSIFIED
UNCLASSIFIED
M&S can save significant time in testing-but the model must be useful
•Lab testing reveals issues
•Limited realism
Laboratory
I2WD
Pt Mugu
•Most data in a given time
•Difficult to account
for all effects
Modeling &Simulation
•Most realistic
•Least cost effective
APG YPG
China Lake
WSMR
Theater
OpenAir
RF Propagation
CraneVehicle
IntegrationNear Earth
Propagation
(U) Integration of M&S with T&E
9
UNCLASSIFIED
UNCLASSIFIED
• (U) JIDO has centralized the conduct of all JIDO test activities in a single office
• (U) The JIDO Integrated Evaluation Team (JIET) is the nucleus of JIDO test planning– (U) Membership includes: JIDO T&E (facilitator)
Service representatives
• OTA, operators
Program manager
Scientists
(U) Rapid Test & Evaluation Team
(U) Interagency acceptance of the final test plan
10
UNCLASSIFIED
UNCLASSIFIED
(U) Rapid Test & Evaluation Methodology
JIDO Initiative Evaluation Team
Initiative Evaluation Plan
Traceability Matrix
Individual Evaluations
Intro
Assessment Overview
Assessment Issues
MOEs
M&S Strategy
Traceability Matrix
Threat & Intel
Competitive Strategies Group
Data Collection Events
Suitability, Survivability & Design Issues
Estimate of Supportability
Eval Schedule
Executive Summary
Background
PurposeRequirementSystem OverviewSystem Evaluation TimelineData Sources & References
Findings, Analysis & Discussion
EffectivenessSuitabilitySupportability
Recommendations
OperationsSupport ArchitectureFuture Planning
Success Stories
Provide JIDO leadership with
recommendations and the “so what” analysis
based on all data collection
Data Collection Events
Data Elem
ents/Metrics
All initiatives will get an IEP which will
outline the depth of the test efforts
Tracker for all data metrics of interest
correlated with data collection schedule. What will we know,
when?
11
UNCLASSIFIED
UNCLASSIFIED
• (U) Rapid solution development and deliverynecessitates the establishment of rapid test and evaluation capabilities to properly assess each rapid initiative
• (U) Preparation time saves execution time– Established test protocols/plans – Fully equipped test ranges– Prepositioned funding– Useful modeling and simulation– Obtain stakeholder buy-in for the test plan
• (U) Integrated test facilities provide significant value
(U) Summary
(U) A simple, direct approach can be very effective
12
UNCLASSIFIED
UNCLASSIFIED
13
UNCLASSIFIED
UNCLASSIFIED
• (U) JIDO has centralized the conduct of all JIDO test activities in a single office– (U) Theater Support Test (TST) program – provides on-call T&E
capabilities to Combatant Commands (CCMDs) and Services to address current RFIs in support of TTP development, and the fielding of material solutions
– (U) Op SABRE – annual force protection electronic countermeasures compatibility test series conducted in cooperation with FVEY partners
– (U) T&E support to JIDO Rapid Acquisition Initiatives – rapid evaluation support of JIDO initiative development to address warfighter requirements against improvised threats devices
(U) Rapid Test & Evaluation Examples
14
UNCLASSIFIED
UNCLASSIFIED
• (U) Theater Support Test– (U) Open-air, hardware-in-the-loop laboratory, modeling & simulation
test capabilities used to rapidly respond to CCMD RFIs– (U) 500+ RFIs answered to provide test data to support TTP changes Counter UAS (C-UAS) system performance and compatibility
• (U) Rapid fielding of new C-UAS capabilities necessitates the need to rapidlyrespond to performance and compatibility RFIs from CCMDs and Services
• (U) Ex. Warfighter SIG endorsement of OIR Memo directing JIDO to conduct C-UAS compatibility testing for DoD
CREW performance & compatibility trials AEA performance & compatibility Rifleman Radio Green Hash – rapidly executed testing in order to confirm
material and TTP solutions• (U) Op SABRE
– (U) Conducted since 2005 among the FVEY partners to respond to national,CCMD and Service RFIs and inform TTP development to mitigate compatibilityconcerns
(U) Rapid Test & Evaluation
15
UNCLASSIFIED
UNCLASSIFIED
• (U) JIDO Rapid Acquisition Initiatives– (U) MARVIN AEB responded immediately to RFIs resulting from operational
assessments in CENTCOM AOR• Testing replicated false alarm conditions and provided data and
recommended TTP changes to mitigate the impact to performance• Testing explored the use of MARVIN as a solution beyond the designed
problem set– (U) Chameleon Rapidly responded to CCMD RFIs to test and assess mitigation measures
for specific improvised threat; test data confirmed viability of simple riskmitigation measure that was adopted
(U) Rapid Test & Evaluation
16
UNCLASSIFIED
UNCLASSIFIED 12
(U) Warfighter Support Portal (WaSP)
• (U) Origins of WaSP– Theater Support Web Tool (TSWT)
was created as a reachback capabilityto operators in theater
– Offers a quick turnaround on requests for information(RFIs)
– Allows storage, retrieval, and access to RFI data• (U) Tools
– Drupal and MySQL• (U) Key Products
– Maintenance of a direct portal link withtheater to enable rapid test requests
– Complete test report archive– JDEP account administration– C-UAS Compatibility Chart hosting– C-SUAS Working Group
(U) https://wasp.navair-rdte.navy.mil
17
UNCLASSIFIED
UNCLASSIFIED
RAM
P:
CE
Ente
rpris
e pr
oduc
es a
ctio
nabl
e sy
stem
per
form
ance
info
rmat
ion
JEM
S &
othe
r In
fo S
harin
g
Test Data
Test Agencies:
e.g., ATEC, other DT, OA
services
Test
Pla
ns
da
ta
OA DataOperational
Units
• IEP• Traceability Matrix• CE Report:
• Cumulative Relevant T&E results• Proven capes/limitations• Risks/Issues• Recommendations
• Baseline capability
Evaluation Div Chief
Eval Div Tech Advisor
Test & Evaluation Enterprise
IET
IET
Cor
e
CE Programs
Team (IET Lead)
J9
J2
J3
TEC
Dev Rep
M&S
AST Chair
PM Rep
Appr
oval
Cha
in fo
r R
elea
se b
eyon
d C
E
IET Coordinates within context of IPTs to address metrics testing with respect to operational risk, Red/blue teaming, leveraging M&S, Intel, and
CONOPS
Develops and Continuously
Updates
JIDO System
Integrator
Initiative PMs
T&E Coord & MGT
Metrics
CE Test Operations
Team
CE Infrastructure
Team
PrioritiesJIDO
Capability Manager
Eval Data
Current ProductsR
elea
se
Senior Leader/FDO
System D
ata
Initiative&
ThreatD
ataC
E An
alys
is T
eam