u.s. army test and evaluation command - home - …...u.s. army test and evaluation command mr. david...
TRANSCRIPT
U.S. Army Test and Evaluation
Command
Mr. David Jimenez
Executive Technical Director –
Deputy to the Commander
ITEA System of Systems Conference
Reducing Risk in 2020 through
Test & Evaluation
28 January 2015
ATEC: Reducing Risk in 2020
through Test & Evaluation
Optimizing T&E Data
• Obtaining and Retaining Information
• Rapid Analysis
• Greater Breath of Analysis
• Advanced Cybersecurity T&E
2
iknnn
i
ttinnnttn
ikn
i
ttintn
ikn
i
ttin
ikn
i
ttin
ikn
i
ttin
eeinnni
nnneen
eeini
ne
eeini
n
eeini
n
eeini
ntR
1544
22544335
44
22435
33
32323
22
32322
11
111
0
1)(
544
544)1(
5
0
1)(
4
4
0
1)(
3
3
0
1)(
2
2
0
1)(
1
1
!!
!1
!!
!
!!
!
!!
!
!!
!)(
Gartner’s 2015 Trends:
#4 – Advanced,
Pervasive, and Invisible
Analytics
2025 Test & Evaluation and Big Data
Goals:• Utilize knowledge, information, and data to
achieve core mission and business objectives.
• Faster, more Accurate Decision Making
• Cost Optimization
• Quicker Responses to Requests for Information
• More Holistic Test and Evaluation
• Automated tracking items or status
• Make useful big data capabilities available to everyone, but tailored to specific needs.
3
Sustainment of data for long term use
(Archival)
Discoverability and Access to data
Analytics of historical and current information
Derive context to inform decision
making
Common Core Requirements:
2025 T&E
Leveraging Historical Data
Faster, More Sophisticated
Analytical Tools
Modeling & Simulation
Design of Experiments
Cloud Computing
Use Case 1 – Evaluation Context
4
Evaluation Context
AMPVPrevious Tests
Capability Gaps
Historical Test Data and Analysis
Variations from future version
M113
Capability Gaps
Historical Test Data and Analysis
Variations from AMPV
Bradley
Capability Gaps
Historical Test Data and Analysis
Variations from AMPV
Use Historical Information to Inform T&E
TaskFor the Armored
Multi-Purpose Vehicle
(AMPV):
• Develop an
Evaluation Strategy
• Test/Fix/Test
• Evaluate
• Analysis and
Recommendations
5
Categorize into
functional
groups,
review historical
changes
Identify and
categorize
major
replacement
components
Analysis included
reviewing over 18M
miles of field data
and test data
Leveraging Historical Data
Determining Risk Areas Prior to Test
Risk Areas Risk Areas
Test Data Field Data
GOAL: Analyze historical reliability risk
areas observed during system test and
fielding to inform future reliability testing
Improved Analysis Techniques
Wheeled Vehicle TestingImproved analysis
techniques and evaluation
methodologies are leading
to more efficient reliability
testing. 3,000 miles
avoided and $135,000 in
savings
Increased use of vehicle
instrumentation has
saved wheeled vehicle
test program 15,000
miles and $400,000
6
Design of Automotive Reliability
Test and Evaluation (DART)
Full-up System Level Testing (FUSL) alone is inefficient for reliability growth. M&S and compressed FUSL testing
are valuable test tools that should be utilized early and often to accelerate understanding
Align government reliability testing with the systems engineering “V” to include developmental testing of components
and subsystems with defined entrance/exit criteria prior to FUSL testing (crawl – walk – run)
Tailor testing to full range of operational usage, including extreme natural environments (XNET)
Multi-organization working group leveraging insight from industry and academia
to further improve effectiveness and efficiency of automotive reliability T&E
Focus: Cooperative Research & Dev, Compressed T&E Methodology, Automotive Rel T&E Guidance, Contracting, Policy
Materiel SolutionAnalysis
TechnologyDevelopment
Engineering & Manufacturing
Production & Deployment
Post Fielding, ECPs
Robust
subsystem PS&T &
Physics of Failure
Robust PS&T,
Compressed
FUSL, XNET
Design for
Reliability
Best Practices Language
Sys RBDModel
FMECA
Increased Utilization of available data (MLLP, SDC, GCSS-A, Black Box)
FUSL
Operational
Testing
Vision for Reliability T&E in a Complex World (unknowable)
B CATargeted and
Informed Testing
7
Building Tools to Make Better Use of
the Data We Collect
• AEC developed visualization tools to make
better use of large data sets obtained from
data acquisition systems to support
evaluations of system effectiveness,
suitability, and survivability.
•Data sets are mined to:
• Provide additional context and details
of test events.
• Better understand mission impact of
test events.
•Support in-depth root-cause analysis of
events.
•Next Steps: Data integration and
visualization initiative to add geospatial
context to system status.
Instrumentation
Data Sets
are 100 GB
per test day!
Visual “fingerprints” allow
evaluators to see what manual
data collection methods could
not.
8
Meeting the Challenge of
System Complexity
As systems and systems of systems increase in
complexity, new methods are required to ensure
adequate test and evaluation in spite of resource
constraints.
AEC is meeting the challenge of complexity with:
• Advancements in Data Collection,
Reduction, and Analysis (DCRA), including
automated data acquisition systems.
• Scientific Test and Analysis Tools (STAT),
including Design of Experiments.
•Advancements in Empirical Simulation
Methods – especially for understanding
statistical risks at the system of system
level.
AEC utilizing advanced
methodology to ensure
test adequacy when the
test configuration of a
system differs from the
field configuration.
9
Calculating Reliability for Systems
of Systems with Redundancy
iknnn
i
ttinnnttn
ikn
i
ttintn
ikn
i
ttin
ikn
i
ttin
ikn
i
ttin
eeinnni
nnneen
eeini
ne
eeini
n
eeini
n
eeini
ntR
1544
22544335
44
22435
33
32323
22
32322
11
111
0
1)(
544
544)1(
5
0
1)(
4
4
0
1)(
3
3
0
1)(
2
2
0
1)(
1
1
!!
!1
!!
!
!!
!
!!
!
!!
!)(
Failure times
Ideal Distribution
Eliminating unnecessary testing through thoughtful
application of analytical techniques
Traditional exponential distribution assumption
gives MTBF requirement of 683 hrs
Direct analytical calculation gives MTBF
requirement of 226 hrs— Can validate with a
shorter test
Optimizing Resources in a System
of Systems Test Event
Subsystems
A-Kit
B-Kit
C2
Test data collected at a
subsystem level, then
combined using a Block
Diagram approach
Option 2:
2000 hours on A-kits
800 hours on B-kits
1000 hours on C2
Producer risk*: 25%
Option 1:
1500 hours on A-kits
1200 hours on B-kits
2000 hours on C2
Producer risk*: 75%
*Probability system will fail test even if it meets requirements
Using advanced techniques to optimize test
design and minimize risk11
AnalyzeStatistically to Model
Performance
What Conclusions?
PlanSequentially for Discovery
Goals, Responses and Factors?
Designto Control Risks and
Span Battlespace
How Many? Which Points?
Executeto Control Uncertainty
and Reduce Bias
How to Sequence?
General Factorial3x3x2 design
2-level Factorial23 design
Fractional Factorial23-1 design
Response SurfaceCentral Composite design
Design-Expert® Software
Cm (pitch mom)
Design points above predicted value
Design points below predicted value
0.0611074
-0.0831574
X1 = A: Alpha
X2 = D: Ail
Actual Factors
B: Psi = 0.00
C: Canard = 0.00
E: Rudder = 0.00
12.00
16.00
20.00
24.00
28.00
-30.00
-15.00
0.00
15.00
30.00
0.02
0.02675
0.0335
0.04025
0.047
C
m (
pitch
mo
m)
A: Alpha D: Ail
Pitching
Moment
Angle of
Attack
Aileron
Deflection
Design of Experiments and Statistical
Analysis
TRIAL A B C
2 HI LO LO
8 HI HI HI
5 LO LO HI
4 HI HI LO
1 LO LO LO
3 LO HI LO
7 LO HI HI
6 HI LO HI
12
System of Systems ExampleSystem X Initial Operational Test with System Y
• Design facilitated side-by-side
comparison between modernized and
legacy aircraft with focus of evaluation
on the additional capability provided
when teaming with unmanned aircraft
systems.
• Primary response variables include
target affiliation range, engagement
range, and mission success.
13
In this sample scenario,
Statistical design verified
that the target acquisition
range for the system of
systems team was
significantly greater than
autonomous capability.
With UAS Without UAS
Aircraft Type Mission Type Day Night Day Night
LegacySystem X
Recon 1 1
Security 1 1 1
Close Combat Attack 1 1 1
Interdiction Attack 1 1
Modernized System Y
Recon 1 1
Security 1 1 1
Close Combat Attack 1 1 1
Interdiction Attack 1 1
Communication System Analysis ExampleOver-the-air Connection Pathways Between Two Nodes
Sample System
At Range 0:
• 68% Terrestrial – One Hop
• 14% Terrestrial – Multi Hop
• 7% Celestial
• 3% No Connection
At Range 2:
• <3% Terrestrial – One Hop
• 18% Terrestrial – Multi Hop
• 60% Celestial
• 14% No Connection
Statistical
Modeling Allowed
AEC to Examine
Connection Path
as a Function of
Range (Sample)
13
• Statistical modeling & summary statistics
indicated the threat factor had no
significant impact on missed distance
• Program obtained 2-Star approval to
accept analysis of threat in DT and
eliminate threat as factor in OT
Guided Projectile SystemDevelopmental Test Design and Analysis
• 32-shot DOE allowed evaluation of multiple
key performance requirements in single test
• DOE supported several test site constraints
allowing design to be executed in allotted time
Missed
Distance
Level 1 Level 2
(n=19) (n=13)
Mean 22 30
CEP 20 32
DOE implementation & analysis resulted in
~$Ks program cost savings
ICD Requirement Design Factors & Levels
Operate between Minimum and
Maximum Specified Temperatures→ Temp: Low, Med, High
Operate in Hostile Environment → Threat: Level 1, Level 2
Operate with Multiple Fuze Modes → Fuze Mode: A, B, C
Operate between Minimum and
Maximum Specified Ranges→ Range: Low, Med, High
SME suggested
Operate over a Range of Offsets → Offset : Low, High
Operate over a Range of QE → QE: Low, Med, High
Objective Electronic Warfare Testing
Sequence
B
Laboratory Characterization
and Field Test Activities
EW Vulnerability Assessment to inform
operational test design
Operational Testing:
integrating EW into
threat arsenal
Enhanced OT data
reduction/analyses with
M&S
An EW Vulnerability
Assessment is a cooperative
effort using M&S to investigate
scenarios and improve likelihood
of effects on SoS missions
during OT.
Electronic Warfare
integrated into threat
maneuver force during
OT to try to
deny/degrade
Command and Control.
C
OTRR IOT
M&S Development contributes to improved
• Integration of disparate databases;
• Data visualization/playback;
• Link quality/performance analyses;
• Terrain and geometry effects; and
• Expanded evaluation and
understanding. Leverage M&S to inform testing which in
turn will validate simulation
Operational TestDevelopmental Test
Integrated DT/OT
17
Modeling:
Inform, Analyze and Improve
RF Coverage
Connectivity
Playback
Each event provides
• Opportunity to expand evaluations
• Additional data to enhance model
18
Risk Management Framework (RMF)
Objective Cybersecurity Testing Sequence
B
Cooperative
Vulnerability &
Penetration
Assessment (CVPA)
(Step 4)
Operational Testing
with TCNO (Step 5)
Adversarial
Assessment (AA)
A Cooperative Vulnerability
& Penetration Assessment
is an overt effort to identify
vulnerabilities in preparation
for OT.
TCNO emulates
an enemy force
during OT to try to
subvert cyber
protection.
C
OTRR IOT
Developmental Test CVPA and
AA Characterization and Field
Test Activities at significant
Software Updates
Developmental Test
Integrated DT/OT Operational Test
19
Cybersecurity T&E approach (IAW AR 25-2, DoDI 8510.01, and DASD(DT&E) & DOT&E guidance*) mitigates software and security risks of fielding unproven platform equipment. Applicable data will be leveraged whenever available.
Cybersecurity Test and Evaluation ApproachMajor Software Drops - Example
New Software
Existing EvaluationNew Integration
New Hardware
Su
bsyste
m E
xam
ple
s
- Computing Systems- Improved Displays- New Processor Units- Maneuver Control Enhancements
- Cross Domain Solution Adjustments- Enhanced Training- Improved Vehicle Management- Improved Communications Manager
- CREW Device - Tactical Communication Devices- Battle Command Systems- Power Distribution Systems
OEM Cybersecurity
Testing
Software Drop
Post-OEM Testing
DT Cybersecurity
Testing
Software Drop
Post-DT
OT Cybersecurity
Testing
Software Drop
Post-OT
Software Lifecycle
Maintenance
“Cybersecurity requirements must be identified, tailored appropriately, and included in the acquisition, design, development, developmental and operational testing and evaluation, integration, implementation, operation, upgrade, or replacement of all DoDPlatform Information Technology IAW DoDI 5200.44 and DoDI 5200.39, this instruction, and other cybersecurity-related DoD guidance, as issued.”
(Ref: DoDI 8510.01)
20
2/5/2015
Visualization of Network
ATEC: Reducing Risk in 2020 through
Test & Evaluation
Questions/Discussion