1Deputy Under Secretary of the Army Test and Evaluation Office
Big Datain
Army Test and Evaluation
Prepared for ITEA System-of-Systems Engineering Workshop26 January 2017COL Patrick Walden, Senior Military Advisor for Army Test and Evaluation
2Deputy Under Secretary of the Army Test and Evaluation Office
Army Test and Evaluation Enterprise challenges and opportunities
• Continue Tri-Service Cooperation & Joint Invest.
• Meeting New data challenges
― Modeling and Simulation― Emerging Technology testing
• Spectrum • Budget uncertainty• Speed of informing
(performance)• Depth of analysis
• Workforce― Career Path Professional Dev.― Professional Outreach― Hiring Authorities
• DoD T&E Oversight (DT and OT)
• T&E Governance Authorities (NDAA 2003)
• Support to Experimentation (informing requirements)
• Support to Prototyping
3Deputy Under Secretary of the Army Test and Evaluation Office
Force 2025 and T&EService OTAs must determine whether or not systems are effective, suitable, and survivable in support of unified land operations in an operational environment dominated by:• Increased momentum of
human interaction• Potential overmatch• Importance of cyber and
space• Dense urban areas
(megacities)• Ubiquitous media• WMD proliferation• CEMA !• Big Data !
Increasing complexity on the battlefield increases complexity in T&E. Demand for data – and the means to use it effectively – is also increasing.
See TRADOC PAM 525-3-1 “The U.S. Army Operating Concept (AOC): Win in a Complex World”found at: http://www.arcic.army.mil/Concepts/operating.aspx CEMA – Cyber Electromagnetic Activities
4Deputy Under Secretary of the Army Test and Evaluation Office
Big Data Changed Everything
Implications for Army Operating Concept and Force 2025: The Force 2025 Soldier will not have known a world without analytics.Our Surroundings
We expect to be able to access analytics – instantly and on demand – to measure and understand our complex world.
Our neighbors
Our interests
Our health
Our wealthOur wealth
Even image searches
5Deputy Under Secretary of the Army Test and Evaluation Office
Big Data Has CausedAn Evolution in T&E
Yesterday Today
Discrete data sets (usually associated with a single test); small overall file size
Large data sets collected over a test program (may include data from contractor tests, simulators, hardware/software-in-the-looplaboratories, M&S, fielded system, and similar systems)
Meaning derived from expert observations Meaning derived from continuous observation
Workforce has expertise in the system under test
Workforce has expertise in analytics
Evaluation products consumed by small, specialized audience
Evaluation products consumed by broad audience with diverse interests
Central evaluation question: “Did it meet requirements?”
Central evaluation question:“What are the system’s strengths and limitations over the range of conditions found on a complex, interoperating battlefield?”
To focus on the “Why and How” of a system’s operational effectiveness, operational suitability, and survivability, increases the demand for deep analytics.
6Deputy Under Secretary of the Army Test and Evaluation Office
T&E Big Data Challenges
Free and shared among responsible practitioners
Support model validations
Amounts of data straining analytical resources More reliance on supercomputing Need tools to make short order of analysis
(visualization, sage, and frame capture)
T&E Big Data
Leverage Advances in Instrumentation
Capabilities
T&E Cadre of the Future Requires Data Scientists and Data Analysts
7Deputy Under Secretary of the Army Test and Evaluation Office
Considerations for Big Data Analytics
Cons:Available data may be underutilized due to awareness gaps:
• What capabilities already exist?• What lessons have already been learned? • What opportunities exist?
Utilizing big data requires careful planning:• Information system and data management design • Data Collection, Reduction, Analysis (DCRA)• Archiving and sustainment (“context” of the event)
Utilizing big data requires appropriate tools:• Even small data sets are unmanageable without right tools• Tool development requires planning, time, and resources
“I paid for all this data. What can I do with it?”
Awareness
Planning
Tools
8Deputy Under Secretary of the Army Test and Evaluation Office
The Big Data Community
Field
T&E
User Needs
Materiel Development
S&T(6.1/6.2/6.3)
Big Data
“Big Data” is a common resource of the Services’ analytical community.
Diverse analytical organizations contribute to and draw from it:
• Data acquisition methods• Computational resources• Models, simulations, laboratories, tools• Historical data • Expertise
Important questions going forward:• Who manages it for stakeholders?• Who sustains it?• How do we establish business rules for
increased collaboration?• Can we obtain synergies through
collaboration?
9Deputy Under Secretary of the Army Test and Evaluation Office
Value of ‘Deep’ Knowledge: Example
Bad Event
Analysis by Service OTAs
& Others
Increased Survivability
10Deputy Under Secretary of the Army Test and Evaluation Office
Big Data Analysis Approach
Week’s worth of test data (~100 GB) processed within 2-3 days
1) Download vehicle data files
2) Process data for each week of test
3) Review report for reliability highlights
Run Course Identification ScriptsGPS coordinates used to ID course
Generates summary file containing metadata for each file (Vendor,
Vehicle ID, Course, Date, Miles, & Hours
Run Data Collector ScriptsCombine files from similar vehicle, course and date
Generates files with concatenated channel data and flags the files
containing incomplete data
Run Report Generator1) Displays summary of mileage and hours2) Compares accelerations, temperatures, and speeds, across multiple vehicles3) Displays plots of major channels for each unique vehicle, course, and date combination
Generates .pdfreport
11Deputy Under Secretary of the Army Test and Evaluation Office
Leveraging the Big Data Space:Use Historical Data to Right-size Future Test
Big Data
Field
T&E
User Needs
Materiel Development
S&T(6.1/6.2/6.3)
Risk Areas = Priority Test Areas
ATEC and AMSAA analyzed 18 million miles of Stryker field and T&E data to develop reliability “risk areas.”
Insights will be used to shape test scope on future versions of the systems.
+
Field Test
Subsystem XAssembly Y
Component ABSub-assembly Z
Block interface WWidget subsystem
Assembly CaseSub-component ABC
Nuts and bolts AssemblyMain element
Subsystem Box KSuperstructure Link
Block assemblyCrankstick BetaShaft Structure
Drive Component WidgetXYZ Interface
12Deputy Under Secretary of the Army Test and Evaluation Office
Leveraging the Big Data Space:Developing Cybersecurity Metrics
Big Data
Field
T&E
User Needs
Materiel Development
S&T(6.1/6.2/6.3)
ATEC leveraging Network Integration Evaluation (NIE) events to develop models, methodologies, and metrics for cybersecurity T&E.
Insights will be used to enable earlier-in-life cycle assessments and requirements development.
0.1% Person is untrustworthy
Resource worth $1000
Threat Model for Untrustworthy Insiders
13Deputy Under Secretary of the Army Test and Evaluation Office
Leveraging the Big Data Space:Improving System Survivability
Big Data
Field
T&E
User Needs
Materiel Development
S&T(6.1/6.2/6.3)
ATEC combined insights about ballistic events on vehicles in theater from:- Intelligence community’s trend analyses- On-board vehicle instrumentation- Ballistic response data from live fire testing.- Modeling and simulation
Insights used to improve:- Current and future system survivability designs- Test Scope- Test and evaluation methodology- Instrumentation and simulation designs
14Deputy Under Secretary of the Army Test and Evaluation Office
2025 T&E and Big Data GoalsGoals:• Utilize knowledge, information, and data to
achieve core mission and business objectives.― Faster, more Accurate Decision-Making― Cost Optimization― Quicker Responses to Requests for Information― More Holistic Test and Evaluation― Automated tracking items or status
• Make useful big data capabilities available to everyone, but tailored to specific needs.
Sustainment of data for long term use
(Archival)Discoverability and
Access to dataAnalytics of
historical and current information
Derive context to inform decision
making
Common Core Requirements:
15Deputy Under Secretary of the Army Test and Evaluation Office
New Data Scientists & Data Analysts
Visual Information
-1084-
IT Management
-2210-
Computer Engineering
-0854-
Mathematical Statistics
-1529-
Expertise in engineering; expertise in data systems, data structures, data mining and programming languages.
Expertise in scientific inquiry into complex relationships and processes using multi-disciplinary analysis tools and techniques – particularly modeling and simulation.
Expertise in statistical tools and techniques; expertise in applied mathematics.
Expertise in applying visual design principles to communicate complex information to diverse audiences.
Expertise in data architectures, information systems, and data management.
Computer Science-1550-
Operations Research
-1515-Expertise in high-speed computing systems, data acquisition systems, algorithm analysis and development, and information processing display, control and transfer.
WANTED: Cadre of Data Scientists and Data Analysts
16Deputy Under Secretary of the Army Test and Evaluation Office
Conclusions
• Big Data analysis : Terabytes of Data Greater Insights ? High potential to leverage learn /understand behaviors of complex systems High potential of over-analysis for sake of over-analysis
• New generation of Data Scientists needed• Real data-driven evidence to investigate anomalies - attribution• Investments required:
New methods and tools to quickly process and analyze Big Data Support the enterprise decision processes Develop a sharing culture – DOD data policy evolutions
Big Data will change our T&E enterprise – in ways we don’t completely grasp yet.