test & evaluation/science & technology program unmanned and autonomous systems test (uast)...

Test & Evaluation/Science & Technology Program Unmanned and Autonomous Systems Test

(UAST)FY2009 Program Execution Review

T&E/S&T Program FY2009 End-of-Year Program Execution Review 2

Objectives

Project Status

UAST Accomplishments

UAST Lessons Learned

Summary


UAST Technology Overview

• T&E capability shortfalls

– Predicting UAS Intelligence and Behavior

– Emulating Mission & Environmental Complexity

– Assessing UAS Effects and Capabilities

– UAST Protocols and Design

– UAS Test Bed and Environment

– UAST Reference Data (Ground Truth)

– UAST Tools and Techniques

• Major technology challenges (in the S&T solution development)

– Develop algorithms for predictive behavior by integrating genetic algorithm, probability theory, and autonomous behavior theory

– Construct models to analyze UAS decision operations in complex and uncertain environment

– Develop methodologies and metrics to evaluate distributive control and situational awareness

– Develop test protocols and algorithms to support on-board management system evaluation and UAS shutdown

– Develop realistic test bed components, and environments to support UAS evaluation approaches to unfolding missions via range assessments

– Compile ‘truth data’ to support range safety, and protocols development

Autonomy

Knowledge

Interaction

Net-Centric Comms (GIG)

Platforms Sensors WeaponsCro

ss-la

yer

Mgm

tan

d C

ontr

ol

UAS Layered Architecture


UAST Technology Overview (cont.)

• How does the focus area prioritize the technology challenges (state your rationale)?– Mapping of T&E needs to POR or NPOR as defined

by:– UAST WG – DoD Roadmaps– SPO/ Project Managers

– By technology having broad applications across UAS Domains

Autonomy

Knowledge

Interaction


Platforms Sensors Weapons

Cro

ss-l

ayer

Mgm

tan

d C

ontr

ol

Terrain, Weather, Comms, Threats, Friendlies, NeutralsAir, Space, Surface, Subsurface

Met

rics

and

Ana

lysi

s

UAS Architecture Layer Emulations

Virtual, Constructive,And Live (mixed)

Mission and Behaviors

KnowledgeInjection

Collaboration and Actuation

Environment(Physical, Tactical, Social, etc.)

Autonomy

Knowledge

Interaction


Platforms Sensors WeaponsCro

ss-la

yer

Mgm

tan

d C

ontr

ol

UAS Layered Architecture


Roadmap leverage of accepted reference models for testing UAS

• NIST Performance Decomposition Model

• As opposed to componentized UAS view presented in roadmap

• Another alternative would be a test framework over ALFUS

Predict

EmulateAssess


Operational UAST Challenge


Autonomy-Base UAST Challenge


Cognitive Autonomous System Tester (CAST)

Key Future Events & Dates

• Start/Finish : October 2008/September 2010• PDR: July 2009• Phase 2 Component Demo: September 2009• Phase 2 Scientific Report: September 2009• CDR: January 2010• Final Integrated CAST Demo: August 2010• Final CAST Scientific Report: September 2010• Final CAST Transition Report: September 2010

Project DescriptionThe overall CAST methodology learns each mission’s goals, learns vehicle-specific characteristics and behaviors, and identifies vehicle behavioral anomalies. In addition, a self-organizing knowledge representation system will organize and categorize the information in a way that permits the tester to test complex emergent behaviors and anomalous behaviors to be differentiated and evaluated during highly difficult missions and mission failure modes.

Transition Partners & Dates• Army Research Lab-BCT (FCS) Mission Based

Applications. 4Q FY10• Army Advanced Technology Directorate (AATD),

TBD

Deliverables• CAST integration strategy report• Stand-alone toolkit for UAS autonomous behavior

testing• Final Project report

Project 1 of 13Project 1 of 13


High-Fidelity Communications, Modeling & Analysis (HFCMA)


• Start/Finish : January 2008/May 2010• Phase 2 Documentation: September 2009• Collaborate with transition partner: December 2009• Define User Cases: January 2010• Incorporate Selected assets Demo: February 2010• Effectiveness of interaction and multiple assets report

April 2010• Final Integrated report: May 2010

Project DescriptionHFCMA is a Live/Virtual/Constructive T&E framework that merges Live and Constructive assets, including heterogeneous collaborative UxVs with the ability to mirror One to Many Live UxV’s performance in the V/C simulation. T&E Use Case is a) multiple vehicles interacting, b) each with collateral systems and c) a test system that must coherently stimulate the vehicles under test and capture limits of UxVs whole system performance and integrity. HFCMA is concerned with testing the dynamic limits of autonomy

Transition Partners & Dates

• Incorporate use cases for air, ground & sea• Verify adequacy with transition partner• ATEC-BCT(FCS) Robotic convoy 1Q/FY10• NAVAIR/452 FTS-Scan Eagle/Predator 2Q/FY10• NUWC-Undersea environments TBD

Deliverables• LVC framework for test data recovery and analysis• Delivery of prototype test framework• Framework training manual• Final project report



Remote Embedded System Test (REST)


• Start/Finish : October 2007/December 2009• Field Demonstration: August 2009• Final Report: October 2009 • Conduct Field Test November, 2009 • Final Report: December, 2009

Project DescriptionREST is the integrated set of technologies that replaces a wired instrumentation system with a wireless, self powered, self organizing, self healing network of sensor nodes designed to be manufactured (embedded and assembled) into the platform structure and provide continuous test and evaluation capability as well as integrate into the platform Integrated System Health Manager.REST is designed to provide UAST capability for multiple platform types and DoD services.

Transition Partners & Dates• Advanced Composite Cargo Aircraft (ACCA)

Program, Phase III FY09 (REST TRL 6 Baseline Technology) – Installed and evaluated in ACCA Fuselage Fatigue Test Article (Mar CY2010)

• Vulture (DARPA), In-Work

Deliverables• Prototype TRL 6 system components• Flight and Ground Demonstration• Final Report and White Papers



Software Configurable Multi-Channel Transceiver (SCMT)

Key Future Events & Dates• Start/Finish : August 2008/August 2009• Transceiver Completion Report: August 2009• Transceiver link Demonstration: October 2009

Project DescriptionFlexible multi-band command and control communications system supports test data capture and command/control applications during complex multi-agent (UAS) test scenarios. Effectively implements a software-customizable transceiver system that is frequency agile with modulation and protocol independence. SCMT encompasses four transmitters and four receivers in a single unit the size of a commodity disk drive programmable for any frequency band between 1 MHz and 6 GHz.

Transition Partners & Dates• Phase 1 completion Oct 2009. Must coordinate with

SET to determine if project will continue• Test community interest: 452nd Test Wing, NAVAIR,

ATEC

Deliverables• Project will provide a 4-channel SCMT demonstration

prototype• Set of capability demonstrations for core functionality• Aeronautical TM modulation formats



Micro-beacon Tracking of Autonomous Systems (MTAS)

Key Future Events & Dates• Start/Finish : October 2007/September 2009• System Design Review: June 2008• Field Demonstration: August 2009• Final Report: October 2009

Project DescriptionEffective evaluation of a UAS performance depends on ground truth about the world, the vehicle and what the vehicle senses. MTAS is designed to provide vehicle localization data utilizing small self-contained, and inexpensive RF micro-beacons affixed to test vehicles. Portable ground stations receive the signal and calculate range to each test vehicle.This supports multiple autonomous systems, low latency, sub-meter positioning performance, and a full performance operation in GPS-denied environments.

Transition Partners & Dates• Brigade Combat Team Modernization (BCTM)

Experiment (RCX), 4Q/FY09

Deliverables• Three prototype beacons• Full scale system demonstration at Ft. Devens airfield • Final project report



Virtual Autonomous Teaming Tool(VATT)

Key Future Events & Dates• Start/Finish : April 2008/November, 2010 • End of Phase 2 Demo: February, 2010• Start of Phase 3: February, 2010• End of Phase 3: November, 2010

Project Description. VATT is a tool used for autonomous vehicle simulation, teaming, and decision making analysis. It aims to develop a VATT configuration for T&E applications, that can predict how modifications to external stimuli will produce changes within the UAS autonomy logic processes. VATT environment strives to seamlessly integrate simulation subsystems with real UAS hardware subsystems. Simulated systems could replicate multiple platforms for the development of mutli-platform behaviors thus supporting repeatable events and detailed system/ event logging.

Transition Partners & Dates• RTTC; John Hardison- 4 Q FY10

Deliverables • Requirements Document• VATT-T Software Toolkit• Final Report



Standards, Software and Hardware for Multi-Resolution Maps/Models (MRMM)

Key Future Events & Dates• April 2008/December 2010: Start/Finish• Sept 27-28 : Final Large Prototype Static Test• Oct 15-16 : Final Small Prototype Static Test• Nov 5-6 : Final Large Prototype Dynamic Test• Nov 26-27: Final Small Prototype Dynamic Test• Dec 10-11: Final Demo Phase II

Project DescriptionHigh precision site data is an essential tool in deriving ground truth. Comparison not only of the vehicle’s position with actual location but with what the vehicle senses in its surroundings. This S&T project is based upon analytical studies and laboratory studies to physically validate theoretical foundations using software and standards for the implementation of multiresolution models for unmanned autonomous systems testing. The analysis, models, software and standards will be evaluated through both hardware emulation and experimental evaluation.

Transition Partners & Dates• Army Research Laboratory, 4 Q/FY10• Air Force Research Laboratory, 4 Q/FY10Other dates of interest:• ARL Test Collaboration TBD LIVEFIRE Program• AFRL/NASA Phase II SBIR – TBD Helicopter flight

demonstration

Deliverables• Software module libraries modified for hardware

deployment.• Ground / Air testing of laptop payload• Detail design specifications of Prototype FPGA-

based I/O Board.



An Integrated Agent-based Framework for Simulated, Virtual and Live Testing of Teams of

Unmanned Vehicles (IABF)

Key Future Events & Dates• Start/Finish : September 2008/September 2010• CDR: April 2008• Phase 1 Demo: September 2008• Phase 2 Demo: August 2009• Final Report and User Documentation

Project DescriptionIABF will allow for systematic and structured T&E of technologies in the cognitive/social domain using a combination of simulation, hardware in the loop testing and live testing. Testers will quickly and efficiently be able to develop test plans and validate UAS Team performance with respect to:• C2 navigation approaches utilizing NLOS techniques• Collision avoidance• UAS Team coordination and performance • Fault tolerance under various failure modes and bandwidth constraints

Transition Partners & Dates• Picatinny Arsenal, ARDEC,1Q FY10• Naval Research Lab (NRL),4QFY09• OneTEss/PEOSTRI/OTC, 4QFY09

Deliverables• DCTF Software and libraries• Phase I : Indoor demonstration of Test-framework• Phase II: Outdoor advanced Technology Demo• Final report and User documentation



Reconfigurable Wireless Measurement System (RWMS)

Key Future Events & Dates• Start/Finish: January 2009/Sep 2009• LVC & RWMS Demonstration: August 2009• Phase 2: Prototype & Documentation: Sept. 2009

Project DescriptionGround truth measurement infrastructure to support UAS testing and vehicle safety analysis. The measure system consists of components that can provide an alternative source of data to compare with those organic to the system under test. Three levels of modularity, namely, i) physical interface (face plate), ii) hardware stack, and iii) computational fabric (FPGA based) are employed to provide maximum flexibility

Transition Partners & Dates• ATEC White Sands Missile Range, 1QFY10 • NAVAIR, 1QFY10 • 412 FTS EDWARDS, 1QFY10

Deliverables• RWMS prototype (HW and SW)• Documentation of the miniaturized RWMS design

RWMS Demo in 2 or more UAS test case Scenarios• Final report and analysis of demonstrated capability



Flexible Command and Control Systems (FCCS)

Key Future Events & Dates• Start/Finish: October 2007/December 2009• Acquisition of telemetry source: June 2008• Delivery of Second Telemetry Source: Sept. 2008• Initial Demonstration: December 2008• FCCS workbench demonstration: August 2009• Delivery of Final Documentation: December 2009

Project DescriptionFCCS is a user facing application framework that provides C2 capabilities, its primary functions are to reduce the amount of time to display meaningful data to the user and improve the C2/HRI functionality of the targeted robotic device. When a new UAS is presented for testing, The C2 system must be able to be rapidly adapted to the unique qualities of the new UAS. The C2 system must also have a standardized, well-documented framework so that UAS developers can prepare their UAS for integration with the C2 system before arriving at the test facility.

Transition Partners & Dates• Global Observer 412 FTS EDWARDS, 1QFY10

Deliverables• Prototype Demonstration • FCCS Documentation • FCCS Final System



A Prescriptive and Adaptive Testing Framework (PATFrame)

Key Future Events & Dates• Start/Finish : October 2009/September 2012• PDR: September 2010• Event Workshop 1&2: September 2010• CDR: March 2011• Event Workshop 3: March 2011• Phase2 :Progress Report Tool: September 2011• Event Workshop 4: September 2011• Final Demonstration: September 2012• Final Demonstration: September 2012• Final out brief & Deliverables: September 2012

Project DescriptionPATFrame is a decision support tool encompassing a prescriptive and adaptive framework for UAS SoS Testing. PATFrame will be implemented using a software dashboard that will enable improved decision making for the UAS T&E community. The technology addresses the Modeling & Architecture and Testbeds & Test Environments areas in the UAST capability framework.

Transition Partners & Dates• U.S. Army Operational Test Command (USAOTC),

Ft. Hood, TBD• Naval Air System Command (NAVAIR), Patuxent

River, MD, TBD• Air Force: 303rd Aeronautical Systems Wing, Wright-

Patterson Air Force Base, OH, TBD

Deliverables• Tool delivery (Software)• Final demonstration • Final Documentation



Synthetic Collective Unmanned Underwater Laboratory (SCUUL)

Key Future Events & Dates• Start/Finish : October 2009/October 2012• Demo underwater motion capture system: Oct 2010• Demo Model-based simulation and hardware

prototype: October 2011• Phase 2 Scientific Report: October 2011• Prototype deployment and demonstration: 2012

Project DescriptionSCULL applies dynamic systems theory to design reduced order models of cooperative UUV navigation, sampling, performance, and control. UUV technology includes operations in remote/inaccessible, and dynamic environments; autonomous trajectory generation, tracking, and coordination. The value added to UUV testers is (1) tests to guide UUV development and deployment; (2) help to define T&E for cooperating underwater vehicles; and (3) provide T&E with a capability for fielding autonomous systems that are suitable, effective, and survivable.

Transition Partners & Dates• Naval Undersea Warfare Center (NUWC), 1Q FY12

Deliverables• Interim report on reduced-order modeling of

cooperative behavior• Interim & final reports on model-based simulation

framework• Prototype micro-UUV platform and software



M&S for UAST

Key Future Events & Dates• Start/Finish : October 2008/December 2009• Progress report of current tasks: December 2008• Ontology/Taxonomy task begun: January 2009• Protocols Document: December 2009• Software solution for Ontology researched and

identified: December 2009

Project DescriptionProposed technology is a developmental support tool that focuses on a collection of entities for UAS testing. This support tool is known as an Ontology/Taxonomy application containing knowledge about existing UASs. Real time positioning of equipment to monitor UAS experiments via the internet for T&E observations occurring within the UAST Enclave. The ontology/taxonomy will be implemented as a decision support system to improve UAS T&E outcomes. Appropriate UAS choice for the mission & environment.

Transition Partners & Dates• UAST WG/ UAST Prototype Testers, 4Q FY10

Deliverables• Development of Test Protocols• Evaluation tool• Ontology/Taxonomy working application• Final Report

( ,t, x, y, z)

( ’, t’, x’, y’, z’)

( ”, t ”,”, x”, y”, z”)UA

ST

N

etw

ork

UAST Technical Topic Areas = Framework

Modeling andArchitecture

UAST Analytics

Test Beds & Test Environments

WSMR

JPL

Aspects & Protocols

A

B

Obstacles

UAS

LU Lab

Ft. HoodSNL

Ft. Bliss

Picatinny



UAST FY09 Achievements

Project Description Benefit T&E

Remote Embedded System Test (REST)

Development of alternative energy harvesting and storage technology. Self healing power harvesting Implemented and demonstrated switching between vibration and thermal energy harvesting modes as well as dynamic power summing from vibration and thermal.

Self powered wireless structure sensor network replacement for wired structure network provides for continuous T&E at low installation weight and O&M.

Micro Beacon Track of Autonomous System (MTAS)

First available T&E/S&T technology which addresses high accuracy full I performance TSPI in GPS denied environments. Demonstrated on manned and small UGV (static and moving vehicles with up to 3 vehicles tracked simultaneously.

•Sub-meter 3d positional accuracy- .68m•Position fix latency-1.0 s•1km2 tracking range•Applicable to manned or unmanned systems and dismounted soldiers

Integrated Agent-based Framework (IABF)

Unified platform independent distributed control framework for simulated and mixed mode testing (live-simulated) of unmanned systems teams.

Extensive use of M&S with integrated comm/networking capability, allowing support for a broader and more robust set of testing in a more realistic simulation

environment.


UAST FY09 Achievements (cont.)

Project Description Benefit T&E

Reconfigurable Wireless Measurement System (RWMS)

Miniaturized RWMS to provide unobtrusive real-time situational measurements and perform override/shutdown actions if safety is compromised. Report the kinematics, monitor the operating environment, UAS health monitoring.

Non-intrusive, user-defined data interface device for real time data collection of UxV system and sub-system levels for both truth and perceived truth data. Addresses the need for multi-platform modular,scalable data collector tool. Candidate solution for ATEC’s Field Data Collector.Utility for TENA in a Resource Constrained Environment (TRCE).

Software Configurable Multi-Channel Transceiver (SCMT)

Emerging and next-generation test instrumentation increasingly require network communication to manage complex and dynamic test scenarios, particularly for UAST. Adapting wireless communication infrastructure to accommodate challenging testing needs can benefit from reconfigurable radio technology.

SCMT enables four transmitters and four receivers in a single unit, programmable for any frequency band between 500 MHz and 6 GHz. Multiple communications links needed for interface to different systems. Real-time reprogramming of transceiver will allow versatility of system.


UAST FY 09 Lessons Learned

• UAST Working Group MUST shift focus from 'marveling about UxS's' to 'T&E'ing UxS's.' We must devise a UAST framework so that the S&T that will enable various 'paths' through the framework can be highlighted and justified then the ROI of S&T projects can be measured.

• We can move from a focus on a) measuring the autonomy algorithm in the payload to b) measuring the autonomy of the Whole System and doing so sufficiently to Predict, with stated Confidence, the anticipated Suitability, Safety, Effectiveness and Survivability of the whole system. "How can you 'prove' that your systems design will solve the customer's problem before you build and prove that design?"

• The UAST process must be synchronized with PMO's view of the world and the TRMC process. Drop 'ConOps' and talk in terms of Use Cases and Use Case Portfolios. Then S&T projects must state Objective in terms of Use Case hot spots and a) S&T intent regarding ameliorating the hot spots and b) requirements allocated to other of TRMC's S&T activities.

• Use Cases must state the T&E implications of the whole UxS. T&E implications are objectives and concepts as stated at the Effects --> Capabilities -->Algorithm/Timeline level (not System Shall statements) across development, prototype, IOC, production


Accelerating UAS Deploy through Test


Safety in the Battlespace


Three Phases of Startups

1. Glimpse of “Market”

2. Learning enough to match market with technologies

3. Create value proposition for garnering ROI

– Variety– Extent– Ambiguity of Operational

Necessity; Fast Tracking; Need for Testing for Safe, Suitable, Effective, and Survivable UAS

We are currently finishing Phase II We are currently finishing Phase II


Three Questions

• Are we progressing fast enough to help the Warfighters?– Prototypes, Demos, Beta-Testing

• How can we accelerate S&T activity for UAST?– Use cases– Connections to UAS programs: workshops, outreach, conferences,

web site, magazine, flyer– Other? AUVSI, Robotic Rodeo, PERMIS, NIST, SAE, INCOSE, IEEE,

NDIA, UARCs, Academia• These rules for triage?

– Must address a T&E need– Must require S&T– Should have a big payoff– Should have wide applicability– Should have a transition partner


Summary

• UAST Working Group– MUST devise a UAST ROI framework– Use-case mapping of test needs for safe, suitable, survivable and effective UAS – Technology forecasting

•CTEIP Mitigation – Solicit needs from Warfighters, Testers, Developers, PMs– Transition partner development – How do we accelerate S&T investments into T&E capability?

•UAST overarching role – Filling in the gaps for testing UAS that are being deployed using rapid acquisition

practices– Focus on autonomy T&E– Focus on safety of UAS

•Stakeholder collaboration (COI-UAST) – Use Cases must state the T&E implications of the whole UxS – T&E implications are objectives and concepts as stated at the – Effects --> Capabilities -->Algorithm/Timeline level (not System Shall statements) across

development, prototype, IOC, production

test & evaluation/science & technology program unmanned and autonomous systems test (uast)...

Documents

test slide

test program fy2009

year program execution

test protocols

test framework

design uas test bed

uas intelligence

uas effects