Team Exploration vs Exploitation with Finite Budgets

David Castañón

Boston University

Center for Information and Systems Engineering

Introduction

• Exploration vs exploitation is a classic tradeoff in decision problems with uncertainty- Exploit available information vs improving information- Numerous applications in finance, adaptive control, machine

learning, …

• Interested in paradigms for teams of agents - Search and exploit information with limited resources- Task partitioning, motivated by applications (e.g. surveillance)

• Objective: techniques for team control of activities- Improve coordination among team members- Allow for variations on human roles in team- Understand aspects of task partitioning for mixed

human/automata teams

Experimental Platform

• Multiple robots search for and perform tasks - Can provide varying levels of operator control: human-automata teams- Control information displayed, risk to each operator using video

• Possible model for interesting problems

Boeing BU

Illustration: Dynamic Search/Classify Problems

• Multiple agents with different fields of regard

• Multiple sites to search for potential objects using noisy measurements, classify them

• Agents have search modes/exploitation modes, finite budget

Related Work

• Quickest detection problem - Noisy measurements of alternative hypotheses

- Trade off decision accuracy versus time (cost) to make decision

• Results: optimal strategy is to make decision when log-likelihood ratio leaves a threshold interval- Sequential Probability Ratio Test- Log-likelihood ratio drift-diffusion model (Cohen-Holmes, …)- Correlates well with human decision strategies on sequential

repeated tasks

• Single sensing mode, single site

Multi-Armed Bandits

• Robbins (50s), Gittins (70s), many others

• Independent Machines- Each machine has individual state with random dynamics- State evolves when machine is played, stationary otherwise- Random payoff depending on state of machine

• Objective: infinite horizon sum of discounted rewards- Repeated decisions among finite alternatives

• Result: Optimal policy based on Gittins indices, computed independently for each machine based on current state- Select machine with largest current Gittins index to play next

Human Exploration/Exploitation inMulti-armed Bandit Problems

• Multi-armed bandit paradigm- Complex choice task with simple structure to normative solution- Can model human choice with heuristics/simple strategies that

approximate Gittins indices and include parameters for human variability

• Daw et al (05,06): Time varying environments, propose alternative models for exploration/ exploitation using soft-max and other random decision rules

• Steyvers et al (09), Zhang et al (09): Finite horizon total reward paradigm, with models using a latent variable to encourage exploration vs exploitation

• Yu-Dayan (05), Aston-Jones & Cohen (05), others: Propose mechanisms underlying brain activity in aspects of exploration vs exploitation

• Cohen et al (07): Highlight limitations of the multi-armed bandit paradigm and discuss directions for future work

Limits of Bandit Paradigm

• Stationary environments- Tasks do not evolve unless acted on- Likelihood of success at task plus follow-on task is time invariant- Set of possible tasks is time-invariant

• Single action per time- Not geared to team activities

• Infinite horizon objective- Unbounded resources

• Single type of action per bandit- Can’t vary choice

Interested in other paradigms that remove these limitations

A Different Paradigm: Spatial Resource Allocation

• Multiple agents with finite resources- Multiple actions per agent- Different action types per agent/location- Similar to classical search theory- Focus on effectiveness of collective team actions

• Problem: M team members, N locations, actions xij from member j to location i:

Solution mechanism: Pricing of resources + Duality- Allocation, but no tradeoff in exploration vs exploitation

Extension: Dynamics

• Allow learning of information based on outcomes of action- Actions on cell i from agent j at time t: xij(t) result in information as to the content of

cell- Different actions different quality of information, different effort- Bayesian integration of information from multiple actions, times- Exploration: inexpensive actions to identify valuable locations (e.g. detect activity)- Exploitation: expensive actions to collect value (e.g. identify objects, etc)

• Allow dynamic arrivals and departures of objects in cells

• Problem: Normative solution of such problems requires full stochastic dynamic programming - Very large information space: probability measures on product space of possible

cell contents per time- Combinatorial explosion in number of control actions Complex problem to solve, even for simple versions

Problem Setup

• N cells, T decision stages- State of cell i at stage t: xi(t) can evolve according to a Markov chain (arrivals,

departures) independently across cells

• M agents, each capable of allocating resource Rj(t) in each stage t by choosing actions (modes) to act on cells- Decisions uijm(t) = 1 if agent j uses action m on cell i at stage t, 0 otherwise- cijm are resources needed for action: constraint- Some actions m are exploratory, others exploit

• Action m on cell i at stage t yields noisy measurement yijm(t) of current cell content- Evolves information state: i(t) is probability distribution over content of cell i at

stage t- Conditionally independent likelihoods p(yijm(t)|xi(t))

• Additional decision at stage t: identify- For each cell, can identify content correctly and get reward (content-dependent)- Objective: Collect total cumulative reward given resources at each time

Constraint Relaxation

• As stated previously, optimal solution requires feedback strategies based on the product space of information states across all states- State of cell i at stage t: xi(t) can evolve according to a Markov chain (arrivals,

departures) independently across cells- Can solve conceptually, but hard to do except for very small number of cells

• Reformulate problem: relax resource constraints at future times- Replace sample path constraints with average constraints- Can be optimistic: will allocate more resources on difficult problems than available,

balance by allocating less resources on easy problems

- Resulting problem has special structure that allows for easier solution- Solution provides optimistic bounds on performance of original problem

Results: Duality

• Theorem: - Given resource prices j(t) for agents at each stage, stochastic optimization

problem decouples into N independent cell problems- Optimal solution can be found in terms of feedback strategies that use only

information on the current information state of a cell to select actions for that cell- Overall information state is product of marginal information state for each cell

• Implication: efficient solution algorithms- Merge pricing approaches from resource allocation with single cell subproblem

solutions that use stochastic dynamic optimization- Reduce joint N-cell optimization problem into N decoupled problems,

coordinated by prices- Replaces combinatorial optimization across cells by pricing mechanisms

• May provide tractable models for human choice in resource allocation and optional stopping- Replace detailed enumeration of outcomes with price estimates

Feasible Decisions: Model-Predictive Control

• Strategies designed using relaxed constraints may run out of resources in future stages- Approximate dynamic programming technique requires on-line computation of

decisions instead of off-line computation of strategies

• Restore feasibility by re-solving with receding horizon given most recent information- Adds robustness to model errors through replanning

ResourcePrice

Update

Cell 1Subproblem

Cell NSubproblem

PricesResourceUtilization

Team Computation

• Interesting problem: agents negotiate based on local problems to agree on prices - Concave maximization problem, but non-differentiable challenging to

establish converging algorithms- Can truncate negotiation with heuristics current approach- Topic of current PhD effort

• Mixed Initiative Variations- Human team members as equals control subset of resources, negotiate on

prices with automata- Human team members as leaders select actions for own resources, automata

select complementary actions for others- Humans as controllers impose constraints (e.g. cell responsibility allocations) on

automata

• Algorithms have been developed to implement above variations and explore potential experiments

Experiment 1

• Team of 3 agents, 100 cells, partial overlap in coverage

• 3 levels of resources per agent

• 3 sensing modes per agent- Search, low-res ID, high-res ID

• 3 object types, 1 of them high-valued- Vary relative error for

missing high-value- Potential initial arrival per cell

no departures

• Discrete-valued observations- Tentative classifications

• Strategy Parameters- Horizon, truncation strategy

• Problem definition studied by Hero et al (07)using stochastic dynamic programming

• Large number of cells, few of them occupied- Want to find occupied cells- Cell i state Ii = 1 if occupied, 0 otherwise- Prior probability Ii = 1 given

• Allocate energy to cells controls signal/noiseratio of measurement

• Two-stage problem: initial search, followed by refined

location of occupied cells • Adapt second stage allocations based on first stage search• Limited energy budget

Experiment 2: Adaptive Radar Search

Algorithms and Results

• Equal allocation among cells- Motivated by standard operations in wide-area Synthetic Aperture Radar (SAR)

• Model Predictive Control using relaxed resource constraints- Two stage allocation: low energy image to detect pixels of interest (high intensity

reflections), followed by higher energy in pixels of interest

• Results similar to prior work by Hero et al (2007) with simpler algorithm 2 orders of magnitude faster

Paradigm Extension: Mobile Agents

• Viewable sites depend on agent positions- Slower time scale control- Focus on trajectory selection and mode- Sequencing of sites critical to set up future sites

• Mobile agents: trajectory and focus of attention control- Models where electronic steering is not feasible- Sequence-dependent setup cost for activities

• Additional uncertainty: risk of travel- Visiting a site accomplishes task that gains task value- Traversing among sites can result in vehicle failure and loss

Experimental Platform for Research

• Multiple robots search for and perform tasks - Can provide varying levels of operator control: human-automata teams- Control information displayed, risk to each operator using video

Future Activities

• Evaluate algorithms on experiments with dynamic arrivals/departures

• Develop algorithms for motion-constrained mobile agents

• Implement experiments involving tasks with performance uncertainty in robot test facility- Vary tempo, size, uncertainty, information

• Implement autonomous team control algorithms to interact with humans in alternative roles- Supervisory control, Team partners, others

• Extend existing algorithms to different classes of tasks- Area search, task discovery, risk to platforms

• Collaborate with MURI team to design and analyze experiments involving alternative structures for human-automata teams