1

The Role of Organizational The Role of Organizational Control in Scaling AI Control in Scaling AI

SystemsSystems

Victor R. LesserVictor R. Lesser

Computer Science DepartmentComputer Science DepartmentUniversity of Massachusetts, AmherstUniversity of Massachusetts, Amherst

IJCAI 2009IJCAI 2009Pasadena, California Pasadena, California

July 16, 2009July 16, 2009

2

ThanksThanks Raj Reddy – for his support, encouragement Raj Reddy – for his support, encouragement

and mentoringand mentoring Lee Erman – my early colleague and closest Lee Erman – my early colleague and closest

friend for over 40 yearsfriend for over 40 years My wonderful graduate students – for their My wonderful graduate students – for their

creativity, hard work and collegialitycreativity, hard work and collegiality A special thanks to my first graduate student, A special thanks to my first graduate student,

Dan CorkillDan Corkill Multi-Agent Systems community – who have Multi-Agent Systems community – who have

been a welcoming homebeen a welcoming home My wife and children – who have created a My wife and children – who have created a

richness in my personal liferichness in my personal life

3

OutlineOutline BackgroundBackground Examples of Organizational ControlExamples of Organizational Control

Distributed sensor networksDistributed sensor networks Distributed search in a peer-to-peer IRDistributed search in a peer-to-peer IR Multi-agent reinforcement learning for Multi-agent reinforcement learning for

distributed resource allocation distributed resource allocation

What are the Major Research TopicsWhat are the Major Research Topics SummarySummary

4

How to Construct Societies of How to Construct Societies of Sophisticated AI Systems that Sophisticated AI Systems that

Work Together EffectivelyWork Together Effectively

• Limited BandwidthLimited Bandwidth

• Lack of Global ViewLack of Global View

• Decentralized ControlDecentralized Control

• Autonomous, Autonomous, Asynchronous Asynchronous SubsystemsSubsystems

• Need for CooperationNeed for Cooperation

Why is this AI rather than Distributed Systems?

5

Why This Model for Building Why This Model for Building Intelligent Systems vs. Intelligent Systems vs. A Monolithic Approach?A Monolithic Approach?

Geographical Distribution of Information, Geographical Distribution of Information, Resources, Expertise Resources, Expertise Privacy in sharing information, fee-based servicesPrivacy in sharing information, fee-based services

Modularity for Ease of Development, Modularity for Ease of Development, Debugging, Modification, EvolutionDebugging, Modification, Evolution

Do we need to add in some picture of Do we need to add in some picture of an example system -- Coordinatorsan example system -- Coordinators

6

7

What is the Control Problem What is the Control Problem

Managing Interdependencies among Managing Interdependencies among

Agent ActivitiesAgent Activities What tasks to do, when, where, howWhat tasks to do, when, where, how

What information to communicate, What information to communicate, when, to whomwhen, to whom

How to do this in a globally optimal wayHow to do this in a globally optimal way

Difficulty

Limited communication and computational resources

Ubiquity of uncertainty – uncertain, out-of-date, incomplete information

8

A Model for A Model for Computation in the 21Computation in the 21stst

CenturyCentury

Scaling to 100’s to 1000’s of agentsScaling to 100’s to 1000’s of agents Complex organizational relationships Complex organizational relationships

among agentsamong agents Organizationally situated agentsOrganizationally situated agents

Operate in a “satisficing” modeOperate in a “satisficing” mode Managing uncertainty as an integral Managing uncertainty as an integral

part of network problem solvingpart of network problem solving Highly adaptive and reliableHighly adaptive and reliable

Network of cooperating, intelligent agents Network of cooperating, intelligent agents (people/machines)(people/machines)

9

What is the Lecture What is the Lecture AboutAbout

Organizational control as one way to Organizational control as one way to approach the scaling of AI Systemsapproach the scaling of AI Systems Organizational control is a multi-level Organizational control is a multi-level

approach in which long-term approach in which long-term organizational goals and roles are used as organizational goals and roles are used as guidelines for agents’ detailed operational guidelines for agents’ detailed operational decisionsdecisions. .

Presenting interesting research topics Presenting interesting research topics associated with organizational controlassociated with organizational control

10

Multi-Layer Control Multi-Layer Control Approach Approach

Organizational ControlOrganizational Control Global and long-term perspective Global and long-term perspective

on system performance on system performance Long-term (a-temporal) directivesLong-term (a-temporal) directives

Operational ControlOperational Control Limited and dynamic perspectiveLimited and dynamic perspective Short-term (temporal) decision in Short-term (temporal) decision in

the context of organizational the context of organizational directivesdirectives

ORGANIZATIONAL

GROUP

LOCAL

Approximate, Distributed Optimization for the Global Control Problem

Organizational ControlOrganizational Control

Organizational Design

Guidelines, Roles, Goals...etc.

• Communication and action spaces limited by roles

• Reward functions modified to prefer certain actions aids in coordination

• Global reward function broken into limited-locality reward components

MDP- based operational decisions

Organization-Aware Operational Control

MDP Agent

= Group-coordinatio

n action

Why Does Organizational Why Does Organizational Control WorkControl Work

Exploits Repetitive and Nearly-Exploits Repetitive and Nearly-Decomposable Nature of Problem SolvingDecomposable Nature of Problem Solving

Exploits Knowledge of the EnvironmentExploits Knowledge of the Environment Expectations of Task Arrivals, Problem-Solving Expectations of Task Arrivals, Problem-Solving

Behavior and OutcomesBehavior and Outcomes

13

Efficiency through Assumptions

14

Drivers for Organization FocusDrivers for Organization Focus“Bounded Rationality”“Bounded Rationality”

Organizational Control provides a framework Organizational Control provides a framework for dealing with computational issues of scalefor dealing with computational issues of scale

Decrease non-local information and reasoning Decrease non-local information and reasoning

necessarynecessary

Acting in accordance with guidelines leads to Acting in accordance with guidelines leads to effective coordination decisionseffective coordination decisions

Shift from an Agent-Centric, Operational Shift from an Agent-Centric, Operational View of Coordination to an Organization-View of Coordination to an Organization-

Centric OneCentric One

17

Example System Example System - 1- 1

Adaptive, Real-Time Adaptive, Real-Time Distributed Sensor Distributed Sensor

Network (2004)Network (2004)

(Bryan Horling, Roger Mailler, Regis (Bryan Horling, Roger Mailler, Regis Vincent)Vincent)

18

DARPA: Distributed Sensor DARPA: Distributed Sensor Network Challenge ProblemNetwork Challenge Problem•Small 2D Small 2D

Doppler radar Doppler radar units (30’s)units (30’s)– Scan Scan one of one of

threethree 120 120 sectors at a sectors at a timetime

• Commodity Commodity processor processor associated associated with each with each radarradar

•Communicate Communicate short short messages messages using one of 8 using one of 8 radio channelsradio channels

•Triangulate Triangulate radars to do radars to do trackingtracking

19

CASA - Monitoring for CASA - Monitoring for Severe Weather (2008)Severe Weather (2008)

Network of short-range (30 Network of short-range (30 km), overlapping, adaptive km), overlapping, adaptive weather-sensing radarsweather-sensing radars SSmall fielded system in mall fielded system in

OklahomaOklahoma

Goal: Detect low-lying weather Goal: Detect low-lying weather phenomena such as tornadoes phenomena such as tornadoes within 60 secondwithin 60 second

20

How to Control the DSNHow to Control the DSN

ScalabilityScalability: Hundreds of sensors, multiple : Hundreds of sensors, multiple targets, constrained communicationtargets, constrained communication

What if there were no (formal) organization?What if there were no (formal) organization? Who decides if a target is new?Who decides if a target is new? Who tracks a target?Who tracks a target? How do trackers obtain sensor information?How do trackers obtain sensor information?

These operational control decisions could be These operational control decisions could be made individually by each agent, but through made individually by each agent, but through organization can be made easierorganization can be made easier

21

DSN Organizational DSN Organizational ControlControl

Partitioned Partitioned EnvironmentEnvironment SectorsSectors Constrains info. Constrains info.

propagationpropagation Reduces Reduces

information loadinformation load Exploits localityExploits locality

Agents assigned Agents assigned rolesroles Sensor Sensor

(Scan/Track) (Scan/Track) Sector ManagerSector ManagerTrack ManagerTrack Manager

Limits sources of Limits sources of informationinformation

Facilitates data Facilitates data retrievalretrieval

22

Partitioning of NodesPartitioning of Nodes

• The environment is first partitioned into sectors.The environment is first partitioned into sectors.• Sector managers are then assignedSector managers are then assigned..

23

Competition for Sensor Competition for Sensor AgentsAgents

• Sector members send their capabilities to their managers.Sector members send their capabilities to their managers.• Each manager then generates and disseminates a scan schedule.Each manager then generates and disseminates a scan schedule.

24

SRTA: SRTA: Soft Real-Time Agent Soft Real-Time Agent

ArchitectureArchitecture Mapping Org and Mapping Org and

Dynamic Coordination Dynamic Coordination Guidelines into Guidelines into Operational DecisionsOperational Decisions

Guidelines into detailed Guidelines into detailed resource allocationsresource allocations

Resolve conflicts locally Resolve conflicts locally not resolvednot resolved

Resource Modeler

Conflict Resolution Module

Task Merging

Problem solver

Periodic Task Controller

TÆMS Library

Cache Check

DTC-Planner

Partial Order Scheduler

Parallel Execution Module

Learning

UpdateCache

CacheHit

Linear Plan

TAEMS-Plan Network/Objective

Goal Description/Objective

Parallel ScheduleScheduleFailure

Results

Update Expectations

Schedule Failure

OtherAgents

Schedule

Resource Uses

MultipleStructures

Negotiation(e.g. SPAM)

Commitments/Decommitments

Schedule failure/Abstract view

25

Track Manager SelectionTrack Manager Selection

• Nodes in the scan schedule perform scanning actions.• Detections reported to Manager and a Track Manager selected.

26

Managing Conflicted Managing Conflicted ResourcesResources

• Track Manager discovers and coordinates with tracking nodes.• New tracking tasks may conflict with existing tasks at the node.

27

Mediator ViewMediator ViewInterdependency GraphInterdependency Graph

SPAM: Mediation-Based Negotiation

M20M33

M0 M8

M7

M25M14

S15

S32

S53S18

S25,S20

S7

S18

S5 S8

S12, S22

S2, S14

M20M33

M8

M7

M25

1

1

11

2

1

1

1

World View- Multi-Linking of Resource Allocations

World View- Multi-Linking of Resource Allocations

29

What Does What Does Organizational Control Organizational Control

AccomplishAccomplish Managing Resource ContentionManaging Resource Contention

Sensors, processors, communicationSensors, processors, communication Centralizing Information in Sector Centralizing Information in Sector

ManagerManager Handling data correlation with multiple Handling data correlation with multiple

trackstracks Fault ToleranceFault Tolerance Communication Locality for TrackingCommunication Locality for Tracking

30

DeviationDeviation

(high = bad)(high = bad)

RMS ErrorRMS Error

(low = good)(low = good)

Organizational Trade-Organizational Trade-OffsOffs

How big should sectors be?How big should sectors be? Empirical evidence: between 5-10 Empirical evidence: between 5-10

sensorssensors This would vary, depending on sensor This would vary, depending on sensor

and environmental characteristicsand environmental characteristics

31

Example System Example System - 2- 2

Information Information Retrieval in a Peer-Retrieval in a Peer-

to-Peer Network to-Peer Network (2007)(2007)

(Haizheng Zhang, Bryan (Haizheng Zhang, Bryan Horling)Horling)

32

Information Retrieval Information Retrieval in a Peer-to-Peer in a Peer-to-Peer

NetworkNetwork

American Patent DB Wall Street JournalAssociated Press News

Insider trading Insider trading stories?stories?

A

B

C

D E

F

GI

H

J

K

Generate incrementallyand distributively an appropriateorganization for effective retrieval

Problem DescriptionProblem Description::

1. Improve IR performance2. System performance

GoalGoal:

33

Organization for Peer-to-Organization for Peer-to-Peer Content RetrievalPeer Content Retrieval

Initial and Initial and Unstructured Peer-to-Unstructured Peer-to-

Peer NetworkPeer Network

Nearly-Decomposable Nearly-Decomposable Hierarchy of Content Hierarchy of Content

MediatorsMediators

34

Content-Based Hierarchical Content-Based Hierarchical Agent OrganizationsAgent Organizations

Group agents of similar contentGroup agents of similar content Limit subset of agents to be probedLimit subset of agents to be probed Add lateral links to quickly locate diverse Add lateral links to quickly locate diverse

contentcontent

Incremental construction of the organization Incremental construction of the organization as new agents join networkas new agents join network

A two-phase search algorithmA two-phase search algorithm Locate relevant hierarchical agent clusters Locate relevant hierarchical agent clusters Perform searches in clustersPerform searches in clusters

35

Two-Phase Search ProtocolTwo-Phase Search Protocol

36

Internal Agent StructureInternal Agent Structure

Search Engine

DocumentCollection

Local Search

Agent Control Unit Agent

View

Neighbor 1

Neighbor 2

Neighbor n

...

QrsQrs

QlsQls

Q1Q1

Q2Q2

QnQn

Load Balance Unit

...

qqikik

qqjkjk

qqkk

qqkk

Resource Selection

Local Queries

qk1

qqknkn

qqk2k2

q'k1

q'q'knkn

q'q'k2k2

37

Experimental Results (Search Experimental Results (Search Quality versus Number of Quality versus Number of

Messages)Messages)(TREK 921 Nodes)(TREK 921 Nodes)

Message Number and Search Quality

0

500000

1000000

1500000

2000000

2500000

3000000

Categories

Categories

Mes

sage

Num

ber

0

102030

405060

708090

100

Sear

ch Q

uality Message Number

Search Quality

PBN PBY HTN HTY HBN HBY

Hierarchical BalancedWith Load Control

38

Example System Example System - 3- 3

Distributed Distributed Resource Allocation Resource Allocation for Computational for Computational

ServicesServices(2009)(2009)

(Chongjie Zhang, Sherief (Chongjie Zhang, Sherief Abdallah)Abdallah)

39

Integrate Organizational Integrate Organizational Control into Multi-Agent Control into Multi-Agent

LearningLearning Convergence in large-scale settings is Convergence in large-scale settings is

challenging — speed, likelihood and challenging — speed, likelihood and quality.quality.

Non-stationary Non-stationary learning learning environmentenvironment

Partial view and no Partial view and no global reward global reward signalsignal

Communication Communication delaydelay

40

Organization-Based Organization-Based Control FrameworkControl Framework

5

41

Integrate Supervisory Integrate Supervisory Information into Multi-Information into Multi-

Agent LearningAgent Learning

Policy Update

Action Selection

Policy

Reward Action State Policy Update

Action Selection

Policy

Reward Action State

Supervisory Policy

Adaptation

ReportGenerator

Rules andSuggestions

Adapted Policy

Abstracted state and reward

(a) Multi-Agent Reinforcement Learning (MARL) (b) MARL under Supervision

Reward

42

Experiments: Distributed Experiments: Distributed Task Allocation Problem Task Allocation Problem

(729 agents)(729 agents)

27 X 27 Agent Network

43

What Do These Examples What Do These Examples Tell UsTell Us

Organizational Control can be used in Organizational Control can be used in scaling of very different types of AI scaling of very different types of AI problem solvingproblem solving

Flexibility and adaptability of control Flexibility and adaptability of control decisions at all levels is importantdecisions at all levels is important

Very early in our understanding of how to Very early in our understanding of how to effectively exploit this approacheffectively exploit this approach

44

How to Create an How to Create an OrganizationOrganization

Top-DownTop-Down

Emergent / Emergent / Self-OrganizingSelf-Organizing

Some CombinationSome Combination

45

What Constitutes an What Constitutes an OrganizationOrganization

What is the Role of Institutional What is the Role of Institutional MechanismsMechanisms Computational artifacts for controlComputational artifacts for control

What Type of AgentsWhat Type of Agents CooperativeCooperative Self-interestedSelf-interested Semi-cooperativeSemi-cooperative

What is an Organizationally Situated AgentWhat is an Organizationally Situated Agent

46

Relationship between MAS and Relationship between MAS and Organizational Structuring from a Organizational Structuring from a business/sociological perspective?business/sociological perspective?

Are emotions effective computational Are emotions effective computational

mechanisms?mechanisms? Skepticism – limits effect of info distraction Skepticism – limits effect of info distraction Boredom – avoid over-learning of routine Boredom – avoid over-learning of routine

taskstasks Self-interest – decision making without Self-interest – decision making without

global impact global impact

MAS and Human MAS and Human OrganizationsOrganizations

47

Can you Automate the Can you Automate the Organizational Design Organizational Design

Process?Process? Theory behind organizational designTheory behind organizational design

The nature of sub-problem interdependenciesThe nature of sub-problem interdependencies

Designing for multi-attributed nature of Designing for multi-attributed nature of organizational performanceorganizational performance Reliability, fail-softness, adaptabilityReliability, fail-softness, adaptability

Predicting the performance of a Predicting the performance of a computational organizationcomputational organization

Specialness of the search process for Specialness of the search process for finding a good organizationfinding a good organization Repetitiveness of structureRepetitiveness of structure

48

The Human in the LoopThe Human in the Loop

How can computational organizations How can computational organizations be controlled by peoplebe controlled by people

How can human and computational How can human and computational organizations interactorganizations interact

What is the implication for how we see What is the implication for how we see ourselves and others ourselves and others

49

SummarySummary

Organizational Control is Organizational Control is important in how we think about important in how we think about scaling AI systemsscaling AI systems

Organizational Control is an Organizational Control is an intrinsically interesting problem intrinsically interesting problem that deserves our intellectual that deserves our intellectual attentionattention