university of waterloo - multi-agent influence diagramsklarson/teaching/f08-886/talks/... · 2008....
TRANSCRIPT
![Page 1: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/1.jpg)
Multi-Agent Influence Diagrams
CS 886: Multi-Agent Systems
“Multi-Agent influence diagrams for representing and solving games” [1]
Authored by D. Koller and B. Milch
Tyler Nowicki
Computer ScienceUniversity of Waterloo
![Page 2: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/2.jpg)
2
Why do we like MAIDs?
![Page 3: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/3.jpg)
3
Why do we like MAIDs?
Well.. these MAIDs won't make dinner,but
they do clean up our games!
MAID form games can be more compact and readable than an extensive form game.
![Page 4: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/4.jpg)
4
Road Example
2n agents own plots on a road that is being built. When the road reaches a pair of plots the agents put up 1 of 3 buildings.
MAID Form● # variables ~ 6n
Extensive Form● # nodes ~ 32n (leaves)
2 4 6 80
500
1000
1500
Agents
Siz
e
![Page 5: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/5.jpg)
5
Topics● Bayesian Networks
– Flow of influence● Influence Diagrams● Multi-Agent Influence Diagrams
– Utility & Nash Equilibrium● Finding a Nash Equilibrium● Advantages & Disadvantages● References & Questions
![Page 6: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/6.jpg)
6
Topics● Bayesian Networks
– Flow of influence● Influence Diagrams● Multi-Agent Influence Diagrams
– Utility & Nash Equilibrium● Finding a Nash Equilibrium● Advantages & Disadvantages● References & Questions
![Page 7: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/7.jpg)
7
Bayesian Networks
● Probabilistic graphical model of a● Set of variables x1,...,xn where● xi is restricted to a finite set dom(xi)
– Also called var(xi)● Arcs connect variables
– Arrows imply parent/descendant relationship
● Conditional Probability Distribution (CPD)– Given node X and its parents Pa(X)– Pr(X | Pa(X)) gives a distribution over the domain
of X for each parent.
![Page 8: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/8.jpg)
8
Example
Earthquake
Radio
Burglary
Alarm
Call
b0 | b1
0.99 | 0.01P(B)
a0 a1
c0 | c1
0.95 | 0.050.30 | 0.70
P(C|A)
a0 | a1
b0,e0
b0,e1
b1,e0
b1,e1
0.99 | 0.010.70 | 0.300.20 | 0.800.05 | 0.95
P(A|B,E)
e0 | e1
0.95 | 0.05P(E)
e0 e1
r0 | r1
0.99 | 0.010.65 | 0.35
P(R|E)
Reproduce of [1], Figure 1.
![Page 9: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/9.jpg)
9
Topics● Bayesian Networks
– Flow of influence● Influence Diagrams● Multi-Agent Influence Diagrams
– Utility & Nash Equilibrium● Finding a Nash Equilibrium● Advantages & Disadvantages● References & Questions
![Page 10: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/10.jpg)
10
Flow of Influence
● Influence flows through the BN● Flow is activated or blocked by observed
variables● When flow between two variables is blocked,
they are independent (aka d-separated)● Let E be evidence or observed variables
Only 3 Cases to Consider– Let x, y, z be variables
![Page 11: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/11.jpg)
11
Case 1Head-Tail Variable
● x → z → y– Active if z E, block otherwise.– Given z variables x and y become independent.
z E p(x,y,z) = p(x)p(z|x)p(y|z)z E p(x,y|z) = p(x|z)p(y|z)
x yz
![Page 12: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/12.jpg)
12
Case 2Tail-Tail Variable
● x ← z → y– Active if z E, block otherwise.– Given z variables x and y become independent.
z E p(x,y,z) = p(x|z)p(y|z)p(z)z E p(x,y|z) = p(x|z)p(y|z)
x y
z
![Page 13: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/13.jpg)
13
Case 3Head-Head Variable
● x → z ← y– Active if z or a descendant is in E , block
otherwise.– Given z variables x and y become dependent.– Without z, x and y are independent.
z E p(x,y,z) = p(x)p(y)z E p(x,y|z) = p(x)p(y)p(z|x,y)
x y
z
![Page 14: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/14.jpg)
14
Topics● Bayesian Networks
– Flow of influence● Influence Diagrams● Multi-Agent Influence Diagrams
– Utility & Nash Equilibrium● Finding a Nash Equilibrium● Advantages & Disadvantages● References & Questions
![Page 15: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/15.jpg)
15
Influence Diagrams
● Variables:– Decision (rectangle)– Chance (oval)– Utility (diamond)
● Decision variables are a choice (bet)– Filled in with a CPD and– Becomes a chance node.
● Chance variables are defined by the game– Cards, dice, lady luck.
● Utility variables do not have children– Payoff is defined by the game.
![Page 16: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/16.jpg)
16
Topics● Bayesian Networks
– Flow of influence● Influence Diagrams● Multi-Agent Influence Diagrams
– Utility & Nash Equilibrium● Finding a Nash Equilibrium● Advantages & Disadvantages● References & Questions
![Page 17: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/17.jpg)
17
Multi-Agent Influence Diagrams
Each players decisions and utilities are specified in the same game.
ExampleAlice reasons about her building plans.
![Page 18: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/18.jpg)
18
Tree Killer Example
![Page 19: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/19.jpg)
19
MAID Terminology
● Given the MAID.● A strategy profile is a set of CPD for a set
of decision variables.● Applying to = [] results in chance
variables for the strategy profile.● A strategy is fully mixed if the strategy
defines a CPD for each decision variable.
![Page 20: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/20.jpg)
20
Topics● Bayesian Networks
– Flow of influence● Influence Diagrams● Multi-Agent Influence Diagrams
– Utility & Nash Equilibrium● Finding a Nash Equilibrium● Advantages & Disadvantages● References & Questions
![Page 21: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/21.jpg)
21
Calculating Utility
● Where Ua be the set of utility variables for
agent a.
● Goal is to select an optimal to maximize utility for agent a.
EU a=∑U ∈U a
∑u∈dom U
PM [ ]U=u⋅u
![Page 22: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/22.jpg)
22
Nash Equilibrium
● Let ε be a● Let be a partial strategy profile over ε● is optimal if for all partial strategies '
EU a− ,EU a− , '
![Page 23: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/23.jpg)
23
Computing NE?
Exponential blow-up prevents us from simply turn the MAID into a extensive form game.
Lets look at that flow again.
![Page 24: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/24.jpg)
24
Topics● Bayesian Networks
– Flow of influence● Influence Diagrams● Multi-Agent Influence Diagrams
– Utility & Nash Equilibrium● Finding a Nash Equilibrium● Advantages & Disadvantages● References & Questions
![Page 25: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/25.jpg)
25
Finding a Nash Equilibrium
Basic Idea
● Transform the MAID into a relevance graph.● Pick an arbitrary fully mixed strategy.● Compute the optimal strategy for each
decision rule in the relevance graph according to the topological ordering.
● Strategy has been optimized!
![Page 26: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/26.jpg)
26
Relevance Graph
● What is a topological order?● How do we construct the relevance graph?
![Page 27: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/27.jpg)
27
Constructing aRelevance Graph
● Consists of decision nodes of MAID .● Edges are formed only when two nodes are
strategically reachable in .● Is Acyclic!● Topological ordering (or ancestral ordering)
is derived from the relevance graph.
![Page 28: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/28.jpg)
28
Constructing aRelevance Graph
● For each variable – For all other variables '
● Determine if ' is s-reachable from ● Then add edge ' to the graph
● Or Shachter's Bayes-Ball runs in linear time.
![Page 29: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/29.jpg)
29
s-reachable
● Let U be any descendant utility variable of . ● Let * be a virtual parent of ' ● ' is s-reachable from if there exists an
active path from * to U given and Pa().
![Page 30: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/30.jpg)
30
Topics● Bayesian Networks
– Flow of influence● Influence Diagrams● Multi-Agent Influence Diagrams
– Utility & Nash Equilibrium● Finding a Nash Equilibrium● Advantages & Disadvantages● References & Questions
![Page 31: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/31.jpg)
31
Advantages & Disadvantages
How does the requirement for a topological ordering affect the usefulness of this
method?
![Page 32: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/32.jpg)
32
Advantages & Disadvantages
Influence diagram must be acyclic● Single-player perfect recall games● Multi-player perfect information games● But only some imperfect information games!
![Page 33: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/33.jpg)
33
Advantages & Disadvantages
● Games with cycles must be separated into a set of strongly connected components.
● Compact, readable, intuitive graphs.● Requires care when designing the graph.● Finds one Nash Equilibrium
– Left up to reader to find multiple.
![Page 34: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/34.jpg)
34
Topics● Bayesian Networks
– Flow of influence● Influence Diagrams● Multi-Agent Influence Diagrams
– Utility & Nash Equilibrium● Finding a Nash Equilibrium● Advantages & Disadvantages● References & Questions
![Page 35: University of Waterloo - Multi-Agent Influence Diagramsklarson/teaching/F08-886/talks/... · 2008. 9. 23. · 35 References 1.D. Koller and B. Milch, Multiagent influence diagrams](https://reader035.vdocument.in/reader035/viewer/2022070218/6124b2f91252d44ced116cd6/html5/thumbnails/35.jpg)
35
References
1. D. Koller and B. Milch, Multiagent influence diagrams for representing and solving games, Games and Economic Behavior, 45(1), p 181-221, 2003.
2. Mudgal, C., Vassileva, J. An Influence Diagram Model for Multi-Agent Negotiation, Proceedings of the Fourth International Conference on MultiAgent Systems (ICMAS-2000), p 451-452, 2000.
3. D. Koller, Structured models of complex decision problems, Invited talk at the First International Congress of the Game Theory Society, 2000.(http://robotics.stanford.edu/~koller/)
4. Sargur Srihari, Lecture Notes on Graphical Models, CSE 574, University at Buffalo, Fall 2007 & 2008.(http://www.cedar.buffalo.edu/~srihari/CSE574/)