@let@token on independence and factorization - logic.at fileon independence and factorization radim...

On Independence and Factorization

Radim JirousekInstitute of Information Theory and Automation

Academy of Sciences of the Czech Republic

DIPLEAP

November 26-28, 2010

Radim Jirousek Independence and Factorization

What does the independence really means andhow it is modeled in different uncertainty calculi?

Probabilistic Independence – Events

Consider two events A and B.

They are independent if

P(A ∩ B) = P(A) · P(B).

Is this definition correct?

Does this definition comply with our intuition?

Both answers are: not exactly

P(A ∩ B) = P(A) · P(B).

What is intuitive on application of multiplication? Why not touse another T-norm?

Is it O.K. when it may happen that an event is independent ofits complement (negation)?

P(A ∩ B) = P(A) · P(B).

O.K. Let us use another definition (Coletti, Scozzafava):

Definition

Events A and B are independent if

0 < P(A|B) = P(A) < 1.

This definition is intuitive.This definition introduces asymmetric relation! Is it O.K.?

Definition

Event A is independent of event B if

0 < P(A|B) = P(A) < 1.

``````

Definition

0 < P(A|B) = P(A) < 1.

Definition

0 < P(A|B) = P(A) < 1.

``````

Definition

0 < P(A|B) = P(A) < 1.

This definition is intuitive.

This definition introduces asymmetric relation! Is it O.K.?

Definition

0 < P(A|B) = P(A) < 1.

``````

Definition

0 < P(A|B) = P(A) < 1.

This definition is intuitive.This definition introduces asymmetric relation!

Is it O.K.?

Definition

0 < P(A|B) = P(A) < 1.

``````

Definition

0 < P(A|B) = P(A) < 1.

Definition

0 < P(A|B) = P(A) < 1.

``````

Definition

0 < P(A|B) = P(A) < 1.

Definition

0 < P(A|B) = P(A) < 1.

``````

Example

P(B|A) = P(B) = 13

P(A|B) = P(A) = 1

This is a coherent system of conditional probabilities:

B 13 0

B 23 0

Example

P(B|A) = P(B) = 13

P(A|B) = P(A) = 1

This is a coherent system of conditional probabilities:

B 13 0

B 23 0

What can be deduced from the independence A ⊥⊥ B?

P(A|B) = P(A)

Since P(A|B) · P(B) = P(A ∩ B) we get

P(A) · P(B) = P(A ∩ B).

It means that the product in the generally used definition isbecause we use division (for normalization) in the definition ofconditional probability.

P(A|B) = P(A)

P(A) · P(B) = P(A ∩ B).

P(A|B) = P(A)

P(A) · P(B) = P(A ∩ B).

Probabilistic Independence – Variables

Variables X ... valuesY ... values

Definition

Variable X is independent of variable Y if for all couples i , j

0 < P(X = ai |Y = bj) = P(X = ai ) < 1.

So we get again that if X ⊥⊥ Y then for all couples i , j

P(X = ai ,Y = bj) = P(X = ai ) · P(Y = bj).

Definition

0 < P(X = ai |Y = bj) = P(X = ai ) < 1.

Definition

0 < P(X = ai |Y = bj) = P(X = ai ) < 1.

Probabilistic Independence – X ⊥⊥ Y'

× × × ×

P(X ,Y )

Necessary condition for X ⊥⊥ Y :

Each value of P(X ,Y ) can be computed from the respectivevalues of P(X ) and P(Y ).

× × × ×

P(X ,Y )

× × × ×

P(X ,Y )

Probabilistic Conditional Independence

Events:

Definition

Event A is conditionally independent of event B given event C if

0 < P(A|B,C ) = P(A|C ) < 1.

Variables:

Definition

Variable X is conditionally independent of variable Y givenvariable Z if for all combinations of values (ai , bj , ck)

0 < P(X = ai |Y = bj ,Z = ck) = P(X = ai |Z = ck) < 1.

Events:

Definition

Event A is conditionally independent of event B given event C if

0 < P(A|B,C ) = P(A|C ) < 1.

Variables:

Definition

Variable X is conditionally independent of variable Y givenvariable Z if for all combinations of values (ai , bj , ck)

0 < P(X = ai |Y = bj ,Z = ck) = P(X = ai |Z = ck) < 1.

Variables:

Factorization lemma

X ⊥⊥ Y |Z iff there exist functions φ and psi such that for allcombinations of values (ai , bj , ck)

P(X = ai ,Y = bj ,Z = ck) = φ(ai , ck) · ψ(bj , ck)

and0 < P(X = ai |Z = ck) < 1.

Pros: Each value of P(X ,Y ,Z ) can be computed from the respec-tive values of marginal distributions P(X ,Z ) and P(Y ,Z ).

Cons: We exclude situations when the probability distributions de-scribe a logical dependence.

Variables:

Factorization lemma

and0 < P(X = ai |Z = ck) < 1.

Variables:

Factorization lemma

and0 < P(X = ai |Z = ck) < 1.

Variables:

Factorization lemma

and0 < P(X = ai |Z = ck) < 1.

What is the difference betweenconditional independence and factorization?

Factorization is a necessary conditionfor conditional independence.

What is the difference betweenconditional independence and factorization?

Factorization is a necessary conditionfor conditional independence.

Probabilistic Graphical Models

Definition

We say that probability distribution π is graphical with graph G ifit factorizes with respect to C1,C2, . . . ,Cm, i.e. if there existnonnegative functions φ1, φ2, . . . , φm such that

π(x1, x2, . . . , xn) = φ1(xii∈C1) · φ2(xii∈C2) · . . . · φm(xii∈Cm).

4 3π(x1, x2, x3, x4) = φ1(x1, x2) · φ2(x2, x3)

·φ3(x3, x4) · φ4(x1, x4).

Definition

We say that probability distribution π is a Bayesian network withDAG G if it factorizes with respect to G , i.e. if

π(x1, x2, . . . , xn) =n∏

π(xi |xjj∈pa(i)

which is equivalent tofor all i = 2, . . . , n (topological ordering)

π↓1,...,i factorizes with respect to 1, . . . , i − 1, (pa(i) ∪ i).

Definition

We say that probability distribution π is a Bayesian network withDAG G if it factorizes with respect to G , i.e. if

π(x1, x2, . . . , xn) =n∏

π(xi |xjj∈pa(i)

which is equivalent tofor all i = 2, . . . , n (topological ordering)

π↓1,...,i factorizes with respect to 1, . . . , i − 1, (pa(i) ∪ i).

How to extend these definitions into D-S theory?

Simple D-S factorization - unconditional case'

Basicassignment

&%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!#

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

However

2N × 2M << 2N×M

Basicassignment

&%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!#

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

However

2N × 2M << 2N×M

Basicassignment

&%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!#

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

However

2N × 2M << 2N×M

Basicassignment

&%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!#

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

However

2N × 2M << 2N×M

Basicassignment

&%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!#

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

However

2N × 2M << 2N×M

Basicassignment

&%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!# &%'$"!#

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Factorization in Dempster-Shafer theory

Frame of discernment Ω = X1 × X2;

Basic assignmentm : 2X1×X2 −→ [0, 1]

Definition - simple disjoint factorization

m factorizes with respect to 1, 2 if there exist set functions

φ : 2X1 −→ [0,+∞],

ψ : 2X2 −→ [0,+∞],

such that for all A ⊆ X1 × X2

m(A) =

φ(A↓1) · ψ(A↓2) if A = A↓1 × A↓2

0 otherwise.

Frame of discernment Ω = X1 × X2;

Basic assignmentm : 2X1×X2 −→ [0, 1]

Definition - simple disjoint factorization

m factorizes with respect to 1, 2 if there exist set functions

φ : 2X1 −→ [0,+∞],

ψ : 2X2 −→ [0,+∞],

such that for all A ⊆ X1 × X2

m(A) =

φ(A↓1) · ψ(A↓2) if A = A↓1 × A↓2

0 otherwise.

Simple D-S factorization - unconditional case

Basicassignment

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

m(A) = φ(A↓X ) · ψ(A↓Y )

Simple D-S factorization - unconditional case

Basicassignment

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

m(A) = φ(A↓X ) · ψ(A↓Y )

m(A′) = 0

Example

Table: Basic assignments m1 and m2.

A ⊆ X1 φ(A) A ⊆ X2 ψ(A)

a 0.2 b 0.6

a 0.3 b 0

aa 0.5 ab 0.4

Example

ab a ⊗ b 0.12

ab a ⊗ b 0

ab a ⊗ b 0.18

ab a ⊗ b 0

ab, ab a ⊗ X2 0.08

ab, ab X1 ⊗ b 0.3

ab, ab 0

ab, ab X1 ⊗ b 0

ab, ab a ⊗ X2 0.12

ab, ab, ab 0

ab, ab, ab, ab X1 ⊗ X2 0.2

Example

ab a ⊗ b 0.12

ab a ⊗ b 0

ab a ⊗ b 0.18

ab a ⊗ b 0

ab, ab a ⊗ X2 0.08

ab, ab X1 ⊗ b 0.3

ab, ab 0

ab, ab X1 ⊗ b 0

ab, ab a ⊗ X2 0.12

ab, ab, ab 0

ab, ab, ab, ab X1 ⊗ X2 0.2

Example

ab a ⊗ b 0.12

ab a ⊗ b 0

ab a ⊗ b 0.18

ab a ⊗ b 0

ab, ab a ⊗ X2 0.08

ab, ab X1 ⊗ b 0.3

ab, ab 0

ab, ab X1 ⊗ b 0

ab, ab a ⊗ X2 0.12

ab, ab, ab 0

ab, ab, ab, ab X1 ⊗ X2 0.2

Example

ab a ⊗ b 0.12

ab a ⊗ b 0

ab a ⊗ b 0.18

ab a ⊗ b 0

ab, ab a ⊗ X2 0.08

ab, ab X1 ⊗ b 0.3

ab, ab 0

ab, ab X1 ⊗ b 0

ab, ab a ⊗ X2 0.12

ab, ab, ab 0

ab, ab, ab, ab X1 ⊗ X2 0.2

Example

ab a ⊗ b 0.12

ab a ⊗ b 0

ab a ⊗ b 0.18

ab a ⊗ b 0

ab, ab a ⊗ X2 0.08

ab, ab X1 ⊗ b 0.3

ab, ab 0

ab, ab X1 ⊗ b 0

ab, ab a ⊗ X2 0.12

ab, ab, ab 0

ab, ab, ab, ab X1 ⊗ X2 0.2

Example

ab a ⊗ b 0.12

ab a ⊗ b 0

ab a ⊗ b 0.18

ab a ⊗ b 0

ab, ab a ⊗ X2 0.08

ab, ab X1 ⊗ b 0.3

ab, ab 0

ab, ab X1 ⊗ b 0

ab, ab a ⊗ X2 0.12

ab, ab, ab 0

ab, ab, ab, ab X1 ⊗ X2 0.2

Example

ab a ⊗ b 0.12

ab a ⊗ b 0

ab a ⊗ b 0.18

ab a ⊗ b 0

ab, ab a ⊗ X2 0.08

ab, ab X1 ⊗ b 0.3

ab, ab 0

ab, ab X1 ⊗ b 0

ab, ab a ⊗ X2 0.12

ab, ab, ab 0

ab, ab, ab, ab X1 ⊗ X2 0.2

Example

ab a ⊗ b 0.12

ab a ⊗ b 0

ab a ⊗ b 0.18

ab a ⊗ b 0

ab, ab a ⊗ X2 0.08

ab, ab X1 ⊗ b 0.3

ab, ab 0

ab, ab X1 ⊗ b 0

ab, ab a ⊗ X2 0.12

ab, ab, ab 0

ab, ab, ab, ab X1 ⊗ X2 0.2

Frame of discernment Ω = X1 × X2 × . . .× Xn;K , L ⊆ 1, 2, . . . , nBasic assignment m : 2X1×...×Xn −→ [0, 1]

Definition - simple factorization

m factorizes with respect to K , L if there exist set functions

φ : 2XK −→ [0,+∞],

ψ : 2XL −→ [0,+∞],

such that for all A ⊆ XK∪L

m↓K∪L(A) =

φ(A↓K ) · ψ(A↓K ) if A = A↓K ⊗ A↓L

0 otherwise.

Frame of discernment Ω = X1 × X2 × . . .× Xn;K , L ⊆ 1, 2, . . . , nBasic assignment m : 2X1×...×Xn −→ [0, 1]

Definition - simple factorization

m factorizes with respect to K , L if there exist set functions

φ : 2XK −→ [0,+∞],

ψ : 2XL −→ [0,+∞],

such that for all A ⊆ XK∪L

m↓K∪L(A) =

φ(A↓K ) · ψ(A↓K ) if A = A↓K ⊗ A↓L

0 otherwise.

Extension - Join of sets

Definition

A ⊆ XK , B ⊆ XL.

A⊗ B = x ∈ XK∪L : x↓K ∈ A & x↓L ∈ B.

Generally:C ↓K∪L ⊆ C ↓K ⊗ C ↓L.

Extension - Join of sets

Definition

A ⊆ XK , B ⊆ XL.

A⊗ B = x ∈ XK∪L : x↓K ∈ A & x↓L ∈ B.

Generally:C ↓K∪L ⊆ C ↓K ⊗ C ↓L.

Factorization lemma in Dempster-Shafer theory

Let K , L ⊆ 1, 2, . . . , n. Basic assignment

m : 2X1×...×Xn −→ [0, 1]

factorizes with respect to K , L iff

XK\L ⊥⊥ XL\K |XK∩L.

Bayesian network in Dempster-Shafer theory

Frame of discernment Ω = X1 × X2 × . . .× Xn;

Acyclic directed graph (DAG) G = (1, 2, . . . , n,E ) (topologicalordering of nodes);

Basic assignment m : 2Ω −→ [0, 1]

Definition - Bayesian network

m is a Bayesian network with graph G if for all i = 2, 3, . . . , nmarginal basic assignment m↓1,2,...,i simply factorizes withrespect to i = 1, 2, . . . , i − 1, (pa(i) ∪ i).

Graphical models in Dempster-Shafer theory

Simple undirected graph G = (1, 2, . . . , n,E ) with cliquesC1,C2, . . . ,Cm;

Definition - graphical model

m is graphical with graph G if there exist set functionsφ1, φ2, . . . , φm such that for all A ⊆ Ω

m(A) =

φ1(A↓C1) · φ2(A↓C2) · . . . · φm(A↓Cm) if A ∈ R(G )

0 otherwise,

R(G ) denotes the set of subsets of Ω complying with G .

Simple undirected graph G = (1, 2, . . . , n,E ) with cliquesC1,C2, . . . ,Cm;

Definition - graphical model

m is graphical with graph G if there exist set functionsφ1, φ2, . . . , φm such that for all A ⊆ Ω

m(A) =

φ1(A↓C1) · φ2(A↓C2) · . . . · φm(A↓Cm) if A ∈ R(G )

0 otherwise,

R(G ) denotes the set of subsets of Ω complying with G .

Sets complying with G

Definition of R(G )

R(G ) = A ⊆ Ω : ∀(L,M) ∈ S(G ) (A = A↓L ⊗ A↓M),

where S(G ) is a set of all couples (L,M) such that

L ∪M = 1, 2, . . . , n;L ∩M is a minimal separating set in G

each pair of nodes i ∈ L \M, j ∈ M \ L is separated by L ∩Min G .

Definition of R(G )

R(G ) = A ⊆ Ω : ∀(L,M) ∈ S(G ) (A = A↓L ⊗ A↓M),

where S(G ) is a set of all couples (L,M) such that

L ∪M = 1, 2, . . . , n;L ∩M is a minimal separating set in G

each pair of nodes i ∈ L \M, j ∈ M \ L is separated by L ∩Min G .

R(G ) = A ∈ X1 × X2 × X3 : A = A↓1,2 ⊗ A↓1,2.

4 3 R(G ) = A ∈ X1 × X2 × X3 × X4 :

A = A↓1,2,3 ⊗ A↓1,3,4 &

A = A↓1,2,4 ⊗ A↓2,3,4.

5R(G ) = A ∈ X1 × X2 × X3 × X4 × X5 :

A = A↓1,2,3⊗A↓1,3,4⊗A↓2,3,5.

R(G ) = A ∈ X1 × X2 × X3 : A = A↓1,2 ⊗ A↓1,2.

4 3 R(G ) = A ∈ X1 × X2 × X3 × X4 :

A = A↓1,2,3 ⊗ A↓1,3,4 &

A = A↓1,2,4 ⊗ A↓2,3,4.

5R(G ) = A ∈ X1 × X2 × X3 × X4 × X5 :

A = A↓1,2,3⊗A↓1,3,4⊗A↓2,3,5.

R(G ) = A ∈ X1 × X2 × X3 : A = A↓1,2 ⊗ A↓1,2.

4 3 R(G ) = A ∈ X1 × X2 × X3 × X4 :

A = A↓1,2,3 ⊗ A↓1,3,4 &

A = A↓1,2,4 ⊗ A↓2,3,4.

5R(G ) = A ∈ X1 × X2 × X3 × X4 × X5 :

A = A↓1,2,3⊗A↓1,3,4⊗A↓2,3,5.

Theorem on an independence structure of Bayesian networks

Let G = (V ,E ) be a DAG in which nodes i and j are d-separatedby a set C . If basic assignment m is a Bayesian network withgraph G then

xi ⊥⊥ xj | xC [m].

Theorem on an independence structure of graphical models

Let G = (V ,E ) be a simple graph in which disjoint (nonempty)subsets of nodes A and B are separated by C ⊂ V . If basicassignment m is graphical with G then

xA ⊥⊥ xB | xC [m].

Theorem on an independence structure of Bayesian networks

Let G = (V ,E ) be a DAG in which nodes i and j are d-separatedby a set C . If basic assignment m is a Bayesian network withgraph G then

xi ⊥⊥ xj | xC [m].

Theorem on an independence structure of graphical models

Let G = (V ,E ) be a simple graph in which disjoint (nonempty)subsets of nodes A and B are separated by C ⊂ V . If basicassignment m is graphical with G then

xA ⊥⊥ xB | xC [m].

Thank you for your attention

@let@token on independence and factorization - logic.at fileon independence and factorization radim...

Documents