building bayesian networksmarinav/teaching/bdminai/bdmaibuildbn.pdf · dynamic bayesian network for...

Building Bayesian Networks

Lecture3: Building BN – p.1

The focus today . . .

Problem solving by Bayesian networks

Designing Bayesian networksQualitative part (structure)Quantitative part (probability assessment)

Simplified Bayesian networksIn structure: Naïve Bayes, Tree-AugmentedNetworksIn probability assessment: Parent divorcing, CausalIndependence


Problem solving

Bayesian networks: a declarative (knowing what)knowledge-representation formalism, i.e.,:

mathematical basis

problem to be solved determined by (1) enteredevidence E (including potential decisions); (2) givenhypothesis H : P (H | E)

Examples:

Description of population (or prior information)

Classification and diagnosis: D = arg maxH P (H | E) i.e.D is the hypothesis with maximum P (H | E)

Prediction

Decision making based on what-if scenario’s


Prior information

Gives description of the population on which the assessedprobabilities are based, i.e., the original probabilities before newevidence is uncovered

Marginal probabilities P (V ) for every vertex V , e.g.,P (WILSON’S DISEASE = yes)


Diagnostic problem solving

Gives description of the subpopulation of the original population orindividual cases

Marginal probabilities P ∗(V ) = P (V | E) for every vertex V , e.g.,P (WILSON’S DISEASE = yes | E) for entered evidence E (red vertices,with probability for one value equal to 1)


Prediction of associated findings

Gives description of the findings associated with a given class orcategory, such as Wilson’s disease

Marginal probabilities P ∗(V ) = P (V | E) for every vertex V , e.g.,P (Kayser-Fleischer Rings = yes | E) with E evidence


Design of Bayesian network

Current design principle: start modelling qualitatively(different from traditional knowledge-based systems)

causalgraph

qualitativeprobabilistic network

quantitativenetwork

Bayesiannetwork

refinement

variables/relationships

domains ofvariables/qualitative

probabilisticinformation

numericalassessment

experts/dataset

evaluation


Terminology

NOYES

FLU

noyes

FEVER

noyes

SARS

noyes

VisitToChina

noyes

DYSPNOEA

<=37.5>37.5

TEMP

Parent SARS of Child FEVER

SARS is Ancestor of TEMP

DYSPNOEA is Descendant of VisitToChina

Query node, e.g., FEVER

Evidence, e.g., VisitToChina and TEMP

Markov blanket, e.g.,for SARS: {VisitToChina,DYSPNOEA,FEVER,FLU}


Causal graph: Topology (structure)

cause1

causen

... effect

Identify factors that are relevant

Determine how those factors are causally related toeach other

The arc cause→ effect does mean that cause is a factorinvolved in causing effect


Causal graph: Common effect

cause1

causen

... effect

An effect that has two or more ingoing arcs from othervertices is a common effect of those causes

Kinds of causal interactionSynergy: POLUTION −→ CANCER←− SMOKING

Prevention: VACCINE −→ DEATH←− SMALLPOX

XOR: ALKALI −→ DEATH←− ACID


Causal graph: Common cause

cause

effect1

effectn

...

A cause that has two or more outgoing arcs to othervertices is a common cause (factor) of those effects

The effects of a common cause are usually observables(e.g. manifestations of failure of a device or symptomsin a disease)


Causal graph: Example

FLU

PNEUMONIA

FEVER

MYALGIA

TEMP

FEVER and PNEUMONIA are two alternative causes offever (but may enhance each other)

FLU has two common effects: MYALGIA and FEVER

High body TEMPerature is an indirect effect of FLU andPNEUMONIA, caused by FEVER


Check independence relationship

FLU

PNEUMONIA

FEVER

MYALGIA

TEMP

Conditional independence: X ⊥⊥ Y | Z

{FEVER} ⊥⊥ {MYALGIA} | {FLU}

? | {FEVER}

{PNEUMONIA} ⊥⊥ {FLU} | ?{PNEUMONIA} 6⊥⊥ {FLU} | {FEVER}


Choose variables

Factors are mutually exclusive (cannot occur togetherwith absolute certainty): put as values in the samevariable, or

Factors may co-occur: multiple variables

DISEASEpneu/flu

FEVER

MYALGIA

(a) Single variable

FLUyes/no

FEVER

PNEUMONIAyes/no

MYALGIA

(b) Multiple variables


Choose values

Discrete valuesMutually exclusive and exhaustiveTypes:

binary, e.g., FLU = yes/no, true/false, 0/1ordinal, e.g., INCOME = low, medium, highnominal, e.g., COLOR = brown, green, redintegral, e.g., AGE = {1, . . . , 120}

Continuous values

Discretization (of continuous values)Example for TEMP:[−50,+5)→ cold[+5,+20)→ mild[+20,+50]→ hot


Probability assessment

qualitativeorders andequalities

indegreeof vertex

divorcing/causalindependence

quantitativeassessment

experts/dataset

probabilitydistribution

compareprediction with

literature

compare withtest dataset

checkconsistency

> 4

≤ 4

EVALUATION

8

>

>

>

>

>

>

>

>

>

>

>

>

>

<

>

>

>

>

>

>

>

>

>

>

>

>

>

:


Expert judgements

Qualitative probabilities:Qualitative orders:

AGE P (General Health Status | AGE)

10-69 good > average > poor70-79 average > good > poor80-89 average > poor > good≥ 90 poor > average > good

Equalities:

P (CANCER = T1|AGE = 15− 29) =P (CANCER = T2|AGE = 15− 29)


Expert judgements (cont.)

Quantitative, subjective probabilities:

P (GHS | AGE)

AGE good average poor

10-69 0.99 0.008 0.00270-79 0.3 0.5 0.280-89 0.1 0.5 0.4≥ 90 0.1 0.3 0.6


A bottleneck in Bayesian networks

FEV ER

FLU

RABIES EARINF

BRONCH

The number of parameters for the effect given n causesgrows exponentially: ≥ 2n (for binary causes)

Unlikely evidence combination:P (fever|flu, rabies, ear_infection) =?

Problem: for many BNs too many probabilities have to beassessed


Special form Bayesian networks

Solution: use simpler probabilistic model, such that either

the structure becomes simpler, e.g.,

naive (independent) form BNT ree-Augmented Bayesian Network (TAN)

or,

the assessment of the conditional probabilities becomessimpler (even though the structure is still complex), e.g.,

parent divorcingcausal independence BN


Independent (Naive) form BN

C

E1

· · ·E2

Em

C is a class variable

Ei are evidence variables and E ⊆ {E1, . . . , Em}. Wehave Ei ⊥⊥ Ej | C, for i 6= j. Hence, using Bayes’ rule:

P (C | E) =P (E | C)P (C)

P (E)with:

P (E | C) =∏

E∈E

P (E | C) by cond. ind.

P (E) =∑

C

P (E | C)P (C) marg. & cond.


Example of Naive Bayes


Example of Naive Bayes (1)

Learning phase

Rain

Overcast

Sunny

NoYesOutlook

Rain

Overcast

Sunny

NoYesOutlook

Cool

Mild

Hot

NoYesTemperature

Cool

Mild

Hot

NoYesTemperature

Normal

High

oYesHumidity

Normal

High

oYesHumidity

Weak

Strong

NoYesWind

Weak

Strong

NoYesWind

P =Yes) = P =No) =



Testing phase (inference)

Play Tennis

Outlook Temp Humidity Wind

Play Tennis


Play Tennis




Testing phase (inference)

Play Tennis


P Yes x P x Ye P Yes x ∝∝ Sunny es Cool Yes High es Strong Yes Yes

0.0053

No x ∝ Sunny o Cool o High No Strong No No 0.0206

P Yes x P No x x PlayTennis No

Note

P Yes x P Yes x P Yes x P No x

Play Tennis


Play Tennis


P Yes x P x Ye P Yes x ∝∝ Sunny es Cool Yes High es Strong Yes Yes

0.0053

No x ∝ Sunny o Cool o High No Strong No No 0.0206

P Yes x P No x x PlayTennis No

Note

P Yes x P Yes x P Yes x P No xLecture3: Building BN – p.25

Tree-Augmented BN (TAN)

C

E1

E3E2

E5

E4

Extension of Naive Bayes: reduce the number ofindependent assumptions

Each node has at most two parents (one is the classnode)


Divorcing multiple parents

Surgery

DrugTreatment

GeneralHealth

Survival

(a) Original network

Surgery

DrugTreatment

GeneralHealth

Post-therapySurvival

Survival

(b) Divorced network

Reduction in number of probabilities to assess:

Identify a potential common effect of two or more parentvertices of a vertex

Introduce a new variable into the network, representingthe common effect


Causal Independence

C1 C2 . . . Cn

I1 I2 . . . In

Ef

with:

cause variables Cj, intermediate variables Ij , and theeffect variable E

P (E | I1, . . . , In) ∈ {0, 1}

interaction function f , defined such that

f(I1, . . . , In) =

{

e if P (e | I1, . . . , In) = 1

¬e if P (e | I1, . . . , In) = 0


Causal independence: Noisy OR

C1 C2

I1 I2

EOR

Interactions among causes, as represented by thefunction f and P (E | I1, I2), is a logical OR

Meaning: presence of any one of the causes Ci withabsolute certainty will cause the effect e (i.e. E = true)

P (e|C1, C2) = ?


Causal independence: Noisy OR (cont.)

C1 C2

I1 I2

EOR

P (e|C1, C2) =∑

I1,I2

P (e|I1, I2, C1, C2)P (I1, I2|C1, C2)

= ?



C1 C2

I1 I2

EOR

I1 I2 P (e | I1, I2) f(I1, I2) = I1 ∨ I2

0 0 0 ⇒ ¬e

0 1 1 ⇒ e

1 0 1 ⇒ e

1 1 1 ⇒ e

P (e|C1, C2) =X

f(I1,I2)=e

P (e|I1, I2)Y

k=1,2

P (Ik|Ck)

= P (i1|C1)P (i2|C2) + P (¬i1|C1)P (i2|C2) + P (i1|C1)P (¬i2|C2)


Noisy OR: Real-world example

Dynamic Bayesian network for predicting the development ofhypertensive disorders during pregnancy

VascRisk (Vascular risk) has 11 causes and its original CPTrequires the estimation of 20736!!! entries. Practically impossible!

Solution: use noisy OR to simplify it


Causal independence: Noisy AND

C1 C2

I1 I2

EAND

Interactions among causes, as represented by thefunction f and P (E | I1, I2), is a logical AND

Meaning: presence of all causes Ci with absolutecertainty will cause the effect e (i.e. E = true);otherwise, ¬e

P (e|C1, C2) = ?


Are Bayesian networks always suitable?

“Essentially, all models are wrong but some are useful”George Box, Norman Draper (1987),

Empirical Model-Building and Response Surfaces, Wiley

Problem (modelling) objective, e.g., for functionapproximation or pure numeric prediction without aneed to explain the results a “black box” model such asneural networks can be sufficient

Sufficient knowledge about the problem (domainexperts, literature, data)

Complexity of the problem e.g., is it decomposable


Refining causal graphs

Model refinement is necessary.

How:ManualAutomatic

What:Probability adjustmentRemoving irrelevant factorsAdding previously hidden, unknown factorsCausal relationships adjustment, e.g., including,removing independence relations


building bayesian networksmarinav/teaching/bdminai/bdmaibuildbn.pdf · dynamic bayesian network for...

Documents