judea pearl computer science department ucla judea direct and indirect effects
TRANSCRIPT
Judea Pearl
Computer Science Department
UCLA
www.cs.ucla.edu/~judea
DIRECT AND INDIRECT EFFECTS
QUESTIONS ASKED
• Why decompose effects?• What is the semantics of direct and indirect effects?• What are the policy implications of direct and
indirect effects?• Can path-analytic techniques be extended to
nonlinear and nonparametric models?• When can direct and indirect effect be estimated
consistently from experimental and nonexperimental data.
1. Direct (or indirect) effect may be more transportable.2. Indirect effects may be prevented or controlled.
3. Direct (or indirect) effect may be forbidden
WHY DECOMPOSEEFFECTS?
Pill
Thrombosis
Pregnancy
+
+
Gender
Hiring
Qualification
EFFECT-DECOMPOSITIONIN LINEAR MODELS
X Z
Y
ca
b
effect Indirect effect Direct effect Total
a bc
Definition: ))(),(|( zdoxdoYEx
a
COUNTERFACTUALS:STRUCTURAL SEMANTICS
Notation: Yx(u) = y Abbreviation: yx
Formal: Y has the value y in the solution to a mutilated system of equations, where the equation for X is replaced by a constant X=x.
u
Yx(u)=y
Z
W
X=x
u
Y
Z
W
X
Probability of Counterfactuals:
FunctionalBayes Net
))(|()())(()( xdoyPu
uPyuxYPyxYP
tindependen- ))(),(|(
))(|(
DETEIE
ZzdoxdoYEx
DE
xdoYEx
TE
TOTAL, DIRECT, AND INDIRECT EFFECTS HAVE SIMPLE SEMANTICS
IN LINEAR MODELS
X Z
Y
ca
b z = bx + 1
y = ax + cz + 2
a + bc
bc
a
z = f (x, 1)y = g (x, z, 2)
????
))(),(|(
))(|(
IE
zdoxdoYEx
DE
xdoYEx
TE
X Z
Y
SEMANTICS BECOMES NONTRIVIALIN NONLINEAR MODELS
(even when the model is completely specified)
Dependent on z?
Void of operational meaning?
Indirect Effect?
NEED OF FORMALIZATION
X Z
AND
Y
=
What is the direct effect of X on Y?
Starting from X=0, (and Z=0 and Y=0)
Total Effect: Change X from 0 to 1, and test the change in Y.
Controlled DE: Keep Z constant at Z=0, or Z=1, and change X=0 to X=1.
Controlled IE: None.
Natural DE: Keep Z constant at its current value, and change X to 1.Natural IE: Keep X at 0, but set Z to what it would be if X were 1.
TWO CONCEPTIONS OF DIRECTAND INDIRECT EFFECTS:
Controlled vs. Natural
X Z
AND
Y
=
``The central question in any employment-discrimination case is whether the employer would have taken the same action had the employee been of different race (age, sex, religion, national origin etc.) and everything else had been the same’’
[Carson versus Bethlehem Steel Corp. (70 FEP Cases 921, 7th Cir. (1996))]
x = male, x = femaley = hire, y = not hirez = applicant’s qualifications
LEGAL DEFINITIONS TAKE THE NATURAL CONCEPTION
(FORMALIZING DISCRIMINATION)
NO DIRECT EFFECT
',' ' xxxx YYYYxZxZ
Starting from X=x*, (and Z=Zx*(u) and Y= Yx*(u))Total Effect: TE(x,x*;Y) = E(Yx) – E(Yx*)
Controlled DE: CDEZ(x,x*;Y) = E(Yxz) – E(Yx*z)Controlled IE: None.
Natural DE: NDE(x,x*;Y) = E(YxZx*) – E(Yx*)
Natural IE: NIE(x,x*;Y) = E(Yx*Zx) – E(Yx*)
TWO CONCEPTIONS OF AVERAGE DIRECT AND INDIRECT EFFECTS:
POPULATION-LEVEL DEFINITIONS
X Z
y = f (x,z,u)
Y
u2u3
u1
Probabilisticcausal model:
P(u)M,
(all other parents of Y)
z = f (x, 1)y = g (x, z, 2)
X Z
Y
THE OPERATIONAL MEANING OFAVERAGE DIRECT EFFECTS
“Natural” Direct Effect of X on Y:The expected change in Y per unit change of X, when we keep Z constant at whatever value it attains before the change.
In linear models, NDE = Controlled Direct Effect
][001 xZx YYE
x
POLICY IMPLICATIONS(Who cares?)
f
GENDER QUALIFICATION
HIRING
What is the direct effect of X on Y?
The effect of Gender on Hiring if sex discrimination is eliminated.
indirect
X Z
Y
IGNORE
z = f (x, 1)y = g (x, z, 2)
X Z
Y
THE OPERATIONAL MEANING OFINDIRECT EFFECTS
“Natural” Indirect Effect of X on Y:The expected change in Y when we keep X constant, say at x0, and let Z change to whatever value it would have under a unit change in X.
In linear models, NIE = TE - DE
][010 xZx YYE
x
Example:
Theorem: If there exists a set W such that
GRAPHICAL CONDITION FOR EXPERIMENTAL IDENTIFICATION
OF AVERAGE NATURAL DIRECT EFFECTS
zw
xzxxz wPwzZPwYEwYEYxxNDE,
** )()|()|()|()*;,(
)()|( ZXNDWWZYXZG and
HOW THE PROOF GOES?
)(,*;, ** xZx YEYEYxxNDEx
xzWZYW xxz and all for If |*
Proof:
)()|(
),|()(
*
*, *
wWPwWzZP
wWzZYEYE
x
w zxxzZx x
)()|(
)|()(
*
, *
wWPwWzZP
wWYYEYE
x
w zxzZx x
Each factor is identifiable by experimentation.
GRAPHICAL CRITERION FORCOUNTERFACTUAL INDEPENDENCE
xzWZY xxz and all for |*
XZGWZY )|(
U3
U1
X
Y
Z
U2
U3
U1
X Z
Y
U2
U3
U1
X
Y
U2
ZXZG
21 UU
21 UU
GRAPHICAL CONDITION FOR NONEXPERIMENTAL IDENTIFICATION
OF AVERAGE NATURAL DIRECT EFFECTS
Identification conditions1. There exists a W such that (Y Z | W)GXZ
2. There exist additional covariates that render all counterfactual terms identifiable.
zw
xzxxz wPwzZPwYEwYEYxxNDE
,** )()|()|()|(
)*;,(
Corollary 3:The average natural direct effect in Markovian models is identifiable from nonexperimental data, and it is given by
where S stands for all parents of X (or another sufficient set).
IDENTIFICATION INMARKOVIAN MODELS
X ZExample:S =
Y
s z
sPsxzPzxYEzxYEYxxNDE )()*,|()*,|(),|()*;,(
z
xzPzxyEzxYEYxxNDE *)|()*,|(),|()*;,(
How effective would the drug be if we eliminate its side-effect (Headache)?
POLICY QUESTION ANSWERED BY NATURAL DIRECT EFFECT
Outcome
Drug
Aspirin
X
Y
Z
W Headache
z
xYExzPzxYEYxxNDE *)|(*)|(),|()*;,(
• NIE(x,x*;Y) = Expected increase in sales, if we bluff the competitor into believing that X is about to change from x* to x.
• For Markovian models:
POLICY-BASED INTERPRETATION OF INDIRECT EFFECTS
X
Y
Z
(Sales)
(Advertisement Budget) (Competitor’s Budget)
z
xzPxzPzxYEYxxNDE *)|()|()*,|()*;,(
Theorem 5: The total, direct and indirect effects obeyThe following equality
In words, the total effect (on Y) associated with the transition from x* to x is equal to the difference between the direct effect associated with this transition and the indirect effect associated with the reverse transition, from x to x*.
RELATIONS BETWEEN TOTAL, DIRECT, AND INDIRECT EFFECTS
);*,()*;,();,( * YxxNIEYxxNDEYxxTE
Y
Z
X
W
x*
z* = Zx* (u)
Nonidentifiable even in Markovian models
GENERAL PATH-SPECIFICEFFECTS (Def.)
)),(*),(();,(* ugpagpafgupaf iiiii
*);,();,( **gMMg YxxTEYxxE
Y
Z
X
W
Form a new model, , specific to active subgraph g*gM
Definition: g-specific effect
SUMMARY OF RESULTS
1. New formulation of path-specific effects, based on signal blocking, instead of value fixing.
2. Path-analytic techniques extended to nonlinear and nonparametric models.
3. Conditions for estimating direct and indirect effects from experimental and nonexperimental data.
4. Estimability conditions hold in Markovian models.5. Graphical techniques of inferring effects of
nonstandard policies, involving signal blocking.