![Page 1: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/1.jpg)
ECON 626: Applied Microeconomics
Lecture 7:
Power
Professors: Pamela Jakiela and Owen Ozier
![Page 2: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/2.jpg)
Lecture 7, Part 1:
Power in Randomized Trials
![Page 3: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/3.jpg)
Power
• Power:probability of rejecting... the null, when... the alternative is true.
• In randomized trials:probability of having a statistically significant coefficient ontreatment when there is, in fact, an effect of treatment.
• A “power calculation” is... a sample size calculation.This means predicting... the standard error.
![Page 4: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/4.jpg)
Coin toss example
• “Null” Hypothesis: the coin is fair50% chance of heads, 50% chance of tails.
• Structure of the data:Toss the coin a number of times, count heads.
• The test:“Fail to reject” null if within some distance of mean under the null;“Reject” otherwise.
• If we only had 4 tosses of the coin, what cutoffs could we use?Could fail to reject under any of these conditions:
I (A) never
I (B) when exactly the mean (2 heads)
I (C) when within 1 (1, 2, or 3 heads)
I or (D) always.
• We don’t want to reject the null when it is true, though;How much accidental rejection would each possible cutoff give us?
![Page 5: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/5.jpg)
Distribution of possible results
0.06
0.25
0.38
0.25
0.06
0.1
.2.3
.4Pr
obab
ility
0 1 2 3 4P(2)=.38; P(1...3)=.88; P(0...4)=1
Distribution of numbers of heads in 4 tosses of a fair coin
![Page 6: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/6.jpg)
Types of error
Test result
“Reject Null,” “Fail to Reject Null,”Find an effect! No evidence of effect.
Truth:There is an effect Great! “Type II Error”
(low power)Truth:
There is NO effect “Type I Error” Great!(test size)
The probability of Type I error (given the null) is the “size” of the test.By convention, we are usually interested in tests of “size” 0.05.
The probability of Type II error is also very important;If P(failure to detect an effect|there is an effect) = 1− κ,then the power of the test is κ. (I/II distinction: “non-mnemonic” -CB)
Power depends on anticipated effect size; we typically want power ≥ 80%.
![Page 7: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/7.jpg)
Coin Toss example
Calculating size and power with “reject null of fairness if HH or TT” rule:
ProbabilitiesHH or TT HT or TH
“Reject Null,” “Fail to Reject Null,”Find an effect! No evidence of effect.
Unfair coin 1 (pH = 0.1) 0.01 + 0.81 = 0.82 0.09 + 0.09 = 0.18power=0.82
Unfair coin 2 (pH = 0.2) 0.04 + 0.64 = 0.68 0.16 + 0.16 = 0.32power=0.68
Fair coin (pH = 0.5) 0.25 + 0.25 = 0.50 0.25 + 0.25 = 0.50size=0.50
![Page 8: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/8.jpg)
Not enough data even for meaningful test size
• There is no way* to create such a test with four coin tosses so thatthe chance of accidental rejection under the “null” hypothesis(sometimes written H0) is less than 5%, a standard in social science.* (Except the “never reject, no matter what” rule. Not very useful.)
• What about 20 coin tosses?
![Page 9: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/9.jpg)
Distribution of possible results
0.00 0.00 0.00 0.00 0.00
0.01
0.04
0.07
0.12
0.16
0.18
0.16
0.12
0.07
0.04
0.01
0.00 0.00 0.00 0.00 0.00
0.0
5.1
.15
.2Pr
obab
ility
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20P(10)=.18; P(9...11)=.5; P(8...12)=.74; P(7...13)=.88; P(6...14)=.96
Distribution of numbers of heads in 20 tosses of a fair coin
![Page 10: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/10.jpg)
Power with 20 tosses
0.0
5.1
.15
.2Pr
obab
ility
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20Heads
Fair 75pct heads
Power: about 0.62
![Page 11: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/11.jpg)
Power with 30 tosses
0.0
5.1
.15
.2Pr
obab
ility
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30Heads
Fair 75pct heads
Power: about 0.80
![Page 12: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/12.jpg)
Power with 40 tosses
0.0
5.1
.15
Prob
abilit
y
0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40Heads
Fair 75pct heads
Power: about 0.90
![Page 13: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/13.jpg)
Power with 100 tosses
0.0
2.0
4.0
6.0
8.1
Prob
abilit
y
0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100Heads
Fair 75pct heads
Power: about 0.9997
![Page 14: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/14.jpg)
Rejecting H0 in critical region
Significance level (test size) α
Area: α/2 Area: α/2
0Pr
obab
ility
dens
ity
-1.96 0 1.96
Null distribution (False) rejection probability α
![Page 15: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/15.jpg)
Under an alternative:
Suppose true effect were 1 SE (standard error):0
Prob
abilit
y de
nsity
0 1
Null Under 1 SE effect
![Page 16: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/16.jpg)
Under an alternative:
Suppose true effect were 1 SE (standard error):0
Prob
abilit
y de
nsity
0 1 1.96
Null Under 1 SE effect
![Page 17: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/17.jpg)
Under an alternative:
Power would only be approximately 0.170
Prob
abilit
y de
nsity
0 1 1.96
Null Under 1 SE effect
![Page 18: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/18.jpg)
Under an alternative:
Suppose true effect were 3 SEs (standard errors):0
Prob
abilit
y de
nsity
0 3
Null Under 3 SE effect
![Page 19: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/19.jpg)
Under an alternative:
Suppose true effect were 3 SEs (standard errors):0
Prob
abilit
y de
nsity
0 1.96 3
Null Under 3 SE effect
![Page 20: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/20.jpg)
Under an alternative:
Power would be approximately 0.850
Prob
abilit
y de
nsity
0 1.96 3
Null Under 3 SE effect
![Page 21: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/21.jpg)
Power calculation, visually
How the power calculation formula works
Effect (in SE units)
Area: 1-α(α=size)
Area: κ(κ=power)
t1-κ
t1-α/2
0Pr
obab
ility
dens
ity
0
Null dist'n Alternative dist'n
Note: see the related figure in the Toolkit paper.
![Page 22: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/22.jpg)
The formula: for power κ and size α,
Effect > (t1−κ+tα/2)SE (β) Notation: t1−p = pth percentile of the t dist’n.
Note that the formula above works no matter the design.Usually: α = 0.05, κ = 0.80, N is large, so:
Minimum Detectable Effect ≈ (0.84 + 1.96)SE (β) ≈ 2.8SE (β)
We focus on sample size. But how would imperfect compliance orbaseline data affect this? Below, I continue for the standard RCT case.
MDE = (t1−κ+tα/2)
√1
P(1− P)
√σ2
N≈ (z1−κ+zα/2)
√1
P(1− P)
√σ2
N
In practice (Stata): sampsiNote: Stata uses z rather than t distribution (skirting D.O.F. issue).We could also flip this equation around:
⇐⇒ N = (z1−κ + zα/2)2 ·(
1
P(1− P)
)·(
σ2
MDE 2
)
![Page 23: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/23.jpg)
The formula: for power κ and size α,
Where do these numbers come from, σ2 and the effect size?Two basic options:
• Consider standardized effect sizes in terms of standard deviations
• Draw on existing data: What is available that could inform yourproject?
![Page 24: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/24.jpg)
What if treatment is assigned by groups?
We have been thinking here of randomizing at the individual level.But in practice, we often randomize larger units.Examples:
• Entire schools are assigned to treatment or comparison;we observe outcomes at the level of the individual pupil
• Classes within a school are assigned to treatment or comparison;we observe outcomes at the level of the individual pupil
• Households are assigned to treatment or comparison;we observe outcomes at the level of the individual family member
• Sub-district locations are assigned to treatment or comparison;we observe outcomes at the level of the individual road
• Bank branch offices are assigned to treatment or comparison;we observe outcomes at the level of the individual borrower
What does this do?It depends on how much variation is explained by the group eachindividual is in.
![Page 25: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/25.jpg)
What happens to the variance of the estimator?
Suppose yi = βti + εi . We compare the means of those with ti = 1 tothose with ti = 0. Departure point: iid εi having variance σ2
ε , and equalnumbers of observations in treatment and control (N/2 in each):
β =1
N/2
∑T
yi −1
N/2
∑C
yi
β = β +1
N/2
∑T
εi −1
N/2
∑C
εi
Var(β) =1
N/2σ2ε +
1
N/2σ2ε =
4
Nσ2ε
SE (β) =√
4
√σ2ε
N
This is the formula from before, with P = 1/2:√1
P(1− P)
√σ2
N
![Page 26: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/26.jpg)
What happens to the variance of the estimator?
Now suppose yi = βti + εi , but εi = νg + ηig for groups g of fixed sizeng . We still compare the means of those with ti = 1 to those with ti = 0.Departure point: within a group, treatment is either 1 or 0; iid νg havingvariance σ2
ν , iid ηig having variance σ2η, so that σ2
ε = σ2ν + σ2
η, and equalnumbers of observations in treatment and control (still N/2 in each).Define the “the intra-cluster correlation,” ρ:
ρε =σ2ν
σ2ν + σ2
η
=σ2ν
σ2ε
Two other ways of writing this will be convenient:
σ2ν = ρεσ
2ε
σ2η = (1− ρε)σ2
ε
![Page 27: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/27.jpg)
What happens to the variance of the estimator?
Let’s think through two pieces of εi and the variance of their sums; wewill need this in just a moment. Within a single study arm (so considerN/2 observations). First, the simple case, ηig :
Var
(1
N/2
∑arm
ηig
)=
1
(N/2)2 Var
(∑arm
ηig
)
=1
(N/2)2 Var
(N/2)∑1
ηig
=
1
(N/2)2
(N
2
)σ2η
=1
N/2σ2η
![Page 28: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/28.jpg)
What happens to the variance of the estimator?
Let’s think through two pieces of εi and the variance of their sums; wewill need this in just a moment. Within a single study arm (so considerN/2 observations). Now, the slightly more complicated case, νg :
Var
(1
N/2
∑arm
νg
)=
1
(N/2)2 Var
(∑arm
νg
)
=1
(N/2)2 Var
(N/2ng )∑1
ngνg
=
1
(N/2)2 n2gVar
(N/2ng )∑1
νg
=
1
(N/2)2 n2g
(N
2ng
)σ2ν
=ng
N/2σ2ν
![Page 29: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/29.jpg)
What happens to the variance of the estimator?
As before,
β = β +1
N/2
∑T
εi −1
N/2
∑C
εi
β = β +1
N/2
∑T
νg +1
N/2
∑T
ηig −1
N/2
∑C
νg −1
N/2
∑C
ηig
Var(β) =ng
N/2σ2ν +
1
N/2σ2η +
ng
N/2σ2ν +
1
N/2σ2η
=4ng
Nσ2ν +
4
Nσ2η
=4
N
(ngσ
2ν + σ2
η
)=
4
N
(ngρεσ
2ε + (1− ρε)σ2
ε
)=
4
Nσ2ε ((ng − 1)ρε + 1)
SE (β) =√
4
√σ2ε
N
√(ng − 1)ρε + 1 =
√1
P(1− P)
√σ2
N
√(ng − 1)ρε + 1
![Page 30: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/30.jpg)
The formula
Scale the effective standard error by:
Design Effect (“Moulton factor ′′) =√
1 + (ngroupsize − 1)ρ
ρ (“rho”) is the intra-class correlation.In practice (Stata): loneway and sampclus
Recall earlier formula:
MDE = (t1−κ + tα/2)
√1
P(1− P)
√σ2
N
√1 + (ngroupsize − 1)ρ
We could also flip this equation around (swapping z for t):
⇐⇒ N = (z1−κ+zα/2)2·(
1
P(1− P)
)·(
σ2
MDE 2
)·(1 + (ngroupsize − 1)ρ)
![Page 31: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/31.jpg)
Estimation example: clustered standard errors
Stata:
Vcluster = (X ′X )−1nc∑j=1
u′juj(X ′X )−1
whereuj =
∑jcluster
eixi
Angrist and Pischke 8.2.6:
Ωcl = (X ′X )−1
(∑g
X ′g ΨgXg
)(X ′X )−1
where
Ψg = aeg e′g = a
e2
1g e1g e2g ... e1g engge2g e1g e2
2g ... e2g engg... ... ... ...
engg e1g engg e2g ... e2ngg
![Page 32: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/32.jpg)
Estimation example: clustered standard errors
But remember, in the simplest case, X ′g is either:[1 1 ... 11 1 ... 1
]or
[0 0 ... 01 1 ... 1
]So
X ′g
e2
1g e1g e2g ... e1g engge2g e1g e2
2g ... e2g engg... ... ... ...
engg e1g engg e2g ... e2ngg
Xg
Count the terms. diagonal: ng ; off-diagonal: ng (ng − 1).Diagonal terms have expectation σ2
ε ,while off-diagonal terms have expectation σ2
ν = ρσ2ε .
The matrix product then has expectation:
σ2εng (1 + (ng − 1)ρ)
[1 11 1
]or σ2
εng (1 + (ng − 1)ρ)
[0 00 1
]
![Page 33: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/33.jpg)
Estimation example: clustered standard errors
So:
E
[(∑g
X ′g ΨgXg
)]= σ2
ε (1 + (ng − 1)ρ)
N2
N2
N2 N
This is a familiar matrix - it is X ′X !
and thus
E[Ωcl
]=E
[(X ′X )−1
(∑g
X ′g ΨgXg
)(X ′X )−1
]
= (1 + (ng − 1)ρ) (X ′X )−1 σ2ε
![Page 34: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/34.jpg)
Intra-cluster correlation ρ (greek letter “rho”)
But where does this ρ number come from before you have endline data?Two basic options:
• Consider what might be reasonable assumptions
• Draw on existing data (again):What is available that could inform your project?
![Page 35: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/35.jpg)
Intra-class correlations we have known
Data source ICC (ρ)Madagascar Math + Language 0.5Busia, Kenya Math + Language 0.22Udaipur, India Math + Language 0.23Mumbai, India Math + Language 0.29Vadodara, India Math + Language 0.28Busia, Kenya Math 0.62Busia, Kenya Language 0.43Busia, Kenya Science 0.35
Duflo, Glennerster, and Kremer (2006) Using Randomization in Development Economics Research:
A Toolkit
Data source ICC (ρ)US Elementary Math, unconditional 0.22US Elementary Math, rural only, unconditional 0.15US Elementary Math, rural only, conditional on previous scores 0.12
Hedges & Hedberg (2007), Intraclass correlations for planning group randomized experiments in
rural education.
![Page 36: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/36.jpg)
More variations
For discussion or further reading:
• Imperfect compliance with treatment;
• Alternative tests
• Small numbers of groups
• “A first comment is that, despite all the precision of these formulas,power calculations involve substantial guess work in practice.”
May be discussed in later lectures:
• Multiple treatments, multiple testing, attrition
Next:
• Actual mechanics of randomization; covariates; stratification
![Page 37: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/37.jpg)
Lecture 7, Part 2:
Design and Balance in Randomized Trials
![Page 38: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/38.jpg)
Besides statistics, registration
Economics: since 2012
![Page 39: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/39.jpg)
Besides statistics, registration
Other disciplines: this example since 2000
Other registries include non-RCTs, focus on specific fields, etc.
![Page 40: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/40.jpg)
Besides statistics, documentation: CONSORT
![Page 41: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/41.jpg)
Besides statistics, documentation: CONSORT
![Page 42: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/42.jpg)
CONSORT-style example from QJE
![Page 43: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/43.jpg)
CONSORT-style example from ECRQ
![Page 44: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/44.jpg)
Besides statistics, documentation: CONSORT
![Page 45: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/45.jpg)
Besides statistics, documentation: CONSORT
Some of these are clearly more applicable to economics than others.
![Page 46: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/46.jpg)
Besides statistics, documentation: CONSORT
![Page 47: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/47.jpg)
Besides statistics, documentation: CONSORT
![Page 48: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/48.jpg)
Bruhn and McKenzie - Approach
A survey of practitioners, then six datasets:
• Microenterprise profits in Sri Lanka
• Employment survey in Mexico
• Indonesia Family Life Survey: children in school
• Indonesia Family Life Survey: household expenditure
• Learning & Educational Achievement Project (Pakistan): math test
• Learning & Educational Achievement Project: height z-score
![Page 49: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/49.jpg)
Bruhn and McKenzie - Approach
Then, five randomization methods:
• Randomization (single random draw)
• Stratification
• Pair-wise matching
• Rerandomization: redraw if anything is significant
• Rerandomization: minimum maximum t statistic
![Page 50: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/50.jpg)
Bruhn and McKenzie - Approach
Really important: choosing the variables.
“The set of outcomes we have chosen spans a rangeof the ability of the baseline variables to predict futureoutcomes. At one end is microenterprise profits in SriLanka, where baseline profits and 6 baseline individualand firm characteristics explain only 12.2 percent of thevariation in profits 6 months later. ... The math testscores and height z-scores in the LEAPS data have themost variation explained by baseline characteristics, with43.6 percent of the variation in follow-up test scores ex-plained by the baseline test score and 6 baseline charac-teristics.”
![Page 51: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/51.jpg)
Bruhn and McKenzie - Recommendation 1
“Better reporting of the method of random assignment is needed.This should include a description of:a. Which randomization method was used and why.b. Which variables were used for balancing?c. For stratification, how many strata were used?d. For rerandomization, which cutoff rules were used?This is particularly important for experiments with small samples, wherethe randomization method makes more difference.”
(Obvious in retrospect?)
![Page 52: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/52.jpg)
Bruhn and McKenzie - Recommendation 2
“Clearly describe how the randomization was carried out in practice.a. Who performed the randomization?b. How was the randomization done (coin toss, random numbergenerator, etc.)?c. Was the randomization carried out in public or private?”
![Page 53: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/53.jpg)
Bruhn and McKenzie - Recommendation 3
“Re-think the common use of rerandomization.Our simulations find pair-wise matching to generally perform as well, orbetter, than rerandomization in terms of balance and power, and likererandomization, matching allows balance to be sought on more variablesthan possible under stratification. Adjusting for the method ofrandomization is statistically cleaner with matching or stratification thanwith rerandomization.”
![Page 54: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/54.jpg)
Bruhn and McKenzie - Recommendation 4
“When deciding which variables to balance on, strongly consider thebaseline outcome variable and geographic region dummies, inaddition to variables desired for subgroup analysis.”
![Page 55: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/55.jpg)
Bruhn and McKenzie - Recommendation 5
“Be aware that over-stratification can lead to a loss of power inextreme cases. This is because using a large number of strata involves adownside in terms of loss in degrees of freedom when estimating standarderrors, possibly more cases of missing observations, and odd numberswithin strata when stratification is used.”
![Page 56: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/56.jpg)
Bruhn and McKenzie - Recommendation 6
“As ye randomize, so shall ye analyze.” (Include dummies for stratain analysis.) “Similarly, pair dummies should be included for matchedrandomization, or linear variables used for rerandomizations.”
![Page 57: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/57.jpg)
Bruhn and McKenzie - Recommendation 7
“In the ex post analysis, do not automatically control for baselinevariables that show a statistically significant difference in means.The previous literature, and our simulations, suggest that it is a betterrule to control for variables that are thought to influence follow-upoutcomes, independent of whether their difference in means isstatistically significant or not. ... One should still be cautious in the useof ex post controls, given the potential for finite-sample bias if treatmentheterogeneity is correlated with the square of these covariates.”
![Page 58: ECON 626: Applied Microeconomics - Economics.ozier.comeconomics.ozier.com/econ626/lec/econ626-L07-handout-2019.pdf · 1 p = pth percentile of the t dist’n. Note that the formula](https://reader036.vdocument.in/reader036/viewer/2022071013/5fcc000051fb6e1c377d9a3c/html5/thumbnails/58.jpg)
McKenzie (2012)
“The vast majority of randomized experiments in economics rely on asingle baseline and single follow-up survey. While such a design is suitablefor study of highly autocorrelated and relatively precisely measuredoutcomes in the health and education domains, it is unlikely to beoptimal for measuring noisy and relatively less autocorrelated outcomessuch as business profits, and household incomes and expenditures.Taking multiple measurements of such outcomes at relatively shortintervals allows one to average out noise, increasing power. When theoutcomes have low autocorrelation and budget is limited, it can makesense to do no baseline at all. Moreover, I show how for such outcomes,more power can be achieved with multiple follow-ups than allocating thesame total sample size over a single follow-up and baseline. I alsohighlight the large gains in power from ANCOVA analysis rather thandifference-in-differences analysis when autocorrelations are low.”