-
Sampling & Power Calculations in Randomized Field Experiments
Craig McIntosh, IRPS/UCSDPrepared for The Asia Foundation Evaluation WorkshopSingapore, March 25, 2010
-
*Sampling:The purpose of sampling is to draw a study group that is representative of some known and interesting population. For a program with a defined beneficiary group, you may not want to include average individuals in the sample because they are irrelevant.For a study to make claims about impacts on the population as a whole, it must track a sample representative of the population.
So, how do you draw a random sample? Not so easy in countries without deep census data.
-
*Drawing a random sample:Assume we dont have an individual-level census data source. How can this be done?Most expensive and best way: a listing exercise goes through the entire village and gathers very basic demographic information on all households.From this listing you can then sample households/individuals for detailed surveys using any rule, because the sample can be weighted back to the population using the listing data.Can also pursue an every third house type of rule, can work well but has some methodological problems (unintentional violations of random sampling, lack of weights, etc.).Use a knowledgeable informant to prepare village lists and then sample from these lists.Cluster sampling.
-
*Cluster sampling:This is a multi-tiered sampling rule, conducted as follows:Using a data source that has basic information on population by geographic areas, randomly pick Primary Sampling Units (PSUs) such that probability of selection is proportional to population.Within the selected PSUs, you then conduct some type of listing exercise to understand the population within each PSU.Then sample households from the listing so that the sample within each PSU is representative of the PSU itself.This creates a representative sample at the national level while minimizing both listing and logistical (travel) costs. However:Not representative at a sub-national level, andSampling of PSUs may not line up well with the geographical rollout of a specific program.
-
*Sampling weights:A sample that is directly representative of a population can be used unweighted. However, often we choose to oversample certain kinds of units:Rare types who are of particular interest given the impact the program is trying to measure (politically engaged, entrepreneurs, at risk for disease, etc.)For a voluntary program, we may want to oversample the kinds of people who look as if they will take up the program so that we observe as many of them as possible.Having oversampled certain units we must use sampling weights to retain the representivity of the sample to the population.This oversampling does not affect the expected value of the outcome but it gives you a more precise (less noisy) measure of outcomes within these of interest groups.Then, how many people do you need to track to measure impacts?
-
*Power & Significance.How many observations is enough?Not a straightforward question to answer.Even the simplest power calculation requires that you know the expected treatment effect ETE, the variance of outcomes and the treatment uptake percentage From here, you need to pick a power (the probability that you reject when you should reject, and thus avoid Type II error), typically and (one-tailed).Then, pick significance (the probability that you falsely reject when you should accept, and thus commit Type I error),typically and (two-tailed).
With these you can calculate the minimum sample size as a function of the desired test power.
-
*Minimum Sample Size:
Then,
And so you can get away with a smaller sample size if you have:High expected treatment effectsLow variance outcomesTreatment & control groups of similar sizes (p=.5)A willingness to accept low significance and power.
-
What you Need for a Power Calculation
Significance level-This is often conventionally set at 5%. - Lower levels (less likely to reject a false positive), we need more sample size to detect the effectPower Level-A power level of 80% says: 80% of the time, if there is a true effect you will be able to detect it in a given sample-Larger sample More PowerThe mean and the variability of the outcome in the comparison group-From previous surveys conducted in similar settings-The larger the variability is, the larger the sample needed for a given powerThe effect size that we want to detect-What is the smallest effect that should prompt a policy response? - The smaller the expected effect size the larger sample size needed
-
How to Determine Effect SizeWhat is the smallest effect that should justify the program to be adopted (in terms of cost-benefit)?Sets minimum effect size we would want to be able to test forCommon danger: use an effect size that is too optimistic too small of sample size How large an effect you can detect with a given sample depends on how variable the outcomes is. Example: If all children have very similar diarrhea prevalence without a program, a very small impact will be easy to detectThe Standardized effect size is the effect size divided by the standard deviation of the outcomeCommon effect sizes are: .20 (small); .40 (medium); .50 (large)
-
*Power & Significance:
Left-hand curve is the distribution of beta hat under the null that it is zero, Right-hand curve is the distribution of beta hat if the true effect size is beta.Significance comes from the right tail of the left-hand curve, Power comes from the left-tailed distribution of the right-hand curve.
(source: Duflo & Kremer Toolkit)
-
Precise outcomes
Chart4
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
10
30
60
120
180
170
220
130
30
40
00
11
012
012
016
020
013
011
09
03
03
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
mean 50
mean 60
Number of Villagers Exposed to MalariaBlue = treatment
Frequency
Low Standard Deviation
Sheet4
46.9976784087470
37.2231683185370
52.4425730771522
62.764735402632
61.9835021906622
67.331331037671
28.1641236041282
47.6581875671482
60.9502252599611
39.1329935053393
43.0979583951432
33.0956767255333
31.5308910911324
40.2237050264404
42.2649294604422
28.8206878293293
44.3207512842446
45.9595243126464
51.3485305317513
46.3450704854462
46.7300936987474
46.297594862465
63.4264155349632
49.1471554495491
48.138423507481
44.8679260342454
69.7221197595704
58.656729733593
73.7565473071744
43.4509332888434
66.6145582625670
33.8760231796342
55.3894837038552
59.0219145932593
69.1891558643692
49.154829311490
44.7620494822450
56.7513838076572
46.1867615639460
57.5761136031582
35.5581336305360
41.527624843420
34.7842900635352
46.3712298268460
49.6752080733501
50.2811702871500
46.7728399497470
71.9450157601720
32.575172907733
42.635230228243
24.22419255624
64.476700016564
37.202363627137
43.464200542843
57.577136784758
54.667117536955
58.746087587559
55.957417670356
36.281500241636
38.842614584439
56.939944796657
53.226364242453
40.601622812441
47.590521161348
51.315356712451
55.577976480756
51.387149950551
40.890387380241
68.84845914969
54.871981218555
50.722388904351
58.298411557958
58.620077096659
43.634685324744
40.768083080241
61.111887943161
37.988212524238
34.411078912734
57.11324901257
56.384061634956
72.056883608572
64.437546269664
63.039039004163
51.129603788351
50.019508661450
54.537014319855
49.744852630150
39.453249327239
32.251938490732
58.283313945958
54.442244971954
56.179061529156
52.13473185852
39.73069068640
62.381951819462
46.887868292147
41.600782323142
41.788718034542
45.710072653346
45.466384916745
Sheet1
valuemean 50mean 60valuemean 50mean 60valuemean 50mean 60
300030003000
310031003100
322032003200
332033003300
342034003400
351135003510
362036003600
372137003710
381138003820
393039003920
402040004020
413141004120
424042004250
434043004310
442244004450
453345104551
466146304640
474247604781
4831481204881
4923491804980
5043501705061
5152512205123
5220521305275
531453305332
541454405421
554055005545
564356115687
5736570125725
5847580125847
5947590165928
6003600206026
6123610136106
6225620116213
633563096314
642364036417
650465036506
660366006614
672467006702
680268006804
692169006905
700170007000
710071007101
722172007200
730273007300
741074007400
750075007500
760376007601
770077007700
78178007800
79179007900
80080008000
81381008100
82082008200
83083008300
84184008400
85085008500
86086008600
87087008700
88088008800
89089008900
90090009000
Sheet1
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
mean 50
mean 60
Number
Frequency
High Standard Deviation
Sheet5
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
mean 50
mean 60
Number
Frequency
Low Standard Deviation
Sheet7
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
00
mean 50
mean 60
Number
Frequency
Medium Standard Deviation
Sheet9
66.5623453338670
56.6444306665570
58.5494923749590
53.991343672540
58.4543364917580
37.6646291139381
58.6645889294590
46.708007782471
61.5657974473621
65.0302332966650
73.1708475237730
53.8070459393541
52.5203610474530
61.4636498236610
57.6856997819582
57.9168933351583
75.7720023708761
60.3768491297602
58.5402155289591
78.3032170753783
53.6926542432543
72.7113708004732
59.8466137185600
59.3558276401594
76.101557776764
63.6263145375640
64.3705881581643
57.6487515596586
66.8332724368677
59.3136498233597
52.6392320049533
49.0246726864493
62.0370066522625
50.18662038505
79.3957930605793
58.7278442859594
44.1236785869443
56.6468521961574
69.5723180493702
56.1702997098561
53.4613583719531
46.3361608732460
63.1611534723631
58.3629095368582
65.7125362218660
53.62626113540
56.5382244227573
62.8593831303630
56.689712134571
62.1222149371621
35.2617441118350
59.3113533492593
62.4685959943620
81.1649421544810
44.7476829018451
44.4136709726440
36.9523128634370
71.7225226859720
83.7436324824840
56.1002458804560
49.9916019785500
65.479932951965
66.589880285967
51.479330674351
68.092911230168
66.602249413867
57.9254880658
44.51849023745
70.127769201170
67.634344001468
56.495148479956
53.09213762953
68.626716407969
63.468278464463
65.562787919266
80.858897187381
62.733690962563
48.93340489349
64.79632262865
47.714613780648
65.655897439366
58.434975623258
62.988713276963
63.57082399264
61.313037500961
57.280610841757
46.526336215347
81.208234102281
75.976957001876
70.488224688770
45.220028975245
48.826925800849
65.326774044165
60.735428784561
59.910119186160
61.612283995262
41.183009367741
57.432701093257
50.620631251751
50.024366525250
Sheet8
51.5621208149520
50.0357272256500
48.9833486325490
48.4957594279480
49.0737342159490
50.261218247500
50.2365482032500
48.8936906448490
45.712278178460
46.7639269016470
50.3006198313500
52.6035195333530
50.6633490557510
50.6333561942510
48.834177859490
49.5841608325501
47.7187962941483
51.8372656996526
48.36266852074812
49.22318238534918
49.60524291965017
45.28394255324522
49.54499344315013
49.8679459168503
51.1556812751514
49.8785233402500
51.9424714992521
46.7312123772470
49.8424800651500
50.9221798791510
48.0860184223480
49.1095228324490
49.9050305632500
46.5371534927470
47.1606666804470
54.3186810217540
48.3907400747480
48.4523674357480
46.8429438013470
50.9023165148510
51.4216675481510
48.3375118973480
51.1437691683510
49.0433093444490
50.6900813787510
48.3697171047480
51.3527733245510
48.9379921317490
51.3600924831510
51.2249029169510
49.4692871041490
48.5710360306490
51.5505247575520
50.9635573406510
50.7099265531510
49.8627299647500
48.9925640875490
46.2330184644460
53.93272785540
52.8293061405530
52.2297672331520
51.54124336452
54.260691639554
49.896144850
49.426372596649
51.850607986952
50.018742412150
50.53055600851
46.675869624647
49.702938566850
51.075773070651
48.649147983149
47.661871020748
53.134391590753
51.775611053752
51.643115865652
51.610105755452
48.979167230649
51.614127995752
47.81106907948
50.006043592350
48.70644842349
51.298269580751
50.509189703751
52.302144821452
50.940426616651
49.125207068649
51.751236595752
51.03287675351
51.270254870251
54.399262252354
48.263751876948
46.357983036446
50.85597548651
48.177845554948
48.779521774849
56.232876330656
50.532140802551
50.341690338250
49.40065663449
Sheet6
65.0056496548650
57.3209696691570
60.8248753147610
58.7707633409590
63.3077071748630
53.2927312329530
62.3775396585620
55.0287224237550
55.9214619594560
58.5760314303590
59.0897845237590
54.2942349662540
68.5711235442690
44.5087586436450
63.3339210859630
57.4822912918571
69.304158084690
49.7818815953501
58.874643552591
58.586890797590
69.8715190688701
64.4599050852643
63.5056245906645
65.9267858887662
59.6263682078601
59.0135643202595
68.5255305748697
68.9733293862695
56.2703191159567
57.0830049278578
51.8423782219526
57.5707351041586
58.5415433912593
60.9120776667614
52.1062362773527
68.2275801105686
58.3764564604584
47.2046100791472
65.7252736951664
56.2178367241565
51.0154292593510
62.336387297621
59.834517439600
75.5964517035760
60.7276730685610
64.807507139650
60.3975469554601
71.2473026093710
51.1998475510
64.382231964640
56.7147141433570
60.4325124792600
69.1942274594690
55.0461642584550
66.9574298323670
51.5543185125520
58.255666469580
60.0562272362600
55.0468190946550
69.6492021839700
62.5066856324630
64.557912234365
67.589892447968
67.524954526268
50.931446518551
65.59492036766
69.76300725670
55.817915987656
56.04949607656
57.632883150858
63.749742063564
69.713294275670
61.498674464561
47.508781487148
58.376456460458
61.405564944461
65.994452294566
61.66261543262
52.568850757453
62.701867742963
67.035341695967
64.696962605565
57.088693817257
51.758299984252
68.459310265668
64.098224053564
58.130306266858
55.354005477655
65.173369572765
58.921791757359
64.330254341864
61.341009010561
64.66460278465
55.584139469456
55.63872279456
57.878528574858
60.015838850260
55.218399817555
51.860850059352
63.863131041764
Sheet2
57.8872290032580
63.997341739640
41.8400590104420
47.380170953470
53.8276266423540
47.9565291142481
51.918901944520
55.9200601754561
49.5412531462502
50.89907644512
49.3804090082492
48.2460894898482
55.9029116528565
62.1690391097621
63.2923014462635
48.255666469485
46.2407628118464
56.4707228375568
49.7504664861508
51.1038764569518
48.2206873028486
47.3194485392472
38.7579906499397
55.7383158492563
52.4437190405522
43.2019409243434
49.2311541064498
37.3942942941372
39.6987173997404
46.9776922626472
48.3450106811482
47.2073965182470
40.1883984421401
51.717539817521
45.894231688461
44.0191491987440
43.9607732813441
58.234633242580
37.5407047512380
44.5702757032450
59.2255231721590
48.6759280546490
55.6755334299560
48.2216422723480
56.3112702264560
58.0967492977580
44.9119774101450
56.5492758949570
41.4970612735410
60.3314596345600
54.7269486458550
46.4211565384460
46.6091559145470
42.4881285551420
53.0384671845530
49.8818020686500
55.0049948186550
42.350476532420
40.584274201410
49.9951842256500
66.3006188814660
46.219514741846
37.552546372238
41.632694218442
48.714938556149
56.120521902656
54.456276201454
46.907140484847
48.558091647349
52.309184310452
55.76228558256
47.436602825547
54.869129952655
52.980691533653
44.577874531445
52.217507244552
48.801220044749
44.440427144644
49.273254616149
41.967820278742
35.224494594135
51.520938894852
54.51027972355
59.687864803760
44.561981111545
44.960469393845
50.121410721550
48.319253791148
44.067975421844
51.641640210452
49.249203028749
49.687977378950
58.432107279158
58.581328074859
53.24909251553
44.198274180244
46.858286976947
39.100360835439
56.980853868357
47.819300006148
Sheet3
59.8887915353600
62.4386326913620
56.7375879351570
63.1650597521630
62.9446437111630
57.1744364302570
60.1323633114600
59.1549111655590
59.6527185431600
57.721724867580
63.0190221878630
61.7803813535620
61.7776528694620
61.6489116206620
61.2549412531610
58.7638193488590
57.1493070916570
60.1222451829600
61.4004717741610
58.5832528182590
55.865218807560
58.2806411936580
56.8930023897570
63.8092912291640
58.069506546580
63.0841874832630
60.5399124348611
64.58589056516512
58.2221197575812
60.19730350696016
58.26174644115820
63.46626620746313
59.59150500226011
59.7765416992609
60.0340423867603
57.6960725689583
61.0236226444610
64.6473724069650
61.3082967598610
57.4373486111570
57.4145794113570
60.7304970495610
60.8712572708610
58.8826175468590
61.7597858459620
58.1402197591580
60.2419415068600
61.1905694919610
61.1931297195610
59.6292399373600
62.035030775620
60.5662991498610
63.3443939174630
59.4220661392590
57.314716893570
59.1637696439590
62.6348971005630
60.4973480827600
61.8746868591620
58.65912514590
57.3555986799570
59.407532413959
56.724791344657
63.93909431364
58.918033270559
63.133345671863
56.566912159357
60.250265657160
58.784678609859
62.054380274862
64.590147000365
58.096873241658
60.861007265561
64.420144250664
59.045880940759
61.825633262362
58.606556346259
59.709280018660
57.824943420758
60.273721525460
57.7830066258
59.437523001759
61.128667008761
61.366267952161
58.617067831259
60.020577317660
59.142014530759
57.159411577757
59.859510353460
63.034892870363
60.190384525960
59.791932623360
62.480674083962
59.292842858359
56.950027707557
62.154447429362
60.399593318460
58.082148522458
59.711599230160
57.661570887458
-
Some noise
Chart5
00
00
00
00
00
10
00
10
20
20
20
20
50
10
50
51
40
81
81
80
61
23
75
32
21
45
87
25
47
28
26
06
13
14
17
06
14
02
04
05
00
01
00
00
00
00
01
00
00
00
00
00
00
00
00
00
00
00
00
00
00
mean 50
mean 60
Number
Frequency
Medium Standard Deviation
Sheet4
46.9976784087470
37.2231683185370
52.4425730771522
62.764735402632
61.9835021906622
67.331331037671
28.1641236041282
47.6581875671482
60.9502252599611
39.1329935053393
43.0979583951432
33.0956767255333
31.5308910911324
40.2237050264404
42.2649294604422
28.8206878293293
44.3207512842446
45.9595243126464
51.3485305317513
46.3450704854462
46.7300936987474
46.297594862465
63.4264155349632
49.1471554495491
48.138423507481
44.8679260342454
69.7221197595704
58.656729733593
73.7565473071744
43.4509332888434
66.6145582625670
33.8760231796342
55.3894837038552
59.0219145932593
69.1891558643692
49.154829311490
44.7620494822450
56.7513838076572
46.1867615639460
57.5761136031582
35.5581336305360
41.527624843420
34.7842900635352
46.3712298268460
49.6752080733501
50.2811702871500
46.7728399497470
71.9450157601720
32.575172907733
42.635230228243
24.22419255624
64.476700016564
37.202363627137
43.464200542843
57.577136784758
54.667117536955
58.746087587559
55.957417670356
36.281500241636
38.842614584439
56.939944796657
53.226364242453
40.601622812441
47.590521161348
51.315356712451
55.577976480756
51.387149950551
40.890387380241
68.84845914969
54.871981218555
50.722388904351
58.298411557958
58.620077096659
43.634685324744
40.768083080241
61.111887943161
37.988212524238
34.411078912734
57.11324901257
56.384061634956
72.056883608572
64.437546269664
63.039039004163
51.129603788351
50.019508661450
54.537014319855
49.744852630150
39.453249327239
32.251938490732
58.283313945958
54.442244971954
56.179061529156
52.13473185852
39.73069068640
62.381951819462
46.887868292147
41.600782323142
41.788718034542
45.710072653346
45.466384916745
Sheet1
valuemean 50mean 60valuemean 50mean 60valuemean 50mean 60
300030003000
310031003100
322032003200
332033003300
342034003400
351135003510
362036003600
372137003710
381138003820
393039003920
402040004020
413141004120
424042004250
434043004310
442244004450
453345104551
466146304640
474247604781
4831481204881
4923491804980
5043501705061
5152512205123
5220521305275
531453305332
541454405421
554055005545
564356115687
5736570125725
5847580125847
5947590165928
6003600206026
6123610136106
6225620116213
633563096314
642364036417
650465036506
660366006614
672467006702
680268006804
692169006905
700170007000
710071007101
722172007200
730273007300
741074007400
750075007500
760376007601
770077007700
78178007800
79179007900
80080008000
81381008100
82082008200
83083008300
84184008400
85085008500
86086008600
87087008700
88088008800
89089008900
90090009000
Sheet1
mean 50
mean 60
Number
Frequency
High Standard Deviation
Sheet5
mean 50
mean 60
Number
Frequency
Low Standard Deviation
Sheet7
mean 50
mean 60
Number
Frequency
Medium Standard Deviation
Sheet9
66.5623453338670
56.6444306665570
58.5494923749590
53.991343672540
58.4543364917580
37.6646291139381
58.6645889294590
46.708007782471
61.5657974473621
65.0302332966650
73.1708475237730
53.8070459393541
52.5203610474530
61.4636498236610
57.6856997819582
57.9168933351583
75.7720023708761
60.3768491297602
58.5402155289591
78.3032170753783
53.6926542432543
72.7113708004732
59.8466137185600
59.3558276401594
76.101557776764
63.6263145375640
64.3705881581643
57.6487515596586
66.8332724368677
59.3136498233597
52.6392320049533
49.0246726864493
62.0370066522625
50.18662038505
79.3957930605793
58.7278442859594
44.1236785869443
56.6468521961574
69.5723180493702
56.1702997098561
53.4613583719531
46.3361608732460
63.1611534723631
58.3629095368582
65.7125362218660
53.62626113540
56.5382244227573
62.8593831303630
56.689712134571
62.1222149371621
35.2617441118350
59.3113533492593
62.4685959943620
81.1649421544810
44.7476829018451
44.4136709726440
36.9523128634370
71.7225226859720
83.7436324824840
56.1002458804560
49.9916019785500
65.479932951965
66.589880285967
51.479330674351
68.092911230168
66.602249413867
57.9254880658
44.51849023745
70.127769201170
67.634344001468
56.495148479956
53.09213762953
68.626716407969
63.468278464463
65.562787919266
80.858897187381
62.733690962563
48.93340489349
64.79632262865
47.714613780648
65.655897439366
58.434975623258
62.988713276963
63.57082399264
61.313037500961
57.280610841757
46.526336215347
81.208234102281
75.976957001876
70.488224688770
45.220028975245
48.826925800849
65.326774044165
60.735428784561
59.910119186160
61.612283995262
41.183009367741
57.432701093257
50.620631251751
50.024366525250
Sheet8
51.5621208149520
50.0357272256500
48.9833486325490
48.4957594279480
49.0737342159490
50.261218247500
50.2365482032500
48.8936906448490
45.712278178460
46.7639269016470
50.3006198313500
52.6035195333530
50.6633490557510
50.6333561942510
48.834177859490
49.5841608325501
47.7187962941483
51.8372656996526
48.36266852074812
49.22318238534918
49.60524291965017
45.28394255324522
49.54499344315013
49.8679459168503
51.1556812751514
49.8785233402500
51.9424714992521
46.7312123772470
49.8424800651500
50.9221798791510
48.0860184223480
49.1095228324490
49.9050305632500
46.5371534927470
47.1606666804470
54.3186810217540
48.3907400747480
48.4523674357480
46.8429438013470
50.9023165148510
51.4216675481510
48.3375118973480
51.1437691683510
49.0433093444490
50.6900813787510
48.3697171047480
51.3527733245510
48.9379921317490
51.3600924831510
51.2249029169510
49.4692871041490
48.5710360306490
51.5505247575520
50.9635573406510
50.7099265531510
49.8627299647500
48.9925640875490
46.2330184644460
53.93272785540
52.8293061405530
52.2297672331520
51.54124336452
54.260691639554
49.896144850
49.426372596649
51.850607986952
50.018742412150
50.53055600851
46.675869624647
49.702938566850
51.075773070651
48.649147983149
47.661871020748
53.134391590753
51.775611053752
51.643115865652
51.610105755452
48.979167230649
51.614127995752
47.81106907948
50.006043592350
48.70644842349
51.298269580751
50.509189703751
52.302144821452
50.940426616651
49.125207068649
51.751236595752
51.03287675351
51.270254870251
54.399262252354
48.263751876948
46.357983036446
50.85597548651
48.177845554948
48.779521774849
56.232876330656
50.532140802551
50.341690338250
49.40065663449
Sheet6
65.0056496548650
57.3209696691570
60.8248753147610
58.7707633409590
63.3077071748630
53.2927312329530
62.3775396585620
55.0287224237550
55.9214619594560
58.5760314303590
59.0897845237590
54.2942349662540
68.5711235442690
44.5087586436450
63.3339210859630
57.4822912918571
69.304158084690
49.7818815953501
58.874643552591
58.586890797590
69.8715190688701
64.4599050852643
63.5056245906645
65.9267858887662
59.6263682078601
59.0135643202595
68.5255305748697
68.9733293862695
56.2703191159567
57.0830049278578
51.8423782219526
57.5707351041586
58.5415433912593
60.9120776667614
52.1062362773527
68.2275801105686
58.3764564604584
47.2046100791472
65.7252736951664
56.2178367241565
51.0154292593510
62.336387297621
59.834517439600
75.5964517035760
60.7276730685610
64.807507139650
60.3975469554601
71.2473026093710
51.1998475510
64.382231964640
56.7147141433570
60.4325124792600
69.1942274594690
55.0461642584550
66.9574298323670
51.5543185125520
58.255666469580
60.0562272362600
55.0468190946550
69.6492021839700
62.5066856324630
64.557912234365
67.589892447968
67.524954526268
50.931446518551
65.59492036766
69.76300725670
55.817915987656
56.04949607656
57.632883150858
63.749742063564
69.713294275670
61.498674464561
47.508781487148
58.376456460458
61.405564944461
65.994452294566
61.66261543262
52.568850757453
62.701867742963
67.035341695967
64.696962605565
57.088693817257
51.758299984252
68.459310265668
64.098224053564
58.130306266858
55.354005477655
65.173369572765
58.921791757359
64.330254341864
61.341009010561
64.66460278465
55.584139469456
55.63872279456
57.878528574858
60.015838850260
55.218399817555
51.860850059352
63.863131041764
Sheet2
57.8872290032580
63.997341739640
41.8400590104420
47.380170953470
53.8276266423540
47.9565291142481
51.918901944520
55.9200601754561
49.5412531462502
50.89907644512
49.3804090082492
48.2460894898482
55.9029116528565
62.1690391097621
63.2923014462635
48.255666469485
46.2407628118464
56.4707228375568
49.7504664861508
51.1038764569518
48.2206873028486
47.3194485392472
38.7579906499397
55.7383158492563
52.4437190405522
43.2019409243434
49.2311541064498
37.3942942941372
39.6987173997404
46.9776922626472
48.3450106811482
47.2073965182470
40.1883984421401
51.717539817521
45.894231688461
44.0191491987440
43.9607732813441
58.234633242580
37.5407047512380
44.5702757032450
59.2255231721590
48.6759280546490
55.6755334299560
48.2216422723480
56.3112702264560
58.0967492977580
44.9119774101450
56.5492758949570
41.4970612735410
60.3314596345600
54.7269486458550
46.4211565384460
46.6091559145470
42.4881285551420
53.0384671845530
49.8818020686500
55.0049948186550
42.350476532420
40.584274201410
49.9951842256500
66.3006188814660
46.219514741846
37.552546372238
41.632694218442
48.714938556149
56.120521902656
54.456276201454
46.907140484847
48.558091647349
52.309184310452
55.76228558256
47.436602825547
54.869129952655
52.980691533653
44.577874531445
52.217507244552
48.801220044749
44.440427144644
49.273254616149
41.967820278742
35.224494594135
51.520938894852
54.51027972355
59.687864803760
44.561981111545
44.960469393845
50.121410721550
48.319253791148
44.067975421844
51.641640210452
49.249203028749
49.687977378950
58.432107279158
58.581328074859
53.24909251553
44.198274180244
46.858286976947
39.100360835439
56.980853868357
47.819300006148
Sheet3
59.8887915353600
62.4386326913620
56.7375879351570
63.1650597521630
62.9446437111630
57.1744364302570
60.1323633114600
59.1549111655590
59.6527185431600
57.721724867580
63.0190221878630
61.7803813535620
61.7776528694620
61.6489116206620
61.2549412531610
58.7638193488590
57.1493070916570
60.1222451829600
61.4004717741610
58.5832528182590
55.865218807560
58.2806411936580
56.8930023897570
63.8092912291640
58.069506546580
63.0841874832630
60.5399124348611
64.58589056516512
58.2221197575812
60.19730350696016
58.26174644115820
63.46626620746313
59.59150500226011
59.7765416992609
60.0340423867603
57.6960725689583
61.0236226444610
64.6473724069650
61.3082967598610
57.4373486111570
57.4145794113570
60.7304970495610
60.8712572708610
58.8826175468590
61.7597858459620
58.1402197591580
60.2419415068600
61.1905694919610
61.1931297195610
59.6292399373600
62.035030775620
60.5662991498610
63.3443939174630
59.4220661392590
57.314716893570
59.1637696439590
62.6348971005630
60.4973480827600
61.8746868591620
58.65912514590
57.3555986799570
59.407532413959
56.724791344657
63.93909431364
58.918033270559
63.133345671863
56.566912159357
60.250265657160
58.784678609859
62.054380274862
64.590147000365
58.096873241658
60.861007265561
64.420144250664
59.045880940759
61.825633262362
58.606556346259
59.709280018660
57.824943420758
60.273721525460
57.7830066258
59.437523001759
61.128667008761
61.366267952161
58.617067831259
60.020577317660
59.142014530759
57.159411577757
59.859510353460
63.034892870363
60.190384525960
59.791932623360
62.480674083962
59.292842858359
56.950027707557
62.154447429362
60.399593318460
58.082148522458
59.711599230160
57.661570887458
-
Very noisy
Chart6
00
00
20
20
20
11
20
21
11
30
20
31
40
40
22
33
61
42
31
23
43
52
20
14
14
40
43
36
47
47
03
23
25
35
23
04
03
24
02
21
01
00
21
02
10
00
03
00
177
178
079
380
081
082
183
084
085
086
087
088
089
mean 50
mean 60
Number
Frequency
High Standard Deviation
Sheet4
46.9976784087470
37.2231683185370
52.4425730771522
62.764735402632
61.9835021906622
67.331331037671
28.1641236041282
47.6581875671482
60.9502252599611
39.1329935053393
43.0979583951432
33.0956767255333
31.5308910911324
40.2237050264404
42.2649294604422
28.8206878293293
44.3207512842446
45.9595243126464
51.3485305317513
46.3450704854462
46.7300936987474
46.297594862465
63.4264155349632
49.1471554495491
48.138423507481
44.8679260342454
69.7221197595704
58.656729733593
73.7565473071744
43.4509332888434
66.6145582625670
33.8760231796342
55.3894837038552
59.0219145932593
69.1891558643692
49.154829311490
44.7620494822450
56.7513838076572
46.1867615639460
57.5761136031582
35.5581336305360
41.527624843420
34.7842900635352
46.3712298268460
49.6752080733501
50.2811702871500
46.7728399497470
71.9450157601720
32.575172907733
42.635230228243
24.22419255624
64.476700016564
37.202363627137
43.464200542843
57.577136784758
54.667117536955
58.746087587559
55.957417670356
36.281500241636
38.842614584439
56.939944796657
53.226364242453
40.601622812441
47.590521161348
51.315356712451
55.577976480756
51.387149950551
40.890387380241
68.84845914969
54.871981218555
50.722388904351
58.298411557958
58.620077096659
43.634685324744
40.768083080241
61.111887943161
37.988212524238
34.411078912734
57.11324901257
56.384061634956
72.056883608572
64.437546269664
63.039039004163
51.129603788351
50.019508661450
54.537014319855
49.744852630150
39.453249327239
32.251938490732
58.283313945958
54.442244971954
56.179061529156
52.13473185852
39.73069068640
62.381951819462
46.887868292147
41.600782323142
41.788718034542
45.710072653346
45.466384916745
Sheet1
valuemean 50mean 60valuemean 50mean 60valuemean 50mean 60
300030003000
310031003100
322032003200
332033003300
342034003400
351135003510
362036003600
372137003710
381138003820
393039003920
402040004020
413141004120
424042004250
434043004310
442244004450
453345104551
466146304640
474247604781
4831481204881
4923491804980
5043501705061
5152512205123
5220521305275
531453305332
541454405421
554055005545
564356115687
5736570125725
5847580125847
5947590165928
6003600206026
6123610136106
6225620116213
633563096314
642364036417
650465036506
660366006614
672467006702
680268006804
692169006905
700170007000
710071007101
722172007200
730273007300
741074007400
750075007500
760376007601
770077007700
78178007800
79179007900
80080008000
81381008100
82082008200
83083008300
84184008400
85085008500
86086008600
87087008700
88088008800
89089008900
90090009000
Sheet1
mean 50
mean 60
Number
Frequency
High Standard Deviation
Sheet5
mean 50
mean 60
Number
Frequency
Low Standard Deviation
Sheet7
mean 50
mean 60
Number
Frequency
Medium Standard Deviation
Sheet9
66.5623453338670
56.6444306665570
58.5494923749590
53.991343672540
58.4543364917580
37.6646291139381
58.6645889294590
46.708007782471
61.5657974473621
65.0302332966650
73.1708475237730
53.8070459393541
52.5203610474530
61.4636498236610
57.6856997819582
57.9168933351583
75.7720023708761
60.3768491297602
58.5402155289591
78.3032170753783
53.6926542432543
72.7113708004732
59.8466137185600
59.3558276401594
76.101557776764
63.6263145375640
64.3705881581643
57.6487515596586
66.8332724368677
59.3136498233597
52.6392320049533
49.0246726864493
62.0370066522625
50.18662038505
79.3957930605793
58.7278442859594
44.1236785869443
56.6468521961574
69.5723180493702
56.1702997098561
53.4613583719531
46.3361608732460
63.1611534723631
58.3629095368582
65.7125362218660
53.62626113540
56.5382244227573
62.8593831303630
56.689712134571
62.1222149371621
35.2617441118350
59.3113533492593
62.4685959943620
81.1649421544810
44.7476829018451
44.4136709726440
36.9523128634370
71.7225226859720
83.7436324824840
56.1002458804560
49.9916019785500
65.479932951965
66.589880285967
51.479330674351
68.092911230168
66.602249413867
57.9254880658
44.51849023745
70.127769201170
67.634344001468
56.495148479956
53.09213762953
68.626716407969
63.468278464463
65.562787919266
80.858897187381
62.733690962563
48.93340489349
64.79632262865
47.714613780648
65.655897439366
58.434975623258
62.988713276963
63.57082399264
61.313037500961
57.280610841757
46.526336215347
81.208234102281
75.976957001876
70.488224688770
45.220028975245
48.826925800849
65.326774044165
60.735428784561
59.910119186160
61.612283995262
41.183009367741
57.432701093257
50.620631251751
50.024366525250
Sheet8
51.5621208149520
50.0357272256500
48.9833486325490
48.4957594279480
49.0737342159490
50.261218247500
50.2365482032500
48.8936906448490
45.712278178460
46.7639269016470
50.3006198313500
52.6035195333530
50.6633490557510
50.6333561942510
48.834177859490
49.5841608325501
47.7187962941483
51.8372656996526
48.36266852074812
49.22318238534918
49.60524291965017
45.28394255324522
49.54499344315013
49.8679459168503
51.1556812751514
49.8785233402500
51.9424714992521
46.7312123772470
49.8424800651500
50.9221798791510
48.0860184223480
49.1095228324490
49.9050305632500
46.5371534927470
47.1606666804470
54.3186810217540
48.3907400747480
48.4523674357480
46.8429438013470
50.9023165148510
51.4216675481510
48.3375118973480
51.1437691683510
49.0433093444490
50.6900813787510
48.3697171047480
51.3527733245510
48.9379921317490
51.3600924831510
51.2249029169510
49.4692871041490
48.5710360306490
51.5505247575520
50.9635573406510
50.7099265531510
49.8627299647500
48.9925640875490
46.2330184644460
53.93272785540
52.8293061405530
52.2297672331520
51.54124336452
54.260691639554
49.896144850
49.426372596649
51.850607986952
50.018742412150
50.53055600851
46.675869624647
49.702938566850
51.075773070651
48.649147983149
47.661871020748
53.134391590753
51.775611053752
51.643115865652
51.610105755452
48.979167230649
51.614127995752
47.81106907948
50.006043592350
48.70644842349
51.298269580751
50.509189703751
52.302144821452
50.940426616651
49.125207068649
51.751236595752
51.03287675351
51.270254870251
54.399262252354
48.263751876948
46.357983036446
50.85597548651
48.177845554948
48.779521774849
56.232876330656
50.532140802551
50.341690338250
49.40065663449
Sheet6
65.0056496548650
57.3209696691570
60.8248753147610
58.7707633409590
63.3077071748630
53.2927312329530
62.3775396585620
55.0287224237550
55.9214619594560
58.5760314303590
59.0897845237590
54.2942349662540
68.5711235442690
44.5087586436450
63.3339210859630
57.4822912918571
69.304158084690
49.7818815953501
58.874643552591
58.586890797590
69.8715190688701
64.4599050852643
63.5056245906645
65.9267858887662
59.6263682078601
59.0135643202595
68.5255305748697
68.9733293862695
56.2703191159567
57.0830049278578
51.8423782219526
57.5707351041586
58.5415433912593
60.9120776667614
52.1062362773527
68.2275801105686
58.3764564604584
47.2046100791472
65.7252736951664
56.2178367241565
51.0154292593510
62.336387297621
59.834517439600
75.5964517035760
60.7276730685610
64.807507139650
60.3975469554601
71.2473026093710
51.1998475510
64.382231964640
56.7147141433570
60.4325124792600
69.1942274594690
55.0461642584550
66.9574298323670
51.5543185125520
58.255666469580
60.0562272362600
55.0468190946550
69.6492021839700
62.5066856324630
64.557912234365
67.589892447968
67.524954526268
50.931446518551
65.59492036766
69.76300725670
55.817915987656
56.04949607656
57.632883150858
63.749742063564
69.713294275670
61.498674464561
47.508781487148
58.376456460458
61.405564944461
65.994452294566
61.66261543262
52.568850757453
62.701867742963
67.035341695967
64.696962605565
57.088693817257
51.758299984252
68.459310265668
64.098224053564
58.130306266858
55.354005477655
65.173369572765
58.921791757359
64.330254341864
61.341009010561
64.66460278465
55.584139469456
55.63872279456
57.878528574858
60.015838850260
55.218399817555
51.860850059352
63.863131041764
Sheet2
57.8872290032580
63.997341739640
41.8400590104420
47.380170953470
53.8276266423540
47.9565291142481
51.918901944520
55.9200601754561
49.5412531462502
50.89907644512
49.3804090082492
48.2460894898482
55.9029116528565
62.1690391097621
63.2923014462635
48.255666469485
46.2407628118464
56.4707228375568
49.7504664861508
51.1038764569518
48.2206873028486
47.3194485392472
38.7579906499397
55.7383158492563
52.4437190405522
43.2019409243434
49.2311541064498
37.3942942941372
39.6987173997404
46.9776922626472
48.3450106811482
47.2073965182470
40.1883984421401
51.717539817521
45.894231688461
44.0191491987440
43.9607732813441
58.234633242580
37.5407047512380
44.5702757032450
59.2255231721590
48.6759280546490
55.6755334299560
48.2216422723480
56.3112702264560
58.0967492977580
44.9119774101450
56.5492758949570
41.4970612735410
60.3314596345600
54.7269486458550
46.4211565384460
46.6091559145470
42.4881285551420
53.0384671845530
49.8818020686500
55.0049948186550
42.350476532420
40.584274201410
49.9951842256500
66.3006188814660
46.219514741846
37.552546372238
41.632694218442
48.714938556149
56.120521902656
54.456276201454
46.907140484847
48.558091647349
52.309184310452
55.76228558256
47.436602825547
54.869129952655
52.980691533653
44.577874531445
52.217507244552
48.801220044749
44.440427144644
49.273254616149
41.967820278742
35.224494594135
51.520938894852
54.51027972355
59.687864803760
44.561981111545
44.960469393845
50.121410721550
48.319253791148
44.067975421844
51.641640210452
49.249203028749
49.687977378950
58.432107279158
58.581328074859
53.24909251553
44.198274180244
46.858286976947
39.100360835439
56.980853868357
47.819300006148
Sheet3
59.8887915353600
62.4386326913620
56.7375879351570
63.1650597521630
62.9446437111630
57.1744364302570
60.1323633114600
59.1549111655590
59.6527185431600
57.721724867580
63.0190221878630
61.7803813535620
61.7776528694620
61.6489116206620
61.2549412531610
58.7638193488590
57.1493070916570
60.1222451829600
61.4004717741610
58.5832528182590
55.865218807560
58.2806411936580
56.8930023897570
63.8092912291640
58.069506546580
63.0841874832630
60.5399124348611
64.58589056516512
58.2221197575812
60.19730350696016
58.26174644115820
63.46626620746313
59.59150500226011
59.7765416992609
60.0340423867603
57.6960725689583
61.0236226444610
64.6473724069650
61.3082967598610
57.4373486111570
57.4145794113570
60.7304970495610
60.8712572708610
58.8826175468590
61.7597858459620
58.1402197591580
60.2419415068600
61.1905694919610
61.1931297195610
59.6292399373600
62.035030775620
60.5662991498610
63.3443939174630
59.4220661392590
57.314716893570
59.1637696439590
62.6348971005630
60.4973480827600
61.8746868591620
58.65912514590
57.3555986799570
59.407532413959
56.724791344657
63.93909431364
58.918033270559
63.133345671863
56.566912159357
60.250265657160
58.784678609859
62.054380274862
64.590147000365
58.096873241658
60.861007265561
64.420144250664
59.045880940759
61.825633262362
58.606556346259
59.709280018660
57.824943420758
60.273721525460
57.7830066258
59.437523001759
61.128667008761
61.366267952161
58.617067831259
60.020577317660
59.142014530759
57.159411577757
59.859510353460
63.034892870363
60.190384525960
59.791932623360
62.480674083962
59.292842858359
56.950027707557
62.154447429362
60.399593318460
58.082148522458
59.711599230160
57.661570887458
-
Design Factors to Take into AccountAvailability of a BaselineA baseline can help reduce needed sample size since:
Removes some variability in data, increasing precisionCan been use it to stratify and create subgroups
The level of randomization Whenever treatment occurs at a group level, this reduces power relative to randomization at individual level
-
*Clustered Treatment Designs:It is often natural to implement randomization at a unit more aggregated than the one at which data is available.Examples:School- or village-level randomization of programs using studentsMarket- or city-level tests of political messages using votersPolice precinct interventions studied using individual surveysConducting randomization at a more aggregated level creates a loss of power (relative to doing it at the individual level) called the design effect, strongest if outcomes were anyway correlated at that aggregate level.
Upshot: the power of the test has more to do with the number of units over which you randomize than it does with the number of units in the study.
-
*Clustered Treatment Designs:Look at the difference between the minimum detectable effect:Without clustered design:
With clustered design:
( is the number of equally-sized clusters, is theintra-cluster correlation, and is the obs per cluster.) As goes to 0, DE disappears. To 1, power from J only.
-
ImplicationsIt is extremely important to randomize an adequate number of groups.
Typically the number of individual within groups matter less than the number of groups
Big increases in power usually only happens when the number of groups that are randomized increase
If you randomize at the level of the district, with one treated district and one control district, you have 2 observations!
-
*Power calculations in practice:Use software!Numerous free programs exist on the internet:Optimal Designhttp://sitemaker.umich.edu/group-based/optimal_design_softwareEdgarhttp://www.edgarweb.org.uk/G*Powerhttp://www.psycho.uni-duesseldorf.de/aap/projects/gpower/Many of these use medical not social science descriptions of statistical parameters and can be confusing to use.Make sure to use power calculator with the ability to handle clustered designs if your study units are not the same as the intervention units.Reality check: You very frequently face a sample size constraint for logistical reasons, and then power calculations are relevant only to provide you the probability that youll detect anything.
-
*Conducting the Actual Randomization.
In many ways simpler than it seems. Easy to use random-number generator in Excel or Stata.
For straight one-shot randomizations:List the ids of the units in the randomization frame.Generate a random number for each unit in the frame.Sort the list by the random number.Flip a coin for whether the first or the second unit on the list goes into the treatment or the control.Assign every other unit to the treatment.Enjoy!
-
*Stratification & Blocking.Why might you not want to do a single-shot randomization?
Imagine that you have a pre-observable continuous covariate X which you know to be highly correlated with outcomes. Why rely on chance to make the treatment orthogonal to this X? You can stratify across this X to generate an incidence of treatment which is orthogonal to this variable by construction.
What if you have a pre-observable discrete covariate which you know to be highly correlated with outcomes, or if you want to be sure to be able to analyze treatment effects within cells of this discrete covariate?You can Block on this covariate to guarantee that each subgroup has the same treatment percentage as the sample as a whole.
The expected variance of a stratified or blocked randomized estimator cannot be higher than the expected variance of the one-shot randomized estimator.
-
*Conducting Stratified or Blocked Randomizations.Again, not very hard to do. For stratified or blocked randomizations:Take a list of the units in the randomization frame.Generate a random number for each unit in the frame.Sort the list by stratification or block criterion first, then by the random number.Flip a coin for whether you assign the first unit on the list to the treatment or the control.Then alternate treatment status for every unit on the list; this generates p=.5.For multiple strata or blocks:Sort by multiple strata or blocks and then by the random number last, then follow same as above.
-
*Post-randomization tests:Quite common for researchers to write loops which conduct randomizations many times, check for balance across numerous characteristics, keep re-running until balance across a pre-specified set of statistics has been realized.Debate exists over this practice.Of course, generates a good-looking t-table of baseline outcomes. Functions like a multi-dimensional stratification criterion. However:T-tests of difference based on a single comparison of means are no longer correct, andIt is not easy to see how to correct for the research design structure in the estimation of treatment effects (Bruhn & McKenzie, 2008).
-
*Experimental Estimation of Spillover Effects:In the presence of spillover effects, the simple treatment-control difference no longer gives the correct treatment effect. Spillover effects cause trouble for designs where the treatment saturation is blocked, but there are a couple of easy ways to use or create variation to measure them directly. Miguel & Kremer, Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities.Deworming program randomized at the school levelControlling for the number of pupils within a given distance of an untreated unit, they look at how outcomes change as a function of the number of these pupils that were randomly assigned to treatment.Because treatment is randomized, localized intensity of treatment is incidentally randomized.Baird, McIntosh, & zler, Schooling, Income, & HIV Risk in Malawi.Conditional Cash Transfer Program run at the Village levelSaturation of treatment in each village directly randomized, so we can compare untreated girls in treatment villages to the control as a function of the share of girls in the corresponding village that were treated.
-
*Randomizing things that alter the probability of treatment: Local Average Treatment Effects. Angrist, Imbens, and Rueben, Identification of Causal Effects using Instrumental Variables, JASA 1996:Paper tests for the health impact of serving in Vietnam using the Vietnam Draft lottery number as an instrument for the probability of serving.Identification problem is the endogeneity of the decision to serve in Vietnam. Draft lottery number is a classic instrument in the sense that it has no plausible connection with subsequent health outcomes other than through military service and is not itself caused by military service (exclusion), and strongly drives participation (significance in first stage).Only atypical element is that the first-stage outcome is binary.LATE estimates a quantity similar to the TET.Rather than giving a treatment effect for all compliers, it gives the treatment effect on those units induced to take the treatment as a result of the instrument.LATE cant capture impacts of serving on those would have registered no matter what, nor of those who dodged the draft.
Therefore provides a very clean treatment effect on a very unclear subset of the sample. You not know which units identify the treatment.
-
*Application of LATE in the field:In principle, very attractive method:Dont have to randomize access to the treatment, just some which alters the probability of taking the treatment.Opens up the idea that promotions, publicity, pricing could be randomized in simple ways to gain experimental identification
Problem in practice:It turns out to be pretty tough to promote programs well enough to gain first-stage significance. Many field researchers have tried LATE promotions in recent years, very few papers are coming out because the promotions have not been sufficiently effective.Even if they work, need to ask whether this newly promoted group represents a sample over which you want treatment effects. Sometimes it does, sometimes it doesnt.
-
*Internal vs. External Validity in Randomized Trials:Randomized field researchers tend to be meticulous about internal validity,somewhat dismissive of external validity.To some extent, what can you say? You want to know whether these results would hold elsewhere? Then replicate them there!However, this is an attitude much excoriated by policymakers: Just tell us what works and stop asking for additional research money!
So, given that it is always difficult to make claims about external validity from randomized trials, what can you do?The more representative the study sample is of a broader population, the better.The more heterogeneous the study sample is, the more ability you have to (for example) reweight the sample to look like the population and therefore use the internal variation to project the external variation.Beware of controlling the treatment in a randomized trial to such a perfect extent that it stops resembling what the project would actually look like when implemented in the field. Remember that your job is to analyze the actual program, warts and all.
-
ConclusionsSampling and sample size are questions that must be considered in tandem.Power calculations involve some guess workSome time we do not have the right information to conduct it very properlyHowever, it is important to do them to:Avoid launching studies that will have no power at all: waste of time and moneyDetermine the appropriate resources to the studies that you decide to conduct (and not too much)If you have a fixed budget, can determine whether the project is feasible at all Lots of good software now exists to help us do this.