Sampling & Power Calculations in Randomized Field Experiments

Craig McIntosh, IRPS/UCSDPrepared for The Asia Foundation Evaluation WorkshopSingapore, March 25, 2010

*Sampling:The purpose of sampling is to draw a study group that is representative of some known and interesting population. For a program with a defined beneficiary group, you may not want to include average individuals in the sample because they are irrelevant.For a study to make claims about impacts on the population as a whole, it must track a sample representative of the population.

So, how do you draw a random sample? Not so easy in countries without deep census data.

*Drawing a random sample:Assume we dont have an individual-level census data source. How can this be done?Most expensive and best way: a listing exercise goes through the entire village and gathers very basic demographic information on all households.From this listing you can then sample households/individuals for detailed surveys using any rule, because the sample can be weighted back to the population using the listing data.Can also pursue an every third house type of rule, can work well but has some methodological problems (unintentional violations of random sampling, lack of weights, etc.).Use a knowledgeable informant to prepare village lists and then sample from these lists.Cluster sampling.

*Cluster sampling:This is a multi-tiered sampling rule, conducted as follows:Using a data source that has basic information on population by geographic areas, randomly pick Primary Sampling Units (PSUs) such that probability of selection is proportional to population.Within the selected PSUs, you then conduct some type of listing exercise to understand the population within each PSU.Then sample households from the listing so that the sample within each PSU is representative of the PSU itself.This creates a representative sample at the national level while minimizing both listing and logistical (travel) costs. However:Not representative at a sub-national level, andSampling of PSUs may not line up well with the geographical rollout of a specific program.

*Sampling weights:A sample that is directly representative of a population can be used unweighted. However, often we choose to oversample certain kinds of units:Rare types who are of particular interest given the impact the program is trying to measure (politically engaged, entrepreneurs, at risk for disease, etc.)For a voluntary program, we may want to oversample the kinds of people who look as if they will take up the program so that we observe as many of them as possible.Having oversampled certain units we must use sampling weights to retain the representivity of the sample to the population.This oversampling does not affect the expected value of the outcome but it gives you a more precise (less noisy) measure of outcomes within these of interest groups.Then, how many people do you need to track to measure impacts?

*Power & Significance.How many observations is enough?Not a straightforward question to answer.Even the simplest power calculation requires that you know the expected treatment effect ETE, the variance of outcomes and the treatment uptake percentage From here, you need to pick a power (the probability that you reject when you should reject, and thus avoid Type II error), typically and (one-tailed).Then, pick significance (the probability that you falsely reject when you should accept, and thus commit Type I error),typically and (two-tailed).

With these you can calculate the minimum sample size as a function of the desired test power.

*Minimum Sample Size:

Then,

And so you can get away with a smaller sample size if you have:High expected treatment effectsLow variance outcomesTreatment & control groups of similar sizes (p=.5)A willingness to accept low significance and power.

What you Need for a Power Calculation

Significance level-This is often conventionally set at 5%. - Lower levels (less likely to reject a false positive), we need more sample size to detect the effectPower Level-A power level of 80% says: 80% of the time, if there is a true effect you will be able to detect it in a given sample-Larger sample More PowerThe mean and the variability of the outcome in the comparison group-From previous surveys conducted in similar settings-The larger the variability is, the larger the sample needed for a given powerThe effect size that we want to detect-What is the smallest effect that should prompt a policy response? - The smaller the expected effect size the larger sample size needed

How to Determine Effect SizeWhat is the smallest effect that should justify the program to be adopted (in terms of cost-benefit)?Sets minimum effect size we would want to be able to test forCommon danger: use an effect size that is too optimistic too small of sample size How large an effect you can detect with a given sample depends on how variable the outcomes is. Example: If all children have very similar diarrhea prevalence without a program, a very small impact will be easy to detectThe Standardized effect size is the effect size divided by the standard deviation of the outcomeCommon effect sizes are: .20 (small); .40 (medium); .50 (large)

*Power & Significance:

Left-hand curve is the distribution of beta hat under the null that it is zero, Right-hand curve is the distribution of beta hat if the true effect size is beta.Significance comes from the right tail of the left-hand curve, Power comes from the left-tailed distribution of the right-hand curve.

(source: Duflo & Kremer Toolkit)

Precise outcomes

Chart4

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

10

30

60

120

180

170

220

130

30

40

00

11

012

012

016

020

013

011

09

03

03

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

mean 50

mean 60

Number of Villagers Exposed to MalariaBlue = treatment

Frequency

Low Standard Deviation

Sheet4

46.9976784087470

37.2231683185370

52.4425730771522

62.764735402632

61.9835021906622

67.331331037671

28.1641236041282

47.6581875671482

60.9502252599611

39.1329935053393

43.0979583951432

33.0956767255333

31.5308910911324

40.2237050264404

42.2649294604422

28.8206878293293

44.3207512842446

45.9595243126464

51.3485305317513

46.3450704854462

46.7300936987474

46.297594862465

63.4264155349632

49.1471554495491

48.138423507481

44.8679260342454

69.7221197595704

58.656729733593

73.7565473071744

43.4509332888434

66.6145582625670

33.8760231796342

55.3894837038552

59.0219145932593

69.1891558643692

49.154829311490

44.7620494822450

56.7513838076572

46.1867615639460

57.5761136031582

35.5581336305360

41.527624843420

34.7842900635352

46.3712298268460

49.6752080733501

50.2811702871500

46.7728399497470

71.9450157601720

32.575172907733

42.635230228243

24.22419255624

64.476700016564

37.202363627137

43.464200542843

57.577136784758

54.667117536955

58.746087587559

55.957417670356

36.281500241636

38.842614584439

56.939944796657

53.226364242453

40.601622812441

47.590521161348

51.315356712451

55.577976480756

51.387149950551

40.890387380241

68.84845914969

54.871981218555

50.722388904351

58.298411557958

58.620077096659

43.634685324744

40.768083080241

61.111887943161

37.988212524238

34.411078912734

57.11324901257

56.384061634956

72.056883608572

64.437546269664

63.039039004163

51.129603788351

50.019508661450

54.537014319855

49.744852630150

39.453249327239

32.251938490732

58.283313945958

54.442244971954

56.179061529156

52.13473185852

39.73069068640

62.381951819462

46.887868292147

41.600782323142

41.788718034542

45.710072653346

45.466384916745

Sheet1

valuemean 50mean 60valuemean 50mean 60valuemean 50mean 60

300030003000

310031003100

322032003200

332033003300

342034003400

351135003510

362036003600

372137003710

381138003820

393039003920

402040004020

413141004120

424042004250

434043004310

442244004450

453345104551

466146304640

474247604781

4831481204881

4923491804980

5043501705061

5152512205123

5220521305275

531453305332

541454405421

554055005545

564356115687

5736570125725

5847580125847

5947590165928

6003600206026

6123610136106

6225620116213

633563096314

642364036417

650465036506

660366006614

672467006702

680268006804

692169006905

700170007000

710071007101

722172007200

730273007300

741074007400

750075007500

760376007601

770077007700

78178007800

79179007900

80080008000

81381008100

82082008200

83083008300

84184008400

85085008500

86086008600

87087008700

88088008800

89089008900

90090009000

Sheet1

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

mean 50

mean 60

Number

Frequency

High Standard Deviation

Sheet5

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

mean 50

mean 60

Number

Frequency

Low Standard Deviation

Sheet7

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

00

mean 50

mean 60

Number

Frequency

Medium Standard Deviation

Sheet9

66.5623453338670

56.6444306665570

58.5494923749590

53.991343672540

58.4543364917580

37.6646291139381

58.6645889294590

46.708007782471

61.5657974473621

65.0302332966650

73.1708475237730

53.8070459393541

52.5203610474530

61.4636498236610

57.6856997819582

57.9168933351583

75.7720023708761

60.3768491297602

58.5402155289591

78.3032170753783

53.6926542432543

72.7113708004732

59.8466137185600

59.3558276401594

76.101557776764

63.6263145375640

64.3705881581643

57.6487515596586

66.8332724368677

59.3136498233597

52.6392320049533

49.0246726864493

62.0370066522625

50.18662038505

79.3957930605793

58.7278442859594

44.1236785869443

56.6468521961574

69.5723180493702

56.1702997098561

53.4613583719531

46.3361608732460

63.1611534723631

58.3629095368582

65.7125362218660

53.62626113540

56.5382244227573

62.8593831303630

56.689712134571

62.1222149371621

35.2617441118350

59.3113533492593

62.4685959943620

81.1649421544810

44.7476829018451

44.4136709726440

36.9523128634370

71.7225226859720

83.7436324824840

56.1002458804560

49.9916019785500

65.479932951965

66.589880285967

51.479330674351

68.092911230168

66.602249413867

57.9254880658

44.51849023745

70.127769201170

67.634344001468

56.495148479956

53.09213762953

68.626716407969

63.468278464463

65.562787919266

80.858897187381

62.733690962563

48.93340489349

64.79632262865

47.714613780648

65.655897439366

58.434975623258

62.988713276963

63.57082399264

61.313037500961

57.280610841757

46.526336215347

81.208234102281

75.976957001876

70.488224688770

45.220028975245

48.826925800849

65.326774044165

60.735428784561

59.910119186160

61.612283995262

41.183009367741

57.432701093257

50.620631251751

50.024366525250

Sheet8

51.5621208149520

50.0357272256500

48.9833486325490

48.4957594279480

49.0737342159490

50.261218247500

50.2365482032500

48.8936906448490

45.712278178460

46.7639269016470

50.3006198313500

52.6035195333530

50.6633490557510

50.6333561942510

48.834177859490

49.5841608325501

47.7187962941483

51.8372656996526

48.36266852074812

49.22318238534918

49.60524291965017

45.28394255324522

49.54499344315013

49.8679459168503

51.1556812751514

49.8785233402500

51.9424714992521

46.7312123772470

49.8424800651500

50.9221798791510

48.0860184223480

49.1095228324490

49.9050305632500

46.5371534927470

47.1606666804470

54.3186810217540

48.3907400747480

48.4523674357480

46.8429438013470

50.9023165148510

51.4216675481510

48.3375118973480

51.1437691683510

49.0433093444490

50.6900813787510

48.3697171047480

51.3527733245510

48.9379921317490

51.3600924831510

51.2249029169510

49.4692871041490

48.5710360306490

51.5505247575520

50.9635573406510

50.7099265531510

49.8627299647500

48.9925640875490

46.2330184644460

53.93272785540

52.8293061405530

52.2297672331520

51.54124336452

54.260691639554

49.896144850

49.426372596649

51.850607986952

50.018742412150

50.53055600851

46.675869624647

49.702938566850

51.075773070651

48.649147983149

47.661871020748

53.134391590753

51.775611053752

51.643115865652

51.610105755452

48.979167230649

51.614127995752

47.81106907948

50.006043592350

48.70644842349

51.298269580751

50.509189703751

52.302144821452

50.940426616651

49.125207068649

51.751236595752

51.03287675351

51.270254870251

54.399262252354

48.263751876948

46.357983036446

50.85597548651

48.177845554948

48.779521774849

56.232876330656

50.532140802551

50.341690338250

49.40065663449

Sheet6

65.0056496548650

57.3209696691570

60.8248753147610

58.7707633409590

63.3077071748630

53.2927312329530

62.3775396585620

55.0287224237550

55.9214619594560

58.5760314303590

59.0897845237590

54.2942349662540

68.5711235442690

44.5087586436450

63.3339210859630

57.4822912918571

69.304158084690

49.7818815953501

58.874643552591

58.586890797590

69.8715190688701

64.4599050852643

63.5056245906645

65.9267858887662

59.6263682078601

59.0135643202595

68.5255305748697

68.9733293862695

56.2703191159567

57.0830049278578

51.8423782219526

57.5707351041586

58.5415433912593

60.9120776667614

52.1062362773527

68.2275801105686

58.3764564604584

47.2046100791472

65.7252736951664

56.2178367241565

51.0154292593510

62.336387297621

59.834517439600

75.5964517035760

60.7276730685610

64.807507139650

60.3975469554601

71.2473026093710

51.1998475510

64.382231964640

56.7147141433570

60.4325124792600

69.1942274594690

55.0461642584550

66.9574298323670

51.5543185125520

58.255666469580

60.0562272362600

55.0468190946550

69.6492021839700

62.5066856324630

64.557912234365

67.589892447968

67.524954526268

50.931446518551

65.59492036766

69.76300725670

55.817915987656

56.04949607656

57.632883150858

63.749742063564

69.713294275670

61.498674464561

47.508781487148

58.376456460458

61.405564944461

65.994452294566

61.66261543262

52.568850757453

62.701867742963

67.035341695967

64.696962605565

57.088693817257

51.758299984252

68.459310265668

64.098224053564

58.130306266858

55.354005477655

65.173369572765

58.921791757359

64.330254341864

61.341009010561

64.66460278465

55.584139469456

55.63872279456

57.878528574858

60.015838850260

55.218399817555

51.860850059352

63.863131041764

Sheet2

57.8872290032580

63.997341739640

41.8400590104420

47.380170953470

53.8276266423540

47.9565291142481

51.918901944520

55.9200601754561

49.5412531462502

50.89907644512

49.3804090082492

48.2460894898482

55.9029116528565

62.1690391097621

63.2923014462635

48.255666469485

46.2407628118464

56.4707228375568

49.7504664861508

51.1038764569518

48.2206873028486

47.3194485392472

38.7579906499397

55.7383158492563

52.4437190405522

43.2019409243434

49.2311541064498

37.3942942941372

39.6987173997404

46.9776922626472

48.3450106811482

47.2073965182470

40.1883984421401

51.717539817521

45.894231688461

44.0191491987440

43.9607732813441

58.234633242580

37.5407047512380

44.5702757032450

59.2255231721590

48.6759280546490

55.6755334299560

48.2216422723480

56.3112702264560

58.0967492977580

44.9119774101450

56.5492758949570

41.4970612735410

60.3314596345600

54.7269486458550

46.4211565384460

46.6091559145470

42.4881285551420

53.0384671845530

49.8818020686500

55.0049948186550

42.350476532420

40.584274201410

49.9951842256500

66.3006188814660

46.219514741846

37.552546372238

41.632694218442

48.714938556149

56.120521902656

54.456276201454

46.907140484847

48.558091647349

52.309184310452

55.76228558256

47.436602825547

54.869129952655

52.980691533653

44.577874531445

52.217507244552

48.801220044749

44.440427144644

49.273254616149

41.967820278742

35.224494594135

51.520938894852

54.51027972355

59.687864803760

44.561981111545

44.960469393845

50.121410721550

48.319253791148

44.067975421844

51.641640210452

49.249203028749

49.687977378950

58.432107279158

58.581328074859

53.24909251553

44.198274180244

46.858286976947

39.100360835439

56.980853868357

47.819300006148

Sheet3

59.8887915353600

62.4386326913620

56.7375879351570

63.1650597521630

62.9446437111630

57.1744364302570

60.1323633114600

59.1549111655590

59.6527185431600

57.721724867580

63.0190221878630

61.7803813535620

61.7776528694620

61.6489116206620

61.2549412531610

58.7638193488590

57.1493070916570

60.1222451829600

61.4004717741610

58.5832528182590

55.865218807560

58.2806411936580

56.8930023897570

63.8092912291640

58.069506546580

63.0841874832630

60.5399124348611

64.58589056516512

58.2221197575812

60.19730350696016

58.26174644115820

63.46626620746313

59.59150500226011

59.7765416992609

60.0340423867603

57.6960725689583

61.0236226444610

64.6473724069650

61.3082967598610

57.4373486111570

57.4145794113570

60.7304970495610

60.8712572708610

58.8826175468590

61.7597858459620

58.1402197591580

60.2419415068600

61.1905694919610

61.1931297195610

59.6292399373600

62.035030775620

60.5662991498610

63.3443939174630

59.4220661392590

57.314716893570

59.1637696439590

62.6348971005630

60.4973480827600

61.8746868591620

58.65912514590

57.3555986799570

59.407532413959

56.724791344657

63.93909431364

58.918033270559

63.133345671863

56.566912159357

60.250265657160

58.784678609859

62.054380274862

64.590147000365

58.096873241658

60.861007265561

64.420144250664

59.045880940759

61.825633262362

58.606556346259

59.709280018660

57.824943420758

60.273721525460

57.7830066258

59.437523001759

61.128667008761

61.366267952161

58.617067831259

60.020577317660

59.142014530759

57.159411577757

59.859510353460

63.034892870363

60.190384525960

59.791932623360

62.480674083962

59.292842858359

56.950027707557

62.154447429362

60.399593318460

58.082148522458

59.711599230160

57.661570887458

Some noise

Chart5

00

00

00

00

00

10

00

10

20

20

20

20

50

10

50

51

40

81

81

80

61

23

75

32

21

45

87

25

47

28

26

06

13

14

17

06

14

02

04

05

00

01

00

00

00

00

01

00

00

00

00

00

00

00

00

00

00

00

00

00

00

mean 50

mean 60

Number

Frequency

Medium Standard Deviation

Sheet4

46.9976784087470

37.2231683185370

52.4425730771522

62.764735402632

61.9835021906622

67.331331037671

28.1641236041282

47.6581875671482

60.9502252599611

39.1329935053393

43.0979583951432

33.0956767255333

31.5308910911324

40.2237050264404

42.2649294604422

28.8206878293293

44.3207512842446

45.9595243126464

51.3485305317513

46.3450704854462

46.7300936987474

46.297594862465

63.4264155349632

49.1471554495491

48.138423507481

44.8679260342454

69.7221197595704

58.656729733593

73.7565473071744

43.4509332888434

66.6145582625670

33.8760231796342

55.3894837038552

59.0219145932593

69.1891558643692

49.154829311490

44.7620494822450

56.7513838076572

46.1867615639460

57.5761136031582

35.5581336305360

41.527624843420

34.7842900635352

46.3712298268460

49.6752080733501

50.2811702871500

46.7728399497470

71.9450157601720

32.575172907733

42.635230228243

24.22419255624

64.476700016564

37.202363627137

43.464200542843

57.577136784758

54.667117536955

58.746087587559

55.957417670356

36.281500241636

38.842614584439

56.939944796657

53.226364242453

40.601622812441

47.590521161348

51.315356712451

55.577976480756

51.387149950551

40.890387380241

68.84845914969

54.871981218555

50.722388904351

58.298411557958

58.620077096659

43.634685324744

40.768083080241

61.111887943161

37.988212524238

34.411078912734

57.11324901257

56.384061634956

72.056883608572

64.437546269664

63.039039004163

51.129603788351

50.019508661450

54.537014319855

49.744852630150

39.453249327239

32.251938490732

58.283313945958

54.442244971954

56.179061529156

52.13473185852

39.73069068640

62.381951819462

46.887868292147

41.600782323142

41.788718034542

45.710072653346

45.466384916745

Sheet1

valuemean 50mean 60valuemean 50mean 60valuemean 50mean 60

300030003000

310031003100

322032003200

332033003300

342034003400

351135003510

362036003600

372137003710

381138003820

393039003920

402040004020

413141004120

424042004250

434043004310

442244004450

453345104551

466146304640

474247604781

4831481204881

4923491804980

5043501705061

5152512205123

5220521305275

531453305332

541454405421

554055005545

564356115687

5736570125725

5847580125847

5947590165928

6003600206026

6123610136106

6225620116213

633563096314

642364036417

650465036506

660366006614

672467006702

680268006804

692169006905

700170007000

710071007101

722172007200

730273007300

741074007400

750075007500

760376007601

770077007700

78178007800

79179007900

80080008000

81381008100

82082008200

83083008300

84184008400

85085008500

86086008600

87087008700

88088008800

89089008900

90090009000

Sheet1

mean 50

mean 60

Number

Frequency

High Standard Deviation

Sheet5

mean 50

mean 60

Number

Frequency

Low Standard Deviation

Sheet7

mean 50

mean 60

Number

Frequency

Medium Standard Deviation

Sheet9

66.5623453338670

56.6444306665570

58.5494923749590

53.991343672540

58.4543364917580

37.6646291139381

58.6645889294590

46.708007782471

61.5657974473621

65.0302332966650

73.1708475237730

53.8070459393541

52.5203610474530

61.4636498236610

57.6856997819582

57.9168933351583

75.7720023708761

60.3768491297602

58.5402155289591

78.3032170753783

53.6926542432543

72.7113708004732

59.8466137185600

59.3558276401594

76.101557776764

63.6263145375640

64.3705881581643

57.6487515596586

66.8332724368677

59.3136498233597

52.6392320049533

49.0246726864493

62.0370066522625

50.18662038505

79.3957930605793

58.7278442859594

44.1236785869443

56.6468521961574

69.5723180493702

56.1702997098561

53.4613583719531

46.3361608732460

63.1611534723631

58.3629095368582

65.7125362218660

53.62626113540

56.5382244227573

62.8593831303630

56.689712134571

62.1222149371621

35.2617441118350

59.3113533492593

62.4685959943620

81.1649421544810

44.7476829018451

44.4136709726440

36.9523128634370

71.7225226859720

83.7436324824840

56.1002458804560

49.9916019785500

65.479932951965

66.589880285967

51.479330674351

68.092911230168

66.602249413867

57.9254880658

44.51849023745

70.127769201170

67.634344001468

56.495148479956

53.09213762953

68.626716407969

63.468278464463

65.562787919266

80.858897187381

62.733690962563

48.93340489349

64.79632262865

47.714613780648

65.655897439366

58.434975623258

62.988713276963

63.57082399264

61.313037500961

57.280610841757

46.526336215347

81.208234102281

75.976957001876

70.488224688770

45.220028975245

48.826925800849

65.326774044165

60.735428784561

59.910119186160

61.612283995262

41.183009367741

57.432701093257

50.620631251751

50.024366525250

Sheet8

51.5621208149520

50.0357272256500

48.9833486325490

48.4957594279480

49.0737342159490

50.261218247500

50.2365482032500

48.8936906448490

45.712278178460

46.7639269016470

50.3006198313500

52.6035195333530

50.6633490557510

50.6333561942510

48.834177859490

49.5841608325501

47.7187962941483

51.8372656996526

48.36266852074812

49.22318238534918

49.60524291965017

45.28394255324522

49.54499344315013

49.8679459168503

51.1556812751514

49.8785233402500

51.9424714992521

46.7312123772470

49.8424800651500

50.9221798791510

48.0860184223480

49.1095228324490

49.9050305632500

46.5371534927470

47.1606666804470

54.3186810217540

48.3907400747480

48.4523674357480

46.8429438013470

50.9023165148510

51.4216675481510

48.3375118973480

51.1437691683510

49.0433093444490

50.6900813787510

48.3697171047480

51.3527733245510

48.9379921317490

51.3600924831510

51.2249029169510

49.4692871041490

48.5710360306490

51.5505247575520

50.9635573406510

50.7099265531510

49.8627299647500

48.9925640875490

46.2330184644460

53.93272785540

52.8293061405530

52.2297672331520

51.54124336452

54.260691639554

49.896144850

49.426372596649

51.850607986952

50.018742412150

50.53055600851

46.675869624647

49.702938566850

51.075773070651

48.649147983149

47.661871020748

53.134391590753

51.775611053752

51.643115865652

51.610105755452

48.979167230649

51.614127995752

47.81106907948

50.006043592350

48.70644842349

51.298269580751

50.509189703751

52.302144821452

50.940426616651

49.125207068649

51.751236595752

51.03287675351

51.270254870251

54.399262252354

48.263751876948

46.357983036446

50.85597548651

48.177845554948

48.779521774849

56.232876330656

50.532140802551

50.341690338250

49.40065663449

Sheet6

65.0056496548650

57.3209696691570

60.8248753147610

58.7707633409590

63.3077071748630

53.2927312329530

62.3775396585620

55.0287224237550

55.9214619594560

58.5760314303590

59.0897845237590

54.2942349662540

68.5711235442690

44.5087586436450

63.3339210859630

57.4822912918571

69.304158084690

49.7818815953501

58.874643552591

58.586890797590

69.8715190688701

64.4599050852643

63.5056245906645

65.9267858887662

59.6263682078601

59.0135643202595

68.5255305748697

68.9733293862695

56.2703191159567

57.0830049278578

51.8423782219526

57.5707351041586

58.5415433912593

60.9120776667614

52.1062362773527

68.2275801105686

58.3764564604584

47.2046100791472

65.7252736951664

56.2178367241565

51.0154292593510

62.336387297621

59.834517439600

75.5964517035760

60.7276730685610

64.807507139650

60.3975469554601

71.2473026093710

51.1998475510

64.382231964640

56.7147141433570

60.4325124792600

69.1942274594690

55.0461642584550

66.9574298323670

51.5543185125520

58.255666469580

60.0562272362600

55.0468190946550

69.6492021839700

62.5066856324630

64.557912234365

67.589892447968

67.524954526268

50.931446518551

65.59492036766

69.76300725670

55.817915987656

56.04949607656

57.632883150858

63.749742063564

69.713294275670

61.498674464561

47.508781487148

58.376456460458

61.405564944461

65.994452294566

61.66261543262

52.568850757453

62.701867742963

67.035341695967

64.696962605565

57.088693817257

51.758299984252

68.459310265668

64.098224053564

58.130306266858

55.354005477655

65.173369572765

58.921791757359

64.330254341864

61.341009010561

64.66460278465

55.584139469456

55.63872279456

57.878528574858

60.015838850260

55.218399817555

51.860850059352

63.863131041764

Sheet2

57.8872290032580

63.997341739640

41.8400590104420

47.380170953470

53.8276266423540

47.9565291142481

51.918901944520

55.9200601754561

49.5412531462502

50.89907644512

49.3804090082492

48.2460894898482

55.9029116528565

62.1690391097621

63.2923014462635

48.255666469485

46.2407628118464

56.4707228375568

49.7504664861508

51.1038764569518

48.2206873028486

47.3194485392472

38.7579906499397

55.7383158492563

52.4437190405522

43.2019409243434

49.2311541064498

37.3942942941372

39.6987173997404

46.9776922626472

48.3450106811482

47.2073965182470

40.1883984421401

51.717539817521

45.894231688461

44.0191491987440

43.9607732813441

58.234633242580

37.5407047512380

44.5702757032450

59.2255231721590

48.6759280546490

55.6755334299560

48.2216422723480

56.3112702264560

58.0967492977580

44.9119774101450

56.5492758949570

41.4970612735410

60.3314596345600

54.7269486458550

46.4211565384460

46.6091559145470

42.4881285551420

53.0384671845530

49.8818020686500

55.0049948186550

42.350476532420

40.584274201410

49.9951842256500

66.3006188814660

46.219514741846

37.552546372238

41.632694218442

48.714938556149

56.120521902656

54.456276201454

46.907140484847

48.558091647349

52.309184310452

55.76228558256

47.436602825547

54.869129952655

52.980691533653

44.577874531445

52.217507244552

48.801220044749

44.440427144644

49.273254616149

41.967820278742

35.224494594135

51.520938894852

54.51027972355

59.687864803760

44.561981111545

44.960469393845

50.121410721550

48.319253791148

44.067975421844

51.641640210452

49.249203028749

49.687977378950

58.432107279158

58.581328074859

53.24909251553

44.198274180244

46.858286976947

39.100360835439

56.980853868357

47.819300006148

Sheet3

59.8887915353600

62.4386326913620

56.7375879351570

63.1650597521630

62.9446437111630

57.1744364302570

60.1323633114600

59.1549111655590

59.6527185431600

57.721724867580

63.0190221878630

61.7803813535620

61.7776528694620

61.6489116206620

61.2549412531610

58.7638193488590

57.1493070916570

60.1222451829600

61.4004717741610

58.5832528182590

55.865218807560

58.2806411936580

56.8930023897570

63.8092912291640

58.069506546580

63.0841874832630

60.5399124348611

64.58589056516512

58.2221197575812

60.19730350696016

58.26174644115820

63.46626620746313

59.59150500226011

59.7765416992609

60.0340423867603

57.6960725689583

61.0236226444610

64.6473724069650

61.3082967598610

57.4373486111570

57.4145794113570

60.7304970495610

60.8712572708610

58.8826175468590

61.7597858459620

58.1402197591580

60.2419415068600

61.1905694919610

61.1931297195610

59.6292399373600

62.035030775620

60.5662991498610

63.3443939174630

59.4220661392590

57.314716893570

59.1637696439590

62.6348971005630

60.4973480827600

61.8746868591620

58.65912514590

57.3555986799570

59.407532413959

56.724791344657

63.93909431364

58.918033270559

63.133345671863

56.566912159357

60.250265657160

58.784678609859

62.054380274862

64.590147000365

58.096873241658

60.861007265561

64.420144250664

59.045880940759

61.825633262362

58.606556346259

59.709280018660

57.824943420758

60.273721525460

57.7830066258

59.437523001759

61.128667008761

61.366267952161

58.617067831259

60.020577317660

59.142014530759

57.159411577757

59.859510353460

63.034892870363

60.190384525960

59.791932623360

62.480674083962

59.292842858359

56.950027707557

62.154447429362

60.399593318460

58.082148522458

59.711599230160

57.661570887458

Very noisy

Chart6

00

00

20

20

20

11

20

21

11

30

20

31

40

40

22

33

61

42

31

23

43

52

20

14

14

40

43

36

47

47

03

23

25

35

23

04

03

24

02

21

01

00

21

02

10

00

03

00

177

178

079

380

081

082

183

084

085

086

087

088

089

mean 50

mean 60

Number

Frequency

High Standard Deviation

Sheet4

46.9976784087470

37.2231683185370

52.4425730771522

62.764735402632

61.9835021906622

67.331331037671

28.1641236041282

47.6581875671482

60.9502252599611

39.1329935053393

43.0979583951432

33.0956767255333

31.5308910911324

40.2237050264404

42.2649294604422

28.8206878293293

44.3207512842446

45.9595243126464

51.3485305317513

46.3450704854462

46.7300936987474

46.297594862465

63.4264155349632

49.1471554495491

48.138423507481

44.8679260342454

69.7221197595704

58.656729733593

73.7565473071744

43.4509332888434

66.6145582625670

33.8760231796342

55.3894837038552

59.0219145932593

69.1891558643692

49.154829311490

44.7620494822450

56.7513838076572

46.1867615639460

57.5761136031582

35.5581336305360

41.527624843420

34.7842900635352

46.3712298268460

49.6752080733501

50.2811702871500

46.7728399497470

71.9450157601720

32.575172907733

42.635230228243

24.22419255624

64.476700016564

37.202363627137

43.464200542843

57.577136784758

54.667117536955

58.746087587559

55.957417670356

36.281500241636

38.842614584439

56.939944796657

53.226364242453

40.601622812441

47.590521161348

51.315356712451

55.577976480756

51.387149950551

40.890387380241

68.84845914969

54.871981218555

50.722388904351

58.298411557958

58.620077096659

43.634685324744

40.768083080241

61.111887943161

37.988212524238

34.411078912734

57.11324901257

56.384061634956

72.056883608572

64.437546269664

63.039039004163

51.129603788351

50.019508661450

54.537014319855

49.744852630150

39.453249327239

32.251938490732

58.283313945958

54.442244971954

56.179061529156

52.13473185852

39.73069068640

62.381951819462

46.887868292147

41.600782323142

41.788718034542

45.710072653346

45.466384916745

Sheet1

valuemean 50mean 60valuemean 50mean 60valuemean 50mean 60

300030003000

310031003100

322032003200

332033003300

342034003400

351135003510

362036003600

372137003710

381138003820

393039003920

402040004020

413141004120

424042004250

434043004310

442244004450

453345104551

466146304640

474247604781

4831481204881

4923491804980

5043501705061

5152512205123

5220521305275

531453305332

541454405421

554055005545

564356115687

5736570125725

5847580125847

5947590165928

6003600206026

6123610136106

6225620116213

633563096314

642364036417

650465036506

660366006614

672467006702

680268006804

692169006905

700170007000

710071007101

722172007200

730273007300

741074007400

750075007500

760376007601

770077007700

78178007800

79179007900

80080008000

81381008100

82082008200

83083008300

84184008400

85085008500

86086008600

87087008700

88088008800

89089008900

90090009000

Sheet1

mean 50

mean 60

Number

Frequency

High Standard Deviation

Sheet5

mean 50

mean 60

Number

Frequency

Low Standard Deviation

Sheet7

mean 50

mean 60

Number

Frequency

Medium Standard Deviation

Sheet9

66.5623453338670

56.6444306665570

58.5494923749590

53.991343672540

58.4543364917580

37.6646291139381

58.6645889294590

46.708007782471

61.5657974473621

65.0302332966650

73.1708475237730

53.8070459393541

52.5203610474530

61.4636498236610

57.6856997819582

57.9168933351583

75.7720023708761

60.3768491297602

58.5402155289591

78.3032170753783

53.6926542432543

72.7113708004732

59.8466137185600

59.3558276401594

76.101557776764

63.6263145375640

64.3705881581643

57.6487515596586

66.8332724368677

59.3136498233597

52.6392320049533

49.0246726864493

62.0370066522625

50.18662038505

79.3957930605793

58.7278442859594

44.1236785869443

56.6468521961574

69.5723180493702

56.1702997098561

53.4613583719531

46.3361608732460

63.1611534723631

58.3629095368582

65.7125362218660

53.62626113540

56.5382244227573

62.8593831303630

56.689712134571

62.1222149371621

35.2617441118350

59.3113533492593

62.4685959943620

81.1649421544810

44.7476829018451

44.4136709726440

36.9523128634370

71.7225226859720

83.7436324824840

56.1002458804560

49.9916019785500

65.479932951965

66.589880285967

51.479330674351

68.092911230168

66.602249413867

57.9254880658

44.51849023745

70.127769201170

67.634344001468

56.495148479956

53.09213762953

68.626716407969

63.468278464463

65.562787919266

80.858897187381

62.733690962563

48.93340489349

64.79632262865

47.714613780648

65.655897439366

58.434975623258

62.988713276963

63.57082399264

61.313037500961

57.280610841757

46.526336215347

81.208234102281

75.976957001876

70.488224688770

45.220028975245

48.826925800849

65.326774044165

60.735428784561

59.910119186160

61.612283995262

41.183009367741

57.432701093257

50.620631251751

50.024366525250

Sheet8

51.5621208149520

50.0357272256500

48.9833486325490

48.4957594279480

49.0737342159490

50.261218247500

50.2365482032500

48.8936906448490

45.712278178460

46.7639269016470

50.3006198313500

52.6035195333530

50.6633490557510

50.6333561942510

48.834177859490

49.5841608325501

47.7187962941483

51.8372656996526

48.36266852074812

49.22318238534918

49.60524291965017

45.28394255324522

49.54499344315013

49.8679459168503

51.1556812751514

49.8785233402500

51.9424714992521

46.7312123772470

49.8424800651500

50.9221798791510

48.0860184223480

49.1095228324490

49.9050305632500

46.5371534927470

47.1606666804470

54.3186810217540

48.3907400747480

48.4523674357480

46.8429438013470

50.9023165148510

51.4216675481510

48.3375118973480

51.1437691683510

49.0433093444490

50.6900813787510

48.3697171047480

51.3527733245510

48.9379921317490

51.3600924831510

51.2249029169510

49.4692871041490

48.5710360306490

51.5505247575520

50.9635573406510

50.7099265531510

49.8627299647500

48.9925640875490

46.2330184644460

53.93272785540

52.8293061405530

52.2297672331520

51.54124336452

54.260691639554

49.896144850

49.426372596649

51.850607986952

50.018742412150

50.53055600851

46.675869624647

49.702938566850

51.075773070651

48.649147983149

47.661871020748

53.134391590753

51.775611053752

51.643115865652

51.610105755452

48.979167230649

51.614127995752

47.81106907948

50.006043592350

48.70644842349

51.298269580751

50.509189703751

52.302144821452

50.940426616651

49.125207068649

51.751236595752

51.03287675351

51.270254870251

54.399262252354

48.263751876948

46.357983036446

50.85597548651

48.177845554948

48.779521774849

56.232876330656

50.532140802551

50.341690338250

49.40065663449

Sheet6

65.0056496548650

57.3209696691570

60.8248753147610

58.7707633409590

63.3077071748630

53.2927312329530

62.3775396585620

55.0287224237550

55.9214619594560

58.5760314303590

59.0897845237590

54.2942349662540

68.5711235442690

44.5087586436450

63.3339210859630

57.4822912918571

69.304158084690

49.7818815953501

58.874643552591

58.586890797590

69.8715190688701

64.4599050852643

63.5056245906645

65.9267858887662

59.6263682078601

59.0135643202595

68.5255305748697

68.9733293862695

56.2703191159567

57.0830049278578

51.8423782219526

57.5707351041586

58.5415433912593

60.9120776667614

52.1062362773527

68.2275801105686

58.3764564604584

47.2046100791472

65.7252736951664

56.2178367241565

51.0154292593510

62.336387297621

59.834517439600

75.5964517035760

60.7276730685610

64.807507139650

60.3975469554601

71.2473026093710

51.1998475510

64.382231964640

56.7147141433570

60.4325124792600

69.1942274594690

55.0461642584550

66.9574298323670

51.5543185125520

58.255666469580

60.0562272362600

55.0468190946550

69.6492021839700

62.5066856324630

64.557912234365

67.589892447968

67.524954526268

50.931446518551

65.59492036766

69.76300725670

55.817915987656

56.04949607656

57.632883150858

63.749742063564

69.713294275670

61.498674464561

47.508781487148

58.376456460458

61.405564944461

65.994452294566

61.66261543262

52.568850757453

62.701867742963

67.035341695967

64.696962605565

57.088693817257

51.758299984252

68.459310265668

64.098224053564

58.130306266858

55.354005477655

65.173369572765

58.921791757359

64.330254341864

61.341009010561

64.66460278465

55.584139469456

55.63872279456

57.878528574858

60.015838850260

55.218399817555

51.860850059352

63.863131041764

Sheet2

57.8872290032580

63.997341739640

41.8400590104420

47.380170953470

53.8276266423540

47.9565291142481

51.918901944520

55.9200601754561

49.5412531462502

50.89907644512

49.3804090082492

48.2460894898482

55.9029116528565

62.1690391097621

63.2923014462635

48.255666469485

46.2407628118464

56.4707228375568

49.7504664861508

51.1038764569518

48.2206873028486

47.3194485392472

38.7579906499397

55.7383158492563

52.4437190405522

43.2019409243434

49.2311541064498

37.3942942941372

39.6987173997404

46.9776922626472

48.3450106811482

47.2073965182470

40.1883984421401

51.717539817521

45.894231688461

44.0191491987440

43.9607732813441

58.234633242580

37.5407047512380

44.5702757032450

59.2255231721590

48.6759280546490

55.6755334299560

48.2216422723480

56.3112702264560

58.0967492977580

44.9119774101450

56.5492758949570

41.4970612735410

60.3314596345600

54.7269486458550

46.4211565384460

46.6091559145470

42.4881285551420

53.0384671845530

49.8818020686500

55.0049948186550

42.350476532420

40.584274201410

49.9951842256500

66.3006188814660

46.219514741846

37.552546372238

41.632694218442

48.714938556149

56.120521902656

54.456276201454

46.907140484847

48.558091647349

52.309184310452

55.76228558256

47.436602825547

54.869129952655

52.980691533653

44.577874531445

52.217507244552

48.801220044749

44.440427144644

49.273254616149

41.967820278742

35.224494594135

51.520938894852

54.51027972355

59.687864803760

44.561981111545

44.960469393845

50.121410721550

48.319253791148

44.067975421844

51.641640210452

49.249203028749

49.687977378950

58.432107279158

58.581328074859

53.24909251553

44.198274180244

46.858286976947

39.100360835439

56.980853868357

47.819300006148

Sheet3

59.8887915353600

62.4386326913620

56.7375879351570

63.1650597521630

62.9446437111630

57.1744364302570

60.1323633114600

59.1549111655590

59.6527185431600

57.721724867580

63.0190221878630

61.7803813535620

61.7776528694620

61.6489116206620

61.2549412531610

58.7638193488590

57.1493070916570

60.1222451829600

61.4004717741610

58.5832528182590

55.865218807560

58.2806411936580

56.8930023897570

63.8092912291640

58.069506546580

63.0841874832630

60.5399124348611

64.58589056516512

58.2221197575812

60.19730350696016

58.26174644115820

63.46626620746313

59.59150500226011

59.7765416992609

60.0340423867603

57.6960725689583

61.0236226444610

64.6473724069650

61.3082967598610

57.4373486111570

57.4145794113570

60.7304970495610

60.8712572708610

58.8826175468590

61.7597858459620

58.1402197591580

60.2419415068600

61.1905694919610

61.1931297195610

59.6292399373600

62.035030775620

60.5662991498610

63.3443939174630

59.4220661392590

57.314716893570

59.1637696439590

62.6348971005630

60.4973480827600

61.8746868591620

58.65912514590

57.3555986799570

59.407532413959

56.724791344657

63.93909431364

58.918033270559

63.133345671863

56.566912159357

60.250265657160

58.784678609859

62.054380274862

64.590147000365

58.096873241658

60.861007265561

64.420144250664

59.045880940759

61.825633262362

58.606556346259

59.709280018660

57.824943420758

60.273721525460

57.7830066258

59.437523001759

61.128667008761

61.366267952161

58.617067831259

60.020577317660

59.142014530759

57.159411577757

59.859510353460

63.034892870363

60.190384525960

59.791932623360

62.480674083962

59.292842858359

56.950027707557

62.154447429362

60.399593318460

58.082148522458

59.711599230160

57.661570887458

Design Factors to Take into AccountAvailability of a BaselineA baseline can help reduce needed sample size since:

Removes some variability in data, increasing precisionCan been use it to stratify and create subgroups

The level of randomization Whenever treatment occurs at a group level, this reduces power relative to randomization at individual level

*Clustered Treatment Designs:It is often natural to implement randomization at a unit more aggregated than the one at which data is available.Examples:School- or village-level randomization of programs using studentsMarket- or city-level tests of political messages using votersPolice precinct interventions studied using individual surveysConducting randomization at a more aggregated level creates a loss of power (relative to doing it at the individual level) called the design effect, strongest if outcomes were anyway correlated at that aggregate level.

Upshot: the power of the test has more to do with the number of units over which you randomize than it does with the number of units in the study.

*Clustered Treatment Designs:Look at the difference between the minimum detectable effect:Without clustered design:

With clustered design:

( is the number of equally-sized clusters, is theintra-cluster correlation, and is the obs per cluster.) As goes to 0, DE disappears. To 1, power from J only.

ImplicationsIt is extremely important to randomize an adequate number of groups.

Typically the number of individual within groups matter less than the number of groups

Big increases in power usually only happens when the number of groups that are randomized increase

If you randomize at the level of the district, with one treated district and one control district, you have 2 observations!

*Power calculations in practice:Use software!Numerous free programs exist on the internet:Optimal Designhttp://sitemaker.umich.edu/group-based/optimal_design_softwareEdgarhttp://www.edgarweb.org.uk/G*Powerhttp://www.psycho.uni-duesseldorf.de/aap/projects/gpower/Many of these use medical not social science descriptions of statistical parameters and can be confusing to use.Make sure to use power calculator with the ability to handle clustered designs if your study units are not the same as the intervention units.Reality check: You very frequently face a sample size constraint for logistical reasons, and then power calculations are relevant only to provide you the probability that youll detect anything.

*Conducting the Actual Randomization.

In many ways simpler than it seems. Easy to use random-number generator in Excel or Stata.

For straight one-shot randomizations:List the ids of the units in the randomization frame.Generate a random number for each unit in the frame.Sort the list by the random number.Flip a coin for whether the first or the second unit on the list goes into the treatment or the control.Assign every other unit to the treatment.Enjoy!

*Stratification & Blocking.Why might you not want to do a single-shot randomization?

Imagine that you have a pre-observable continuous covariate X which you know to be highly correlated with outcomes. Why rely on chance to make the treatment orthogonal to this X? You can stratify across this X to generate an incidence of treatment which is orthogonal to this variable by construction.

What if you have a pre-observable discrete covariate which you know to be highly correlated with outcomes, or if you want to be sure to be able to analyze treatment effects within cells of this discrete covariate?You can Block on this covariate to guarantee that each subgroup has the same treatment percentage as the sample as a whole.

The expected variance of a stratified or blocked randomized estimator cannot be higher than the expected variance of the one-shot randomized estimator.

*Conducting Stratified or Blocked Randomizations.Again, not very hard to do. For stratified or blocked randomizations:Take a list of the units in the randomization frame.Generate a random number for each unit in the frame.Sort the list by stratification or block criterion first, then by the random number.Flip a coin for whether you assign the first unit on the list to the treatment or the control.Then alternate treatment status for every unit on the list; this generates p=.5.For multiple strata or blocks:Sort by multiple strata or blocks and then by the random number last, then follow same as above.

*Post-randomization tests:Quite common for researchers to write loops which conduct randomizations many times, check for balance across numerous characteristics, keep re-running until balance across a pre-specified set of statistics has been realized.Debate exists over this practice.Of course, generates a good-looking t-table of baseline outcomes. Functions like a multi-dimensional stratification criterion. However:T-tests of difference based on a single comparison of means are no longer correct, andIt is not easy to see how to correct for the research design structure in the estimation of treatment effects (Bruhn & McKenzie, 2008).

*Experimental Estimation of Spillover Effects:In the presence of spillover effects, the simple treatment-control difference no longer gives the correct treatment effect. Spillover effects cause trouble for designs where the treatment saturation is blocked, but there are a couple of easy ways to use or create variation to measure them directly. Miguel & Kremer, Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities.Deworming program randomized at the school levelControlling for the number of pupils within a given distance of an untreated unit, they look at how outcomes change as a function of the number of these pupils that were randomly assigned to treatment.Because treatment is randomized, localized intensity of treatment is incidentally randomized.Baird, McIntosh, & zler, Schooling, Income, & HIV Risk in Malawi.Conditional Cash Transfer Program run at the Village levelSaturation of treatment in each village directly randomized, so we can compare untreated girls in treatment villages to the control as a function of the share of girls in the corresponding village that were treated.

*Randomizing things that alter the probability of treatment: Local Average Treatment Effects. Angrist, Imbens, and Rueben, Identification of Causal Effects using Instrumental Variables, JASA 1996:Paper tests for the health impact of serving in Vietnam using the Vietnam Draft lottery number as an instrument for the probability of serving.Identification problem is the endogeneity of the decision to serve in Vietnam. Draft lottery number is a classic instrument in the sense that it has no plausible connection with subsequent health outcomes other than through military service and is not itself caused by military service (exclusion), and strongly drives participation (significance in first stage).Only atypical element is that the first-stage outcome is binary.LATE estimates a quantity similar to the TET.Rather than giving a treatment effect for all compliers, it gives the treatment effect on those units induced to take the treatment as a result of the instrument.LATE cant capture impacts of serving on those would have registered no matter what, nor of those who dodged the draft.

Therefore provides a very clean treatment effect on a very unclear subset of the sample. You not know which units identify the treatment.

*Application of LATE in the field:In principle, very attractive method:Dont have to randomize access to the treatment, just some which alters the probability of taking the treatment.Opens up the idea that promotions, publicity, pricing could be randomized in simple ways to gain experimental identification

Problem in practice:It turns out to be pretty tough to promote programs well enough to gain first-stage significance. Many field researchers have tried LATE promotions in recent years, very few papers are coming out because the promotions have not been sufficiently effective.Even if they work, need to ask whether this newly promoted group represents a sample over which you want treatment effects. Sometimes it does, sometimes it doesnt.

*Internal vs. External Validity in Randomized Trials:Randomized field researchers tend to be meticulous about internal validity,somewhat dismissive of external validity.To some extent, what can you say? You want to know whether these results would hold elsewhere? Then replicate them there!However, this is an attitude much excoriated by policymakers: Just tell us what works and stop asking for additional research money!

So, given that it is always difficult to make claims about external validity from randomized trials, what can you do?The more representative the study sample is of a broader population, the better.The more heterogeneous the study sample is, the more ability you have to (for example) reweight the sample to look like the population and therefore use the internal variation to project the external variation.Beware of controlling the treatment in a randomized trial to such a perfect extent that it stops resembling what the project would actually look like when implemented in the field. Remember that your job is to analyze the actual program, warts and all.

ConclusionsSampling and sample size are questions that must be considered in tandem.Power calculations involve some guess workSome time we do not have the right information to conduct it very properlyHowever, it is important to do them to:Avoid launching studies that will have no power at all: waste of time and moneyDetermine the appropriate resources to the studies that you decide to conduct (and not too much)If you have a fixed budget, can determine whether the project is feasible at all Lots of good software now exists to help us do this.