review of survey methodology

Review of Survey Methodology

Bradley A. Woodruff, MD MPHInternational Emergency and Refugee Health Branch,

U.S. Centers for Disease Control and Prevention

SUMBER: www.smartindicators.org/.../B%20Woodruff%207-23%20Review%20of...


Review some basic principles in surveys and sampling

Describe some issues which • Often done incorrectly or inefficiently• Remain without strong consensus

Opinions expressed are only my own• Given to provide basis of discussion• Recommendations are strictly personal and not

meant to be adopted by this meeting



1. When to use surveys2. Sampling methods3. Final sampling stage (in multistage sampling)4. Sample size5. Number of clusters6. Select 1 or all eligible persons?7. Judging age eligibility8. Children <6 months of age9. Measuring mortality10. High mortality and low malnutrition


When to use surveys


When to use surveys

A cross-sectional survey is a collection of data from a specific

population at a single point in time.


When to use surveys

Survey Surveillance

Collects data at single point in time

Collects data over time period

Can gather wide variety of data Gathers limited data

Usually collects data on sample of population

Often tries to collect data on every case of illness

Collects data allowing calculation of prevalence or incidence rates

Collects only data for numerator; must get data for denominator from separate source


When to use surveys

Alternate data collection methods Morbidity and mortality surveillance Birth and death registration Qualitative methods Program data Others


When to use surveys

What a single survey can tell you The point prevalence of health or nutrition

outcome Retrospective measure of cumulative

incidence, such as mortality rates Can assess outcomes when other data

collection systems absent or impossible


When to use surveys

What a single survey cannot tell you Cause-effect relationships Why events occur or why things are the way

they are Trends over time

• However, repeated surveys can measure change


When to use surveys

Purposes of a survey Determine need for new program Design new program Evaluate existing program


When to use surveys

Strengths of surveys Can give reasonably accurate estimate of prevalence of health

condition in population Can be replicated to evaluate health outcomes Can be done when other data collection systems (e.g. surveillance)

not feasible

Weaknesses of surveys Difficult to assess cause/effect Often confined to short, simple questions Difficult to answer “why” questions Must be repeated to follow trends over time Sometimes difficult to define population of interest Sampling can be complex and is often not done well


When to use surveys

Surveys cost money, manpower, and time Collect only data useful for program purposes If primary objective is measuring malnutrition

prevalence or mortality rate• Do not add extraneous data collection• However, large part of resources spent in

traveling to selected households


When to use surveys – and where?

National-level data • Requires only one survey for country• Provides overall picture• Can serve as general assessment of nutritional status

NGOs often work on local level• Local data needed for program decisions• Difficult to formulate overall picture from many local

surveys• Lack of standardization inhibits comparison• Difficult to achieve good coverage


When to use surveys - recommendations

Before beginning preparations• Carefully consider if survey necessary before

beginning• Identify specific uses of survey results• Design survey to answer specific questions

Rigidly follow correct sampling procedures


Sampling methods


Sampling methods

The goal of sampling is to estimate some measure in the larger population.

Probability sampling: A selection of elements in a population, such that every element has a known, non-zero probability of being selected.*

* Last, Dictionary of EpidemiologySUMBER: www.smartindicators.org/.../B%20Woodruff%207-23%20Review%20of...

Sampling methods - definitions

Sampling universe Sampling frame Sampling unit Basic sampling unit or elementary unit Sampling fraction Respondent Survey subject Unit of analysis


Sampling methods – simple and systematic sampling

Simple random sampling• Most basic type of sampling• Statistical theory based on SRS

• Calculate p values and confidence limits• Output from most statistics computer programs assume

SRS• Selection of units is random and independent

Systematic random sampling• Similar to simple random sampling• First sampling unit chosen randomly• Systematic selection of subsequent units• Statistics same as simple random sampling


Sampling methods – simple and systematic sampling

Both simple and systematic random sampling require a complete list of all sampling units in sampling universe OR that sampling units be organized to allow systematic selection


Sampling methods – cluster sampling

Cluster sampling is probability sampling in which sampling units at some point in the selection process are collections, or clusters, of population elements*

* Kalsbeek, Introduction to survey sampling



Is almost always multistage Randomly choose geographic areas as primary

sampling units Selection is probability proportional to size Final stage of sampling – choose basic

sampling unite (For example, households or children)



Advantages

Cheaper - basic sampling units closer together Does not need complete list of basic sampling units

Disadvantages

Decreased precision of estimate Calculation of p values and confidence limits more

complicated



D istr ic t A D istr ic t B

S u b d is tric t 1 S u b d is tric t 2 S u b d is tric t 3

H H 1 H H 2 H H 3 H H 4 H H 5

S u b d is tric t 4 S u b d is tric t 5

D istr ic t C D is tr ic t D D is tr ic t E D is tr ic t F

CountryStage 1

Stage 2

Stage 3



Sampling probability proportional to size

A B C DDistrict: E F

NotPPS

231 912 3,099 376 484 763

231 912 3,099 376 484 763PPS


Sampling methods – cluster samplingSelecting sample of districts not affected by severe weather

Total Total childrenNo Province District Population <5 years of age

1 Bayan-Ulgii Tsagaannuur 1,853 2312 Bayan-Ulgii Nogoonnuur 7,094 9123 Bayan-Ulgii Ulgii 25,763 3,0994 Bayan-Ulgii Altantsogts 3,168 3765 Bayan-Ulgii Bugat 3,541 4846 Bayan-Ulgii Bayannuur 5,140 7637 Bayan-Ulgii Tolbo 4,773 6728 Bayan-Ulgii Deluun 8,347 1,2519 Bayan-Ulgii Bulgan 5,916 816

10 Uvs Bukhmoron 2,435 32611 Uvs Davst 1,918 24812 Uvs Ulaangom 24,888 2,45513 Uvs Khovd 2,804 38914 Uvs Umnogobi 4,723 51215 Uvs Ulgii 2,861 43816 Khovd Erdeneburen 3,414 45717 Khovd Khovd 4,834 46418 Khovd Myangad 4,015 45519 Khovd Buyant 3,362 42520 Khovd Jargalant 26,418 2,83721 Khovd Dorgon 3,077 41222 Khovd Chandmana 3,417 54523 Khovd Darvi 3,061 356

etc.


Sampling methods – cluster samplingSelecting sample of districts not affected by severe weather

Total Total children Culmulative numberNo Province District Population <5 years of age children < 5 years

1 Bayan-Ulgii Tsagaannuur 1,853 231 2312 Bayan-Ulgii Nogoonnuur 7,094 912 1,1433 Bayan-Ulgii Ulgii 25,763 3,099 4,2424 Bayan-Ulgii Altantsogts 3,168 376 4,6185 Bayan-Ulgii Bugat 3,541 484 5,1026 Bayan-Ulgii Bayannuur 5,140 763 5,8657 Bayan-Ulgii Tolbo 4,773 672 6,5378 Bayan-Ulgii Deluun 8,347 1,251 7,7889 Bayan-Ulgii Bulgan 5,916 816 8,604

10 Uvs Bukhmoron 2,435 326 8,93011 Uvs Davst 1,918 248 9,17812 Uvs Ulaangom 24,888 2,455 11,63313 Uvs Khovd 2,804 389 12,02214 Uvs Umnogobi 4,723 512 12,53415 Uvs Ulgii 2,861 438 12,97216 Khovd Erdeneburen 3,414 457 13,42917 Khovd Khovd 4,834 464 13,89318 Khovd Myangad 4,015 455 14,34819 Khovd Buyant 3,362 425 14,77320 Khovd Jargalant 26,418 2,837 17,61021 Khovd Dorgon 3,077 412 18,02222 Khovd Chandmana 3,417 545 18,56723 Khovd Darvi 3,061 356 18,923

etc. etc.184 Dornod Matad 2335 267 129,177


Cluster sampling – probability of selection

Number of HHs in district

———————

Number of HHsin country

Number of HHs in subdistrict

———————

Number of HHs in district

15——————


30 x 15——————

Number of HHs in country

x x =

30 x

30 clusters of 15 households each

Probability of selecting district

Probability of selecting subdistrict

within selected district

Probability of selecting HH within selected subdistrict

Total probability of selecting anyHH in country


Sampling methods – recommendations

Choose sampling method appropriate to availability and organization of data on population

Use simple or systematic random sampling when possible if logistic savings from cluster sampling not important• Most refugee camps are small enough to so that cluster

sampling offers little logistic advantage For cluster sampling, always sample probability

proportional to size in all but the last sampling stage


Final sampling stage


Final sampling stage – EPI method

EPI (bottle spinning) method of selecting households


Final sampling stage – EPI method

Simple Widely known Easy to train Results in biased sample


Final sampling stage – selecting households

1

2

34

5

678

109





12

3



1

2

34

5

678

109



Final sampling stage – alternate methods

Less simple Not as widely known Requires somewhat more training Results in less-biased sample



Map and dart throw method of selecting households



1

6

5

4

32

Segmentation method of selecting households



1

2

3

26

14

13

12

10

9

8

7

5

429

21

28

27

25

24

22

2016

15

18

19

1117

30

623

Simple or systematic random selection of households


Final sampling stage - recommendations

Do not use EPI method Continue sampling stages until reach small

enough area to do alternate method Map and segment or list, then do simple or

systematic random sampling


Sample size


Sample size

To estimate sample size for single survey using simple or systematic random sampling, need to know:1. Estimate of the prevalence of the outcome2. Precision desired3. Design effect4. Size of total population5. Level of confidence (always use 95%)


Sample size

Effect of Changing the Estimated Prevalence(assume 95% CI, +/- .05, large population)

0

100

200

300

400

500

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00

Estimated prevalence

Req

uire

d sa

mpl

e si

ze


Sample sizeEffect of Changing the Desired Precision

(assume 95% CI, +/- .05, large population)

0

500

1000

1500

2000

2500

3000

0.00 0.05 0.10 0.15 0.20

Width of Confidence Interval

Req

uire

d sa

mpl

e si

ze


Sample sizeEffect of Changing the Population Size

(assume 95% CI, prevalence=.50, +/- .05)

0

100

200

300

400

500

0 10,000 20,000 30,000 40,000 50,000 60,000 70,000 80,000 90,000 100,000

Population size

Req

uire

d sa

mpl

e si

ze


Sample size

As long as the target population is more than a few thousand people, you do not need to consider it in the sample size.

You do NOT generally need a larger sample size if the population is bigger.


Sample size

Where do get information to make assumption about prevalence?

Prior surveys Qualitative estimates Wild guesses Err toward a prevalence of 50%


Sample sizeEffect of Changing the Estimated Prevalence

(assume 95% CI, +/- .05, large population)

0

100

200

300

400

500

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 1.00

Estimated prevalence

Req

uire

d sa

mpl

e si

ze


Sampling – Sample size

To estimate prevalence with 95% confidence limit and simple or systematic random sampling:

N = 1.962 x (P)(1-P) d2

1.96 = Z value for p = 0.05 or 95% confidence limitsP = Estimated prevalenced = Desired precision (for example, 0.05 for ± 5%)


Cluster sampling

To estimate prevalence with 95% confidence limit and cluster sampling:

N = DEFF x 1.962 x (P)(1-P) d2

DEFF = Design effect1.96 = Z value for p = 0.05 or 95% confidence limitsP = Estimated prevalenced = Desired precision (for example, 0.05 for ± 5%)


Sample size

Examples of design effect for wasting: Mongolia 1.3 Badghis Province, Afghanistan 1.6 Sar-i-Pul Camp, Afghanistan 1.4 Mazar-i-Sharif, Afghanistan 2.0

Source: Mongolia MOH/CDC; UNICEF/CDC; MSF-B; ACF


Sample size - 30 x 30 cluster design

Assumptions:• Prevalence = 50%

• Precision = +/- 5 percentage points

• Design effect = 2

• Non-response = 15%

Why assumptions not valid:• Prevalence rarely so high

• Often do not need such precision

• Design effect often higher or lower than 2

• Non-response rarely so high


Sample size

Larger sample size increases precision (by decreasing sampling error)• It does NOT guarantee absence of bias• Bias may result in very incorrect estimate• If little sampling error, may have confidence in this

wrong estimate• Often difficult to detect bias; cannot quantify bias

Quality control is more difficult the larger the sample size

Therefore, you may be better off with smaller sample size, less precision, but much less bias.


Calculate sample size for every survey• Tells you what precision you may have when finished• Assists in planning logistics• Use this sample size to maximize efficient use of

resources Estimate prevalence somewhat closer to 50% than you

expect Overestimate design effect if using cluster sampling Include population size if more than a few thousand and

requested by software• Can result in smaller sample size• If not sure, enter large number (e.g.100,000)• If calculating by hand, can safely ignore population size

Sample size - recommendations


Number of clusters

Number of clusters

Trade-off between statistical precision and logistic requirements

Usually use 30• Design effect increases rapidly with < 30• Design effect not decrease rapidly with > 30

30 clusters not permit subgroup analysis


Number of clusters

From Binkin N. Rapid nutrition surveys: how many clusters are enough? Disasters 16(2): 97-103.


Number of clusters - recommendations

Use at least 30 clusters Sample size calculations for prevalence will

allow only calculation of estimates for entire sample

Add clusters if subgroup analysis is important Add clusters in some cases if wish to

substantially increase precision


Select 1 or all eligible persons?

Select 1 or all eligible persons? – selecting 1

Advantages May decrease design effect because selected children

come from more households

Disadvantages Must visit more households to recruit subjects Requires additional sampling step Produces biased sample Requires weighted analysis



D istric t A D is tr ic t B

S u b d is tric t 1 S u b d is tric t 2 S u b d is tric t 3

C h ild 1 C h ild 2 C h ild 3

H H 1 H H 2

C h ild

H H 3 H H 4

C h ild 1 C h ild 2

H H 5

S u b d is tric t 4 S u b d is tric t 5

D is tr ic t C D istr ic t D D is tr ic t E D istr ic t F

CountryStage 1

Stage 2

Stage 3

Stage 4



Child123456789

10

Child11121314151617181820

Select 8 households in population where 50% of children alone in household, 50% of children have 1 sibling

HouseholdABCDEFGHIJ

Household

K----L

----M----N

----O

10 children alone(50% of all children)

10 children with siblings(50% of all children)



Child123456789

10

Child11121314151617181820


HouseholdABCDEFGHIJ

Household

K----L

----M----N

----O

5 children alone(63% of sampled children)

3 children with siblings(37% of sampled children)



Child123456789

10

Child11121314151617181820


HouseholdABCDEFGHIJ

Household

K----L

----M----N

----O

5 children alone(45% of sampled children)

6 children with siblings(55% of sampled children)



x x = Number of

HHs in district———————Number of HHs

in country

30 x Number of HHsin subdistrict

———————Number of HHs

in district

15——————


30 clusters of 15 households each

Probability of selecting district

Probability of selecting subdistrict

within selected district

Probability of selecting HH within selected subdistrict

1——————

Number of children in HH

x

Probability of selecting child within

selected HH

30 x 15———————————————

Number of HHs X Number in in country children in HH

Overall probability of selecting any child in

sampling universe



Number of eligible children in selected households differs for different children

Therefore, overall probability of selection differs for different children• Must do weighted analysis

• Requires weighting for all prevalence estimates• Each child in multi-child household represents all the

children in that household

30 x 15———————————————

Number of HHs X Number in in country children in HH


Select 1 or all eligible persons?- recommendations

In most cases, include all eligible persons in selected households

If clustering expected to be very high:• Choose 1 eligible person• Record number of eligible persons in every

selected household• Do weighted analysis


Judging age eligibility


Must select specific age group for survey• Usually 6 - 59 months of age for nutrition

assessment of young children Age often not well known Family records may be lost How to judge age of children in selected

household?• Some use height/length• Exact age not necessary for measuring

wasting



If height cut-off is same as population’s height at 60 months of age (e.g. 100 cm in stunted population):

Includes older, stunted children Excludes younger, taller children Results in same bias toward older, stunted

children


Do not use height/length criteria to judge eligibility unless absolutely necessary

Use local calendar to estimate age• Use one-time important events to determine year

of birth• Use seasonal events to determine month of birth

If must use height/length criteria, how to minimize bias?

Judging age eligibility - recommendations


Children < 6 months of age

Children <6 months of age

Reasons cited for exclusion*: Thought to be protected by breastfeeding Anthropometric measurements substantially influenced by

birthweight and intrauterine factors Small sample size in surveys of children < 5 years of age Reference population less applicable because reference

infants virtually all bottle-fed Survey team members hesitant to manipulate young infants Difficult to weigh and measure

• Imprecision in measurements larger as proportion of measurement

• Most scales accurate to +/- 100 grams

* Golden and Prudhon, Field Exchange



Reasons cited for exclusion: Thought to be protected by

breastfeeding

Anthropometric measurements substantially influenced by birthweight and intrauterine factors

Small sample size

Reference population less applicable because reference infants virtually all bottle-fed17

However: Non-exclusive breastfeeding leads

to exposure to enteric pathogens producing wasting

Breastfeeding and complementary feeding practices still important

Can include and analyze data as collected or oversample infants

Mortality very high for infants defined as wasted using current reference. Therefore, regardless of applicability, current reference useful.



Age of child

(months)

Median weight for boys in reference

population (kg)

50 gm as % of

weight

Median height/length for boys

in reference population (cm)

0.5 cm as % of height/

length1 4.3 1.2% 54.6 1%

6 7.8 0.6% 67.8 0.7%

12 10.2 0.5% 76.1 0.7%

24 12.3 0.4% 85.6 0.6%

36 14.6 0.3% 94.9 0.5%

48 16.7 0.3% 102.9 0.5%

60 18.7 0.3% 109.9 0.5%



0%

5%

10%

15%

20%

25%

30%

35%

40%

< 12 12-23 24-35 36-47 48-59Age group (months)

Prev

alen

ce &

con

fiden

ce in

terv

al

Age-specific wasting prevalence and 95% confidence intervals,by 1 year age groups, Badghis Province, 2002

N = 87 125 149 101 77



May need to be included because Case fatality ratio of wasting much higher in infants

• More important to detect and treat wasting May be at higher risk of wasting than commonly

thought Many nutritional interventions very different in infants

• Normal feeding programs not appropriate


Children <6 months of age - recommendations

Include infants if any suspicion of wasting in this age group

Take length and weight measurements carefully• Measure length to nearest 0.1 cm• Measure weight to nearest 100 grams• Use usual rules for rounding

• Do not routinely round up or down• If possible, use uniscales

• But how do uniscales round? If possible, collect data on breastfeeding in this age

group


Mengukur Mortalitas

Mengukur Mortalitas

Cross-sectional survey gathers data at single point in time

Mortality measurement is rate• Requires counting deaths over period of time

Therefore, must gather data from retrospectively• Ask about deaths during specific period in

the past (recall period)


Recall period• Beginning of period must be well-known date

• Major holiday or festival• Occurrence everyone remembers

• End of period is usually day of survey data collection• Recall period should be short enough to allow

accurate recall & for results to be meaningful• Not usually interested in mortality rate from distant past

• Recall period should be long enough to detect enough deaths for statistical precision

• Often choose about 1 year


Mengukur Mortalitas – Prinsip Umum

Kelahiran & Kematian• Birth and death information reported by living

household member• Method should account for births during

recall period• Detection of deaths must be nearly complete

Denominators of mortality rates are survey sample itself• Does not depend on estimates of population

size




Beginning of recall period

End of recall period(usually when survey data collected)

HH member

Moved into HH during recall period

Birth during recall period

Death during recall period

Birth and death during recall period

Moved out of HHduring recall period

Time



End of recall period(usually when survey data collected)Time

Deaths

Births



At least 3 methods currently used:• Current household census• Previous birth history• Past household census


Mengukur Mortalitas

Measuring mortality – current HH census

Determines number of household members alive on day of survey

Asks number of deaths and births within household during recall period

Ask separately about number of household members and deaths for children < 5 to derive mortality rate for this age group





HH member



HH census






Number of current residents+ ½ number of deaths during recall period– ½ number of births during recall period

Number of deaths during recall periodMortality

rate =

x constant / time period


Measuring mortality – previous birth history

Asks about births to women in household in prior 5 years• Does not take household census

Determines current status for each child born in prior 5 years

Collects data only one children born in prior 5 years, therefore, on children < 5 years of age




End of recall period(usually when survey

data collected)

Older than 5

Under 5, bornbefore recall period

Under 5, bornduring recall period

Under 5, deathduring recall period

Under 5, born anddied during recall period

Older than 5, diedduring recall period

5 years priorto survey

Under 5, left HHduring recall period



Number of births in prior 5 years– ½ number of deaths during recall period+ ½ number of births during recall period

Number of deaths during recall periodamong children born in prior 5 years Mortality

rate(for children

<5 years of age)

=



Measuring mortality – past HH census

1. Asks about number of household members at beginning of recall period

2. Determines number of births during recall period

3. Asks about status of these individuals





HH member


Time





HH census



Number of residents at beginning of recall– ½ number of deaths during recall period+ ½ number of births during recall period

Number of deaths during recall periodMortality

rate =



Measuring mortality - difficulties

1. Manipulation by respondents2. Taboo regarding reporting or counting deaths3. Poor recall of deaths, especially in young children4. Poor recall or definition of live births, especially those >

2 years in past5. Poor recall of date of deaths or births 6. Determining age7. Time required at each household8. Deaths of women or entire household


Measuring mortality - difficulties

Present in method

DifficultyCurrent

HH censusPrior birth

historyPast HH census

Manipulation

Taboo Recall of deaths Recall of births Recall of date of death or birth Determining age Time required Deaths of women or household


Methods need to be studied• Compare to reliable measure of mortality rate,

for example, death registration• Compare to each other

When results reported, technique must be described in detail

Keep in mind potential difficulties and biases

Measuring mortality - recommendations


High mortality and low malnutrition


Some believe that high mortality can mask high malnutrition prevalence

The argument:• Malnourished children at greater risk of death• High mortality preferentially kills

malnourished children• Therefore high mortality lowers apparent

malnutrition prevalence



The history• Paper in 1992 suggested that continued high mortality

may cause plateau in malnutrition prevalence• Some suggest that high mortality can produce normal

or near-normal malnutrition prevalence The reality

• Mortality rate correlated with malnutrition prevalence• When one rises, so does the other

• Very high mortality rate and malnutrition prevalence have coexisted in many emergencies

• Somalia• South Sudan



High mortality cannot explain normal or low malnutrition prevalence


Review of Survey Methodology – Potential recommendations from SMART

When to use surveys Sampling methods Final sampling stage (in multistage sampling) Sample size Number of clusters Select 1 or all eligible persons? Judging age eligibility Children <6 months of age Measuring mortality High mortality and low malnutrition


Additional data to collect



Nutritional status of other age groups• Adult women• Older persons

Infant feeding practices Household food security? Micronutrient status Vaccination status Water/sanitation dll.


Additional data to collect - caveats

Inclusion of only households with children eligible for anthropometric measurement results in biased household sample• Cannot make valid conclusions about all

households in sampling universe Must keep in mind differences between unit

of sampling and unit of analysis Must write valid questions and organize data

collection form correctly



Trade-off between useful data and logistic and time requirements

Collect only data essential to make program decisions


review of survey methodology

Documents

b woodruff

eligible persons

selected children

children alone63

children10 children

children alone50

sampled children3 children

importantadd clusters