Disaster Surveys and Their Validity: Comparing Discrete Distributions

Kishore GawandeTexas A&M University

Gina ReinhardtTexas A&M University

Carol SilvaUniversity of Oklahoma

Domonic BearfieldTexas A&M University

Validity• Internal

• Does the study measure what it’s intending to measure?• A B ?• Randomized controlled experiments

• External• Would this study function the same way in a different

time, place, or context?• Would A B elsewhere?• Representative samples

Survey Experiments• Internal Validity

• A B ?• Randomized assignment into control v. treatment

groups

• External Validity• Would A B elsewhere?• Representative samples polled

• Sampling Frame• Telephone, internet, dual frame

Survey Experiments – Internet Frames• Internal Validity

• A B ?• Randomized assignment into control v. treatment

groups is possible

• External Validity• Would A B elsewhere?• Offer pre-established sampling frames• Known probabilities, easily comparable to populations

of interest

• So! It’s allllll good. Right?

Disasters are…• Unplanned disruptions in social and political

mechanisms and systems (Quarantelli, Lagadec, and Boin 2006)

• Inherently social phenomena (Perry 2006)

• Unpredictable– Planning for their research• Funding, access, IRB clearance

Disasters and Validity• Internal Validity– A B ?• Randomized assignment into control v. treatment (“no

disaster” v. “disaster”) groups– Rarely possible, never ethical– Could be hypothetical, but rarely done well (North and Norris

2006)

• Natural experiment– pre-post comparisons, affected-unaffected comparisons

Disasters and ValidityExternal ValidityWould A B elsewhere?

• Very high exposure to none at all• Populations unknowable, unreachable, inaccessible• Geographic foci offer a place to begin• Victims die or disperse• Access is denied or restricted

Convenience samples, selection bias• Noncoverage (Brodie et al 2006; Jenkins et al 2009; Kessler et al

2007)• Nonresponse (Schlenger & Cohen Silver 2006; Hussain,

Weisaeth, & Heir 2009)

What Shall We Do?• Don’t give up!– Disaster surveys are more prone to problems that plague

all surveys• Three-fold Solution:– Survey a representative sample using online sampling

frames despite widespread displacement of respondents– Assess external validity with a distribution test: the

discrete Kolmogorov-Smirnov (KS) test– Include a question from a well-established and respected

survey, enabling the design of a survey and quasi-experiment regarding an unpredicted event with the intention of post-validation

Why Shall We Do It?• 2011 in the US– Hurricane Irene is the 10th billion-dollar

weather/climate event of the year– Total cost pre-Irene: over $33.5 billion, at least 577

deaths• 2011 in the World– Earthquake, Tsunami, Nuclear Accident in Japan– Earthquakes/tsunamis in Argentina, Chile, India,

Indonesia

Why Shall We Do It?• Hurricanes Katrina and Rita and Wilma– 2005– deadliest and costliest hurricane season on US

record since 1928– $168 billion in damages – estimated 1987 deaths

• So what?– Risk, Sociology, Epidemiology, Public Health– Small samples, convenience, interviews

Why Shall We Do It?• Advances in survey sampling technology

• No

• Surge in disasters• No

• Disasters and their management affect:• Public opinion• Trust and legitimacy• Electoral behavior• Structure, health, safety

What do we show?• Methodological– The discrete Kolmogorov-Smirnov (KS) test is a

simple and effective device for validating and analyzing ordered responses typically found in surveys

• Substantive– Research on unpredicted events does not have to

be restricted to small-N or qualitative work– Disaster experiments need not be held to samples

of convenience

How we’ll show it• Disasters and Validity• Our Solution

• Online sampling frame• Distribution test: the discrete Kolmogorov-Smirnov (KS)

test• Question from a well-established and respected survey

• Our Study• Methods: KS test v. Chi-square test• Apply our solution to our study• Evaluate policy question:

• Likelihood of returning to hurricane- ravaged area

Our Study• Hurricanes Katrina and Rita– August and September, 2005– $150.9 billion– 1952 deaths (Lott et al 2011)

– displaced 4 million people (estimated, NOAA 2011)

• Sample: Hurricane-threatened respondents– Conducted September 2006– Residents of Counties/Parishes at general risk of hurricane

damage– States on Gulf and South Atlantic coast (Texas – North

Carolina)– No more than one county/

parish from the coast

Our Study• Respondents registered in SSI database by

county/parish of residence• Invitation by email• $2.50 for complete survey• Entry in $5000 lottery• Restriction of Floridians to 1000

• Results:• 7024 respondents• 1576 affected by Katrina/Rita directly• 894 evacuated for Katrina (414 displaced)• 994 evacuated for Rita (360 displaced)

Solution, Step 2: Distribution Test• Kolmogorov-Smirnov Test (Continuous)• Empirical CDF of random variable X:

– Where ) = 1 if the ith value of X , 0 otherwise– Measures the proportion of the sample with

values less than or equal to x.

Solution, Step 2: Distribution TestKolmogorov-Smirnov Test (Discrete)- Calculate the two empirical cdfs- Compute the KS statistic:

And if the sample sizes are different: