Checklists for Reporting Social Science Experiments

Alan Gerber

Dave Doherty

Conor Dowling

OutlineDefinition: checklist A list of items in order that should be

reported/performed.

• 1. The benefits of checklists generally, slow adoption and resistance

• 2. The benefits of checklists in research• 3. Checklists for reporting medical research

(CONSORT).• 4. Compare and Contrast Social Science and Medical

Research: how do the differences affect reporting priorities?

• 5. Modifications of CONSORT for social science research

• 6. Next steps

Benefits to Checklists in General/Story of Checklists

• Checklists used in situations where a complicated, step by step task is attempted.

• Pilot safety check. • Medicine.

• Checklists work but often not used even when beneficial.

• Somewhat odd situation.

Benefits to Checklists in Research

• Suppose the checklist said that in section 1 there should be a clear description of the sample (from initial sample through all exclusions/attrition, etc.), sample means, and a clear description of the treatment and treatment assignment.

• Readers will be provided the information they need to evaluate the work.

• Researchers will know what they will need to show and so will be careful to take necessary steps.

Checklists in Medicine

• The problem of inadequate reporting was widespread in medical research and a couple of groups got together to address the problem, merged. – Result: CONSORT statement - checklist.

The Consort List

Comment

• Basically: this is great.

• Lots of things here. Some seem a bit superfluous, but “it can’t hurt.”

• Can we make it better for social sciences?– What should be added/given greater

emphasis?

Medicine versus Social ScienceSources of Bias, Problems of Generalization. A few ideas.

1. Biochemistry versus Choice. For Choices: objectives, actions/resources, beliefs (connect

actions to outcomes).

If the outcome variable is a choice, the “treatment effect” may depend on all of these things.

• “heterogeneous treatment effects”

Implication: may need more detailed accounting of these conditions to interpret results. Lots of context, boring as it is. Perhaps a standardized list of things to be placed into the appendix? Theory and prediction?

2. For field contexts, social processes may be important.

Not clear that there is spillover type problems in randomized medical studies.

Implication: clear reporting of opportunities for treatment group to contaminate control group, discussion of magnitude of bias, evidence related to this.

3. Subtle strategic behavior by subjects makes outcome measures tricky.

If the outcome measures are not comprehensive, it may be hard to accurately interpret the treatment effect.

Relatively clear in medicine, but what about social science?• Medicine: overall mortality, days in hospital• Social Science example: experiment on effect of X on

fraud at the polling place. The opportunities for fraud include: unregistered voters voting, ballot stuffing, poll worker manipulation, etc.

Implication: Report the range of important behavioral effects predicted, whether measured or not. Try to measure as many as possible.

4. Non-standard treatments.

Drug tests: 50 mg of X.

Social Science Experiment: Exposure to Mailing that I designed myself.

In medicine, there are many fewer studies of devices and surgery than of drugs, perhaps because of this problem at least in part.

Implication: Provide treatment materials. Perhaps minimize treatment innovations across studies (avoid gratuitous innovations).

5. Ex-ante versus ex-post hypotheses.

It’s a matter of degree, but the endpoints and subject populations are generally more obvious in medical studies than social science research.

Implication: Clearly report ex ante hypotheses versus ex post hypotheses.

6. List of Other differences

Conclusion1. Checklists appear to work but rarely spring up

and are slow to catch on. 2. There are excellent starting points for social

science reporting already.3. The challenges to the validity and

generalizability of social science research are different in some ways from medical research.

4. Modification/Additions to the medical research checklist can produce a useful checklist for social science research.

5. Talk focused on experiments. The paper version discusses observational work as well.