universityofwashington* …september(28,(2016(5"calendar***! october 3, 2016 class begins –...

September 28, 2016 1

University of Washington Daniel J. Evans School of Public Affairs

SYLLABUS Research Design PPM 502 Fall 2016

Professor Ann Bostrom Parrington 327 Tel 206.685.8198 [email protected] Office hours: Mondays 4pm-‐5pm and by appointment Class meetings: Mondays 12pm to 2:50pm in Parrington 306 Course website: https://canvas.uw.edu/courses/992503 Introduction This course introduces the philosophical and practical basics in research design for policy scientists. While it is an introductory course, it is rigorous. By the end of the term you will be able to: (1) characterize research methods commonly used in the social and behavioral sciences; (2) evaluate research methods applied in published research in your field; (3) design your own research project in a dissertation prospectus or grant proposal, and; (4) identify key ethical issues in the conduct of social and behavioral sciences. The course aims to help you understand how to avoid proposing ”impossible fieldwork to answer unanswerable questions” (to quote King, Keohane & Verba from Rethinking Social Inquiry p 182). Books to purchase (available at the University Book Store and online):

Gerring, John (2011). Social Science Methodology: A Unified Framework. 2nd edition. Cambridge University Press. [Referred to in the syllabus as Gerring] Available as an eBook, isbn: 9781139199919.

Note that this course assumes you have been exposed to the basics of research design elsewhere. If not, you are responsible for familiarizing yourself with the key concepts. If you have not explored research design previously, these two books may be helpful:

Gerring, John and Christenson, Dino. (Draft August 2015) An Applied Guide To Social Science Methodology. Cambridge University Press. Draft available online as of September 24, 2016 at http://blogs.bu.edu/jgerring/files/2016/06/Text-‐16-‐05-‐26.pdf

Neuman, W. Lawrence (2012). Social Research Methods: Qualitative and Quantitative Approaches. 7th edition. Allyn & Bacon /Pearson. [Referred to in the syllabus as Neuman] Also available in paperback (international economy version, Pearson India) as of 2014, and as an eBook at http://www.pearsonhighered.com/educator/product/Social-‐Research-‐Methods-‐Quantitative-‐and-‐Quantitative-‐Approaches-‐CourseSmart-‐eTextbook-‐7E/9780205646197.page

Alternative basic material in research design (not required for the course):

Trochim, William M. The Research Methods Knowledge Base, 2nd Edition. Internet WWW page, at URL: <http://www.socialresearchmethods.net/kb/> (version current as of October 20, 2006). [this is available free online. More recent editions cost a lot, but are improved. See e.g., Trochim, W., Donnelly, J. P., & Arora, K. (2016). Research methods: The essential knowledge base, 2nd edition. Cengage. ISBN-‐13: 9781133954774. Available as an ebook as well.]


All other readings will be made available through the course website, though your libraries will be more complete and interesting if you purchase the original volumes of books referenced. Grading Grades in this course are a function of your preparation, participation, and papers. This seminar will be conducted as a series of discussions, with the exceptional informal lecture or presentation. Read the material well in advance of our discussions so that you have time to think about it beforehand. Participation consists of presenting the assigned papers, conveying your own thoughts and reactions to the readings, and reacting to the presentations and thoughts of others in the class. Critique and explore others’ thoughts and work constructively—the more constructive and creative the better! Assignments and proposals will be evaluated according to criteria determined by the class and informed by National Science Foundation review criteria. To earn an A in the course, attend class every week and demonstrate through your presentations and active participation in the discussion that you have carefully considered the assigned readings. If you absolutely have to miss a class meeting for any reason, please submit typed answers to the weekly discussion questions in advance of the class. Course Requirements

1. Read all of the “required” readings and attend all class sessions. 2. Complete all assignments and exercises. 3. Produce a research design and, after the design is evaluated, a formal proposal—either a

dissertation prospectus or a grant proposal (e.g., for a dissertation grant or a research grant from the National Science Foundation)

4. Most important: all of us must be actively engaged in helping one another formulate and evaluate research issues and solve problems as they emerge. This interactive learning is much more important than the actual reading material or even the research proposal product.

To receive a top grade in this course and benefit most from it, you should:

1. Fulfill all requirements listed above 2. Read both the “required” and the “supplemental” readings. The readings are front-‐loaded, so

it’s especially important to keep on top of them the first few weeks. 3. Present supplemental readings in class (to be arranged with instructor) 4. Develop a high level of substantive knowledge of research and theory pertaining to the topic of

your proposal. 5. Begin communication/correspondence with other researchers concerned with your topic. 6. Complete by the end of the quarter a high quality, theory-‐based empirical research proposal,

complete with literature review, theory overview, methods and expected results. The proposal should be of fundable quality and should be ready to submit to a funding organization or your advisor by the end of the term. The paper should take your level of methodological sophistication at least one step beyond your current level.

Participation and readings presentation (30% of grade): Participate every week and be prepared to lead

discussion of one of the required readings each week. One way of doing this is to prepare a 1-‐page summary of the key points and questions from the reading. See schedule for required preparation.


Assignments and final proposal: Each of the five assignments is designed to promote progress toward your final proposal.

Assignment 1 (10% of grade, due October 17th): Draft your research question and briefly sketch a

design (2 pages double-‐spaced). This is a brainstorming exercise; be creative! (a) Write a researchable question for a policy or management topic that interests you. (b) Develop a primary research design to gather and analyze evidence addressing that question. You should select this primary design from one of the major categories of research approaches (e.g., ethnographic, case study/comparative method, statistical, experimental – choose one with which you are familiar) and (briefly) justify why you chose this design. (c) Interpret how different philosophies of social science might assess your design, and the results it might produce if implemented successfully. Be sure to address falsifiability.

Assignment 2 (10% of grade, due October 31st): Revise and expand your draft research design to include a draft literature review. See Gerring’s assignment (at the end of this syllabus) for further advice on developing your research design. Meet with the instructor individually to discuss your research design at least once before submitting this assignment (i.e., before November 1st).

Assignment 3 (10% of grade, due November 7th): Peer review of two draft research designs (written reviews, approximately 1-‐2 pages each). NSF review criteria must be applied, in addition to any other review criteria determined by the class. In each review, include a one paragraph summary of the proposed research design (in your own words).

Assignment 4 (30% of grade; due December 5th or four days before mock NSF panel): Final

research proposal. Assignment 5 (10% of grade; due at mock NSF panel meeting, date TBD – tentatively Friday

December 9th, 2015): Peer review of two full research proposals and summary of panel discussion for one of those. Extra credit will be awarded for completing a written review of an additional proposal. Consult sample panel reviews on course website.

Specifications for the required research proposal

• You must present a research design to the class and receive approval for the design from me before the proposal is begun in earnest. This is not to constrain your creativity, but to make sure that the research is feasible.

• You may work alone or in teams of two. If you have a co-‐author, that arrangement must be made early (and should be reflected in the research design preparation).

• All proposals must present explicit, testable hypotheses and describe criteria for the test of hypotheses.

• The proposal should include: an “introduction” section that frames the issue and presents a brief statement of the research and theory objectives of the research; a “data and methods” section that details sampling procedures, your approach to analysis, characteristics of the data and so forth (examples will be provided); a “research questions” section that presents hypotheses, relevant theory, and (if appropriate) a model; an analysis and “expected findings” section that presents statistical analysis and anticipated results; and a conclusions or implications section that links your results to those of others and discusses the policy relevance of the anticipated results, along with relevant caveats and possible future directions for research.


• Your literature review should be included in your research questions section and discuss (as pertinent) relevant literature and previous findings (i.e., the state of the literature).

• Note that each variable employed must be described precisely, both conceptually and with regard to measurement.

• A one page summary of the research proposal must be provided after the title page. • Your proposal should meet the most stringent criteria for form, grammar, spelling and clarity. • All tables and figures should be provided in the proposal. No appendices will be allowed. • All material within the body of the paper must be double-‐spaced. • Pages must be numbered. • Tables and figures must be printed, not drawn. • All material cited must be fully referenced. Proposals not citing directly relevant research will be

marked down. • Excluding tables, references and appendices, proposals may be no longer than 25 double-‐spaced

pages. There is no minimum length but 10-‐18 pages is a good target for a draft proposal. • Proposals may employ existing data (questionnaire, interview, or other), or include data collection.

Any proposed data collection or acquisition must take into account human subjects review and other ethical and fiscal exigencies (i.e., it should be feasible).

Specifications for the presentation of research designs You will present your research design at least twice. The first time conceptually (no data analysis strategy necessary), the second time proposed methods and data analysis are required (draft proposal presentation). In each case, you will have ten minutes to present and a discussant will have ten minutes to critique your design.

Specifications for the review of the research proposal The final presentation of proposals will simulate a National Science Foundation review panel. The proposer will be excused for the presentation of his or her own proposal. Each student will have exactly 10 minutes to present another student’s research proposal, including both a summary of the proposed research and a thorough evaluation/critique and suggested action (fund/do not fund). After the 10 minutes, one other panelist will have approximately 3 minutes to summarize his/her evaluation of the proposal. After 3-‐4 additional minutes devoted to questions from the instructor and other students, the panel will reach a final assessment. The date and time for the mock panel will be arranged but it will probably take place a few days after the last regularly scheduled class. All students will receive written panel summaries along with copies of the individual (anonymous) reviews of their proposal.

Timeline for research proposal:

• Initial research question and design sketch due October 17th • Literature review (context for your research question) begin October 17th • Research question panic (someone stole my idea! or, it’s old hat!) October 19st • Data collection/acquisition strategy (i.e. what data address your question?) October 24th • Data panic (i.e. how can I get the right data!) October 24th • Design consulting with Bostrom October 25th through October 29th • Draft research design due to instructor & discussants October 31st • Review and discussion of research designs November 7th • Draft proposal discussions November 28th • Final proposal due to instructor and to discussants December 5th • Final review panel TBD (tentatively Friday December 9th)


Calendar October 3, 2016 Class begins – please do the assigned reading before class. October 10, 2016 Short presentations (methodology of social scientific research) October 17, 2016 Assignment 1 due October 24, 2016 October 31, 2016 Assignment 2 due November 7, 2016 Assignment 3 due, peer discussion of research designs November 14, 2016 November 21, 2016 November 28, 2016 Draft proposal discussions. December 5, 2016 Last official class meeting, Assignment 4 proposal due. December 9, 2016 Tentative date for mock NSF panel (Friday, Dec 9th)

Schedule Week 1: October 3—Causation Week 2: October 10—On the methodology of (social) scientific research Week 3: October 17 – Getting started: Putting research questions in context. Week 4: October 24th -‐ An introduction to research design strategy, sampling and

measurement issues through meta-‐analysis. Week 5: October 31st —Case studies, comparative analysis and causality Week 6: November 7 —Experiments, Quasi-‐experimental design, random assignment,

and the ethics of experiments Week 7: November 14 – From ethnography to survey methodology, Part I Week 8: November 21– From ethnography to survey methodology, Part II Week 9: November 28—Statistical modeling and new approaches to research in the

social sciences Week 10: December 5—TBA (guest speaker invited).


Week 1: October 3—Causation

Introductions, course nuts and bolts, discussion of causation, tests of causal hypotheses, and strategies for designing research.

Note taker and summary: Student Required reading and preparation:

"Cargo Cult Science" excerpted from 'Surely You're Joking Mr. Feynman!' Adventures of a Curious Character by Richard Feynman, Bantam Books: New York, 1986 http://calteches.library.caltech.edu/51/2/CargoCult.htm Atul Gawande, The Mistrust of Science, New Yorker, June 10, 2016: http://www.newyorker.com/news/news-‐desk/the-‐mistrust-‐of-‐science Andrew Gelman’s September 21, 2016 blogpost with a timeline on the reproducibility crisis: http://andrewgelman.com/2016/09/21/what-‐has-‐happened-‐down-‐here-‐is-‐the-‐winds-‐have-‐changed/ American Statistical Association statement on p-‐values: http://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108#/doi/suppl/10.1080/00031305.2016.1154108

Gerring, John (2011). How good is good enough? A multidimensional, best-‐possible standard for research design. Political Research Quarterly, 64(3), 625-‐636. http://blogs.bu.edu/jgerring/files/2013/06/HowGoodIsGoodEnough.pdf Gerring Chapters 1-‐2 (pages xix-‐57)

Strongly recommended:

Henry Brady, “Causation and Explanation in Social Science.” In Janet Box-‐Steffensmeir, Henry Brady, and David Collier, eds., Oxford Handbook of Political Methodology (Oxford University Press, 2008), pp. 217-‐270. The Oxford Handbook is available as an e-‐book through UW libraries. Neuman chapters 1-‐4 (read as necessary, but critically; if you haven’t covered this kind of material previously, you will find this extremely helpful background for the course).

Foundations, in Trochim, William M. The Research Methods Knowledge Base, 2nd Edition. Internet WWW page, at URL: <http://www.socialresearchmethods.net/kb/> (version current as of October 20, 2006). [this is available free online]


Week 2: October 10 – On the methodology of (social) scientific research

Study questions: Does science require falsification? What constitutes a research program? Note taker and summary: Required reading and preparation:

Karl Popper. 1963. "Science: Conjectures and Refutation," in Martin Curd & Jan A. Cover (eds.), Philosophy of Science: The Central Issues, New York: Norton, 1998, pp. 3-‐10. [presenter: ] Karl Popper. 1959. "The Problem of Induction," in Martin Curd & Jan A. Cover (eds.), Philosophy of Science: The Central Issues, New York: Norton, 1998, pp. 426-‐432. [presenter: ] Thomas S. Kuhn. 1962. Chapter IX The Nature and Necessity of Scientific Revolutions, of The Structure of Scientific Revolutions. University of Chicago Press. Chapter IX is available at http://www.marxists.org/reference/subject/philosophy/works/us/kuhn.htm [presenter: ] Lakatos, Imre Lectures 1-‐3 on Scientific Method (pp 19-‐51) in Motterlini, Matteo (Ed), For and Against Method. University of Chicago Press / Chicago & London, 1999. [presenter: ] Imre Lakatos. 1970. "Falsification and the Methodology of Scientific Research Programmes." In Imre Lakatos and Alan Musgrave, (eds.). Criticism and the Growth of Knowledge. Cambridge: Cambridge University Press, pp. 91-‐196. [skim the last half] [presenter: ] Clifford Geertz (1973). "Thick Description: Toward an Interpretive Theory of Culture." In The Interpretive Theory of Culture. Chapter 1, Boulder: Basic Books, pp. 3-‐30. [presenter: ] Max Weber, "Objectivity in Social Science and Social Policy." In Fred R. Dallmayr & Thomas A. McCarthy (eds.), Understanding and Social Inquiry. Notre Dame: University of Notre Dame Press, 1977, pp. 24-‐37. [presenter: ] Gerring chapters 3-‐5 (pages 58 -‐140) [All]

Supplemental resources/reading:

Raadschelders, J. The Future of the Study of Public Administration: Embedding Research Object and Methodology in Epistemology and Ontology. Public Administration Review, Nov/Dec 2011, 916-‐924.


Week 3: October 17 – Getting started: Putting research questions in context. “If I have seen further it is by standing on the shoulders of giants” -‐ Isaac Newton

Study question : What is a good research question?

Note taker and summary: Required reading and preparation:

Howard S. Becker. 1986. “Terrorized by the Literature” In Writing for Social Scientists: How to Start and Finish Your Thesis, Book or Article. University of Chicago Press, pp 135-‐149. [presenter: ]

Brady, Henry E. and David Collier (2004). Chapters 1 and 2 of Rethinking social inquiry: Diverse Tools, Shared Standards. Rowman & Littlefield Publishers. [presenter: ] PS-‐The Political Methodologist vol 6-‐2 Symposium on Designing Social Inquiry Parts I-‐II (by Burton and Brady respectively), Spring 1995, American Political Science Association. [presenter: ] Gerring, John and Craig W. Thomas. Quantitative and Qualitative: A Question of Comparability. International Encyclopedia of Political Science, Bertrand Badie, Dirk Berg-‐Schlosser, Leonardo Morlino (eds), Sage, 2011. [presenter: ]

Two examples of types of theory and research on them— 1. Simon, Herbert A. (2000). Public Administration in Today's World of Organizations and

Markets. PS: Political Science and Politics, Vol. 33, No. 4 (Dec., 2000), pp. 749-‐75. Stable URL: http://www.jstor.org/stable/420911. [presenter: ]

Simon, Herbert A. Rational Decision-‐Making in Business Organizations. Nobel Memorial Lecture, 8 December, 1978. [presenter: ]

2. Poteete, A and Ostrom E (2008). Fifteen Years of Empirical Research on Collective Action in

Natural Resource Management: Struggling to Build Large-‐N Databases Based on Qualitative Research. World Development Vol. 36, No. 1, pp. 176–195. [presenter: ] Ostrom, Elinor. Beyond Markets and States: Polycentric Governance of Complex Economic Systems. Nobel Prize Lecture, 2009. [presenter: ]

Assignment 1 is due. Strongly recommended:

Neuman chapters 5-‐6 Dagobert Manteltasche. “Otto I.Q. Besser-‐Wisser. PS, pp 730-‐731.

Committee on Behavioral and Social Science Research to Improve Intelligence Analysis for National Security, National Research Council. "Chapter 7 Intuitive Theories of Behavior-‐-‐Hal R. Arkes and James Kajdasz." Intelligence Analysis: Behavioral and Social Scientific Foundations. Washington, DC: The National Academies Press, 2011. http://www.nap.edu/openbook.php?record_id=13062&page=143


Week 4: October 24th -‐ An introduction to research design strategy, sampling and measurement issues through meta-‐analysis.

Study questions: What about sampling is important, and why? What is bias?

Note taker and summary: Invited guest speaker: Professor Marieka Klawitter Required reading and preparation:

Gerring chapter 7 (pages 155-‐192). [presenter: ]

William R. Shadish, Thomas D. Cook, and Donald T. Campbell. 2002. “Construct Validity and External Validity.” In Experimental and Quasi Experimental Designs for Generalized Causal Inference. Houghton Mifflin, pp. 64-‐82. [presenter: ] R. Rosenthal and M. R. DiMatteo. Meta-‐Analysis: Recent Developments in Quantitative Methods for Literature Reviews Annu. Rev. Psychol. 2001. 52:59–82. [presenter: ]

http://www.campbellcollaboration.org/resources/research/Methods_Links.php [presenter: ] http://www.cochrane.org/training/cochrane-‐handbook [skim Cochrane handbook] [presenter: ]

Strongly recommended:

Neuman chapters 7-‐8

David Garson’s webpages on Research design (64 pages including references)

[Executive summary] Finding What Works in Health Care: Standards for Systematic Reviews (2011). http://www.nap.edu/catalog.php?record_id=13059 Authors: Jill Eden, Laura Levit, Alfred Berg, and Sally Morton, Editors; Committee on Standards for Systematic Reviews of Comparative Effectiveness Research; Institute of Medicine. National Academy Press, Washington DC.

Becker, H. Tricks of the Trade, chapter on sampling.


Week 5: October 31st —Case studies, comparative analysis and causality

Study question: What is a case study? What are case studies good for? Note taker and summary: Required reading and preparation:

Gerring chapters 8-‐12 (pages 197-‐357) [Presenter: ] (skim chapters 8-‐9, read chapters 10-‐11, skim chapter 12). Gerring, John (2004). What is a case study and what is it good for? American Political Scientist Review, 98 (2), 341-‐354. [Presenter: ] Elman, Colin, John Gerring, and James Mahoney. Case Study Research: Putting the Quant Into the Qual. Sociological Methods & Research August 2016 45: 375-‐391, first published on April 21, 2016 doi:10.1177/0049124116644273 (Introduction to special issue on case study research). John Gerring and Lee Cojocaru. Selecting Cases for Intensive Analysis: A Diversity of Goals and Methods. Sociological Methods & Research August 2016 45: 392-‐423, first published on February 17, 2016 doi:10.1177/0049124116631692 [Presenter: ] Seawright, Jason. The Case for Selecting Cases That Are Deviant or Extreme on the Independent Variable. Sociological Methods & Research August 2016 45: 493-‐525, first published on April 21, 2016 doi:10.1177/0049124116643556 [Presenter: ]

Assignment 2 due. Strongly recommended:

Neuman chapter 14

Ragin, Charles C, Joane Nagel and Patricia White. July 11-‐12, 2003 Workshop on the Scientific Foundations of Qualitative Research. National Science Foundation report, 2004. Read pages 1-‐18, and the appended papers. These are required reading for later in the course:

• Becker (45-‐48) [Becker, Burton, Collier] • Burton (59-‐70) • Collier (71-‐76) • Ragin (109-‐116) [Ragin, Satterfield and Silbey] • Satterfield (117-‐120) and • Silbey (121-‐126).

Rihoux, Benoît (2006) Qualitative Comparative Analysis (QCA) and Related Systematic Comparative Methods: Recent Advances and Remaining Challenges for Social Science Research International Sociology 2006 21: 679.

Seawright, Jason and John Gerring (2008). Case Selection Techniques in Case Study Research: A Menu of Qualitative and Quantitative Options. Political Research Quarterly, Vol. 61, No. 2 (Jun., 2008), pp. 294-‐308.


Week 6: November 7 —Experiments, Quasi-‐experimental design, random assignment, and the ethics of experiments

Study questions: Why random assignment? What characterizes a “good” experiment in the policy sciences?


William R. Shadish, Thomas D. Cook, and Donald T. Campbell. 2002.“Experiments and Generalized Causal Inference.” In Experimental and Quasi-‐Experimental Designs for Generalized Causal Inference. Houghton Mifflin, pp. 1-‐32. [Presenter: ] William R. Shadish, Thomas D. Cook, and Donald T. Campbell. 2002.“Statistical Conclusion Validity and Internal Validity.” In Experimental and Quasi-‐Experimental Designs for Generalized Causal Inference. Houghton Mifflin, pp. 33-‐53. [Presenter: ] William R. Shadish, Thomas D. Cook, and Donald T. Campbell. 2002. “Quasi-‐Experiments: Interrupted Time-‐Series Designs.” In Experimental and Quasi-‐Experimental Designs for Generalized Causal Inference. Houghton Mifflin, pp. 171-‐206. [Presenter: ] Symposium (essays by Gary King, Paul Herrnson, and Kenneth Meier). 1995. “Verification/Replication.” PS 28, pp. 443-‐459. [Presenter: ] The Reproducibility Project: http://openscienceframework.org/project/EZcUj/ [Presenter: ] Revisit readings from Week 1

Assignment 3 due. Strongly recommended:

Gerring Chapters 8-‐11 [these will be very helpful for week 10] Neuman Chapters 6-‐9 Replication data & re-‐analysis: Dataverse, http://gking.harvard.edu/data ; skim e.g., A Method of Automated Nonparametric Content Analysis for Social Science, by Daniel J. Hopkins Georgetown University and Gary King Harvard University [All]


Week 7: November 14 – From ethnography to survey methodology, Part I

Study questions: What information about methods and methodology should be reported in a scientific paper that employs qualitative research? Survey research?

Note taker and summary: Required reading and preparation: (DRAFT – may change) On the scientific foundations of qualitative research.

Wedel JR, Shore C, Feldman G and Lathrop S (2005) Toward an Anthropology of Public Policy. Annals of the American Academy of Political and Social Science Vol 600, pp 30-‐51. [Presenter: ] Robert K Merton. 1987. The Focussed Interview and Focus Groups: Continuities and Discontinuities. Public Opinion Quarterly 51: 550-‐566. [Presenter: ]

Ragin, Charles C, Joane Nagel and Patricia White. July 11-‐12, 2003 Workshop on the Scientific Foundations of Qualitative Research. National Science Foundation report, 2004. Read pages 1-‐18, and the appended papers by

• Becker (45-‐48), [[Presenter: ] • Burton (59-‐70), [[Presenter: ] • Collier (71-‐76), [Presenter: ] • Ragin (109-‐116), [Presenter: ] • Satterfield (117-‐120) [Presenter: ] • Silbey (121-‐126). [Presenter: ]


Week 8: November 21– From ethnography to survey methodology, Part II On scientific advances in survey methodology. Note taker and summary:

Norman M. Bradburn, Seymour Sudman and Brian Wansink (2004). Asking Questions: The Definitive Guide to Survey Design. Wiley. Chapters 1-‐3. [Presenter: ] Jon A. Krosnick and Stanley Presser, Question and Questionnaire Design in Handbook of Survey Research (2nd Edition) James D. Wright and Peter V. Marsden (Eds). San Diego, CA: Elsevier, 2010. [Presenter: ] Michael F. Schober and Frederick G. Conrad, “Survey Interviews and New Communication Technologies” Chapter 1 in Envisioning the Survey Interview of the Future, edited by Frederick G. Conrad and Michael F. Schober. New York, Wiley, 2008. [Presenter: ]

Strongly recommended: Neuman Chapters 10-‐13 AAPOR guidelines for survey research. See especially http://www.aapor.org/Standards-‐Ethics/Best-‐Practices.aspx http://www.aapor.org/Standards-‐Ethics/AAPOR-‐Code-‐of-‐Ethics.aspx http://www.aapor.org/Education-‐Resources/For-‐Researchers/Poll-‐Survey-‐FAQ/Question-‐Wording.aspx [Presenter: ]

Supplemental reading and resources:

Joseph C. Hermanowicz (2002). The Great Interview: 25 Strategies for Studying People in Bed. Qualitative Sociology, Vol. 25, No. 4, Winter. [Presenter: ]

Consortium on Qualitative Research Methods: http://www.maxwell.syr.edu/moynihan/cqrm/The_Institute_for_Qualitative_and_Multi-‐Method_Research/


Week 9: November 28—Statistical modeling and new approaches to research in the social sciences (individual meetings with instructor on proposal this week too)

Study questions: What counts as evidence? Are there better (new) ways of testing social science theories?


King, Gary. "The Changing Evidence Base of Social Science Research." In The Future of Political Science: 100 Perspectives, edited by Gary King, Kay Schlozman and Norman Nie. New York: Routledge Press, 2009. copy at http://j.mp/k5lI9s (http://gking.harvard.edu/gking/files/evbase.pdf) Re modeling: Take a look at the replication data and at least one re-‐analysis at Dataverse: http://gking.harvard.edu/data [Presenter: ]

Automated content analysis: See http://gking.harvard.edu/category/research-‐interests/applications and for an engaging example of this read King, Gary, Jennifer Pan, and Molly Roberts. How Censorship in China Allows Government Criticism but Silences Collective Expression. Working Paper, 2012. [Presenter: ]

John Wilkerson on data scraping / tools for text: http://toolsfortext.wordpress.com/ Loren Collingwood & John Wilkerson (2012): Tradeoffs in Accuracy and Efficiency in Supervised Learning Methods, Journal of Information Technology & Politics, 9:3, 298-‐318. http://dx.doi.org/10.1080/19331681.2012.669191 [Presenter: ] Simulation (e.g., agent-‐based modeling) and systems modeling: Erik W. Johnston, Darrin Hicks, Ning Nan, and Jennifer C. Auer. Managing the Inclusion Process in Collaborative Governance. J Public Adm Res Theory (2011) 21(4): 699-‐721 first published online August 19, 2010 doi:10.1093/jopart/muq045 . [Presenter: ] Committee on Organizational Modeling: From Individuals to Societies, National Research Council, Part II: State of the Art in Organizational Modeling." Behavioral Modeling and Simulation: From Individuals to Societies. Washington, DC: The National Academies Press, 2008. http://www.nap.edu/openbook.php?record_id=12169&page=93 [study the figure] [Presenter: ]


Week 10: December 5th – Assignment 4 due. Invited speaker on psychology and judgment and decision research in public policy or on eSciences for policy research

Friday December 9 (TENTATIVE DATE) – Proposal review panel For the final review panel, each class member will contribute as follows:

(a) Final research proposal submitted at least four days in advance of the panel. (b) Short written reviews of two proposals (assigned by the instructor), to be presented orally to the

review panel (comprised of everyone in the class except the author of that proposal being reviewed).

(c) Lead reviewer on one proposal, required to summarize the panel’s discussion of that proposal in a short paragraph (during the panel).


To guide your readings (especially in Gerring and Neuman), here are comprehensive exam questions in this area from previous years. 2016 (both days one and two were open book in 2016) To be added. 2015 (both days one and two were open book in 2015) Day One: Students must answer all questions below, which are based on the following paper (previously distributed to you): Ladd, H. F., Muschkin, C. G., & Dodge, K. A. (2014). From Birth to School: Early Childhood Initiatives and Third‐Grade Outcomes in North Carolina. Journal of Policy Analysis and Management, 33(1), 162-‐187. Ladd et al. talk about spillover effects. Describe what the authors mean by spillover effects (be specific), address the advantages/disadvantages of their research design with regard to identifying and estimating spillover effects from each of the programs, and discuss whether spillover effects threaten the validity of the authors' conclusions given their efforts to address them. Day Two: Develop a research design that combines qualitative and quantitative elements to test a hypothesis about the interaction of institutions and the policy process (i.e., drawing on one or more institutional and policy process theories). To keep your design manageable you may consider restricting it to one sector, type of policy, type of policy-‐making institution, and/or geographic or cultural area. Your hypothesis must explicitly draw from at least one policy process theory and at least one institutional theory. Be open about the strengths and weaknesses of your research design. You may not use a research design you proposed in a class paper last year. 2014 (closed book) Day One: “Can targeted transfers improve birth outcomes? Evidence from the introduction of the WIC program”, by Hilary Hoynes, Marianne Page, and Ann Huff Stevens, Journal of Public Economics 95 (2011) 813–827. Please answer the following questions regarding this article which was previously distributed.

1. Do the authors' data and methods allow them to effectively test and appropriately discuss the hypotheses or issues under discussion? The authors state, “The validity of our research design hinges on the exogeneity of county WIC start dates.” What do they mean by this?

2. Why are the findings in table 3 estimates of the “intent to treat” impact of WIC? Why do the authors convert these estimates into the impact of the “treatment on the treated?”

3. If the implementing agency had planned to do an evaluation of the impact of the WIC program on birth weights prior to the rollout, what alternative research design would you have proposed at that time? Specify the necessary assumptions about available data to carry out your design.

Day Two:


Develop a research design on a policy problem of your choice (this may not be a design proposed in one of your class papers last year) that tests two competing theories of the policy process, out of the following three: (i) social construction, (ii) diffusion, (iii) network theory. Be sure to identify the key authors for the two theories you compete. 2013 Day One: We would like to receive your critical review of the following paper (previously distributed to you) on the dimensions identified below: Harper, Sam, Strumpf, Erin, Burris, Scott C., Davey Smith, George and Lynch, John, "Do Mandatory Seat Belt Laws Affect Socioeconomic Inequalities in Seat Belt Use?" Downloaded from the Social Science Research Network on July 22, 2013. Critique the article's Research Design. First, explain what question(s) the authors are seeking to answer and why. Do the authors' data and methods allow them to effectively test and appropriately discuss the hypotheses or issues under discussion? What are the main advantages and limitations of their approach? Is there an alternative method (or methods) that would have been better suited to effectively test the authors' hypotheses? If so, briefly explain what it is (they are) and your rationale. Why do you think the authors used the methods employed in this paper rather than alternative methods? Make sure that your answers address the “Selective Recruitment” hypothesis discussed in the article. Day Two: Develop a feasible research design on a policy problem of your choice (this may not be a design proposed in one of your class papers last year or the one addressed in yesterday’s article) that tests two competing theories of the policy process, from the following three: (i) social construction, (ii) diffusion, (iii) network theory. Be sure to identify the key authors for the two theories you compete. 2012 Please answer the following question. Researchers trained in experimental, statistical, and/or ethnographic research designs and methods often completely misunderstand comparative and case study research designs and methods. Describe and explain King, Keohane and Verba's "quasi-‐statistical" approach to comparative and case study research designs and methods. Then review the major criticisms of this quasi-‐statistical approach from competing comparative and case study researchers (Brady and Collier, McKeown, Ragin, Bennett, etc.). What are ideas about comparative and case study research designs and methods offered and promoted by these researchers that are different from (or similar to) the "quasi-‐statistical" approach? Which of these two schools of thought do you find more compelling and useful to your own research program? Why? 2011

Answer one of the questions below.

1) What are the key issues one should consider when conducting applied research/evaluative research using experimental/quasi-‐experimental versus other types of research designs and methods? On what types of applied research and evaluation questions do the different designs and methods shed light? What are the benefits and challenges of these different types of designs and methods? Illustrate these trade-‐offs by discussing at least two different research designs to conduct applied/evaluative research in


an area of interest to you. Conclude with a statement about which research design would be preferable given the area you have chosen.

2) A common error in research is for authors to claim they are doing “thick description” when they are actually deploying other types of case study and comparative methodologies (process-‐tracing, within-‐case comparison, general elimination, counterfactual thought experiments, etc.). How is thick description, as developed by Geertz, different from or similar to these other types of case study methods? Be sure to discuss epistemological issues such as meaning, inter-‐subjective understandings, causality, generalizability, and other issues you believe are key differences. Conclude with a discussion of how you would use different types of case study methods in an area of interest to you.

2010

Write a researchable question for any policy or management topic that interests you. It must be on a different topic than the one you wrote on in the Research Design seminar. Develop a primary research design to gather and analyze evidence addressing that question. You should select this primary design from one of the major categories of research approaches (e.g., ethnographic, case study/comparative method, statistical, experimental) and justify why you chose this design. Then develop a secondary “mixed-‐methods” design that builds on your original proposal. Ideally the mixed-‐methods design would generate different types of data (quantitative or qualitative) and complementary analysis to your primary design. Compare the two designs in terms of the trade-‐offs (methodological, operational, etc.) you think are critical for choosing one design over the other. The purpose of this question is to demonstrate how well you can identify and analyze the relative trade-‐offs among research designs for answering a particular question. Therefore, do not set up one of the designs as a straw person (i.e., set up one of the designs as intentionally weak to demonstrate the superiority of the alternate design). Finally, interpret how different philosophies of social science might assess the two designs, and the results they might produce if implemented successfully.

2009

Please respond to each of the questions below regarding the attached article by DeHart-‐Davis, entitled: “A Theory of Effective Organizational Rules,” JPART 19(2), 2009.

1. The author proposes a new theory of green tape that is juxtaposed with red tape.

a. What is “tape” (as implied by the article)? b. How does “tape” vary? (e.g., is it only red or green)?

2. On p. 377, the author states: “The results of this analysis (table 7) indicate convergent and discriminant [divergent] validity of the green tape construct.”

c. What does she mean by this? d. How does table 7 support her claim? e. What do these results imply about using the measures for both green tape and red

tape?

3. The research design on p.363 describes the 4 cities in the sample. The author is not clear, however, about how she selected these cities.


f. If we assume they were selected non-‐randomly, what does this imply about external validity (generalizability)?

g. Does the author adequately speak to generalizability in the last paragraph on p. 379?

4. Imagine that you are designing a follow-‐up study to test green-‐tape theory.

h. Which hypothesis would you test? (Please identify only one.) i. How would you test it? (Be sure to discuss your sampling strategy, dependent variable,

independent variables, measures, and other considerations you think important.) j. What is the epistemology (philosophy of science) underlying the research design of your

follow-‐up study?

2008

Answer one of the following questions:

a) Write a research question for any policy or management topic that interests you. Develop two competing research designs to answer that single question. One of the designs must use solely qualitative methods; the other, solely quantitative methods. The question must be the same for both research designs. Compare the two designs in terms of any and all trade-‐offs you think are important for choosing one design over the other. Conclude your analysis by selecting either the quantitative or the qualitative design; you may not conclude by choosing both designs as parts of a mixed-‐methods strategy (even if doing so may seem like the best choice an actual research project). The purpose of this question is to demonstrate how well you can identify and analyze the relative trade-‐offs, not to demonstrate that you can develop the best possible research design for your question.

b) Random assignment methodology is considered the “gold standard” for policy and program evaluation. Why? So, if random assignment is so great why isn’t it more widely used in practice? What are its major limitations? What do we know about how effectively various quasi-‐experimental evaluation designs can substitute when random assignment cannot be used? Illustrate your answer with reference to relevant literature, examples and specific designs.


To help you think about your proposal, here is Gerring’s assignment (which I’ve edited somewhat). The original is available under “Resources” for Gerring’s book @ http://www.cambridge.org/gb/knowledge/isbn/item6566290/?site_locale=en_GB The major written work for this class consists of a research proposal on a subject of your own choosing. You will turn in two drafts; however, only the grade on the final draft will count towards your grade. (Turn in all drafts by email attachment and in Word format so that I can insert comments in the text.) This proposal should take approximately the same form as a dissertation prospectus or grant proposal. Indeed, you may consider this assignment as a dress-‐rehearsal for your dissertation or grant proposal. It should include a big theory (what it’s all about; the theoretical interest), a specific hypothesis or set of related hypotheses, and a research design (how you propose to investigate your hypothesis). Be as clear and well-‐organized as possible. Anticipate possible objections. In addition to this brief set of guidelines, you are well-‐advised to consult various sources on writing and publishing listed in the syllabus and on the course website. OBJECTIVES 1. The theory, and the hypothesis, informing the study must be fairly general in scope. It must be of interest to a broader audience in public policy and management. 2. You are strongly encouraged to make a causal argument, rather than a predictive or descriptive one. Predictive arguments may flow from causal arguments (indeed, they may be unavoidable), but they would not typically be the main subject of a proposal in the policy sciences. The reasons for preferring causal over descriptive arguments are more complicated and should be briefly reviewed. First, descriptive inference is in some respects harder (as discussed in the course). Second, we will be talking mostly about causal arguments during the course of the term. Third, most social and behavioral sciences are obsessed with causal arguments, so it is a good idea to figure out how they work. And finally, there is more pay-‐off to you (on the job market or wherever you end up). 3. You must propose a specific hypothesis. Clarify, if it is not entirely clear, what change on X is predicted to result in what change in Y. Of course, you may be unsure about which of several possible hypotheses to focus on. It is natural to begin a dissertation with a high degree of uncertainty. However, it is not possible to write a convincing proposal by merely stating a series of questions. Exploratory proposals are possible, but only if there are some plausible expectations that render the proposal interesting – theoretically and/or substantively. The more specific you can be, the better, for without such specificity it is very difficult to engage questions of method – the primary purpose of the course. 4. A dissertation is a large piece of work so there is space to make more than one argument. The proposal, however, is a very short piece of work and there is space for only one main hypothesis, or a set of hypotheses that are tightly


integrated. Do not suppose that the proposal must incorporate all that you will deal with in the dissertation (and the eventual book or set of articles that you plan to write). 5. Remember that you need not stick with your chosen theory and hypothesis through the rest of your graduate career. This is an exercise, not a final product. Its purpose is largely heuristic, that is, to help you think through the process of conceptualizing and implementing research – and, more specifically, writing a dissertation. It does not matter to me whether you do end up doing what you say you will be doing. Consider the proposal a hypothetical plan of action. It matters whether this plan is workable, but it does not matter if you choose to abandon it or dramatically reformulate it in future years. 6. The literature review must be extensive enough to show the value-‐added of your suggested project. That is, you need to verify whether your idea has already been explored by other scholars, and if so with what results. If your contribution is empirical rather than theoretical, then you need to show that your research design is better than – or adds something significant – to the body of extant empirical work on the subject. Your review should involve printed sources (published books and articles, as well as unpublished papers) but also direct contact with scholars working in the chosen subfield. Sometimes, the latter is the best way to arrive at a determination of whether a topic is truly novel, or merely commonsense, and whether it is workable. You may be lucky enough to find one of these scholars at the University of Washington. But if not (and given the specialization of the academy, this is unlikely) you should consult scholars by email wherever they happen to be, or consider sending an inquiry to the appropriate listserv (after you have conducted your initial literature review and checked the archives of the listserv for similar inquiries). Remember that your prof in this course is not an expert in everything (some might argue that she is an expert in nothing). The most that I can do is to weigh in on the methodological components of your proposal; its substantive contribution is probably not an area in which I will have much to say (and if I do say anything you should take it with a grain of salt). ORGANIZATION

As a summary of your proposal, please include a one page summary, along with your name, the title of the project, and the date, according to the instructions found in the NSF Grant Proposal Guide at http://www.nsf.gov/publications/pub_summ.jsp?ods_key=gpg . This summary should include a brief statement of your objectives (theory and hypotheses), description of your methods (research design), and separate paragraphs describing the intellectual merit of the proposed research (how it will advance public policy and management sciences) and the resulting broader impacts.

This is the format employed by the National Science Foundation for its proposal summaries and you might want to scan the summaries of funded awards (search awards at www.nsf.gov) if you are unsure what the above means or it can be written


in a single page. Of course, this is highly reductive. You will have plenty of chance to explain if your proposal does not fit neatly into these boxes. In the body of the proposal, you might consider the following organization (keeping in mind that this will vary somewhat according to the topic, the state of the literature on this topic, and your argument): Introduction. Introduce the general topic or question of your research. What’s the big picture? You should say something about the everyday or policy significance of the topic if not this is not apparent. General theory and literature review. Clarify what the value-‐added of your study might be, relative to extant work on the subject. There are three ways of establishing this. You may point out that a) this topic is insufficiently studied; b) there are important unresolved questions (debates); or c) the prevailing wisdom on this matter is wrong. (These three tacks are not mutually exclusive.) Do not write pages and pages of literature review. Try to be as concise as possible, while remaining comprehensive, in your review. The best way to do this is usually to structure your discussion by way of substantive points, citing authors as you go. For example, rather than reviewing what Smith (1980), Jones (1999), and Hall (2005) have to say, seriatim, disaggregate the literature on the topic by its substantive findings and/or methods. For example: “There are three approaches to the question of the democratic peace: a) the case study (e.g., Smith 1980), b) the crossnational statistical study (e.g., Jones 1999), and c) the formal model (e.g., Hall 2005).” If the literature on your subject is vast and complicated, you might consider presenting them in tabular format. E.g., Study Finding Sample Method Weakness Smith (1980) Positive Switzerland Case study Jones (1999) Negative All countries,

1960-‐ Large-‐N Xnatl

Hall (2005) Positive None Formal model Hypothesis. Sometimes, the hypothesis can be stated as part of the general theory. Sometimes, it is helpful to introduce it later, as an instantiation of that theory. In any case, if your argument is rather complicated, draw a diagram showing how the major factors in your theory inter-‐relate. As an example, here is a diagram that I constructed for a recent project on democracy and development:


Research design. Next, discuss the nature of the evidence that you will be evaluating and the form of analysis that you will employ. Since this is the main topic of the course, this is the section that I will be paying closest attention to. If there are a relatively small number of cases and a large number of variables, I strongly encourage you to construct a “truth table” in which you score each case (or each case type) on each dimension (i.e., on each independent and dependent variable). This will allow you – and us – to evaluate the evidence in a concise format. Of course, this is not possible in projects that incorporate a large sample or where the scoring of cases is unknown, as with experiments. Be sure you justify your choice of research design. How does your approach differ (or not) from other writers? Why did you choose this research design, and not others? Finally, and very importantly, discuss the possible weaknesses of this research design. Recall that the objective of this course is to teach methodology, not simply to develop good research. This means enhancing methodological self consciousness. If your research design has flaws or limitations (as all do), acknowledge these. Your job is not to identify a perfect research design but rather the one that is “best possible,” under the circumstances – given limited time, resources, access to materials, ethical constraints, and so forth.

1. Economic policy 2. Infrastructure 3. Policy continuity 4. Social peace 5. Environmental policy 6. Education 7. Public health 8. Gender equality 9. Economic growth

1. Policy investments 2. Learning 3. Institutionalization 4. Inclusion 5. Consensus, Stability

Causal pathways

Outcomes

Accumulation of democratic stock

Causal factor

universityofwashington* …september(28,(2016(5"calendar***! october 3, 2016 class begins –...

Documents