introduction to using qualitative methods in impact evaluation ryan sheely kennedy school of...
TRANSCRIPT
DIME – FRAGILE STATESDUBAI, MAY 31 – JUNE 4
Introduction to Using Qualitative Methods in Impact Evaluation
Ryan SheelyKennedy School of GovernmentHarvard UniversityDIME Workshop, DubaiJune 1, 2010
Multiple Reasons for Using Participatory Methods
This morning’s session emphasized social justice and accountability as reasons to utilize participatory methods in impact evaluation
There are are also methodological justifications for integrating participatory assessment and other qualitative measures with randomized impact evaluation
What Are Qualitative Methods?
Qualitative methods are a diverse set of research tools that are primarily concerned with the meaning of social actions, statements, and documents
Both alone and in combination with randomized evaluation and sample surveys, these methods provide leverage with respect to: Describing and categorizing social contexts Interpreting the social behavior, actions,
and outcomes Tracing the specific processes and
mechanisms whereby programs lead to effects
The Diversity of Qualitative Methods
Participatory and Action-Based Research
Interviewing and Discussion
Content Analysis of Documents and Records
The Diversity of Qualitative Methods
Participatory and Action-Based Research Systematic Social Observation Drama and Role Playing Maps, Models, and Diagrams Ranking and Scoring
Interviewing and Discussion
Content Analysis of Documents and Records
The Diversity of Qualitative Methods
Participatory and Action-Based Research
Interviewing and Discussion In-Depth, Semi-Structured Interviews Focus Group Discussions Oral Histories of Individuals, Communities, or
Projects
Content Analysis of Documents and Records
The Diversity of Qualitative Methods
Participatory and Action-Based Research
Interviewing and Discussion
Content Analysis of Documents and Records Diaries or Personal Time/Financial Records Archives and Government Records Newspapers and Civil Society Reports
Analyzing Qualitative Data: The Role of Narrative
Despite the differences between these methods of collecting qualitative data, the common principle is using close observation to understand the meaning of what people say and do
As a result, the writing of field notes and systematic the aggregation of notes into narrative case studies are a central skill in analyzing qualitative data
Combining Qualitative and Quantitative Methods
Qualitative methods can be linked to randomized evaluation
Before Implementation: Use interviews, participation, and observation to design
contextually relevant surveys and behavioral activities Use observation or participation to design interventions
that can test the interaction of different program components
After Implementation: Aggregating and coding the content of interviews,
behavioral observation, and newspapers to cross-check survey data
Analyzing process to understand the lack of statistically significant differences▪ Implementation failure vs. design failure?
Homework: Qualitative Methods and Your Projects
When you meet with your clinic groups this afternoon, consider this question:
What types of qualitative methods would add value to the evaluation of your project?
Focus on three things:1) Identify each question you want to answer2) Choose a set of qualitative methods to
integrate into your evaluation design for each question and how/when you would use each method
3) Why these methods add value to your research design
Qualitative Methods:Methodological Limitations
Small, Nonrepresentative samples Researcher/Facilitator Bias Possibility of Contamination of Control
Groups when combining with Randomized Evaluation
Lack of standards for comparing qualitative and quantitative findings
Qualitative Methods:Practical Limitations Time and Cost of Qualitative Research
Understanding and Documenting Context Requires Time to Develop Mutual Trust Between Researcher and Community
Increased Staffing and Training due to more diverse skill sets
Coordination and Management Problems Associated With Larger Teams
Conclusions: Mixing Qualitative and Quantitative Methods
Qualitative Methods are a diverse set of tools and skills that require training, practice, and careful research design
Qualitative methods can help to better understand why a given program did or did not have an effect Helping to develop better treatments and outcome
measures for randomized evaluations Using local knowledge and cultural context to
interpret the meaning of social behavior and process of intervention
Ideal mix of qualitative and quantiative methods depends on the questions you want to answer and the local context
REFERENCES AND SUGGESTIONS FOR FURTHER READING
Bamberger, Michael, Vijayendra Rao and Michael Woolcock (2010) “Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development”, in Abbas Tashakkori and Charles Teddlie (eds.) Handbook of Mixed Methods (2nd revised edition) Thousand Oaks, CA: Sage Publications
Barron, Patrick, Rachael Diprose and Michael Woolcock (2010) Contesting Development: Participatory Projects and Local Conflict Dynamics in Indonesia New Haven: Yale University Press (in press)
Chambers, Robert. 1997. Whose Reality Counts: Putting the First Last, ITDG Publishing.
Cronk, Lee. 2007. “The influence of cultural framing on play in the trust game: a Maasai example” Evolution and Human Behavior.
Kanbur, Ravi, ed . 2001. Q-Squared? Qualitative and Quantitative Methods of Poverty Appraisal
Malan, N. 2000. “On the Relationship between participatory research and participatory development.” Africanus 2: 64-75.
Ravallion, Martin. 2001. “How Can Qualitative Methods Help in Measuring Poverty,” Qual-Quant: Qualitative and Quantitative Poverty Appraisal- Complementarities, Tensions and the Way Forward, Permanent Black Publishers, 2002.
Rodrik, Dani. 2008. “The New Development Economics: We Shall Experiment, but How Shall we Learn?” Paper Prepared for the Brookings Development Conference, May 29-30 2008.
Sheely, Ryan. 2010. “Community Governance, Collective Action, and the Maintenance of Local Public Goods: Qualitative and Experimental Evidence from Rural Kenya.” Better Governance for Better Health Conference, Stanford University. April 26-27, 2010.
Van der Riet, Mary. 2008. “Participatory Research ant he Philosophy of Social Science.” Qualitative Inquiry 14 (4): 546-565.
White, Howard. 2002. “Combining Quantitative and Qualitative Approaches in Poverty Analysis,” World Development.
Woolcock, Michael (2009) ‘Toward a Plurality of Methods in Project Evaluation: A Contextualized Approach to Understanding Impact Trajectories and Efficacy’ Journal of Development Effectiveness 1(1): 1-14