CRJS 4466PROGRAM & POLICY EVALUATION

LECTURE #5

•Evaluation projects

• Questions?

Qualitative Evaluation Methods

Qualitative Methods – strengths

• Carol Weiss (1998) – the coming of age of qualitative methods

• note the Campbell versus Cronbach controversy here

• qualitative methods are an important technique for under- standing the ‘context’ of success or failure of programs

• qualitative methods most commonly used in process evaluation (formative) or the ‘how’ and ‘why’ of a program and its workings

• ability to represent subtleties and complexities of program functioning and dynamics

Qualitative Methods - strengths

• many evaluators like the interpersonal nature of qualitative methodologies

• qualitative approaches are often ‘action oriented’

• a choice of ‘depth’ over ‘breadth’

Qualitative Methods - limitations

• reliability and validity of measures

• interviewer bias

• anecdotal

• labour-intensive

• costly

• less useful for reporting on program outcomes

Qualitative Methods – Single System Evaluations

• evaluating an intervention with a single client system, usually quantitatively during the course of treatment

• limited or questionable usefulness of qualitative methodology

4. Qualitative Methods – Focused Qualitative Evaluation (FQE)

• reflects the immediacy of much evaluation – not time for extended research, ethnomethodological development

• use of FG and rapid ethnographic assessment

Qualitative Methods – Mixed Methods

• ‘triangulation’, combining both quantitative and qualitative methods

• ‘temporal sequencing’ and qualitative subsamples

• benefit of both breadth and depth

Qualitative Methods – Action (Participatory) Research

• overtly ‘action-oriented’ approach primarily among those conducting qualitative evaluations

• participatory, collaborative, empowering – now an expectation in some evaluation projects

• ‘praxis’ orientation

Qualitative Methods – Research Design

• units of analysis and comparative designs

• gaining access, key informants, and sponsorship

• reciprocity, payback and feedback – and the methodological, ethical concerns here (‘false hope’)

• sampling considerations • deviant case sampling/typical case sampling• maximum variation sampling• snowball sampling• purposive sampling• convenience sampling• random sampling

Qualitative Methods – Research Design

• data collection strategies• on-site observation• participant observation• reflexivity• use of interview guide• use of focus groups• cross-validation among interviewers• use of documents

• when to stop data collection – the issue of saturation

Qualitative Methods – Data Management and Analysis

• use of qualitative data analysis programs: Ethnograph, AskSam QSR (Nudist)

• coding data

• emic (indigenous) and etic (researcher-created) coding

• quality control mechanisms

• reporting – issues related to qualitative methodologies

Table 5-1: Key Differences Between Qualitative and Quantitative Evaluation Approaches

Table 5.2: Summary of Key Questions in Conducting Qualitative Evaluation Assessments and Evaluation Studies

Table 5-3: Sampling Strategies for Qualitative Evaluations

Table 5-4: Open Ended Questions for the Economic Development Agreement Stakeholders Project

Table 5-4: Open Ended Questions for the Economic Development Agreement Stakeholders Project

Table 5-5: Some Basics of Face-to-Face Interviewing

Table 5-6: Helpful Hints as You Analyze Qualitative Data

Table 5-6: Helpful Hints as You Analyze Qualitative Data

Table 5-7: Comparing Qualitative Validity with Experimental and Quasi-Experimental Validity

Table 5-8: Ways of Testing and Confirming Qualitative Findings

Focus Group Question:

“Based on your own experiences of the Criminal Justice Program here at Nipissing University, what do you like about the program? What don’t you like about the program?