Validity and Reliability in Program Evaluation

Janet MaherNovember 1, 2011

Curriculum Development: design and monitor a collaborative process for achieving consensus on core competencies

Participant/Learner satisfaction with learning experience

Participant/Learner outcomes linked to learning experience

The Tasks at Hand

To ensure that study protocols meet the highest possible standards with regard to:

◦Quality of Evidence◦Format (engagement, ease of use)◦Practicality (usefulness in applied

setting)◦Feasibility (ease of implementation)◦Maintenance (amount of time, cost

required to maintain knowledge)

Quality Objective

Background ◦ Historically a disjuncture between quantitative and

qualitative research over rigour and trustworthiness of analysis and by implication of data used 1970-2000: methodological debates developed ‘parallel’

standards for qualitative and quantitative research Since 2000: move back to reviewing both types in terms of

validity and reliability and strategies for achieving them

This presentation focuses primarily on quality of evidence and relies on 2 main sources which are posted on igloo: Morse et al., 2002 in nursing and Golafshani, 2003 in education

Addressing Quality of Evidence

Refers to ◦ Reproducibility of an outcome

Tested by◦ Study protocol sufficiently specifies a procedure

so that others can use it to achieve consistent and stable results with other similar populations

Reliability

Refers to ◦ The extent to which a given test accurately

represents the features of the phenomena it is intended to describe, explain or theorize (Hammersley 1992)

Tested by◦ An agreement between 2 or more efforts to

measure the same thing using different indicators or measures

Validity

Main objective is to persuade peers that the results can be generalized

◦ Protocol should summarize criteria, including interest in typical or critical cases, in a similar fashion to quantitative strategies, including

Sampling strategy Review at several points by different coders/analysts

in data collection not just at end Triangulation—use and document different strategies

to validate Verification strategies—audit trail, member check

Strategies for Achieving Reliability and Validity in Qualitative Research

Suggest we revisit the process of the past 10 days or so collaborating on design of our brand

Sampling strategy—whole core team with shared objectives, range of expertise/competencies, some overlapping

Review at several points by different coders/analysts in data collection not just at the end—moderator(s) intervene and take account of contrary information

Triangulation—not formally, but could be done based on the materials on our forums

Verification strategies—not formally, but documentation is available—would be good if it were all in one place

Review of Core Team Design Exercise: can we meet the test

To summarize, consider advantages of mixed method approach

Look at the other criteria for study tools◦Format (engagement, ease of use)◦Practicality (usefulness in applied

setting)◦Feasibility (ease of implementation)◦Maintenance (amount of time, cost

required to maintain knowledge)

Next Steps

Do you agree on the primary focus on quality of evidence?

Can we commit to careful documentation of process through the igloo community space(s)?

Can you help me with some better search terms around polling and distance education

Thanks Janet

Questions/discussion/help