Product EvaluationProduct Evaluationthe outcome phasethe outcome phase

Do the magic bullets work?Do the magic bullets work?

How do you know when an How do you know when an innovative educational program innovative educational program has “worked”?has “worked”?

What kinds of evidence are What kinds of evidence are needed to make a judgment needed to make a judgment that something “works” in that something “works” in education?education?

Do the magic bullets work?Do the magic bullets work?

Input – Process – ProductInput – Process – ProductHas the process improved? Has the process improved? Has teacher morale improved? Has teacher morale improved? Do teachers consider the Program a Do teachers consider the Program a

Resource or a Demand? Resource or a Demand? Did the test scores improve?Did the test scores improve?Did the test scores improve as a Did the test scores improve as a

result of Program participation?result of Program participation?

CIPPI ModelCIPPI Model

ContextContext InputInputProcessProcessProductProduct ImpactImpact

The goal is to make The goal is to make INFORMEDINFORMED judgments about Program merit or judgments about Program merit or value.value.

The Nature of Evaluation The Nature of Evaluation TasksTasksThe goal is to make The goal is to make INFORMEDINFORMED

judgments about Program merit or judgments about Program merit or value.value.

Making judgments requires a thorough Making judgments requires a thorough understanding of whether Program understanding of whether Program goals and objectives were met.goals and objectives were met.

Were the needs of the target audience Were the needs of the target audience met?met?

Product Evaluation Product Evaluation

The central focus of a Product The central focus of a Product evaluation is the documentation of evaluation is the documentation of Program achievements or Program achievements or accomplishments.accomplishments.

The Product evaluation phase The Product evaluation phase

requires the evaluator to develop and requires the evaluator to develop and implement a plan to collect the implement a plan to collect the outcome information called for on the outcome information called for on the measurable objectives worksheets. measurable objectives worksheets.

The Product Evaluation The Product Evaluation Plan Plan

Define a typical service delivery Define a typical service delivery cycle.cycle.

Is there a typical unit of time that Is there a typical unit of time that makes a natural timeframe for the makes a natural timeframe for the Product evaluation?Product evaluation?

This issue is made easy in This issue is made easy in education (i.e. academic year). education (i.e. academic year).

The Product Evaluation The Product Evaluation Plan Plan Does the Program operate based on Does the Program operate based on

client “spells” or “incidents” rather client “spells” or “incidents” rather than a service delivery cycle? than a service delivery cycle?

Once the typical service delivery Once the typical service delivery cycle is defined, decide when the first cycle is defined, decide when the first “fair test” service delivery cycle has “fair test” service delivery cycle has taken place. Did a “pilot” phase take taken place. Did a “pilot” phase take place?place?

The Product Evaluation The Product Evaluation Plan Plan The plan should include:The plan should include:

– Multiple measures of successMultiple measures of success– Collected at multiple points in timeCollected at multiple points in time– Multiple methods of data collection Multiple methods of data collection

It should be based on the Measurable It should be based on the Measurable Objectives Worksheets outlined Objectives Worksheets outlined during the Input evaluation phase. during the Input evaluation phase.

Evaluation Strategies Evaluation Strategies

Surveys designed to document the Surveys designed to document the subjective judgments of the subjective judgments of the stakeholders about the success of stakeholders about the success of the Program.the Program.

Interviews designed to provide rich Interviews designed to provide rich supplements to the survey data.supplements to the survey data.

Evaluation Strategies Evaluation Strategies

Standardized tests and objective Standardized tests and objective measurements.measurements.

Measures designed to assess Measures designed to assess knowledge gains, attitude changes, knowledge gains, attitude changes, or behavior changes.or behavior changes.

Performance assessment data.Performance assessment data.

Evaluation Strategies Evaluation Strategies

Observations.Observations.Case studies of individual Program Case studies of individual Program

participants or sub-groups. This participants or sub-groups. This method can help take a more in-method can help take a more in-depth look at the subjective depth look at the subjective experience of selected Program experience of selected Program participants.participants.

Collections of Program products and Collections of Program products and other artifacts of Program success.other artifacts of Program success.

Side EffectsSide Effects

What were the intended and What were the intended and unintended (side effects) unintended (side effects) consequences of the Program, for consequences of the Program, for each group of stakeholders, across each group of stakeholders, across a complete service delivery cycle?a complete service delivery cycle?

Document the positive and Document the positive and negative outcomes of participation negative outcomes of participation in the Program.in the Program.

Sub Group AnalysisSub Group Analysis

Do the intended and unintended Do the intended and unintended consequences of Program consequences of Program participation vary by sub-group?participation vary by sub-group?

Ask “Did it work?” But also ask, Ask “Did it work?” But also ask, “For whom did it work, and under “For whom did it work, and under what circumstances?” what circumstances?”

InterpretationInterpretation

Interpret the data gathered Interpret the data gathered according to standards of growth according to standards of growth and improvement that are and improvement that are accepted by the Program and accepted by the Program and stakeholders.stakeholders.

Were the gains or accomplishments Were the gains or accomplishments large enough to justify continuing large enough to justify continuing the Program?the Program?

InterpretationInterpretation

In addition to just gain scores In addition to just gain scores based on pre / post measures, can based on pre / post measures, can your interpretations be enhanced your interpretations be enhanced by any information about:by any information about:

– Continuous improvementContinuous improvement– Trends in improvements made Trends in improvements made

between the measurementsbetween the measurements– Periods of time when the Program Periods of time when the Program

worked better than othersworked better than others

Cost ConsiderationsCost Considerations

Was all of the funding spent Was all of the funding spent properly in accordance with the properly in accordance with the objectives?objectives?

Did the Program follow all the Did the Program follow all the

financial guidelines from the funding financial guidelines from the funding agency?agency?

Could the same benefits be Could the same benefits be achieved for less expense?achieved for less expense?

Cost ConsiderationsCost Considerations

Are these outcomes worth the Are these outcomes worth the investment? investment?

Formal cost-benefit or cost-Formal cost-benefit or cost-effectiveness analysis may be part effectiveness analysis may be part of a Product evaluation as needed.of a Product evaluation as needed.

This may require consultation with This may require consultation with financial professionals.financial professionals.

Summative ConclusionsSummative Conclusions

Make a summary judgment about Make a summary judgment about whether the Program is worth whether the Program is worth repeating for another service delivery repeating for another service delivery cycle. cycle.

Make a summary judgment about what Make a summary judgment about what outcomes can be attributed to the outcomes can be attributed to the Program.Program.

Cause and Effect is hard to establish. Cause and Effect is hard to establish.

Summative ConclusionsSummative Conclusions

Make sure your interpretations of Make sure your interpretations of the Product evaluation are the Product evaluation are grounded in the results of the grounded in the results of the Context, Input, and Process phases Context, Input, and Process phases of the evaluation.of the evaluation.

Formative ConclusionsFormative Conclusions

The Product phase consists of The Product phase consists of Summative evaluation activities. Summative evaluation activities. However, some Formative However, some Formative conclusions can come out of the conclusions can come out of the Product phase.Product phase.

Include recommendations about Include recommendations about Program modification, extension, or Program modification, extension, or export to other settings.export to other settings.