12 chapter
TRANSCRIPT
![Page 1: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/1.jpg)
Designing and Conducting Summative Evaluation
The Evaluation after implementation
![Page 2: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/2.jpg)
Involves _______ data
CollectingAnalyzingSummarizing
![Page 3: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/3.jpg)
For the purpose of
Giving decision makers information on the effectiveness and efficiency of instruction
![Page 4: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/4.jpg)
Effectiveness of Content Instruction solve the problem? Criterion created prior to evaluation? Was the criterion established
in conjunction withthe needs assessment?
![Page 5: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/5.jpg)
Specifically
Did learners achieve the objectives? Learners feeling about instruction? What were the costs? How much time did it take? Was instruction implemented
as designed? What unexpected outcomes?
![Page 6: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/6.jpg)
Alternative Approaches to Summative Evaluation
ObjectivismSubjectivism
![Page 7: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/7.jpg)
Objectivism
Based on empiricism Answering questions on the bases of
observed data Goal based and replicable, uses the
scientific method
![Page 8: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/8.jpg)
Subjectivism
Employs expert judgment Includes qualitative methods
observation and interviews evaluate content
“Goal Free” evaluators haven’t a
clue about the goals
![Page 9: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/9.jpg)
Objectivism (limitations)
Examine only a limited number of factors
May miss critical effects
![Page 10: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/10.jpg)
Subjectivism (limitations)
Are not replicable
Biased by idiosyncratic experiences, perspectives, or the people who do the evaluation
May miss critical effects
![Page 11: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/11.jpg)
Designer Role in Summative Evaluation?
Somewhat controversial
![Page 12: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/12.jpg)
Timing of Summative Evaluation?
Not in the first cycle
![Page 13: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/13.jpg)
Summary Diagram Formative
Design ReviewsExpert ReviewsOne-to-one Eval.Small Group Eval.Field TrialsOngoing Eval.
SummativeDetermine Goals of the EvaluationSelect OrientationSelect DesignDesign or Select Evaluation MeasuresCollect DataAnalyze DataReport Results
![Page 14: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/14.jpg)
Goals of the Evaluation
What decisions must be made? What are the best questions? How practical is it to gather data? Who wants the answer to a question? How much uncertainty?
![Page 15: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/15.jpg)
Orientation of Evaluation Goal-based or goal-free A middle ground? Quantitative or qualitative appropriate? Experimental or naturalistic approach?
![Page 16: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/16.jpg)
Select Design of EvaluationDescribes what data to collect
When the data will be collectedAnd under what conditions Issues to consider:
How much confidence must we have that the instruction caused the learning? (internal validity)
How important is the generalizability? (external validity)
How much control do we have over the instructional situation?
![Page 17: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/17.jpg)
Design or Select Evaluation Measures Payoff outcomes
Is the problem solved? Costs avoided Increased outputs Improved quality Improved efficiency
![Page 18: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/18.jpg)
Design or Select Evaluation Measures (2) Learning Outcomes
Use instrument you’ve already developed for the summative evaluation
But measure the entire program
![Page 19: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/19.jpg)
Design or Select Evaluation Measures (3) Attitudes
Rarely the primary payoff goals Ask about learner attitudes toward
learning instructional materials subject matter
Indices of appeal attention, likeableness, interest, relevance,
familiarity, credibility, acceptability, and excitement
![Page 20: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/20.jpg)
Design or Select Evaluation Measures (4) Level of Implementation
degree to which the instruction was implemented
Costs Cost-feasibility Cost-effectiveness
![Page 21: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/21.jpg)
Alternative Designs
Instruction then posttest Pretest then instruction then posttest
![Page 22: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/22.jpg)
The Report
SummaryBackground
Needs assessment, audience, context, program description
Description of evaluation study Purpose of evaluation, evaluation of the
design, outcomes measured, implementation measures, cost-effectiveness info., analysis of unintentional outcomes
![Page 23: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/23.jpg)
The Report (continued)
Results Outcomes, implementation, cost-
effectiveness info., unintentional outcomes
Discussion causal relationship between program &
results Limitation of study
Conclusion & Recommendations
![Page 24: 12 chapter](https://reader035.vdocument.in/reader035/viewer/2022081419/5560cff6d8b42a08088b4dc8/html5/thumbnails/24.jpg)
Summary Summative evaluation is after
implementation Limitations of subjective and objective
evaluation What to include in the report