![Page 1: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/1.jpg)
Evaluation and Attitude AssessmentBME GK-12 Fellows
12/1/10, 4-5pm 321 Weill Hall
Tom ArchibaldPhD Student, Adult & Extension Education
GRA, Cornell Office for Research on Evaluation
![Page 2: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/2.jpg)
quick overview of evaluation
• merit and worth• program improvement• knowledge generation• very similar to research, with some differences
![Page 3: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/3.jpg)
CORE’s “touchstone” statement on the goal of evaluation:
To obtain accurate, useful insights about the answers to evaluation questions in a manner that
is feasible, is credible to relevant stakeholders, makes strategic use of limited time and resources,
and contributes to our general knowledge, to future evaluations and to program evolution.
![Page 4: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/4.jpg)
a systems approach for evaluation planning
• Programs are viewed (and modeled) as parts of larger systems dynamic and evolving related to other programs in the present connected to past and future programs being perceived differently by different stakeholders
and systems• Evaluation plan development takes this into
account and should help programs and systems evolve
4
![Page 5: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/5.jpg)
5
![Page 6: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/6.jpg)
6
Initiation Development Maturity Translation &Dissemination
RET
Program Phase
Evaluation Phase
REU
Module-Based Education
Step 2.05 - Program and Lifecycle Analysis
![Page 7: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/7.jpg)
7
Step 2.07 - Pathway Modeling
![Page 8: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/8.jpg)
The key elements of Evaluation Plan quality are
1. Consistency with a high-quality program model
2. Fitness of evaluation plan elements to the program and program context
3. Internal alignment of the evaluation plan elements
4. Holistic coherence
8
![Page 9: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/9.jpg)
1. Consistency with a high-quality program model
– A “high-quality model” …• is grounded in knowledge of the program • incorporates perspectives of multiple stakeholders• shows causal pathways (program logic)• reflects careful thought about program “boundaries”• includes program assumptions and key elements of context• is connected to program evidence base (relevant research)
– “consistency with…” means• evaluation questions can be located in terms of model elements• evaluation “scope” makes sense
9
![Page 10: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/10.jpg)
2. Fitness of evaluation questions and other evaluation plan elements to the program and program context
– Evaluation questions are “mined” from the model– Evaluation questions are appropriate for the program’s
maturity and stability, existing state of knowledge, and program needs
– Evaluation focus, methods, and tools meet needs of key stakeholders
– Evaluation plan makes efficient and strategic use of program and evaluation resources
10
![Page 11: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/11.jpg)
3. Internal alignment of the evaluation plan elements
– Measures fits the constructs – Measure is the most strategic option among those that fit– Design is appropriate for lifecycle stage– Design can support claims implied in purpose statement– Sampling and Analysis plans can generate evidence
needed
11
![Page 12: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/12.jpg)
4. Holistic coherence… the elusive element
– Evaluation planning requires myriad decisions about multi-faceted tradeoffs
– Making these decisions well requires a holistic comprehension of the program and the environment and systems that embed it
– The decisions may be invisible in the written plan so some of the resulting quality might be ascertainable, some of it will not be
12
![Page 13: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/13.jpg)
Getting to Measures Worksheet 1. Starting Points: (a) Copy here what you want to know about your program
for this Evaluation Question. (b) Copy here your formal Evaluation Question for this
inquiry. 2. Clarify the “constructs” in your Evaluation Question
above – what exactly do you mean by the activity and/or outcome you are focusing on? Note: revisit your Evaluation Question above and see if it needs to be revised to capture this precision.
![Page 14: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/14.jpg)
3. Using this sharper definition of what’s being evaluated, work
through the following sequence of questions: (a) What would this “look like” or consist of in practice? (List as
many options as you can. Be creative – think outside the “usual” options.)
(b) What might serve as evidence? For the most promising
candidates in (a), write down various kinds of evidence that would be informative about whether this has occurred.
(c) Review the strengths and weaknesses of the options, taking into
account several aspects that matter: “closeness” to the real thing; accuracy and reliability. Write down a short list of the most promising candidates from (b).
![Page 15: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/15.jpg)
4. How could you gather this evidence? Again, think of as many options as you can and list them here. (Possibilities might include video-taping a demonstration, live observation during class sessions, directly asking participants, asking people who know the participants, testing the end-products, etc.) Identify the ways that seem best, taking into account accuracy, feasibility, fit with program context and target population. Indicate the top choices among those on your list.Once again, take a moment to revisit your Evaluation Question phrasing in 1(b) and see if it needs updating.
5. What types of measure would be needed? From your answer(s) to 4,
now ask yourself what type(s) of measure would allow you to gather evidence in this way. Here is a list of possible measure types from which to choose: case study, interview, observation, group assessment (e.g. focus group), expert or peer review, portfolio review, testimonial, test of knowledge or skill, photograph, slide or video, diary or journal, log, document analysis, action cards, simulation, problem story, creative expression, unobtrusive measures.
![Page 16: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/16.jpg)
For more explanation of measure types, see:
Taylor-Powell, E., S. Steele (1996) Collecting Evaluation Data: An Overview of Sources and Methods, Cooperative Extension Publications, University of Wisconsin, Madison, WI. (http://learningstore.uwex.edu/Collecting-Evaluation-Data-An-Overview-of-Sources-and-Methods-P1025C237.aspx)
![Page 17: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/17.jpg)
find v. writeSummary checklist (questions to help make the “find” versus “write” decision): 1.Is the evaluation question assessing an outcome? If yes, do you need
to make a strong claim of effectiveness? 2. Do you need to compare your evaluation results to those of others or
to some external norm? 3. Do stakeholders require a pre-validated measure? 4. Are staff resources available to draft measures in accordance with
best practices? To conduct a high quality search for measures? 5.After a quick scan, does it appear you are likely to be able to find
measure that fits your construct through peers, the Netway, your program system, program funders or professional groups related
![Page 18: Evaluation and Attitude Assessment BME GK-12 Fellows 12/1/10, 4-5pm 321 Weill Hall](https://reader036.vdocument.in/reader036/viewer/2022062810/56815d59550346895dcb6408/html5/thumbnails/18.jpg)