tips and tools for measuring program impacts: from delivering programs to reporting impacts office...

Tips and Tools for Measuring Program Impacts:

From Delivering Programs to Reporting Impacts

Office of Educational Innovation and Evaluation2323 Anderson Avenue, Suite 220Kansas State UniversityManhattan, Kansas 66502Phone: 785-532-5930Fax: 785-532-7185

Part I: Preparing to Evaluate Your Program

TipsFind and review evaluation instruments

Utility and feasibility

Purposes of evaluation: Accountability, Improvement, Knowledge Development, Oversight and Compliance “Who wants to know what about your program?”

Multiple, complementary sources of data

ToolsRubric: Reviewing Evaluation Instruments

Low Medium High Notes

UTILITY Value to PFTConsider: How well will the evaluation instrument gather data that aligns with the PFT action plan?

Alignment with Program CurriculumConsider: How well aligned is the evaluation instrument with the curriculum that was taught?

Part II:Collecting and Managing Data to

Measure Impacts

TipsPlan early – determine when evaluation will

take place in the programming cycle

Review and manage data as you receive it

Analysis quantitative and qualitative by hand, using Excel “Who wants to know what about your program?”

ToolsData Analysis Plan Worksheet

Question or Item Number

What did I hope to learn from this

question?

What analysis technique would assist

in that learning?

How could I present the data to convey what I

learned?

E.g., #2

If participants felt that they increased their knowledge of best management practices for a healthy yard.

Count and percentage of respondents who reported “Agree” or “Strongly Agree”

Bar or pie graph that compares “Agree” and “Strongly Agree” responses to “Disagree,” “Strongly Disagree” and “Unsure” responses

Part III:Presenting Impact Data

TipsKSRE Impact Reports

“Who wants to know what about your program?”

Illustrate your data

Be clear and concise, avoid jargon

Use the active voice

TipsChoose Graphics to Effectively Convey Data

RegionEducation (in Years of Schooling)

≤8 9-11 12 13-15 ≥16

Northeast 25.32 25.29 18.26 18.29 16.34

North Central 32.09 29.04 18.78 24.32 19.02

South 38.81 31.02 19.33 15.66 16.79

West 25.37 21.09 20.29 23.97 17.52

Example #1 – Table NOT formatted effectively

Table 1. Chewing Gum Rates in the United States by Geographic Region and Level of Education*

*Education level based on number of years of schooling. Adapted from a report of the U.S. Department of Chewing Gum.

Education Level

Region

Northeast West SouthNorth

CentralAverage

≤8 25 25 39 32 309-11 25 21 31 29 2712 18 20 19 19 1913-15 18 24 16 24 21≥16 16 18 17 19 17Average 20 22 24 25 23

Example #2 – Table IS formatted effectively

Table 2. Chewing Gum Rates in the United States by Geographic Region and Level of Education*

ToolsWhat Does the Data Mean?

What are the key findings? Write a paragraph to summarize the results.

What do you find interesting about these results? What would your audience find interesting?

E.g., After participating in the program, many participants reported feelings of increased financial abilities. In particular, a majority of participants reported increased skills to track spending (64%), to create a spending plan (71%), and to plan for unexpected expenses (77%). One respondent commented, “The skills I learned from this program will help me stretch my money to the next payday.”

QUESTIONS?

Contact Information

Office of Educational Innovation and Evaluation Kansas State University

2323 Anderson Avenue, Suite 220Manhattan, KS 66502Phone: 785-532-5930

www.ksu.edu/oeie

Sarah Bradford(785) 532-5677

sbradfor@ksu.edu

Amy Hilgendorf(785) 532-5538

aehilgen@ksu.edu