Presents:

Information for participants:• Your microphone will be muted for the formal

presentation. If your audio portion the presentation is not working, try running the Audio Set Up Wizard (under meeting).

How to Engage:1) Use the “Set Status” feature.2) Type questions into the chat box. We will address

them right away or as part of scheduled Q&A breaks. 3) Please send follow up questions to the Project

Evaluator Network Working Group: http://stepcentral.net/groups/267 1

"6 Steps for STEP Evaluation” Presented by Dr. David Blair at St. Edward's University and

Dr. Raymond McGhee at SRI International.

David Blair, St. Edwards UniversityRaymond McGhee, SRI International

6 Steps to get started on your STEP Evaluation

Objectives

Promote an exchange of ideas on relevant evaluation topics in the STEP CoP.

Provide a general framework or roadmap for thinking about STEP evaluation tasks .

Set the stage for continued one-on-one interactions between STEP project evaluators.

DB-3

Overview

Engage project leaders in evaluation design and development

Document project achievements Describe challenges and adjustments made by the

project Use your different advisory committees to guide and

inform your evaluation planning Analyze data to document implementation, lessons

learned and impact Share results in a strategic fashion with key

stakeholders

RM-4

5

Opening Question

Who should participate in evaluation planning and why should they participate?

STEP 1

RM-6

Communicate early and regularly with leadership

Document decisions and next steps Show how evaluation results can be leveraged

to improve implementation Be persistent without being annoying

PI/Project Leader Collaboration

8

Opening Question:

What are the key research questions that your project wants to document?

Why?

STEP 2

DB-9

Key Activities/Number of students impacted Data Collection

Student Characteristics Surveys Focus Groups Journals Impact Data Other

Document Achievements

Successes Experienced Increased GPA, retention, research interest,

cohort building, student satisfaction Project activities integrated at institutions

Off-shoots Different versions

Other project impacts beyond intended goals Socialization, confidence

Document Achievements (cont’d).

Partnership achievements Roles Changes as project proceeds

Unexpected benefits Meetings/Groups Additional resources

Document Achievements (cont’d.)

13

Opening Question:

What can be learned from a project’s misfires or mistakes in implementation?

STEP 3

DB-14

Challenges faced Grant notification Recruitment of participants Funding for additional staff Finding an evaluator Students not taking advantage of resources

Significant changes Document fully

Describe challenges/adaptation/ assessment/sustainability

Intermediate measures and metrics to monitor progress Data Analysis

Quantitative Qualitative Mixed Methods

Describe challenges/adaptation/ assessment/sustainability (cont’d.)

17

Opening Question:

How can your advisory committee(s) best contribute to your project evaluation?

STEP 4

RM-18

19

Advisory Board Involvement

Internal – use them to help facilitate data collection and analysis efforts as well as interpreting the results of data analyses. This board may also help frame results for different stakeholders within the participating institutions.

External – use them to help you identify key audiences for your evaluation results as well as the key messages you want to share based on your evaluation results.

20

Opening Question:

Why is it important for an evaluator to adopt "methodological pluralism" with respect to collecting and analyzing project evaluation data?

STEP 5

DB-21

22

Lessons Learned/Overall Impact

Summary of how well project is meeting project goals Tie back to original goals Work with your external evaluator Fully explain changes and corrections

Actions taken to sustain project Additional funding Integrating within the university culture and

budget

23

Opening Question:

How can we better target our project evaluation findings so that they have the largest impact on institutional practice?

STEP 6

RM24

25

Dissemination of Results

Rationale – share successes to key stakeholders to secure support for additional funding and institutionalization of project efforts. Rationale can be sharpened with external advisory input.

26

Dissemination of Results (cont’d.)

Targets – with input from external advisory committee, identify multiple audiences who could use the results of your evaluation. Include targets at different levels of the institution as well as key external audiences (e.g., industry partners, sister institutions, 3rd Year Review board.) Also determine what messages, metrics, or narratives speak to the concerns of these audiences.

27

Dissemination of Results (cont’d.)

Methods – develop a dissemination plan that harnesses multiple modes of disseminating your evaluation’s results. Include print as well as electronic modes of sharing your work.

28

Dissemination of Results (cont’d.)

Products – based on the audiences identified, develop several dissemination products tailored for each audience. Collaborate PI and external advisory panel’s recommendations to determine the high value products.

29

6 Steps to get You Started

1. Project leader collaboration2. Document achievements3. Describe challenges/adaptation/assessment/

sustainability4. Advisory board involvement5. Document implementation, lessons learned

and impact6. Dissemination of results

30

Conclusion/Questions

Future materials and resources Suggested Checklist to help with plan

preparation General project evaluation references