designing and conducting an evaluation of your project ...€¦ · designing your evaluation 1....

March 6, 2014

Clemencia Cosentino (Mathematica Policy Research)

Catherine Mobley (Clemson University)

Catherine Brawner (Research Triangle Educational Consultants)

Matthew Ohland and Russell Long (Purdue University)

2014 Annual Grantees’ Meeting

National Science Foundation

Science, Technology, Engineering and Math Talent Expansion Program (STEP)

Research Triangle

Educational Consultants

Designing and Conducting an Evaluation

of Your Project that Meets NSF Expectations

Part 1: Describe the steps to design a project

evaluation (40 minutes)

Part 2: Answer questions (10 minutes)

Part 3: Design a project evaluation (small

group activity, 40 minutes)

Workshop Objectives

2

Research Triangle


Why Evaluate?

Questions to Ask Before You Start Designing Your

Evaluation

What to Consider In Designing Your Evaluation

Steps to Design Your Evaluation

An Example

Part 1: Topics to be Covered

3

Research Triangle


Mathematica® is a registered trademark of Mathematica Policy Research.

Why Evaluate?

To comply with funder requirements

– NSF, Your Institution, and other funders

To document project outcomes and impacts

– To use in other proposals, to get buy-in for reforms

at your institution, to justify expenditures, to make a

case for additional resources

To improve practice

– In your courses, department, institution, or wider

community

To contribute to knowledge

– In your field through publications and other

dissemination activities

4

Research Triangle


Mathematica® is a registered trademark of Mathematica Policy Research.

What to Ask Yourself Before Starting

What is the purpose of your evaluation?

Who is your audience?

What is your project trying to accomplish? How?

Do you have questions you want to answer?

What is your evaluation budget?

How much time do you have?

Do you have access to staff with the right skills?

5

Research Triangle


What to Consider in Designing Your Evaluation?

Focus

– Implementation (what was done and how?)

– Outputs (who was served?)

– Outcomes (what was produced?)

– Impacts (what was produced by the project?)

Type

– Formative (how can we improve?)

– Summative (how did we do?)

Method

– Qualitative or quantitative

– Multiple (using different methods)

– Mixed (integrating qualitative and quantitative methods)

6

Research Triangle


Alignment

– Evaluation questions and research methods

Rigor in quantitative research

– Experimental Designs (RCTs)

– Quasi-experimental Designs (matched comparison

groups)

– Other Designs (such as pre-post and descriptive studies)

Rigor in qualitative research

– Appropriate qualitative methods

– Sample selection

– Triangulation

What to Consider in Designing Your Evaluation?

7

Research Triangle


Designing Your Evaluation

1. Create a logic model

2. Draft evaluation questions

3. Select an appropriate design

4. Identify and recruit staff needed

5. Plan the budget, schedule, products

6. Establish a mechanism to monitor progress on

the evaluation

8

Research Triangle


An Example

9

Research Triangle


The problem

– Engineering statics course curriculum is dated; it does not

include active learning or use new technology and software

to engage students and prepare them for high-tech jobs

– Faculty suspect that this is affecting student performance

and their employment prospects

What is your project trying to accomplish?

– Increase student learning

– Better prepare students for future jobs

My New NSF Undergraduate Project

10

Research Triangle


How?

– Revise the curriculum in the engineering statics course to

incorporate active learning activities as well as the use of

new technologies and software

– Provide faculty with professional development on new

pedagogical approach and new technology and software

– Purchase new software, hardware, and other materials

needed

What is the purpose of your evaluation?

– To document whether the revised pedagogical approach

adopted was implemented in the engineering statics course

– If adopted, does it have a positive impact on student

learning and job prospects

– If it has a positive impact, to use findings to lobby for

similar changes in other courses

My New NSF Undergraduate Project (continued)

11

Research Triangle



Who is your audience?

– Faculty and institutional administrators

– Foundations and alumni that may support future

expansion to other courses

– Other institutions that may want to adopt revisions

What will you do in your project?

– The curriculum will be revised by a team of faculty with

help from an expert consultant to adopt more active

learning (such as hands-on activities)

– New software and hardware will be purchased (and other

material as needed) for use in and out of the classroom

– Faculty will be trained to adopt use of active learning and

new technology

12

Research Triangle








6. Establish a mechanism to monitor progress

13

Research Triangle


Problem

Engineering statics curriculum is dated; does not engage students and prepare them for high-tech jobs

Inputs (resources

and partners)

- Purchase new software and hardware

- Partner with a technology company and an education consultant

Outputs (activities

and products)

- Provide faculty PD

- Incorporate active learning activities and new technologies into the curriculum

Outcomes(progress

toward goals)

- Increased student

participation

- Students report greater satisfaction with course

Impact (cause

and effect)

- Students earn better grades in

subsequent course

- Students who take

course have better job placement

rates

Step 1: Create A Logic Model

14

Research Triangle


Problem

Engineering statics curriculum is dated; does not engage students and prepare them for high-tech jobs

Inputs (resources

and partners)

- Purchase new software and hardware

- Partner with a technology company and an education consultant

Outputs (activities

and products)

- Provide faculty PD

- Incorporate active learning activities and new technologies into the curriculum

Outcomes(progress

toward goals)

- Increased student

participation

- Students report greater satisfaction with course

Impact (cause

and effect)

- Students earn better grades in

subsequent course

- Students who take

course have better job placement

rates

Step 1: Create A Logic Model

Implementation Study Outcomes

Study Impact

Study

Accountability Effectiveness

15

Research Triangle









16

Research Triangle


Evaluation Questions?

Do faculty adopt the new pedagogical approach?

Does the revised approach have a positive impact

on student learning in the statics course?

Are students who take the revised course more

successful in the job market?

17

Research Triangle









18

Research Triangle


Evaluation Designs?


• Implementation study

Does the revised approach have a positive impact

on student learning in the statics course?

• Impact study (experimental or quasi-experimental)

Are students who take the revised course more

successful in the job market?


19

Research Triangle









20

Research Triangle



Do you have staff with the evaluation expertise?

– Sociology faculty do implementation studies

– Statistics faculty and graduate students can do

quantitative analysis

– Need expertise in evaluation design to set up the study,

provide limited technical assistance, and review the

results/report to provide feedback

What is your evaluation budget?

– $50,000 per year (10 percent of the project’s budget)

How much time do you have?

– 3 years

21

Research Triangle


Can you tackle your evaluation questions?


• Implementation study

• You conclude that you can do this in house with help from

sociology faculty

Does the revised approach have an impact on student learning in

the statics course?


• You decide to revise some but not all sections of the statics

course (and assign faculty that received PD to the revised

sections). You will randomly assign students to the revised

(treatment) or not revised (control) sections (to be offered at

the same time)

Are students who take the revised course more successful in the

job market?


• Cannot study this given that the project ends in 3 years

22

Research Triangle


Q & A

23

Research Triangle


Small Group Activity

Select a topic for your project

– Retention (of students)

– Partnerships (between 2- and 4-year institutions)

– Outreach (to K-12)

Join a table that is working on that topic

Design your project evaluation

– Use the handouts to guide your team’s work

Report back

– 2-minute presentation from each team

24

Research Triangle


Resources on STEP Central

Go to:

http://stepcentral.net/groups/posts/942/

25

Research Triangle


Thank You

Clemencia Cosentino

– [email protected]

Catherine Mobley


Catherine Brawner


Matthew Ohland


Russell Long


26

Research Triangle


mailto:[email protected]







designing and conducting an evaluation of your project ...€¦ · designing your evaluation 1....

Documents