TAH Project Evaluation

Great - Yet Realistic - ExpectationsJeff Sun, Sun Associates

Our BackgroundInitiative-wide Qualitative Evaluation

District-level initiatives

State-level initiatives

Strategic Planning

STEM, Science, Instructional Technology, Social Studies, New Media

Evaluation of Teacher Professional Development

10 TAH projects (7 of these are current)

Qualitative is the EmphasisMakes the connection between what

projects say they want to do and what participants say that projects have done.

This data is invariably qualitative and accounts for a wide range of effects

Quantitative data helps paint the picture in broad outlines...

...but qualitative data helps you understand what the quantitative data means and to use the evaluation strategically

This Presentation

We want to talk about how to

Create a good evaluation

Use evaluation strategically

Work effectively with evaluators

Four Functions for Evaluation

Evaluation performs four meaningful functions in a project’s life...

As a part of the proposal

To clarifying a project’s work

To provide formative assessment of Progress and Implementation

Summatively, to account for the project’s success (or lack thereof)

First Function - The Proposal

As expressed in the logic map, evaluation helps explain:

Need for the project

The project’s actions in response to those needs

Who the project will serve

What the project expects to produce

All in one tidy package in the proposal text!

Issues Related to Function 1We need time to develop the evaluation plan

Ideally, we come in when the basic work is set, but the details are not fully fixed

We need space to fully describe the evaluation process

It’s hard to create this plan in 2 paragraphs or a half page

We need connect with those who will be actually implementing the project if funded

Proposal-writers will help you get funded...but it’s often no coincidence that they’re nowhere to be found once you get the grant.

Second Function - ClarificationOnce you’re funded, the real work begins

Use the evaluation plan as an organizer for talking with partners about

what the project will be doing

expectations for partners and participants

This discussion clarifies goals and develops a shared vision for project success

Out of this discussion come the project’s performance indicators...which we organize into rubrics

It would be nice to have all of your indicators before funding, as part of the proposal

But realistically, there’s never time for that

Projects need their own indicators

ED’s indicators are not the same and cannot replace your own indicators

Issues Related to Function 2

We can’t help if we aren’t at the table

Often, we have to really bug projects to bring us on board early

Evaluation is too often seen as the thing that happens after the work is done...not as the thing that can get work done.

We read the proposal and base the actual evaluation on what is written

This can be a problem when the proposal was written by the now-absent proposal-writer

The clarification discussion - assuming it happens early enough - is where this problem is resolved

Third Function - Formative

Collecting the data and sharing it as the project progresses

Measuring what the project is doing against what it said that it would do

Using indicator rubrics to account for data that is highly qualitative

Issues Related to Function 3

Our challenge is to identify data collection points that get at the essence of the project and its impact.

This can be one of the most important parts of our work.

Questions continue the process of clarifying the project work through participant feedback on impact

Next most difficult is connecting directly with participants

Fourth Function - Summative

Annual comparison of progress to goals

Sometimes more reporting than “annual” is important

More reporting might be useful in the beginning, less reporting might be necessary as the project matures

Summative data should play a role in the development of new proposals

Issues Related to Function 4

Qualitative data is valuable for projects, but does not seem to be something that ED is interested in.

This is a problem because projects cannot really produce meaningful data that ED is interested in.

So, do we spend extra time (money) producing two sets of data?

Or do we sacrifice the meaningful data to concentrate on producing what we’re told to create?

This has been the on-going debate among TAH evaluators on a national level

Questions? Comments?

What challenges have you faced when considering project evaluation?

How have you used evaluation in different stages of your project? (from planning to reporting)

jsun@sun-associates.com or 978-251-1600 x 204