Research Strand: Program Evaluation

Stephen Kroeger, Christy Borders, Kim Webster

LogicizingUniversity of Cincinnati

The Birth of a Logic Model to Evaluate one 325T Program

Answers Needed The Government Performance and Results Act (GPRA)

essentially asks three questions: What is our program trying to achieve? How will our program’s effectiveness be determined? How is our program actually doing?

The Challenge Clear and logically consistent methods have not been readily

available to help program coordinators make implicit understandings explicit

(McLaughlin & Jordan, 1999).

Measuring Change Providers are often being required to document specific

interventions and expected outcomes and to demonstrate their efforts to increase program effectiveness.

As a result, a great number of providers need to identify services or policies that are designed to lead to positive changes in the services provided to children and families they serve.

Yampolskaya, Nesman, Hernandez, & Koch, 2004

What is a Logic Model process? The Logic Model process is a tool used by program

evaluators and coordinators to develop and tell the performance story for their program.

The Logic Model describes the reasonable linkages among program resources, activities, outputs, stakeholders reached, and short-intermediate-longer term outcomes.

Once this model of expected performance is produced, critical measurement areas can be identified.

(McLaughlin & Jordan, 1999)

Benefits of a Logic Model process Builds a common understanding of the program, identifies

assumptions, and supports team communication Identifies projects that are critical to goal attainment,

redundant, or have inconsistent or implausible linkages among program elements

Communicates the place of a program in the organization or problem hierarchy

Provides a balanced set of key performance measurements and evaluation issues to improve data collection and usefulness

Meets requirements of GPRA(McLaughlin & Jordan, 1999).

Logic Model Models Gugiu & Rodriguez-Campos, 2007 Theory Approach logic model asks four questions: What issues or

problems does the program seek to address? What are the specific needs of the target audience? What are the short- and long-term goals of the program? What barriers or supports may impact the success of the program? W. K. Kellogg foundation logic models

United Way of America logic model consists of resources needed to achieve goals; activities implemented to affect change; outputs are services provided, materials distributed, people served; and outcomes are derived benefits as a result of program activities. Contextual factors may be incorporated when it’s important to note constraints on the program.

Antecedent Target Measurement (ATM) logic models identify antecedent conditions on a problem’s underlying rationale (why); identify antecedent conditions targeted by program activities; determine whether outcomes are reasonable to include in the logic model in light of the evaluation timeframe.

Open Systems logic models focus on establishing collaborative partnerships to achieve strategic objectives and to measure impacts at various levels. Levels include changes that are temporary; sustaining the prior changes and stakeholder buy-in; making broader changes in organizational practices prevent/reduce problems.

Learning Curve During Construction Activities are not Goals We had over 10 activities we thought were goals These activities were categorized into 4 main goals

Outcomes Link Back to Activities Whether Short term, Intermediate or Long term, the

outcomes need to provide data used to evaluate the activity Outcomes Must be Measureable If we can’t quantify or qualify the outcome we won’t know if

the activity was useful or effective The act of establishing outcome measures required that we

modify the activities to produce measureable outcomes What activity would increase retention of new teachers?

Logic Model Changed Our Thinking

Logic Model Changed Our Thinking

Logic Model Changed Our Thinking

What productwill come outof this activity?

Changes orbenefits causedby outputs

Logic Model Changed Our Thinking

Short-term = changes or benefits causedby outputs.

Intermediate = results as an applicationof short-term

Long-term = are these results correct?

Logic Model Changed Our ThinkingData: SurveyCollection: YearlyFeedback: Failure to meet outcome would result in the examination of how we are communicating instruction on EBP

What data?

When/how to collect?

How will this feedback to activities?

Daily Use of the Logic Model

How can we use the Logic Model to inform day-to-day decisions?

Funneling the goals into daily tasks in a manageable format

GoalsActivitiesDaily Tasks

Feedback Loops

Data provides continuous feedback. Resources,activities, outputs, customers, or outcomes may adjust based on the how the data are supporting the project’s goals.

Sample of Daily Graphic

Questions How does a logic model help me? What resources do I need in order to begin using a

logic model? Can you give an example of how the LM helped you

in your project design and evaluation?

References Gugiu, P. C., & Rodriguez-Campos, L. (2007). Semi-

structured interview protocol for constructing logic models. Evaluation and Program Planning, 30, 339-350.

McLaughlin, J. A., & Jordan, G. B. (1999). Logic models: A tool for telling your program's performance story. Evaluation and Program Planning, 22, 65-72.

Yampolskaya, S., Nesman, T. M., Hernandez, M., & Koch, D. (2004). Using concept mapping to develop a logic model and articulate a program theory: A case example. American Journal of Evaluation, 25(2), 191-207.

Research Strand: Program EvaluationLogicizing�University of CincinnatiAnswers NeededMeasuring ChangeWhat is a Logic Model process?Benefits of a Logic Model processLogic Model Models �Gugiu & Rodriguez-Campos, 2007Learning Curve During ConstructionLogic Model Changed Our ThinkingLogic Model Changed Our ThinkingLogic Model Changed Our ThinkingLogic Model Changed Our ThinkingLogic Model Changed Our ThinkingDaily Use of the Logic ModelFeedback LoopsSample of Daily GraphicQuestionsReferences