What Difference do We Want to Make; Developing a Strategic Framework and Measurement

Strategy for Sport Manitoba’s PSO Funding Unit

June 5, 2008

Jennifer Birch-Jones, Senior AssociatePerformance Management Network Inc.

Tel: (613) 236-2320Fax: (613) 236-8644

Email: jennifer.birch-jones@pmn.net

2

Today’s Objectives

Increased awareness and understanding of: performance based planning, measurement and evaluation what difference the PSO Funding Unit is trying to make the interplay btw the PSO Funding Unit’s outcomes and the

outcomes of other Sport Manitoba programs measurement considerations evaluation considerations learning considerations, including ways in which Sport Manitoba

can help build capacity internally and in PSOs Consensus on a strategic framework / logic model Consensus on Next Steps

3

Today’s Approach

Learn from yesterday evening’s session

Confirm key concepts and terms

Build on the good work that already has been done

Opportunity to discuss / explore issues as they relate to success

Interactive – we will build the strategic framework together

Ask lots of questions, especially if any of the terms / jargon not clear

“Why” questions not meant to challenge, but to help get to an outcome / key result

Parking Lot

Logistics

4

Strategic AnalysisWhere are we now?

Monitoring & EvaluationAre we getting there?

Strategic FrameworkWhere do you want to be?

ImplementationHow do we get there?

Strategic Management Cycle

Source: Kent and Wilkinson, Applied Strategic Planning, 1991.

5

Key Terms and Concepts

Performance-based planning combines performance measurement and evaluation concepts and approaches with strategic planning

Move away from a focus on how busy we are (activity-based) to what difference we are making (outcome-based)

Uses an integrated approach to planning, measurement and reporting

Can strengthen the planning process by Clearly articulating and agreeing on what success looks like

and how to measure and report on progress Focusing scarce resources on measuring progress for only the

most important aspects Adapting the level of detail to meet the information needs of

the users, e.g., Board, Minister, CEO, Director, etc.

6

Key Terms and Concepts

Managing for results entails: Knowing what difference you are trying to make

(strategic framework and targets) Knowing what progress / success you are having

(monitoring and evaluation) Finding out why something is not working

(monitoring and evaluation) Making adjustments in your work to improve

performance (continuous improvement / learning)

7

Overview of Process

1. Develop strategic framework2. Develop measurement / ongoing monitoring

strategy Priority performance areas for monitoring Measurement strategy – indicators, data sources, data

collection methods, timing, responsibility, resources, whether baseline data is needed

3. Develop evaluation framework Type of evaluation Evaluation questions Timing of evaluation Evaluation strategy - indicators, data sources, data

collection methods, timing, responsibility, resources, whether baseline data is needed

8

Overview of Process

4. Develop learning strategy – ways in which PSOs can better use and learn from monitoring and evaluation – their own and Sport Manitoba’s

5. Develop / refine existing processes and tools6. Implement, learn, refine and share

Engage PSOs at every opportunity!Logic model / strategic framework is the basis of both

monitoring and evaluationYou can adapt monitoring and evaluation

to meet your needs (not vice versa)!

9

Key Terms and Concepts

A Strategic Framework includes:

Inputs Activities Outputs Goals / Outcomes

Goals / outcomes can be Short-Term Medium-Term and Long-Term Goals / Outcomes

Also known as a results chain, logic model, performance framework

diagram showing the links from the activities through the sequence of outcomes to the final outcomes (or results commitments)

10

Key Terms and Concepts

HOW?WHO?

WHERE?

WHATdo wewant?

WHY?

inputs activities outputsdirect and

intermediateoutcomes

final outcomes

“Operational” “Behavioural Change” “State”

users / clients /co-deliverers /beneficiaries

11

REACH

Primary targets or clients

Partners orCo-deliverers

Stakeholders

Reach is defined as the group, or groups, which are reached by program / service outputs. The reach category can be segmented into different groups, e.g.,

SHORT- AND MEDIUM-TERM OUTCOMES

LONG-TERM OUTCOMES (State level)

OUTPUTS and ACTIVITIES

Action / Adoption Ability / Competence (Capacity Building)

Awareness / Perception / Understanding Active Partner Support (Commitment)

Long-term outcomes should relate to the mission, mandate and key results of the program/service provider.

Direct Outcomes are the initial effect or consequence of an output. Direct outcomes include early behavioural changes in the target group or clients, reflected in changes in awareness, client satisfaction, acceptance and support. These lead to intermediate outcomes, such as new knowledge, improved capability, improved behaviour (compliance) and decision making.

(Operational level)

(Behavioural Change level)

11

Strategic Framework

12

Key Terms and Concepts

Characteristics of a Strategic Framework Sequence of variable length extending from inputs to

final outcomes (results commitments) Outputs mark transition between internal and

external results (outcomes) Partnerships externalize delivery process and

lengthen sequence of outcomes What is achievable is directly related to:

Resources Timeframe

13

HOW?

(Operational)Your operational

environmentYou have direct control over the

behaviors within this sphere

WHAT do we wantby WHOM?

(Behavioral Change)Your environment of direct influence

e.g., people, groups and organizations in direct contact with

your operations

WHY?(State)

Your environment of indirect influencee.g., individuals and/or

communities of interest where you do not make direct contact

Performance needs to be considered in terms of its differing spheres of influence. Actions in the operational sphere should directly lead to changes in targeted groups which should in turn affect the desired ‘state’.

Sources: Van Der Heijden (1996), Montague (2000)

Spheres of Influence

End Outcomes

Immediate Outcomes

Intermediate Outcomes

Outputs

14

Strategic Framework

WHYWHY

HOW

WHAT BY WHOM

REACH

Office of the Auditor General – Basic Results Chain

END OUTCOMES

Office contributes to better managed government programs and better accountability to Parliament and the public.

INTERMEDIATE OUTCOMES

Audits and studies are relevant. Critical issued and problems are addressed by audits and agendas are changed.

Audits result in better informed legislature, government, entities, and public.

IMMEDIATE OUTCOMES

Office support for its role and its work maintained.

Clients and stakeholders engaged in audit process.

OUTPUTS AND OPERATIONAL PROCESSES

Source: Office of The Auditor General, pg. 12, Departmental Performance Report 2001

1515

16

Determine the priority / important performance areas for monitoring and why

Identify what should be the key indicators for these: What would you see or hear if the expected results /

outcomes you have described are being achieved? Determine how, how often and who will collect

this information Determine the means by which to implement

Measurement Considerations

17

Measurement Considerations

Sample Measurement Strategy

Perfor-mance Area / Outcome

Perf.Indicator

Data Source

CollectionMethod

Responsibility

Timing

Increased awareness of the Health Check (HC) program by consumers

The percentage change in the no. of consumers aware of the HC program btw 2008 and 2011

Canadianconsumers

Consumer survey

HC Program staff

Baseline in April 2008,Follow-up in March 2011

18

Measurement Considerations

Performance indicators (or measures) are the specific items of information that track a program’s success.

PIs describe observable, measurable characteristics or changes that represent achievement of an outcome, e.g., number / percentage of Canadian consumers who are aware of the Health Check program

19

Results can be operationally defined as follows:

Such As:

# Proportion / Percentage

Rate

Ratio

Amount

‘Level’

Population Target Group

Program Participant

Client

Individual

Organization

Agency

Community

Awareness

Attitude

Perception / Satisfaction

Knowledge

Skill

Behaviour / Practice

Example:

The percentage of clients Who express positive

satisfaction with the

services / advice

rendered

The Measure of Change

By Whom In What

A Basic Template for Defining Indicators

20

Evaluation Considerations

Over and above the PSO Funding Units ongoing monitoring, is there a need for periodic evaluation?

If so, should the focus be: Formative? Summative? Issue-based? Some combination of the above?

What would you like to learn most from an evaluation of the PSO Funding program?

21

Evaluation Considerations

Non-profit Work and the Nature of Evaluation (Festen and Philbin, 2007)

You do work. When you evaluate how well you do what you do, it’s called a

process evaluation. Your work has results. When you evaluate the

results of your work, it’s called an outcome evaluation.

Lots of work produces multiple outcomes over time. This equals

impact or long-term outcomes.

22

Evaluation Considerations

Formative evaluations: Are associated primarily with analysis of program implementation

and early results / progress Provide feedback so that implementation can be improved and

barriers to improved performance can be identified and removed

Summative evaluations: Deal with whether the program has achieved its’ intended

outcomes. Focuses on;

Relevance / ongoing need Success Cost-effectiveness and Alternatives

23

Evaluation Considerations

Sample Evaluation Strategy

Evaluation Question

Perf.Indicator

Data Source

CollectionMethod

Responsibility

Timing

How successful has PSO funding been in providing a stable environment within which to deliver sport in Manitoba?

Fluctuations in annual funding amounts (overall budget and to PSOs)Extent PSOs view funding as providing a stable environment

PSO Funding Unit Files

PSOs

File Review

Interviews (Sample of 20)

PSO Funding unit staff / Evaluator

Evaluator

Summative evaluation in 2012

Summative Evaluation in 2012

Evaluation Vs Monitoring

Resources are usually a part of the program

Targeted resources are needed for each program evaluation

Attribution is generally assumedAttribution of observed outcomes is usually a key question

Measures are developed & data are usually gathered through routinized process

Measures are usually customized for each program evaluation

Broad issues Issue Specific

OngoingEpisodic

MonitoringProgram Evaluation

Program managers play a key role in developing and using performance measurement / monitoring system

Program evaluators are not programmanagers

Source: Adapted from McDavid & Hawthorn, 2006.

25

Learning Considerations

Considering how the PSO Funding Unit plans to use monitoring and evaluation:

What are the capacity implications for PSOs? Are we being realistic? What can we do to make it easier for PSOs to meet our

PSO Funding Unit accountability requirements? What can we do to help PSOs being more intentional

about using evaluation as a tool for learning? How can we use / share the learnings with our PSOs?

Learnings about how to do monitoring and / or evaluation effectively?

Learnings from the results of the information gathered and analysed from our monitoring and evaluation?

Logic Model Worksheet

HOW? WHAT WE WANT? WHY?

Activities / Outputs Direct Outcomes Intermediate Outcomes Final Outcomes Def’n; An activity is an operation or work process internal to an organisation, which uses inputs to produce outputs. E.g., Training, research, construction, negotiation, investigation, etc… Def’n: An output is what is produced as a consequence of the activity, e.g., build a website (activity) / website (output) Key Activities and outputs of the Office

Def’n: An immediate outcome is an outcome that is directly attributable to a policy, program or initiative's outputs. In terms of time frame and level, these are short-term outcomes and are often at the level of an increase in awareness of a target population. The direct outcome we would expect to see happen / occur as a result of each key activity of the Office.

Def’n: An intermediate outcome is an outcome that is expected to logically occur once one or more immediate outcomes have been achieved. In terms of time frame and level, these are medium term outcomes and are often at the change of behaviour level among a target population. The outcomes which flow from the direct outcomes and which contribute to the final outcomes.

Def’n: Final Outcome is the highest-level outcome that can be reasonably attributed to a policy, program or initiative in causal manner, and is the consequence of one or more intermediate outcomes having been achieved. These outcomes usually represent the raison d'être of a policy, program or initiative. They are long-term outcomes that represent a change of state of a target population. The end outcome(s); what we are ultimately trying to achieve through the contribution of our work.

Def’ns adapted from the Treasury Board Secretariat Results-based Management Lexicon. 26

27

Building the Strategic Framework

What is the time frame for the Strategic Framework?

What level of resources are allocated for the PSO Funding Unit?

Existing / New? O&M? / Other? Staff / FTEs?

What is the reach of the PSO Funding Unit?

27

28

Building the Strategic Framework

Over the longer-term, what is the “why” of the PSO Funding Unit?

What are the key activity areas / outputs?

For each of the key activity areas, what are the immediate outcomes?

What are the intermediate outcomes?

28

29

Building the Strategic Framework

Logic check - Is the logic of the strategic framework clear and robust:

Any logic leaps?

Any key gaps?

Any of the “reach” missing? 29

30

Next Steps

A good performance-based planning and reporting system takes time, patience and persistence.

Strategic discussions along the way are as important as the end product.

Next Steps and Timing

Additional Definitions and Examples

32

Key Definitions and Examples

Activities:An operation or work process internal to an organisation, intended to produce specific outputs (e.g. products or services). Activities are the primary link in the chain through which outcomes are achieved.

Examples:Development of …Preparation of …Coordination of …Identification of …

33

Key Definitions and Examples

Outputs:Direct products or services stemming from the activities of a policy, program, or initiative, and delivered to a target group or population. Usually things you can count.

Examples:Plans for …Tools for …Workshops / training sessions / etc.WebsitesDocumentsEtc.

34

Key Definitions and Examples

Reach:The individuals and organisations targeted and directly affected by a policy, program or initiative.

Examples:Farm organizationsProducersLandownersEtc.

35

Key Definitions and Examples

Outcomes:An external consequence attributed to an organisation, policy, program or initiative that is considered significant in relation to its commitments. Outcomes may be described as: immediate, intermediate or final, direct or indirect, intended or unintended.

Examples of immediate outcomes:Better understanding of … Easy access to more information on …Change in attitudes regarding ...Enhanced participation of … in …Increased capability of …Opportunities for employment for … in ...Etc.

36

Key Definitions and Examples

Examples of intermediate outcomes:Continued employment (after end of project)Use of information / knowledge gained / etc.Improved decision-makingSustained change in (behaviour or practice)Etc.

Examples of final outcomes:Improved socio-economic conditionsIncrease in exportsImproved environment / healthEtc.