An Integrated Approach to Performance Measurement for the CIC Settlement Program

Presented to the Atlantic Region Association of Immigrant Serving Agencies (ARAISA) ConferenceSeptember 22, 2010

30/07/2010 4:11 PM

Objectives of the Presentation

The objective of the presentation is:

1. To provide an overview of an integrated performance measurement framework for the CIC Settlement Program.

2. To report on the work of the Performance Measurement Advisory Group (PMAG)

3. To gather your comments and suggestions about the use of various data collection methods for immediate outcomes

2

3

VisionTogether, we all make a difference in the settlement and integration of newcomers

We can measure and assess the impact of our actions

Acting upon this information permits on-going improvement of services

MonitoringPolicy/ProgramDevelopment

PerformanceMeasurement

Research& Evaluation

Reporting& Planning

• Program level• Outputs• Outcomes• Client feedback• Public opinion polls

on performance

The Program-Policy Continuum

• Departmental planning & performance reports

• Strategic Review• Best Practices

• Objectives• Planned results• Improvements• Design & delivery• Implementation

• Activities• Financials• Operations• Systems

• Broader context• Impact, efficiency,

effectiveness & alternatives

• Periodic evaluations

Evidence Based Decision-Making

Performance Measurement has a unique but complementary role in program-policy development

4

5

What is Performance Measurement?

• It is the systematic gathering of evidence about the outputs (products and services) and outcomes (results) of a program or policy

• It is a cyclical process of measurement, collection, analysis and reporting of information to assess the achievements of a program or policy and facilitate informed decision-making

It answers the key questions:• What do we want to accomplish over the life of the program?• How will we know how well the program is doing?• What evidence is there to support the needs to be addressed by the program?• What needs to be undertaken to improve the program?

What is the performance story?

What needs to be done to improve

policy and programming?

How will we measure success?

How will we collect information?

What are the data and information

telling us?

How will we communicate it?

What are the indicators? What are the objectives

and planned results?

When will we collect

information?

Performance Measurement On-going collection of information on program performance (outputs & outcomes) to guide decision-making and enable reporting on program results

Settlement program• Logic Model with expected Outcomes• Terms and conditions• Funding

Settlement Services Based on priorities for services and programming and directives to manage contribution agreements (left side of the logic model)

Outputs

Immediate Outcomes

Intermediate Outcomes

Ultimate Outcomes

MonitoringFinancial & operational activities exercised by settlement officers over contribution agreements and SPOs.

Accountability, Risk and Audit Framework (ARAF)

6

AuditTo determine whether the Department’s strategy and practices related to

risk management, control, and governance processes are adequate.

EvaluationSystematic collection and analysis of the performance of a program to make judgements about relevance, progress or success and cost-effectiveness to

inform future programming decisions about design and implementation.

A visual representation of the interaction between elements of the June 2008 ARAF that relate to performance measurement of the CIC Settlement Program.

7

OutputsOutputs are direct products or services under a policy, program or initiative, delivered to a target group or population, which serve to achieve outcomes.

Responsibility for gathering data:• Service provider organisations (SPOs) are responsible for providing program outputs to

CIC

Data is captured through these sources:• iCAMS (Immigration-Contribution Accountability Measurement System): collects mostly

quantitative data about settlement services provided• ARS (Automated Reservation System): collects data from Ontario service providers about

language services, which is uploaded to iCAMS• ELT Data Collection Tool: collects data about ELT program• Annual Reports from BC, Manitoba & Quebec

Other methodology that could be used:• SPO-initiated client feedback & other data collection strategies• Website hits & publications produced and distributed

Immediate Outcomes represent the immediate results achieved, are often observable during or directly following the completion of an activity, and are directly attributable to outputs.

8

Immediate Outcomes

Responsibility for gathering the data:• Responsibility falls mostly with SPOs, as immediate outcomes relate to projects

Data is captured through these sources:• Narrative Reporting: to report on project outcomes for monitoring and performance

measurement that were indicated in Schedule 1 of Contribution Agreements: used to specify project results and identify links to program results

• ELT Client Survey: assess the results of ELT programming• Annual Reports from BC, Manitoba & Quebec

Other methods that could be used:• SPO-initiated client feedback & other data collection strategies

• Client feedback forms (distributed at time of service delivery)• CIC-initiated client feedback survey(s)

• General client survey (telephone, online)• Exit Survey• Exit Testing (of knowledge and skills)

Intermediate Outcomes are expected to logically occur once one or more immediate outcomes have been achieved. Their impacts extend beyond program participants.

9

Intermediate Outcomes

Responsibility for gathering the data:• CIC is responsible for collecting data and measuring intermediate outcomes

Data is captured through these sources:• ELT Client Survey

Other methods that could be used:• Client Surveys: gather information from clients about intermediate outcomes as a result

of receiving services• POR (Public Opinion Research): broad-based survey of clients and general public• Follow-up surveys• Focus Testing

Ultimate Outcomes are the highest-level outcomes that can reasonably be attributed to a policy, program or initiative in a casual manner, and is the consequence of achieving one or more intermediate outcomes.

10

Ultimate Outcomes

Responsibility for gathering the data:• CIC is responsible for gathering data from existing sources

Data is captured through these sources:• Existing data sources: Statistics Canada data, Longitudinal Studies, Surveys, etc.

Purpose:• To measure long-term outcomes of newcomers to Canada

Challenges:• Difficult to attribute ultimate outcomes to the Settlement Program.• Ultimate outcomes should be used as a benchmark

• The PMAG was formed in February 2010 at the SIJPPC (Settlement and Integration Joint Policy and Program Council) meeting

• Mandate: To provide CIC with advice on the development of its performance measurement strategy

• Membership includes: Service provider representatives from across Canada and CIC staff (Region and NHQ)

11

Show the impact of our

actions on clients’

settlement and

integration

Performance Measurement Advisory Group (PMAG)

Note:This group may also be called upon to advise on the development and implementation of data capturing methodology

Deliverables and timelines:

• Recommendation of indicators to assess outcomes – June 2010

• Recommendation of outputs – September 2010

• Improve management practices and service delivery and refine methods to

gather outputs and immediate results (Narrative Reports, Client Feedback)

You are Part of the Success...We Need Your Input to Help Move the Work Forward

• Do you have samples of data collection tools/mechanisms you currently use that would show results of Settlement programming?

• What current types of reporting are useful? Not useful?

• What type of information do you want to share about your projects/services?

• How can we improve on gathering data about outputs and immediate outcomes?

12

Discussion

Next Steps

• Continue the work with the PMAG, taking into consideration your comments

• Consult with methodologists to confirm proposed approach to gathering data for – Outputs – Outcomes

• Finalize and implement performance measurement strategy

• Collect data and analyse and report on outputs and outcomes

14

UltimateOutcomes

Key OutcomesA - OrientationB - Language/SkillsC - Labour Market AccessD - Community ConnectionE - Development and Capacity Building

Settlement Program Logic Model – June 2008

Immediate Outcomes

Program Components

CIC Strategic Outcome 3

Outputs

Intermediate Outcomes

Policy Development, Program Design and Management

Policy and Program Development

Settlement Services

Program Implementation and Management

Needs Assessment and Referrals Support Services

Information & Awareness

Services

Employment-related Services

• Initial and on-going needs assessments (including language assessments)

• Referrals to CIC-funded and community settlement services

Enabling services:– Childminding– Transportation

assistance– Provisions for

disabilities

Other support services:

– POE reception services

– Translation– Interpretation– Settlement/ crisis

counselling

• Information products

• Orientation sessions

• Promotion and outreach

• Labour market bridging

• Job search skills training

• Labour market information

• Workplace orientation

23. Successful integration of newcomers into society and the promotion of Canadian Citizenship

1. Policies and programming align with departmental and government priorities

2. Program models are evidenced-based, informed by stakeholder input and address the barriers & needs of both newcomers and communities

3. Standards, tools, resources and program coordination support the effective delivery of services)

4. Services are efficiently delivered 5. Provision of settlement services across Canada that

achieve comparable outcomes

6. Clients, service providers and CIC are aware of newcomer settlement needs

7. Referrals and personalized settlement plans are based on assessed settlement needs

19. Newcomers find employment commensurate with their skills and experience

20. Newcomers enjoy their rights and act on their responsibilities in Canadian society

21. Canadians provide a welcoming community to facilitate the full participation of newcomers into Canadian society

22. Newcomers contribute to the economic, social and cultural development needs of Canada (in PAA)

8. Target population is aware of CIC settlement services9. Timely, useful and appropriate CIC settlement services are available in the Official Language of choice

(in accordance with the Official Languages Act and Policy) 10. Clients obtain the CIC settlement services they need to deal with settlement issues as they emerge

Language Learning &

Skills Development

• Language training

• Other skills/ life-skills training

11. Clients have timely, useful and accurate information needed to make informed settlement decisions

12. Clients understand life in Canada including laws, rights, responsibilities and how to access community resources

13. Clients have the official language skills needed to function in Canadian society

14. Clients have the skills/life-skills needed to function in Canadian society

15. Clients have knowledge of the Canadian work environment and are connected to local labour markets

16. Clients have the skills to find and apply for employment

17. Clients are connected to the broader community and social networks

18. Program participants are aware of newcomers’ needs and contributions and are engaged in newcomer settlement

• Strategic plans• Policy, priorities,

standards and outcomes

• Performance measurement strategy and national reports

• Horizontal coordination

• PT consultations• Research analysis/

and reports• Funding allocation

• Operational plans• Program delivery materials and

tools• Functional guidance & training • Data collection and

regional/local/SPO reports• Regional, local and SPO

coordination• Service delivery capacity

building• Best practices and info sharing• Contribution agreements

Community Connections

• Individual and community-level bridging, e.g.:

– Host/ mentor matches– Volunteers engaged trained

and supported

• Cultural awareness, anti-racism, and welcoming communities services

E

A B C D

Expected Results

A - Orientation

B - Language/Skills

C - Labour Market Access

D – Welcoming Communities

E - Program and Policy Development