Evaluation Function: Managing for

Results

2

Purpose

•  Outline requirements set out in TB Policy on Evaluation (2009), Directive and Standard, and MAF criteria for AOM 6 (Evaluation) relating to management of the evaluation function.

•  Outline performance measures for the evaluation

function to: –  Ensure compliance with TB Policy on Evaluation and MAF

requirements; –  Support annual performance reporting.

3

Context

•  TB Policy on Evaluation (2009) requires conformance with specific directives and standards.

•  Departmental compliance with the Policy on Evaluation is

assessed annually through the MAF, and through a neutral assessment of the function that should be conducted at a minimum of once every five years.

4

TB Evaluation Policy Requirements •  DM is responsible for the department’s evaluation function, and for ensuring

compliance with the requirements set-out in the Evaluation Policy (2009) and Financial Administration Act:

–  Quality of Evaluation Information – evaluations to focus on five core issues related to relevance and performance, and be conducted in accordance with Directives and Standards associated with Evaluation Policy.

–  Governance and Support – independence, control of resources, Departmental Evaluation Committee, Performance Measurement to support evaluation.

–  Coverage Requirements – 100% coverage of direct program spending every five years; evaluation of ongoing grants and contributions programs every five years.

–  Use of Evaluation to Support Decision-Making and Reporting – information included in the development of TB submissions, MCs, RPP/DPR, Strategic Review, MRAP follow-up.

5

MAF Requirements •  MAF Area of Management 6 (Evaluation) objective: “A comprehensive and reliable base of evaluation evidence is created and

subsequently used to support policy and program improvement, expenditure management, Cabinet decision-making and public reporting”.

•  Evaluation function is assessed against four lines of evidence:

1.  Quality of Evaluation Reports – reports consistently address relevance and performance in a sound and credible manner.

2.  Governance and Support to the Evaluation Function – evaluation function is independent and adequately resourced, and the organization ensures that sufficient performance information is made available by program managers to support evaluation.

3.  Evaluation Coverage of Direct Program Spending – organization is moving towards full coverage of its direct program spending.

4.  Use of Evaluation to Support Decision-Making and Reporting – evaluation findings are used to support policy and program improvement, expenditure management, Cabinet decision-making and public reporting.

6

Internal Control Framework •  The Evaluation Division has developed an Internal Control Framework, to monitor compliance with

TB Policy on Evaluation, support MAF, and provide assurance

(C) Control Activities

For each business process : 1) integration with assessment of risks2) consideration of cost and effectiveness of control activities; 3) supporting policies and procedures; and 4) management of information systems

Manage the Evaluation Function

·∙   Management Oversight·∙   Management of Resources

(People, Financial Technology, Facilities)

·∙   Practice Management (standard methodology, tools, control framework)

Undertake Evaluations

·∙   AAFC 5-Year Risk-based Evaluation Plan

·∙   Carry-out Individual Evaluations

·∙   Monitor Evaluations

Contribute to Departmental

Performance Monitoring and Reporting

·∙   Review of Corporate Reporting Related to Performance and Evaluations

·∙   Annual Report on State of Performance Measurement

·∙   Program-level Review and Support

(A) Supporting Environment1. Integrity and Ethical Values2. Governance, Organization and Authority3. Strategic Direction4. Management Philosophy and Operating Style5. People Management – Competencies, Professionalism and Leadership

(B) Risk Assessment and Risk Management

1. Risks to the Evaluation Function

2. Risks Associated with Individual Evaluations

(E) Monitoring1. Ongoing monitoring of the control framework

2. Ongoing monitoring of performance of the evaluation function (e.g. Performance Measurement Framework)

3. Periodic assessments - Annual MAF Assessment, And 5-year Neutral Assessment

(D) Information and Communication

·∙   Dissemination of Evaluation Information and Reports·∙   Information on Effectiveness of the Function

7

Internal Control Framework

•  AAFC’s control framework is: –  Similar to federal MAF placemat, and incorporates many of its management elements. –  Based on internationally recognized Committee of Sponsoring Organizations (COSO) Integrated Control Framework. –  A tool to help senior management evaluate the effectiveness of the evaluation function as a whole

•  The five elements of the Internal Control Framework are not sequential. All of these elements need to be in

place at the same time, to support effective management of the function.

A)  Supporting Environment that fosters a neutral, credible and effective evaluation function. Serves as foundation for all other activities. (Includes DEC, Head of Evaluation that reports directly to DM, evaluators abiding by values and ethics, and evaluation standards in conduct of work.)

B)  Risk Assessment and Risk Management - Involves identification and analysis of the internal and external risks faced by the function and the department that threaten the achievement of established objectives (e.g., risk-based evaluation plan, adequacy of resources to support evaluation etc.) Management is able to use the results of the assessment to manage risks proactively.

C)  Control Activities - describe the type of work carried out by the evaluation function (e.g., quality evaluations, coverage, use in decision-making, governance and support).

D)  Information and Communication - Evaluation information identified, captured and communicated on a timely basis (e.g., reporting, publishing, outreach) throughout the evaluation process.

E)  Monitoring - Monitoring activities are carried out on an ongoing basis, and periodically (e.g., MAF, five-year assessment) to ensure function is operating effectively and in compliance with policy requirements throughout the evaluation process. Enables management to assess performance relative to the objectives and overall effectiveness of the function.

8

Performance Monitoring and Reporting •  The ‘Control Activities’ for the evaluation function are outlined in greater

detail in the logic model for the function. –  The logic model sets out the specific outputs and associated immediate,

intermediate and end outcomes for the function.

•  Underlying performance outcomes, targets and indicators are based on TB Policy on Evaluation (including Directive and Standards), and MAF:

–  Align with the objectives of the evaluation function; –  Map to the AAFC Program Activity Architecture (PAA); –  Provide a foundation for performance agreements with staff, and time reporting

against activities.

•  Together, the logic model and performance measurement plan will enable AAFC to measure the effectiveness of the function on an annual basis, and the achievement of results over a five year period.

9

Logic Model for AAFC Evaluation Function

Undertake evaluations Contribute to performance monitoring and reporting Manage evaluation function

Main Activities

Carry-out Individual

Evaluations

Monitor MRAPs and lessons learned from conducting evaluations

AAFC 5-Year Plan

Review of Corporate Reporting related to

Performance

Assess PM and contribute

to capacity building

Program-level Review and

Support

Management Oversight

Risk Management of Evaluation

Function

Monitoring Performance

of the Function

Practice Management

Management of Resources

Sub- Activities

5-Year Plan

Evaluation products and

reports (Information)

MRAP tracking and follow-up

reports;Lessons learned

Comments/suggestions for departmental

reports

·∙   Annual Report on PM

·∙   PM tools and guidance

Advice and recommendations

to program managers

·∙   Input/Feedback to/from DEC, DM;·∙   Risk Assessment and RM Strategy;·∙   Performance Measurement Reports;·∙   HR plans; Competency framework and training plans;

Financial budgets and reporting; Document management and storage plans;

·∙   Practice methodology and tools, control framework.

Outputs

Immediate Outcomes(1-2 years)

·∙   Achievement of planned coverage·∙   Evaluation information is credible, timely,

neutral) ·∙   Improved monitoring of implementation and

timeliness of actions taken by Program areas on MRAPS

·∙   Operational improvements are made to Evaluation Function

·∙   Improved quality and availability of performance information

·∙   Improved accountability in the area of performance measurement

·∙   Improved advice and support to Program areas with respect to performance measurement strategies

·∙   Improved advice and support to Program areas and senior management with respect to reporting on performance

·∙   Compliance with government and departmental policies, directives and standards

·∙   Risks managed to an acceptable level.·∙   Progress towards a credible, neutral and effective

evaluation function in support of evaluation activities

·∙   Full coverage of direct program spending·∙   Evaluation information is consistently used to support decision making on policy, expenditure management and

program improvements·∙   Evaluation information is consistently used in departmental performance reporting

·∙   Neutral, credible and effective evaluation function in support of evaluation activities

Intermediate Outcomes(3-5 years)

·∙   Evaluation needs of Parliament, the public, Ministers and central agencies and the Deputy Head are adequately supported with respect to policies and programs aimed at producing:

- An environmentally sustainable agriculture, agri-food and agri-based products sector- A competitive agriculture, agri-food and agri-based products sector that proactively manages risk- An innovative agriculture, agri-food and agri-based products sector

End Outcome

(5 years and beyond)

10

Key Performance Measures

•  Key performance measures have been identified for the four key areas of coverage under the TB Policy on Evaluation, and the MAF.

•  These key performance measures will form the basis of an annual

report to DEC on the effectiveness of the evaluation function.

•  They will also support a neutral assessment of the function every five years, as required by the TB Policy on Evaluation.

11

Key Performance Measures – Quality of Evaluation Information

Indicators: Targets: Frequency of Assessment:

Reporting:

Credible: Address relevance and performance, employ appropriate methodologies, describe limitations, findings and conclusions flow logically, include strong recommendations and MRAP Neutral: Use multiple lines of evidence; planning considers potential limitations, biases and conflict of interest Timely: Meet deadlines set-out in Five Year Evaluation Plan

100% compliance with TB Directives, Standards 100% compliance with TB Directives, Standards Reporting deadline met minimum of 75% of the time

Annually MAF Report to DEC

Usefulness of evaluation recommendations and overall satisfaction with evaluations carried out expressed through client satisfaction surveys

Majority of programs surveyed strongly agree or agree

Annually Report to DEC

Lessons learned from individual evaluations and a summary of trends from peer review and client satisfaction surveys

Positive feedback from client surveys Issues raised are addressed

Annually Report to DEC

12

Key Performance Measures – Governance and Support

Indicators: Targets: Frequency of Assessment:

Reporting:

Budgeted evaluation resources commensurate with Five-Year Evaluation Plan

Minimum of 75% of planned evaluations completed for each fiscal year

Annually MAF Report to DEC

Existence of a formal human resources plan linked to resource requirements outlined in the Five-Year Evaluation Plan

75-85% of evaluation positions filled

Annually Report to DEC

Existence of individual learning plans that are linked to the competency profile for evaluators

100% of evaluators have learning plans 75% of learning activities are completed Each evaluator has a minimum of 35 hours of training (based on a combination of “on the job training” and classroom training)

Annually Report to DEC

13

Key Performance Measures - Coverage Indicators: Targets: Frequency of

Assessment: Reporting:

Five-Year Plan aligns with MRRS; supports expenditure management requirements; risk-based; covers all direct program spending; estimates required resources to conduct evaluations

100% Planned coverage of AAFC PAA Majority of evaluations scheduled to respond to senior management information needs

Annually MAF Report to DEC

Five-Year Evaluation Plan submitted to DEC and TBS in accordance with established deadlines

Submitted annually by April 15th Annually MAF Report to DEC

Planned evaluation projects are launched as set out in the Five-Year Evaluation Plan

75% of planned evaluations have terms of reference so that evaluations can be launched on schedule

Annually Report to DEC

Coverage of Direct Program Spending: demonstrated progress towards achieving coverage of all direct program spending over five years (excluding G&C spending)

At least 20% average annual evaluation coverage of direct program spending 100% coverage of program spending over a five-year period

Annually Every five years

MAF Report to DEC

Coverage of Ongoing G&C Spending: demonstrated progress towards regular and systematic evaluation of G&C spending

At least 20% average annual evaluation coverage of G&C spending

Annually MAF Report to DEC

14

Key Performance Measures – Use to Support Decision-Making and Reporting

Indicators: Targets: Frequency of Assessment:

Reporting:

TB Submissions, MCs, RPP/DPR: timely use of evaluation to support organizational management and decision-making

TB submissions/MCs include evaluation findings, when available RPP and DRP include appropriate references to evaluation findings

Annually MAF Report to DEC

Evaluation Follow-Up: explicit acknowledgement and response from management regarding intended follow-up to evaluation results and recommendations; systematic follow-up

100% of MRAPs are monitored and reported to DEC

Annually Semi-annually

MAF Report to DEC Report to DEC

Evaluation Reports are shared with TBS upon DEC approval

100% of evaluation reports are sent to TBS within 7 days upon DEC approval

Annually MAF

Evaluation Reports are posted in both official languages within 90 days following approval

75% of evaluation reports are published within 90 days

Annually MAF

Annual Report on State of Performance Measurement Completed annually by June 30th Annually Report to DEC

15

Conclusion

•  The Evaluation Division has developed a data application to support monitoring and reporting against the evaluation function control framework.

•  Individual performance agreements and learning plans are aligned with the activities outlined in the performance measurement and control frameworks for the function.

•  First report on the effectiveness of the evaluation function was

presented to AAFC’s Departmental Evaluation Committee in June 2011.