2009

AUSTRALASIAN EVALUATION SOCIETY INC.AWARDS FOR EXCELLENCE IN EVALUATION 2009

“There is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in

its success, than to take the lead in the introduction of a new order of things!”

Machiavelli

CATEGORY: BEST EVALUATION POLICY AND SYSTEMS AWARD

Meat & Livestock Australia (MLA) & The Centre for International Economics (CIE)

MEASURING THE VALUE OF MLA PROGRAMS

“Learning by doing”

1

BACKGROUNDAs Australia’s premier research and marketing enterprise for the Australian red meat industry, Meat & Livestock Australia Ltd’s (MLA’s) core activities are: working to improve market access, building demand for Australian red meat, conducting research and development (R&D) to provide competitive advantages from paddock to plate, and partnering with industry to build capability.

MLA is majority funded by compulsory livestock transaction levies, processor levies and matching government funds for R&D work. Other income is generated through activities such as commercialisation of R&D and partnering with industry participants. Currently the red meat and livestock industry is valued at $16 billion per annum.

MLA has constantly sought ways to more accurately and consistently measure and report the benefits and costs of its activities. Periodically, since its inception in 1998, MLA conducted ex-ante and ex-post evaluations using a distinct evaluation approach for each program, however this led to infrequent communication that was not meaningful for key stakeholders.

The largest single contributor to MLA R&D programs is the Australian Government, through matching R&D contributions. The Australian Government is increasingly calling on all Research & Development Corporations (RDCs) to account for the public investment in their programs, requesting verifiable triple bottom line information on R&D program benefits in relation to National Research Priorities.

Although interest in measuring the value of MLA programs had stemmed mainly from the Australian Government, there is also evidence of a growing focus on accountability among industry stakeholders.

By 2005, when MLA had been in existence for six years, the MLA Board deemed it appropriate to systematically assess the value of much of its work.

An evaluation strategy was required for analysis of the net benefits of each program within a coordinated structure capable of identifying new monitoring and measurement parameters going forward.

MLA subsequently developed a comprehensive framework for program evaluation across its diverse marketing, market access and R&D programs. (See appendix, Why Does MLA Need A Framework For Independent Evaluation, for a detailed description.)

“ Rigorous evaluation across MLA’s R&D and marketing programs ensures the net impact of MLA programs can be determined in terms of industry and wider community benefits.”

“ Peer review at various levels has informed the development of an evaluation framework that fosters greater understanding by program managers and increased engagement with key stakeholders.”

LiveCorp program managers aligning programs with strategies.

Sheepmeat councillors participate in peer review of the counterfactual scenario.

2

The MLA Executive Committee developed the following criteria for judging the successful development and implementation of an evaluation framework that would meet this challenge. The Executive Committee pronounced that:

1. A SATISFACTORY EVALUATION FRAMEWORK WOULD:

�be appropriate for the consistent evaluation of marketing, market access and R&D programs

�be appropriate for triple bottom line analysis to evaluate program outcomes against traditional cost-benefit benchmarks and wider societal/environmental considerations

�be sensitive to the influence of the program lifecycle on evaluation processes and criteria

�be comparable between ex-ante and ex-post evaluations

�permit easy identification of key parameters and methodologies for ongoing monitoring and measurement of program Key Performance Indicators (KPIs)

be rigorous, with all evaluations conducted using this framework standing up to scrutiny from both Government and industry

represent a transparent approach, with any measurement techniques applied also being transparent

�be simple and comprehensible to MLA stakeholders

�be an efficient evaluation process that does not drain resources.

2. A SATISFACTORY CORPORATE LEARNING OUTCOME WOULD BE EVIDENT WHEN:

�program managers demonstrate awareness of the benefits of using the framework to identify and monitor more appropriate program KPIs

�program managers demonstrate an understanding of the rules that dictate where equivalent evaluation approaches should be used in marketing, market access and R&D programs, and where there should be departures from this equivalent approach

�program managers demonstrate an understanding of how information on program evaluation can best be incorporated into program proposals

�program managers demonstrate an understanding of how to establish appropriate measurement and monitoring procedures to allow effective and consistent evaluation of new programs

�program managers demonstrate a sense of framework ownership and a willingness to use it in all ongoing work.

3. A SATISFACTORY STAKEHOLDER COMMUNICATION OUTCOME WOULD BE DEMONSTRATED BY:

�a sample set of communication brochures that illustrates how to communicate the outcomes of a program evaluation in simple and relevant terms to various external key stakeholder groups as nominated by the program manager

�presentation of program evaluation outputs in terms that are relevant and understandable to internal stakeholders at various levels including: program investors, MLA executives, other program managers and personnel at individual project levels.

Meat & Livestock austraLia �

evaluationseries

Eating qualityThe industry impact

2.1 improving eating quality

MLA_Eat_Quality.indd 1 22/8/07 5:16:40 PM

Meat & Livestock austraLia �

evaluationseries

Food safety:predictive microbiology

The industry impact

1.1 enhancing product integrity

MLA_Food_Safety.indd 1 22/8/07 5:18:06 PM

Meat & Livestock austraLia �

evaluationseries

Market accessThe industry impact

1.2 Maintaining and liberalising access to world meat markets

MLA_Market _Access.indd 1 25/1/08 2:51:01 PM

evaluationseries

Red meat nutrition marketingThe industry impact

2.2 Enhancing the nutritional reputation of red meat

MLA_RedMeatNutri.2.2.indd 1 9/02/09 1:59 PM

3

SITUATION ANALYSIS

“ Sustainability can be enhanced by increasing …transparency and by implementing sound monitoring and evaluating performance systems. This assures not only continued funding but also probably attracts additional sources of funding.” (International Service for National Agricultural Research)

In an increasingly tight funding environment intensified by international economic decline, RDCs are challenged by growing competition for a share of the public funding pool.

The possibility that Australian Government policymakers and industry investors could follow other countries, who have adopted competitive funding mechanisms to redirect priorities, increase accountability and reduce costs, has placed greater pressure on RDCs to demonstrate a favourable return on stakeholder investment in light of their responsibility “to ensure funding is allocated to achieve the highest return possible”1.

The imperative for RDCs to justify their worth escalates the need for transparency and heightened communication with key sponsors including the Australian Government and peak industry bodies.

While MLA recognises that such communication is dependent on a consistent body of knowledge that can only be collected through thorough evaluation, a system to appraise its performance did not exist prior to 2005, hampering efforts to demonstrate accountability to key investors and alignment with the Australian Government’s National Research Priorities2 and Rural Research and Development Priorities3.

The organisation perceived the situation as a key chance to address this weakness and assume a competitive posture.

MLA drew on its strengths: a professional internal resource base committed to program improvement (effectiveness, efficiency and triple bottom line benefits4); and external expertise in evaluation management, engaging the CIE to assist its development of an evaluation framework that could withstand – and would welcome – intense scrutiny of its programs.

1 MLA Program Evaluation Strategy 2007-10, MLA

2 Department of Education, Employment and Workplace Relations 2003, Australian Government

3 Department of Agriculture, Fisheries and Forestry 2007, Australian Government

4 Atkinson, Dr L 2008, ‘The evaluator as change agent in history: challenges and strategies for success’, Australasian Evaluation Society Inc. paper delivered on September 12, 2008

LiveCorp program manager presenting KPIs. Kate Roberts presenting to program managers on Social Impact Assessment.

“ We have common need for reliable information in directing our investment and need to continue discussion on this issue.” – Professor Beth Woods, Assistant Director-General Innovation, Primary Industries and Fisheries, Queensland Department of Employment, Economic Development and Innovation

“ I have been telling them (other stakeholders) about what we and MLA did (ex-post evaluation), in the lamb industry, and how interesting and useful it was” – Geoff Kroker, Manager Strategy Development Victorian Department of Primary Industries

4

Broad Goal

Purpose

Next Users

End Users

Partners/ Collaborators

NARRATIVE SUMMARY

Build & maintain stakeholder confidence in MLA

1. Reporting to Government2. Engagement with industry

stakeholders3. Guiding future investment4. Building a culture managing for

outcomes

MLA Staff:• MLABoard• ManagingDirector• ExecutiveCommittee• ProgramManagers

Target Stakeholder Audiences:• FederalMinister• ParliamentarySecretary• AustralianGovernment

Department of Agriculture, Fisheries and Forestry (including agencies)

• Politicians• Policymakers• CRRDCCSecretariat• OtherRDCs• PeakIndustryCouncils• IndustryAssociations• ProducerGroups• IndustryLeaders• Industrycynics

Evaluation Collaborators:• ServiceProviders• OtherRDCs• CRRDCCSecretariat• PICEDs&staff• OtherRDCs• Federal&StateGovernmentStaff

MEANS OF VERIFICATION

• MLAMembershipregister• Membershipsurvey• FeedbackatAGM• AnnualreviewofAnnual

Operating Plan (AOP) by PICs• CRRDCCreporting&feedback• Levelsofco-investment,

collaboration, and alignment with NRPs as reported in Annual Report

• Regularindependentmeta-analysis of the results arising from the framework.

• Resultsreportedusingascorecard for tracking progress towards stated purpose and objectives.

• ImprovementinthealignmentofAOP KPIs and industry outcome measures.

• Extentofalignmentispeerreviewed by MLA and PIC staff on an annual basis.

• Regularsurveyoftargetaudiencemembers in Governments, PICs, and CRRDCC to assess value of information

• Collectionofevidencethatinformation has been used in press releases, speeches, reports, and policy documents.

• Evidenceofsharedandendorsedex-ante reviews

• Evidenceofsharedandendorsedex-post reviews

• Evidenceofcommitmenttoandco-investment in collaborative evaluation frameworks

MEASURABLE INDICATORS

• Membershipnumbers• Membershipsatisfaction• PICsupportforMLA• On-goingsupportforlevy• On-goingco-investmentinR&D

by Australian Government• CollaborationbetweenRDCs

1. Reporting to Governmenta. Simple & Relevantb. Meeting compliance requirementsc. Independent & objectived. Understandable & Useful for

Policy2. Engagement with industry

stakeholdersa. Meeting member expectationsb. Simple & Relevantc. Understandable & Useful for

Policy3. Guiding future investmenta. Consistent across programsb. Triple bottom line reportingc. Used to better understand the

value add arising from leveragingd. Comparable ex-ante to ex-post4. Building a culture managing for

outcomesa. Staff have easy access trainingb. Outcome KPIs developed &

reportedc. Measurement is transparent &

objectived. MERI is resource efficiente. Use of program logic in program

planningf. Understandable & Useful for

Program improvementg. Staff are rewardedh. Expert verification

• Recordedevidenceofprogramimprovement

• Valueofevaluationresultsasaninput into decision making and policy development by both industry and Government

• Allpartnersparticipateinthedevelopment of the program logic for all collaborative programs.

• Allpartnersendorseandsupportthe inclusion of measurement & evaluation activities in to program design in order to meet the data needs of the evaluation framework

ASSUMPTIONS

• AustralianGovernmentcontinuesto demand evaluation reporting for continued support of the RDC model

• PICsandProducerscontinuetodemand evaluation reporting to support the RDC model

• Thereisnocatastrophicdisruptionto supply or market access

• OtherRDCscontinuetoremainopen to collaboration and committed to harmonisation of evaluation reporting

• On-goingsupportfromtheMLAExecutive Committee for the MLA Program evaluation Strategy 2007 – 10

• PICstaffandresourceswillbeavailable to undertake these reviews.

• Thatthecurrenthardcopyandweb based versions of the evaluation results continue to reach target audiences.

• Thetargetstakeholderaudienceshave a capability to interpret and use results from evaluations.

• Evaluationcapacitybuildingremains a high priority for all partners and collaborators.

• Evaluationframeworkisefficientand does not become a drain on program resources.

‘People-Focused’ Logic for Capacity building projects: Dr Jess Dart, Workshop # 8, AES Conference 2005

LOGICAL FRAMEWORK MATRIX FOR ADOPTION OF EVALUATION AT MLA (DART, 2005)

5

The MLA Program Evaluation Strategy 2007-10 and the supporting program evaluation schedule drive the evaluation process at MLA (see table right). These are reviewed and agreed with the MLA Executive team, on an annual basis.

The core lines of inquiry for the evaluation framework are best summed up in MLA’s five key evaluation questions. It is important that these questions are sensitive to the influence of the program lifecycle: Are MLA programs adding value? �Are MLA’s collaborations efficient and effective? �Does MLA facilitate innovation from concept

through to commercialisation? �Does MLA facilitate increased industry

capability and investment in innovation? �Is MLA identifying – and correctly managing

– industry risk?1

1 MLA Program Evaluation Strategy 2007-10, MLA

IMPLEMENTATION

IDENTIFYING OPPORTUNITIES FOR PROGRAM IMPROVEMENT ACROSS THE PROGRAM EVALUATION LIFECYCLEThe diagram illustrates how the MLA framework and specific business/corporate functions support programs throughout their various program evaluation lifecycle stages:

ENGAGEMENT:Ex-postanalysis

INPUT:Ex-antereview

OUTPUT:AOP* KPIReview

OUTCOME:Monitor &

Report

BUSINESS SERVICES•Commercialise/Adopt•CorporateCommunications

CORPORATE SERVICES•Evaluation•SAP/Finance/Contracts

ALL MLA PROGRAMS HAVE LIFECYCLESKPI

ENGAGEMENT

IMPA

CT

TIME

Early indicatorse.g Investment + engagement levels

Progress- Ongoing engagement- Satisfaction- Project progress

Outputs- Workshops- Technology- POS material

Potential benefit quantified

Awareness Capability to adopt Adoption Measure impact

Adoption & benefit realised

KPIOUTPUT

KPIINPUT

KPIOUTCOME

MLA EVALUATION SCHEDULE

*AOP: Annual Operating Plan

2007 2008 2009 2010

Enhancing product integrity 1.1

Maintaining and liberalising access to world meat markets 1.2

Improving eating quality 2.1

Enhancing the nutritional reputation of red meat 2.2

Aggressive promotion in the market place (sheep) 2.5

Increasing cost efficiency and productivity – on farm 3.1

Aggressive promotion in the market place (beef) 2.5

Improving industry and market information 3.3

Ensuring sustainability 3.4

Developing new markets and products (beef) 2.3

Developing new markets and products (sheep) 2.3

Promoting industry integrity (beef) 2.4

6

ASSESSMENT“ Over time every MLA program will be subject to thorough, independent evaluation. This process imposes a new discipline on the company. …It is recognised that all staff need to share and benefit from the evaluation process. It is the responsibility of the Executive and Managing Director to illustrate and drive those benefits through the company [and] this will be acted on.” David Palmer, MD MLA, 4 December 2008

Effective implementation of the framework is facilitated by ensuring that the principles of the MLA Evaluation Framework are aligned with the company’s values, vision and strategy.The table below summarises the challenges and issues that have arisen during the implementation phase. MLA management has responded with strategies to increase the effectiveness of the change management process, build evaluation capacity, and create a more evaluation-focused organisational culture.These strategies have included: learning on the job, using a ‘people-focused program logic’ approach to implement the framework, creating a cross-functional team to champion implementation, engaging key stakeholders, linking outcomes to employee performance measures, and leveraging off the Australasian Evaluation Society (AES) network of expertise.

SUMMARY OF CHALLENGES & TACTICS FOR CHANGECHALLENGES TO CHANGE SUCCESSFUL CHANGE TACTICS

1. Making the complex simple

1. Alignment with MLA values, vision, and strategy

2. Embedding a focus on outcomes

2. Engaging key stakeholders in setting better outcome KPIs

3. Maintaining independence and consistency across programs

3. Engaging the Executive Committee

4. Building evaluation capability

4. Creating cross-functional teams, learning by doing, reflecting

5. Developing a learning culture 5. Creating layers of independence and transparency

SHARED GOALSMEAT INDUSTRY

STRATEGIC PLAN

GOVERNMENT RESEARCH PRIORITIES

INDUSTRY CONSULTATION PROCESS

MLA STRATEGIC PLAN

ANNUAL OPERATING PLAN

PROGRAMS

SUB PROGRAMS

WORK PLAN

STAFF PERFORMANCE PLANS

Linking evaluation with strategy

Meat & Livestock austraLia �

Why does MLA need a framework for

independent evaluation?

7

QUALITY

EX-POST EVALUATION APPROVAL PROCESSMLA needs credible, rigorous and consistent evaluation procedures across the entire range of its R&D and marketing programs to ensure program assessments can effectively determine the net impact of MLA programs in terms of both industry and wider community benefits.

Each year the Executive Committee determines the major ex-post evaluations that will make a significant contribution towards meeting objectives of the independent company performance review scheduled for 2010. These appraisals are subject to a scrupulous planning and reporting process to ensure consistent, high quality information is provided to program managers to assist decision-making. The following flow chart demonstrates the approval process that supports this approach.

MLA EX-POST EVALUATION APPROVAL PROCESS

Approval of proposed MLA Program Evaluation Schedule

Manager Evaluation & Program Improvement works with relevant program managers to develop evaluation proposals

Evaluation team develops evaluation proposal

Independent expert verification team

Program Manager forms the evaluation team

Evaluation team develops the evaluation plan

Data collection & analysis

Evaluation team develops draft evaluation report

Independent expert verification team

Evaluation brochure is published and communicated as appropriate

NO

NO

NO

NO

NO

YES

YES

YES

YES

YES

Manager Evaluation

Executive Committee

Executive Committee

8

LEADING EDGEThe MLA Evaluation Framework embodies three simple principles which guide program managers in their development of individual program KPIs.These are consistent with the Council of Rural Research and Development Corporations’ Chairs (CRRDCC) evaluation guidelines and are based on the Commonwealth Department of Finance and Administration approach to program evaluation:

1Setting a baseline - establishing the change in industry and public benefits that would

occur if MLA does nothing.

2Showing evidence of causality – providing the evidence that proves MLA program

outcomes have led to more industry and public benefits when compared to the baseline.

3Attribution – estimating the proportion of additional benefits that can be reasonably

attributed to the contribution made by MLA programs relative to other influencing factors.

These three simple principles are generic. They are valid in every context within which MLA programs operate. They can be consistently applied using either a ‘bottoms up’ or ‘tops down’ evaluation approach.

‘Bottoms up’ is always the preferred way to conduct ex-post evaluations. This approach is based on reviewing the theory of action for the program and the observed changes in the relevant KPI measures compared to the baseline. This evidence is then used to substantiate the benefits claimed by the program. It is a very transparent process and any claims arising can withstand close scrutiny: �because the baselines are produced using real

data; �because there is clear evidence of causal links

between program outcomes and observed changes in industry performance; and

�because the theory of action means that MLA’s role is well understood and therefore attribution is more objective.

The decision to go ‘tops down’ for an ex-post evaluation is normally due to a combination of factors such as: a lack of evidence linking program outcomes (e.g. adoption figures) to the benefits being claimed by the program; and/or situations in which a large number of parties has contributed to the achievement of claimed benefits, such as a multi-RDC project.

Unfortunately, although it often turns out to be the only practical option, the ‘tops down’ approach to ex-post evaluation is perceived to be more arbitrary and less transparent than the ‘bottoms up’ approach: �because establishment of the baseline is

reliant on industry expert opinion; �because causal linkages between program

outcomes and observed changes in industry performance are based solely on justified assumptions;

�because there is a need to test attribution assumptions using a sensitivity analysis such as breakeven modelling; and

�because multi-party collaboration means that attribution has to be negotiated and therefore is usually a very subjective process.

There have been numerous issues and lessons arising from the application of the MLA Evaluation Framework to the programs evaluated so far: limited evidence for causal linkage between program outcomes and observed industry impact for both marketing and research; negotiating the attribution of observed industry benefits across key stakeholder contributions.

Framework implementation has been an evolutionary ‘learning by doing’ process. Peer review at various levels (detailed under ‘Quality’) has informed the framework’s refinement and improvement. This iterative approach has culminated in the development of a framework that can be adapted so as to be fit for purpose (i.e. marketing vis-à-vis market access vis-à-vis R&D), while maintaining the rigour, transparency, simplicity and resource efficiency necessary to allow increased understanding by program managers and engagement with relevant stakeholders.

9

CONTRIBUTION TO KNOWLEDGEThe CRRDCC released a set of “Guidelines for Evaluation” in May 2007 which were revised in May 2009. The guidelines assist evaluation of the impact of R&D programs funded by RDCs and were established by the CRRDCC to report to Government on the: �collective returns from RDCs to industry �collective public and spillover returns from

RDCs �public and spillover returns that are conditional

on public contributions to RDCs.It is intended that the processes and procedures set out in the CRRDCC framework will represent the minimum evaluation requirement of each RDC. A systematic evaluation of a sample of investments will constitute an ‘evaluation’ pool and an analysis of the overall economic, social and environmental returns is provided on an annual basis.

EFFECTIVENESS The discipline behind the process of negotiating attribution of benefits to program stakeholders has had the greatest impact on MLA staff and MLA investment partners.The MLA Evaluation Framework process for apportioning benefits across program stakeholders typically:a. considers what other stakeholders were

involved; b. considers the contribution of other

stakeholders; andc. rules that the total estimate should incorporate

all stakeholders involved and add up to 100 per cent.

The process of determining attribution involves verification with all program stakeholders.

EQUITYMLA has adopted the Australasian Guidelines for the Ethical Conduct of Evaluation (see http://www.aes.asn.au/). These guidelines are consistent with the current MLA Code of Business Conduct & Ethics (2006), but can be specifically applied to the execution of the evaluation work plan outlined in the MLA evaluation strategy 2007 – 2010.

One of the key objectives of the MLA Evaluation Framework is to enable an outcomes-focused performance culture that rewards measurement and achievement of KPIs.An effective evaluation framework provides findings that can be used for program decision-making, program improvement and optimal resource allocation. Moreover, it empowers MLA staff to further sharpen the focus on achieving real outcomes that align with the objectives of the MLA Strategic Plan 2008 – 2012 (detailed under Assessment).

The MLA Executive & Senior Managers Forum is a recent initiative that is partly the result of implementation of the evaluation framework (see diagram below). Its objectives are to:

1Ensure all staff gain a broader understanding of, and have greater input into, the company’s

objectives and activities

2Ensure alignment of industry and government priorities with MLA strategy and program

implementation

3Ensure program measures align with MLA strategy measures to allow clear reporting to

stakeholders.

Implementation of the evaluation work schedule promotes a collaborative culture of robust program review and improvement within MLA. It will allow successes to be rewarded and replicated, and facilitate knowledge-share of lessons learned. This enables changes that underpin MLA’s continuous improvement in the delivery of its programs and services.

The MLA Strategic Plan 2008 – 2012 works in concert with this evaluation work schedule to deliver a culture of collaborative inquiry and participative reflection within MLA and with our investment partners.

ProjObjectivesActivity KPIsContracted

Activities

MLA measuresAOP Objective KPIs

Strategy 1Strategy 2Strategy 3Strategy n

AOP StrategyStrategy KPIs

Projects

Industry OutcomesMLA measure KPIsEach KPIsfor AOP

ObjectivesAccumulate

(Over 5 years)

KPIs Cascading Down – from industry outcome to Strategic Plan to AOP to project to contract activity

KPIs Cascading Up – from the Project to AOP to Strategic Plan levels

Improving MLA governance throughstrategic alignment of KPIs

10

USE OF INTERNAL AND EXTERNAL RESOURCESThe role of external evaluators (typically the CIE) is to conduct evaluation in a way that builds the capacity of MLA by supporting the evaluation team during any or all of the evaluation phases. This role includes:

� facilitating the scope & scale of the evaluation �advising on appropriate data sources and methods of

collection �designing and implementing various data collection

tools and instruments �drafting the evaluation report.

Using internal staff in evaluations brings corporate knowledge of both the program and MLA into the evaluation process. Internal teams have the advantage of being able to quickly modify the design of an evaluation when it becomes apparent that an evaluation activity is not working.

Using external evaluators brings specific expertise in evaluation design, data collection and analysis or reporting to the evaluation. External evaluators also bring independence to the process.

The ex-post evaluations of most MLA programs are conducted by CIE. The program manager is required to manage the process to ensure that the findings and lessons learned are relevant and useful. This involves actively negotiating the evaluation plan with the contractor/consultant, monitoring data collection and analysis, and approving the final evaluation report in consultation with the independent expert verification team and the Executive Committee.

The activity diagram above illustrates the context and interaction between each of the components of a successful evaluation framework at MLA. It illustrates that reporting to government and engagement with peak industry councils/industry – the function of the MLA evaluation brochure series – contribute to the overarching objective of building and maintaining stakeholder confidence in MLA.

Program improvement is an equally important outcome of the framework. Its embodiment will be observed as guiding improvement in future investment strategy and the creation of a more outcomes-focused management culture at MLA.

The program evaluation schedule is now past the half-way mark and there are very promising signs of sustained program improvement at MLA (e.g. adoption of program logic for planning etc.).

The chart above illustrates the results of a subjective assessment against criteria for MLA’s progress towards desired reporting and program improvement outcomes. The comparison between 2005 and 2008 suggests good progress has been made, particularly with respect to reporting to the Australian Government.

CONCLUSIONWe will know that we have been successful when: �evaluations are meaningful to end-users �MLA staff manage for outcomes �evaluation is core to MLA values.

In essence, implementation of the MLA Evaluation Framework has been all about establishing patterns of behaviour that permit learning and reflection to occur:

�At the strategic level – the evaluation strategy sets the timeframe and processes for learning and reflection. Human knowledge is deeply contextual; it is triggered by circumstance and need, and is revealed in action. “We only need to know what we know when we need to know it”.

�At an operational level – there is a range of ‘learning by doing’ processes that work both in series and parallel. These begin with development of an evaluation proposal for approval by the Executive Committee and finish with final production, stakeholder/peer review workshop, and approval of the evaluation brochure by the MLA Board.

The role of MLA’s Manager Evaluation & Program Improvement is to work with each of the relevant stakeholders (guided by people-focused program logic, detailed on p4) to facilitate the adoption of these patterns of behaviour.

There is strong evidence that attitudes are gradually changing and an evaluative culture is beginning to emerge. This is not an ‘all of company at the same time’ approach, but rather a ‘bit by bit – program by program’ evolution unfolding over the course of the evaluation strategy’s implementation.

SUSTAINABILITY

Culturemanaging

foroutcomes

MLA Board

Evaluation reporting Program improvement

Reportingto

Government

Guiding future

investment

Stakeholder confidence in

MLA

AOP KPIReport

Engagementwith

PICs/industry

KPIReporting

Baselines & measurement

Context and interactions of evaluation at MLA

Levels of achievement for framework objectives*

Ideal

Culture managingfor outcomes

Usefulness as a guidefor future investment

Satisfactory PICResponse

Satisfactory GovernmentResponse

0% 20% 40% 60% 80% 100%

MLA 2005

100%

25%

25%

25%

25%

32%

33%

42%

56%

MLA 2008

*2005 levels represents compliance with minimum statutory requirements.

i

APPENDIXMEASURING PROGRAM VALUE THROUGH EX-POST EVALUATION

PAGE 1

Meat & Livestock austraLia �

Why does MLA need a framework for

independent evaluation?

ii

� Meat & Livestock austraLia

introductionPerformance evaluation is a critical success factor that allows Meat & Livestock australia (MLa) to demonstrate its delivery of world-class services and solutions in partnership with the red meat industry and government.

in 2005 MLa engaged the centre for international economics (cie) to conduct an independent review of the way MLa evaluated its performance and consequently developed a more effective evaluation framework. implementing the recommendations of this review ensures accountability to stakeholders including red meat producers (MLa members), the red meat industry, peak industry bodies, third party investment partners, and the commonwealth Government. the framework allows MLa to assess the industry impact of its programs and their compliance with government priorities.

assessing actual program outcomes against the principal goals – to build demand, increase market access, develop competitive advantage and build industry capability – is called ex-post evaluation. it enables MLa to report back to management and to be answerable to stakeholders.

assessing anticipated program outcomes against goals is called ex-ante evaluation, and this helps MLa to predict the impact of its programs in advance.

independent evaluation of programs across MLa’s portfolio of work is essential to accountability and supports internal management of activities, helping to ensure that a learning process is instilled within MLa. Backing this is an evaluation work schedule developed by MLa to ensure that the framework is consistently applied across all programs and that it involves both ex-post and ex-ante evaluation.

the framework will also report on alignment of MLa program outcomes with the new rural research and Development Priorities, which complement and are informed by the australian Government’s National research Priorities.

Why Does MLa NeeD a fraMeWork for iNDePeNDeNt evaLuatioN?

APPENDIXMEASURING PROGRAM VALUE THROUGH EX-POST EVALUATION

PAGE 2

iii

Meat & Livestock austraLia �

how does the MLa framework for independent evaluation work?MLa programs are evaluated at various stages throughout their lifecycle for their efficiency, economy and effectiveness in achieving objectives based on established criteria. each stage has specific implications for the level and type of evaluation. evaluation also depends on the program’s reporting requirements.

Beyond the basic indicators – inputs and outputs – the MLa evaluation framework traces outcomes, impacts and benefits, providing a comprehensive assessment of program performance (chart 1).

a set of objectives is derived from the australian red Meat industry strategic Plan 2004-2009 and MLa’s own strategic planning processes. these program or project objectives determine the inputs required to achieve the objectives. Monitoring inputs – investment

Objectives

Marketreturns

Benefit-costratio

Triple bottom line

Environmental Social

Inputs

Outputs

Outcomes

KPIs

Integratedframework

Chart 1: Program evaluation overview

Benefits to stakeholders Program evaluation is important for several reasons:

1. Responsibility for investing funds on behalf of the industry. MLa makes investments today to produce future benefits. to ensure funding is allocated to achieve the highest return possible, MLa must evaluate programs ex-ante to determine the expected program benefits or pay-offs and to draw comparisons.

2. Efficient operation. efficient operation is dependent on effective project management, which can only be achieved with program monitoring and evaluation.

3. Learning from the experience of different programs. evaluation allows MLa to learn both from successful programs and from those which need further improvement. the accumulated knowledge provides insights that enable funding allocations to be channelled into higher pay-off areas across MLa’s range of activities.

four key criteria ensure the evaluation is robust:

1. Consistency across programs – this ensures programs can be compared and allows MLa to consistently apply its evaluation framework across the entire program lifecycle and across all programs.

2. Calculation of marginal return from investments – this involves determining whether or not funds can be transferred from one program to another to achieve a higher overall return from the total investment in MLa’s activities.

3. Declaration of the key assumptions – this allows a ‘counterfactual’ case to be established which determines what would have happened in the absence of outcomes generated by the program.

4. Estimation of the spill-over benefits arising from program outcomes – this measures the indirect benefits enjoyed by consumers, the community and other enterprises outside the industry value chain.

APPENDIXMEASURING PROGRAM VALUE THROUGH EX-POST EVALUATION

PAGE 3

iv

Financial modelAssessment andranking of benefit

Chart 2: Integrated framework

Value chain model Global trade modelOther triple bottom line:• environmental• social

Integrated frameworkQuantifiable

Notquantifiable

costs or total dollars spent – at this level produces simple key performance indicators (kPis) that are valuable for short-term project management and monitoring.

inputs lead to outputs – tangible results of inputs, such as the number of butchers trained in better meat display – and these provide another opportunity to develop clear kPis that measure the quality of the output, not just whether it was achieved.

outputs lead to outcomes – the changes in practice or behaviour resulting from adoption of outputs, such as increased sales and ultimately better returns to producers attributable to improved butcher training. outcome kPis measure adoption as well as resulting changes, and measure the extent to which expected results are achieved. Deriving outcomes often requires detailed knowledge about both consumer and producer behaviour, and involves translating outputs into the key differences they make to industry supply and demand relationships. in the case of food safety or animal welfare, the evaluation process translates the output into a risk reduction.

once a supply, demand or risk outcome is identified, it is possible, through the integrated framework, to quantify the benefits and costs to the industry that can be aggregated across programs (chart 2).

the environmental and social outcome components of the triple bottom line can sometimes be quantified and included formally in program cost and benefit assessments. however they are often not quantifiable, particularly in dollars. such benefits are reported qualitatively in case studies.

for MLa to meet its evaluation requirements it must have a thorough appreciation of how the red meat industry interacts. there are four key elements to ensure the evaluation framework accounts for these interactions:

1. A value-chain model of the industry

2. MLA’s global meat industries model

3. A framework to handle triple bottom line environmental and social issues

4. A financial model (to calculate net present values of benefit-cost ratio streams) that allows the effectiveness of the current MLA portfolio to be evaluated against what could be achieved if it was reallocated to maximise MLA’s return on its total investment budget.

further information• For access to the MLA Program Evaluation

Framework manual, the MLA Strategic Plan 2007-2011 or information on MLA programs, visit the MLA website: www.mla.com.au

• To download More From Less: the Strategic Direction for the Australian Red Meat Industry 2004-2009, visit the Red Meat Advisory Council website: www.rmac.com.au

• The Council of Rural Research and Development Corporation Chair Guidelines for Evaluation are available on the Rural RDC website: www.ruralrdc.com.au

Level 1, 165 Walker St, North Sydney NSW 2060 AUSTRALIA Tel +61 2 9463 9333 Fax +61 2 9463 9393 www.mla.com.au

Published by Meat & Livestock Australia LimitedABN 39 081 678 364August 2007© Meat & Livestock Australia (2007)ISBN 9781741911626Disclaimer: Where possible care is taken to ensure the accuracy of the information contained in this publication. However MLA cannot accept responsibility for the accuracy or completeness of the information or opinions contained in the publication. You should make your own enquiries before making decisions concerning your interests. Reproduction in whole or part of this publication is prohibited without prior consent and acknowledgment of Meat & Livestock Australia.

Meat & Livestock Australia acknowledges the matching funds provided by the Australian Government to support the research and development detailed in this publication.

APPENDIXMEASURING PROGRAM VALUE THROUGH EX-POST EVALUATION

PAGE 4