Getting more focus on effect and

effectiveness

Director Inge Laustsen,The State of Bahia Court of Audit, Brazil, 26th November 2012

Agenda

Who am I, where do I come from?Why this subject?What have we done in Denmark?Effect focus in audits of the educational sector – two examplesGood advice on getting focus on effect

Who am I?

• Master’s degree in political science, auditor since 1998, in performance audit since 1999

• Director since 2004 for a section with approximately 10 auditors. Responsible for planning and execution of performance audit reports in the areas of education, science, employment and culture

• Responsible for the delivery of 3 performance audit reports every year – and follow up on prior studies

• Member of Performance Audit Subcommittee since 2008

• Participant in a peer review of the performance audit function in the SAI of India

My organisation - and performance audit

• Denmark has a population of approximately 5,6 mio. people

• The National Audit Office of Denmark (Rigsrevisionen) employs approximately 270 persons

• About 70 are engaged fulltime with performance audit. All graduates with in social science. We deliver 17-18 performance audit reports every year

• The tasks and authority of the Public Accounts Committee (PAC) are described in the Constitution and in the Act on the Public Accounts Committee. The PAC comments on the reports presented by Rigsrevisionen and requests that Rigsrevisionen carries out certain examinations

Why this subject?

•The definition of performance audit (ISSAI 3100):

•“Performance auditing is an independent and objective examination of government undertakings, systems, programmes or organisations, with regard to one or more of the three aspects of economy, efficiency and effectiveness, aiming to lead to improvements”

Definition of the 3 Es

• “the economy of activities in accordance with sound administrative principles and practices, and management policies”

• “the efficiency of utilisation of human, financial and other resources, including examination of information systems, performance measures and monitoring arrangements, and procedures followed by audited entities for remedying identified deficiencies”

• “the effectiveness of performance in relation to the achievement of the objectives of the audited entity, and the actual impact of activities compared with the intended impact”

What have we done in Denmark?• We started to conduct performance audit around 1975. In the

beginning the audits mainly focused on topics evolving from financial audit

• In 2000 we were specialized in departements for financial audit and departements for performance audit – performance audit were thereby strengthened

• After the division we got better control of the production of reports, the strategic planning was improved, guidelines and procedures were created to support the process of the individual audit and reporting was highly improved

• Over the years compliance or economy topics were incorporated in the audits but topics related to efficiency and effectiveness were rare

Peer review in 2006

“The NAOD's performance audit objectives tend to have a compliance focus and, to a lesser extent, address the areas of economy, efficiency, and effectiveness. This compliance focus may influence the ability of the NAOD to contribute to modernization and change in the public service.”

What did we do to get more focus on the 3 Es?

• Mission: Rigsrevisionen audits the state accounts on behalf of the Danish Parliament and supports the development of efficient administration in order to create maximum value for the citizens

• Strategic objective: we promote quality, effectiveness, efficiency and economy in the administration

• Focus areas: examining whether the adminstration implements the decisions made by the parliament, taking into consideration quality as well as economy; examining significant areas where the risk of low effectiveness is high, i.e. areas involving many actors, or areas where many large projects are launched

• Performance targets: 75 per cent of our preliminary studies should be about at least one of our three focus areas, i.e. effectiveness, efficiency and financial management

More focus on effect and effectiveness

• Enhanced focus on effect and effectiveness in planning and selecting audit topics

• Important with a common understanding of, what we mean by focusing on effect and effectiveness

• Development of our own guide to inspire and to conduct studies of effect and effectiveness

• Definition of the different types of studies, and description of which tools and methods to use

• Doing it – launched 3 ”flagship” studies in the couple of years

Types of studies – and our role

• Goal-attainment studies – goals compared with short term effect

• Effect studies – has the expected effect been reached, are there other effects

• Cost-effectiveness studies – relation between effect and the costs to obtain it

• Our role is mainly as ”driver” in getting the audited entities to focus on effect, we will make our own studies on rare occasions

Investment in PhD education

• The report was about the investment in the PhD education, i.e. the decision made by the Folketing to double the intake of PhD students gradually over the years 2005 to 2012 and establish PhD schools. The objective of this investment was to increase the national research capacity and thereby provide the basis for continued growth and welfare in Denmark.

• The problem: The size of the investment and its educational implications, which include implementation of extensive organisational changes at the universities

• The objective of the study was to assess the progress of the implementation of the PhD investment. The audit questions: Have the universities and the Ministry of Science implemented the PhD investment? And Has the Ministry of Science, Technology and Innovation prepared and followed up on the investment in the PhD education?

Investment in PhD education

• Objective was achieved, all universities achieved their objectives, the organizational changes were implemented

• But only an input objective was set, no output or effect objectives had been considered

• The objective was not based on any sound analysis of where the PhDs would be expected to get employment afterwards

• Important to choose the right way to analyse effect – e.g. wage level and employment rate

• There was a growing number of international Phds students, and many of these foreign students had so far left the country after graduating degree – risk of becoming a net exporter of PhDs

Investment in PhD education

• “In 2012 the Folketing will decide how to continue the investment in the PhD education. Considering the size of the investment and its educational implications, Rigsrevisionen is of the opinion that the Ministry of Science should provide the Folketing with a sound basis for this decision. The ministry should, for instance, collect more information on the extent to which and within which areas the private business sector is recruiting PhDs, as this sector is expected to employ the majority of the new PhD graduates. The ministry should also look into the effect of the increased number of PhD graduates from the universities on Danish economy, and assess the development in PhD immigration and migration. The ministry has stated that this knowledge will be obtained. “

95 per cent objective set for secondary education

• The report is about the Ministry of Children and Education's management of the effort made to ensure that at least 95 per cent of a youth cohort should complete a secondary education by 2015 (intermediate objective of 85 per cent in 2010)

• The problem: The objective was supported by a large effort with a lot of initiatives. The overall objective was reached in 2010, but not in the area targeted by initiatives

• The objective of the examination was to assess whether the Ministry of Children and Education had worked towards achievement of the objective.

• One of the audit questions: Had the Ministry of Children and Education any knowledge of the effect of the effort made to ensure that more young people complete a secondary education ?

95 per cent objective set for secondary education

• The overall objective was not split up in sub-objectives. Neither was any idea put forward, on how much the single initiatives should achieve in correspondance with the overall objective

• Most initiatives were followed, but only to be informed on whether they were implemented – and that regardless of Parliaments expressed expectation to recieve knowledge about the effect

• Because no objectives were set up, no baseline measured – no knowledge about effect was produced

• Recommendation: The mayor initiatives, the most costly or the most important should be followed more closely. Set objectives, be specific about the theory of change. Use indicators to tell if the initiative is on the right track, if is difficult to prove causal effect

95 per cent objective set for secondary education

• ”Rigsrevisionen recommends that the ministry should operationalize the 95 per cent objective and in the future set objectives for the various elements of the effort and initiatives – individually or group based – in order to assess also in what way the initiatives contribute to achievement of the objective and to ensure follow-up on the effect of the initiatives.”

• “It is Rigsrevisionen’s overall assessment that an improved level of knowledge of the effect of the effort and initiatives in combination with enhanced quality of data on the actual educational status of young people would put the Folketing in a better position to decide on possible adjustments of the effort.”

Good advice – on getting focus on effect

•Strategic choice and commitment

•Common understanding of what we are aiming at, when and how

•Focus on effect in areas where it is needed and possible

•Make it simple – it doesn’t have to be rocket science

•Good communication skills – explain why it is important to focus on effect – for the sake of the parliament, the administration as well as the taxpayer

Performance Audit Subcommittees focus on effect

• Develop standards and guidelines for performance audit (ISSAI 3000 and 3100 with appendix). Recently produced a serie of good practise guidance

• Performance audit standards are mainly focused on the performance audit process

• We are now discussing how to get more focus on the 3 Es, the essence of performance audit – e.g. how we can share knowledge on selection of topics, choice of criteria and methods – and do it in a costefficient way

• How can we share knowledge on getting more focus on effect and effectiveness?

Useful links

• Homepage of the National Audit Office of Denmark (Rigsrevisionen): http://www.rigsrevisionen.dk/composite-6.htm

• Mission and strategy: http://www.rigsrevisionen.dk/composite-2849.htm

• Peer review report: http://www.rigsrevisionen.dk/composite-1030.htm

• Phd report: http://www.rigsrevisionen.dk/media(1840,1033)/07-2010.pdf

• 95 % report : http://www.rigsrevisionen.dk/composite-2883.htm

• 3 ”flagship” reports: http://www.rigsrevisionen.dk/composite-2453.htm, http://www.rigsrevisionen.dk/composite-1254.htm, and http://www.rigsrevisionen.dk/composite-1253.htm

• Homepage of the Performance Audit Subcommittee: http://www.psc-intosai.org/composite-170.htm, and http://www.psc-intosai.org/composite-239.htm

• My email adress: il@rigsrevisionen.dk