A Primer on Performance AuditingBy

Muhammad Akram KhanFormer Deputy Auditor General of Pakistan

makram1000@gmail.com

AbstractPerformance auditing is an expansion in the scope of traditional auditing. It examines the management of resources in the framework of economy, efficiency and effectiveness. It aims at improving management of resources and sharpening of accountability of the public managers. Performance auditing techniques follow the familiar cycle of planning, executing and reporting. However, each performance audit assignment is unique as it relies on distinct set of criteria, derived from various sources such as project or program plans, generally accepted management practices and technical specifications of assets and services. Besides, performance auditing gathers evidence from diverse sources beyond the internal documents. The performance audit report presents achievements of the management along with any suggestions for overcoming the shortcomings noticed during the audit.

___________________________________________________________________1. Introduction

(a) Evolution of performance auditing

Performance auditing is a recent expansion in the scope of financial auditing. Traditionally, financial auditing has been concerned with financial control and accuracy of accounts. Generally, a single auditor used to check the accounts. The scope of audit covered checking of hundred per cent transactions. For centuries, it had been like that. With the advent of Industrial Revolution and Joint Stock Company, it became difficult for a single person to check hundred percent transactions. It led to at least two changes. First, the single-person auditing gave way to teams of auditors. Second, hundred percent checking became impossible and the auditors adopted sample testing. Simultaneously, the auditors realized that it was not possible to certify accuracy of accounts without hundred percent checking. These developments were quite slow by themselves. But they were slower in government auditing. In the government sector, for centuries, the auditors had been concerned with regularity of expenditure and compliance with regulations. They used to report to the rulers about any frauds or leakages in revenue collection and any waste of public funds. But this role remained in its rudimentary form for centuries. The government auditors did not react to the developments that were taking place in the private sector accounting and auditing. There was a state of complacency. But developments in concepts of democracy and public administrations jerked the government auditors out of their slumber in early seventies of the last century. There was a noticeable change in the scope and approach of government auditing.

It originated from USA, Canada and some countries of Europe like Sweden and West Germany. The elected representatives of the people started demanding information on the efficacy and effectiveness of the public expenditure. They expected a greater accountability of public managers in the management of public resources. Government auditors in developed countries responded to this challenge by developing techniques and methods for auditing performance of the government. This ultimately led to the birth of ‘performance auditing’ as we see it today.

The idea of performance auditing spread like a wild fire among Supreme Auditing Institutions. In 1977 Ninth Congress of the International Organization of Supreme Audit Institutions (INTOSAI) in Lima drew attention to performance auditing. Several countries

amended their audit laws to expand the scope of audit to include performance auditing during eighties of the last century. Next two decades saw immense intellectual activity in methodology development and research in the field of performance auditing.

(b) Objective and scope of the paper

The objective of the present paper is to a give a general description of the concept, methodology and application of performance auditing in the public sector. The next section discusses issues relating to definition, scope and approach of performance auditing. Section three discusses standards relating to performance auditing. Sections four to eight discuss the familiar phases of auditing: planning and executing. Section nine takes up the issue of reporting in performance auditing. The last section consists of concluding remarks and points out some current issues in performance auditing. The style and format of the paper conforms to the need of a general reader and avoids technical debates on various issues.

2. What is performance auditing? Despite a general consensus on the need for expanding the scope of audit, there is no universally agreed definition of performance auditing. However, most of the definitions agree that the performance auditing focuses on the economy, efficiency and effectiveness in the management of public resources. For example:

INTOSAI (2004, 11) has defined the performance auditing as follows: ‘Performance auditing is concerned with the audit of economy, efficiency and effectiveness and embraces:

(a) audit of the economy of administrative activities in accordance with sound administrative principles and practices, and management policies; (b) audit of the efficiency of utilisation of human, financial and other resources, including examination of information systems, performance measures and monitoring arrangements, and procedures followed by audited entities for remedying identified deficiencies; and (c) audit of the effectiveness of performance in relation to achievement of the objectiveness of the audited entity, and audit of the actual impact of activities compared with the intended impact’.

Some other definitions are available at US Governmental Accountability Office (2007, 17), Canadian Comprehensive Auditing Foundation (CCAF, 1991, 8), Australian National Audit Office (1986, 75), Waring, C.G. & S.L. Morgan (2007, 323), Everard & Wolter (1989) and Khan, M.A. (2005, 19). The hard core of performance auditing is the framework of economy, efficiency and effectiveness.

(a) The three Es

Economy refers to acquisition of resources at the lowest cost keeping in view the objectives of the organization. It implies that the resources should be acquired at the right cost, at right time, at right place, in right quantity and of right quality. Economy should be perceived with reference to achievement of the goal. Economy in an absolute sense may mean not to spend anything at all. But in a relative sense it has to be related to the purpose of spending. Thus ‘economy’ means spending only that much which is barely essential to achieve the goal. Methods for measuring economy include comparison with benchmark costs, such as private sector charges, historical costs, costs incurred by a similar public sector organization or budget allocation. Economy also applies to physical resources such as space used, number of vehicles, number of computers and photocopiers. Performance audit measures economy in the use of physical resources against audit criteria, which could be diverse. For example, for space, it could be space

standards of the government, for vehicles, computers and photocopiers, it could be prescribed ratios for the number of staff, etc.

Efficiency means optimum utilization of resources keeping in view the objectives of the organization. It implies maximizing output from the given resources or minimizing input for the given outputs. Analysis of efficiency presumes that standards of input and output are available. But in a large number of cases they are not available and the auditor has to work with the auditee management to determine the agreed standards. The most commonly used standards, however, are planned outputs for given inputs laid down by the auditee itself. Other methods for determining standards of efficiency are efficiency levels of other units within the organization, comparison with the past years, or with other organizations in the public sector or private sector, or international organizations operating in similar circumstances, etc.

Effectiveness refers to the extents the objectives have been achieved. In auditing effectiveness, the auditors distinguish among outputs, outcomes and impacts. The outputs pertain to results of certain inputs produced by the organization. They are generally within the organization. The outcomes relate to the results external to the organization over a short to medium term. Impact refers to the effect of the outcomes on the society in the long run. For example, a certain project visualizes installing tube-wells in rural areas for reducing the level of sub-soil water and protecting the land from water-logging and salinity and increasing the income of farmers. In this example, the number of tube-wells installed would be the output of the project. Reduction in the sub-soil water level and number of acres of land protected would be two outcomes of the project. The increase in the level of farmers' income over the long run would be the impact. In performance auditing, the auditors are supposed to audit all the four elements: inputs, outputs, outcomes and impact. However, generally, they are unable to audit the impact as that can be measured over a long period of time. This can be done only if the audit is undertaken after several years of the project completion. In that case, usually, the data on other three elements would have become irrelevant. Thus, despite theoretical claims, performance auditors hardly ever audit the impact of projects or programs in practice.

Even when the auditors are focusing on outputs only, there are several performance measurement aspects they can examine. For example, what is the quality of the output? What is the rate of error? How reliable is the service? How do users rate utility of an output? To that extent they would be auditing the effectiveness of the project or program.

(b) Objectives of performance auditing

Twelfth Congress of INTOSAI (Sydney, 1986) defined the objectives of performance auditing as follows:

Provision of a basis for the improvement of public sector management

Improvement in the quality of information on the results of public sector management that is available to policy makers, legislators and the general community

Encouragement of public sector management to introduce process for reporting on performance

Provision for more adequate accountability

The above statement clarifies that the performance auditing is a means for (i) improving management practices in the public sector and (ii) sharpening the accountability process of public managers.

(c) Approach of performance auditing

Traditionally, the term ‘audit’ sparks in the mind a sort of activity that is mainly concerned with finding faults, blaming others and searching for weaknesses. For centuries, auditors have been doing that to earn this image. However, performance auditing has departed from this oft-trodden path. It has adopted an innovative approach.

Performance auditing uses the concept of a reasonable manager. Instead of taking a focused look at the weaknesses of the management, the performance auditors try to stand in the shoes of the managers and ask the basic question: what would a reasonable manager do in the given circumstances? The answer to this question transforms their perspective from an adversary to a friend of management. The performance auditors do not try to take the benefit of hindsight wisdom. They take a balanced view and report on the achievements of the management, giving credit where it is due. In fact, performance audit reports begin with the achievements of the management rather than its failures.

There are other innovations that performance auditing has introduced in the audit approach. Financial auditors seldom collect data from external sources except for verification of debtor- creditors’ balances and reconciliation of bank balances. However, the performance auditors do not remain restricted to internal records of the organization. For example, the performance auditors may obtain information on markets and prices from external sources, such as Internet, published journals, and even by market surveys. They may go out in the field and see the operations, satisfying themselves on questions of economy and efficiency.

Performance auditing can be carried out at the level of an organization, a program or a project. It can also be a government-wide study of any particular issue such as human resource management, travel management, or cash management in the whole of government.

Performance auditing follows a normative approach. It does not stop short at identifying instances of waste, inefficiency and ineffectiveness. It goes a step further and makes recommendations for remedial action. Performance auditors do not question the policies directly. However, they trace performance of the projects or programs to policies, if the cause of poor performance lies in faulty policies.

(d) Comparison of performance auditing and financial auditing

The above discussion leads us to summarize the differences between financial auditing and performance auditing. Table 1 below presents that.

Table 1: Comparison of financial auditing and performance auditing

Aspect Financial Auditing Performance Auditing

Purpose Assessment of truth and fairness in financial statements

Assessment of economy, efficiency and effectiveness in management of resources with a focus on improving management and accountability of managers

Approach Finding errors and omissions, Measuring the performance,

based on internal records and evidence

based on internal as well external evidence

Criteria Accounting standards, rules, regulations. Standard criteria for all audits

Applicable rules, regulations, and generally accepted management practices, and technical standards, etc. Unique in each case

Knowledge requirements Accounting, auditing Collective competence of the team in accounting, auditing, management, law, and other disciplines depending on the nature of the auditee operations

Methods Standardized method Vary from audit to audit

Reports Annual report, more or less standardized

Ad hoc reports depending on the completion of audit and reporting policies of the auditee

3. Standards for performance auditingThere are no universally accepted standards for performance auditing. INTOSAI have issued Guidelines for the Implementation of Performance Audits (2004). It has also set up a Performance Audit Standards Sub-committee (2005) to develop standards in light of the Implementation Guide. But as yet it has not issued any standards for performance auditing, though it has issued INTOSAI Auditing Standards (1989) for government auditing in general. The Government Accountability Office (USA) has issued standards of performance auditing in its famous Yellow Book (2007). However, they are meant for application in the US. Despite all this, there is a general consensus on following practices relating to performance audit. Whenever, authentic internationally acceptable auditing standards come into being they would have a substantial resemblance to these practices. Following is a summary of these practices:

Performance audit should have a defined mandate.

Performance audit should be formally planned.

The criteria for performance audit should be made known to the auditee and if possible agreed with them.

Performance audits should be properly supervised.

Performance audit report should be based on relevant, sufficient and competent evidence.

Performance audit should report positive achievements along with any weaknesses and make recommendations for improvement.

Performance auditors should share the draft findings and conclusions with the auditee and incorporate their reaction in the final audit report.

Performance auditors should collectively have competence to audit the assignment.

The performance auditors should be independent and have freedom to select audit areas within the mandate.

The performance auditors should observe ethical code of integrity, objectivity, confidentiality, neutrality, and due care.

4. Planning for a performance audit assignment: preliminary survey reportPlanning for a performance audit assignment is an integral part of the annual audit plan of an audit office. For the sake of brevity we shall not discuss the process of audit planning of an audit office. Needless to say that every audit office prepares an annual plan based on such factors as risk assessment, evaluation of internal controls, political priorities communicated by the legislature or the government, audit cycle, past audit findings and availability of human and financial resources. We are assuming that an audit plan for the office exists and it has identified a particular assignment for performance auditing. The question is: how would the auditors plan this audit? The following discussion attempts to answer this question.

Planning for performance audit culminates in the form of a preliminary survey report (PSR). The objective of the PSR is twofold: (i) to understand the audit entity, determine the objectives and scope of audit, lay down audit criteria, identify issues of potential significance and determine the resources and time budget for the audit. (ii) to make an initial assessment whether the audit office should continue with the conduct of the audit in question in view of the value that it is likely to add. It provides a decision point in the audit process early enough for the audit management to decide if continuing with the performance audit would be appropriate. The auditors take following steps to develop a PSR:

(a) Know the audit entity and subject of audit

The first step in performance audit planning is to understand the objectives, and operations of the organization, and expected outputs, outcomes and impact of the project or program to be audited. For purpose of familiarization, the auditors study the basic documents relating to such areas as internal controls, budgets, operational plans, management information system, claimed achievements, obvious problem areas, significant risks, any initiatives, major procurements and recruitments. They also interview key personnel including, quite often the chief executive officer. They try to get a good grasp of the management information and monitoring system by reviewing a sample of important reports being produced periodically.

(b) Quantify the objectives of audit entity, project or program

A primary concern of the auditors is to see the objectives of the audit entity or project or program in a quantified form. If they are not in that format, the auditors engage the management for doing that. In absence of the quantified and measurable objectives the job of performance auditors becomes immensely difficult. If the auditors do not succeed in this endeavor, it may be a time to re-think about the continuance of the audit any further.

(c) Carry out risk assessment

Risk assessment involves auditors’ judgment about what could go wrong? What is the probability that it will go wrong? What would be the impact if does go wrong? The auditors seek answers to these questions by studying the basic documents, reviewing internal controls,

interviewing key personnel and observing the auditee operations. Risk assessment shows the auditors the way forward for scoping the audit. They would generally focus on those risk areas where the likelihood of happening a risk and its impact is higher as compared to low risk areas.

(d) Determine the audit objectives

At this stage, the auditors try to define their own objectives. The audit objectives broadly define the extent of audit examination and the approach to be adopted by the auditors. All performance audits try to focus on economy, efficiency and effectiveness. However, in each assignment there could be other significant objectives as well. The auditors list down all these objectives.

(e) Identify the issues of potential significance

Next step is to determine the scope of audit. The auditors try to identify issues of potential significance. This is a judgmental area. However, the auditors exercise this judgment in light of such factors as risk assessment, control evaluation, money value, past audit findings, expected savings, any management concerns learned during the familiarization stage, and project outputs and outcomes. It is so important an activity for the PSR that rest of the audit work revolves around these issues.

(f) Lay down the audit criteria

In performance auditing, the auditors have to lay down audit criteria for each issue of potential significance. It is unlike financial auditing where the audit criteria are determined by centuries old practice and even mention of audit criteria seems superfluous. In performance auditing, there is no single source of audit criteria. The auditors have to undertake a sort of research to determine the benchmarks against which they would assess the performance of the auditee. In their research, they may review such sources as follows:

Objectives of the organization, project or program

Generally accepted management practices

Past performance standards

Standard operating procedures (SOPs) of the organization

Rules and regulations applicable to the organization

Sector studies

Comparison with similar organizations, projects or programs

State-of-the-art studies

Academic pronouncements of the profession

Performance specifications of manufacturers of an equipment

Good practice in performance auditing requires that the audit criteria be shared with the auditee management and their agreement obtained at this stage of planning.

(g) Estimate the time and financial budget and nominate the audit team

The PSR contains an estimate of the time budget, and time schedule for various audit activities. It also nominates the auditors for the assignment. Financial budget would include staff and non-staff cost and any fees to be paid to consultants, if they are hired.

5. Executing the performance audit

Developing the audit program

The first step in the execution of performance audit is to plan for the field work for gathering evidence on issues of potential significance identified at the stage of the PSR. It takes the form of developing an audit program. The auditor in-charge writes the audit program for the rest of the team. The audit program consists of a set of written instructions for the auditors about the procedures to be followed during the audit execution. For each issue of potential significance, the auditors mention the audit objectives and criteria which they had already laid down in the PSR to provide a logical link to their previous work. The substantive part of the audit program consists of audit procedures to be followed and the type of evidence to be collected for supporting or rejecting their earlier perceptions about the audit issue. The last part of the program requires the auditors to conclude on each issue. The audit program is a document that remains under revision throughout the field work as the auditors may have to modify the procedures in light of evidence that they collect.

6. Sources of Evidencea) Direct observation/ photos and videos

The most common method of collecting evidence is by field inspection and direct observation. This is a time-tested technique and perhaps no audit can be said be complete if the auditors did not physically inspect the situation. In financial auditing and compliance auditing, it usually relates to physical verification of assets. In performance auditing of projects and programs the area of physical inspection is much wider. The auditors go to the field and see for themselves the actual outputs of the projects or programs. It may involve demonstration of actual functioning of the facilities created by the management. The performance auditors can take pictures and make videos if they consider that it would strengthen the audit conclusion. In case of videos and tape recorders the auditors have to seek permission of the concerned authorities or persons involved. In case of observation of persons actually in action in a certain situation, the auditors ensure that they record the evidence that is representative of the most of the situations. They make sure that they do not get the blame of being biased.

b) Literature searches

The auditors undertake literature searches with a view to familiarizing themselves with the area under audit, for enlisting main issues involved and for spanning the debate on the subject. The literature searches could be general about the sector under study. For example, if auditors are planning to audit waste management, they search for the literature on the subject so that they understand the applicable standards and regulations on the subject. The most common source of literature search these days is Internet. However, most audit offices also have libraries that contain documents on areas of their interest. The literature search could also relate to past studies of the project, program or organization to be audited. They are more specific and mostly available with the management.

c) Interviews

The auditors interview key auditee personnel for gathering information. In large projects such interviews are conducted at three levels: top, middle and operational. The information from the top is usually more authoritative but broad in nature. As the level moves downward more details are available. Collecting information through interviews have its own risks. There could be communication gaps between the interviewer and interviewee; the auditor may make a mistake in making a record of the interview and the interviewee may not own the record of the interviewer later on. Therefore, before

accepting information collected in interviews, it is corroborated from other sources. Sometimes the auditors prefer to send a written note of the interview to the interviewee for confirmation. If they receive the confirmation no further verification may be necessary. At times the interviewees do not respond. The failure to respond to this note by the auditee staff leaves a doubt as to whether they have agreed with the note or would not disagree with it later. Therefore, these notes are used with care. Preferably, before relying on them, they are confirmed from other sources.

d) Reviewing auditee files

This is, again, the most common source of audit evidence. The auditee information could be in hard files. It could also be in electronic databases. The auditors should have authority and be conversant with the techniques to have access to the client databases. The information in the client files and databases could relate to the actual performance of the project or program. The auditors may have to select a sample for their work and that too can be done from the information in client databases.

e) Questionnaires

Auditors prepare questionnaires for soliciting information. A good questionnaire is not suggestive, does not pre-empt the respondent from giving a genuine reply, nor does it limit the scope of reply unnecessarily. The questionnaires requiring the respondent to work hard are likely to get a poor response.

f) Use of expert studies

Sometimes studies by experts or committees are available on different issues or sectors. The auditors make use of these studies also. However, they prefer to test some of the results of these studies before relying on them.

g) Use of consultants

Performance auditors may take the help of consultants in certain areas, especially, those that require technical knowledge. In such cases they check the advice of the consultants for its inherent validity, for sources of its data and for its practicality before they decide to adopt it in the audit report.

h) Departmental manuals

Departmental manuals often contain valuable information. The manuals contain rules, regulations and procedures. Some of the procedures emphasize economy and efficiency in operations. Auditors adopt them as their criteria. Any deviation from these procedures becomes basis for audit opinion.

i) Surveys

Some times the auditors are interested in getting information that is not available in the client files. For example, they may like to formulate an opinion about adequacy of a service being provided to a certain clientele. This can be ascertained only from the clientele. This type of information can be collected only through surveys. Surveys are expensive techniques. They should be considered only when it is absolutely essential to undertake them. The surveys could be through mail questionnaires, in-person interviews, telephonic interviews, e-mails, or circulars through departmental channels. In any case, the designing of survey questionnaires requires specialized expertise to avoid bias and to make the survey results amenable to analysis. Since surveys cannot be administered to the entire population, they usually involve selection of sample and decision on the size of sample. The auditors would require help of statisticians in this work. Conducting surveys

through e-mails has become easier now. There are Internet based service providers who circulate the questionnaires to the given e-mail addresses, collate and analyze the responses electronically for a small fee.

Market research

It involves collection of data about environment in which an organization or project is operating. It also means gathering data on similar projects or programs. The cost for this type of work is often quite high. It also requires knowledge of such disciplines as economics, sociology, psychology, statistics and EDP besides training in auditing and accounting. Sometimes the auditors hire consulting firms to do this specialized work for them.

It is not an exhaustive list of the sources of evidence for performance auditing. Evidence collection is an organic activity. The creativity of auditors may suggest to them certain unique sources of information. They may resort to these sources if they do not conflict with their professional conduct.

Ensuring quality of evidence

It is generally agreed in the auditing profession that the audit evidence should be: (a) valid (b) relevant (c) sufficient (d) timely (e) economical and (f) objective. Valid evidence means that there should be no doubt about the facts. Relevant evidence means that the evidence should be directly related to the point at issue. Sufficient evidence means it should be persuasive enough to enable the auditors express their opinion. Timeliness of evidence means that it should be available within the timeframe of the audit. Need not be said, that the audit evidence should not require incurring of unnecessary expenditure. Lastly, the evidence should be objective and unbiased and not intended to target an individual or entity.

7. Data Analysis in Performance AuditingThe auditors can analyze the data collected through fieldwork in a number of ways, depending upon the objective of audit, nature of data and the cost of carrying out the analysis. Following are more common methods of data analysis:

(a) Benefit-cost analysis

Benefit-cost analysis is often used in public sector projects and programs where information on profitability is generally not available. The cost-benefit analysis applies the following types of tools:

Net present value (NPV): The auditors assume a certain rate of interest (which is mostly the rate that the planning agencies of the country use or which the lenders of public debt apply to the debt given to the country. The estimated cost and benefits streams of projects are discounted at the assumed rate of interest and a net present value (difference of the discounted benefits and discounted costs) is arrived at. The auditors adopt the NPV of the project worked out in the project appraisal at the planning stage as their benchmark. During audit, they work out the NPV on the basis of actual data. Any increase in the actual NPV over the estimated NPV is considered positive and a decrease is considered negative. The auditors then report the results of this analysis in the audit report. Done manually, the analysis is quite laborious. However, it is now possible to carry out this analysis through computer by using Excel software.

Internal rate of return (IRR). It is a rate of discount on which the NPV is zero. The idea is to find out a rate of interest on the capital being invested in a project that would give just enough benefits as its cost. Any rate higher than that would make the NPV negative and not acceptable. If we are able to determine the IRR of a project, we can always decide whether project loan can be accepted at a certain rate of interest. Any loan which is available at a rate higher than IRR would be disadvantageous. IRR could be worked out using only financial data. It is called financial internal rate of return (FIRR). However, most of the public sector projects do not have complete financial data. They are usually justified on the basis of economic benefits. Using economic benefits and costs, it is called economic internal rate of return (EIRR). Whatever the type of IRR, it is generally worked out at the project appraisal stage by using estimated costs and benefits. During performance audit, the auditors work out IRR by using actual data. The IRR worked out on the basis of actual data is compared with the IRR in the project planning document. Any increase in the actual IRR over the estimated IRR is considered as an evidence of the success of the project.

(b) Correlation and regression analysis: Correlation and regression analysis correlates two variables to determine the effect of one on the other. For example, the auditors may like to know the relationship of education level with the earning level of persons. They would collect data on these tow variables and by using statistical analysis determine coefficient of correlation. The analysis involving multiple variables is also possible but becomes complex. The auditors would require help of statisticians.

(c) Ratio analysis: Ratio analysis is the most commonly used methods of data analysis. It involves comparison of data over a base value. The base value can be a time period or any other value. Ratio analysis has its own short-comings. The auditors can make gross errors in using data analysis if they do not see the analysis in its proper context. If, for example, the data being compared is insignificant with the total population, the result may covey a wrong message. For example, if in a total population of 7500 payments, only 10 payments were made by cheque and five of them had errors, we cannot say that 50 percent the cheques were incorrect. Though factually true, it conveys a wrong message. Similarly, ratio analysis where the denominator is zero would convey a wrong message. For example, if error in cheques written in a base period was zero and it rises to 5 next month, the ratio would be infinity, which is wrong.

(d) Simple statistical measures: Simple statistical measures such as mean, mode, median, range and standard deviation are also applied by the auditors according to purpose of analysis. Mean refers to simple arithmetic average. Mode refers to the most common value of the variables. Median refers to the mid-point value. Standard deviation refers to the deviation or spread of the data beyond the normal distribution and can help understand the extent to which the data was away from the normal spread. The auditors can also use range. However, it requires some caution. For example, they may note that the range of absenteeism is between one to 29 days. However, it is possible that only one person stayed away for 29 days. Others remained away between one to five days. Using a simple range in this situation would be misleading. The auditors should isolate the awkward data and smooth out the range for the rest.

(e) Analysis of Cost Over-run: Auditors often come across situations of cost over-run in projects. The usual explanation they get from the management is that the cost over-run was beyond their control because it was caused by such factors as inflation,

foreign exchange rate fluctuations, additional taxes and duties, etc. For validating the assertions made by the management the auditors deepen their analysis. They analyze the cost over-run into two segments: (a) cost over-run due to uncontrollable factors such as inflation, foreign exchange rate variations, and taxes and (b) cost over-run due to controllable factors, such as waste, theft and careless planning in the start of the project, etc. The auditors take the following course of action for this analysis:

i. Uncontrollable cost over-run:

Cost overrun due to inflation: For local cost components, the auditors obtain information on price indices published by government or other reliable organizations and apply the rate of inflation on the planned data and then deduct from it the original planned cost data of cost for each year to obtain the extent of cost due to inflation. For example, suppose the planned cost data pertained to year 2004 and actually execution of the project started in year 2006. The auditors should apply the rate of inflation for years 2005 and 2006 to respective costs incurred in these years and then deduct the original 2004 data in the project plan from the aggregate of 2005 and 2006. The difference would be increase in cost due to inflation. For imported goods and material, the auditors compare the estimated costs in the plan and the invoices of the foreign exporters to compare any increase in the prices of imported goods and materials.

Cost over-run due to foreign exchange rate fluctuation: The auditors take the original cost data for those items having a foreign exchange component, such as import of machinery or raw material. They should then determine any change in the foreign exchange rate of the local currency and the foreign currency and apply the change to the original cost data for those costs which had a foreign currency component. The difference gives the extent of cost over-run due to foreign exchange rate fluctuation. For example: suppose the original cost data had a 40 percent component of imported material. The rate of exchange in Pakistan Rupee, for example, and US dollar was 1 US $ = Rs 60 at the time of project planning. In year 2006, when actual imports took place the rate of exchange was 1 US $= Rs 63. The auditors can multiply the original cost in terms of dollars with Rs 3 and arrive at the extent of cost over-run due to rate of exchange.

Cost over-run due to additional tax burden: The auditors can calculate the additional tax burden since the time of project planning by applying the tax or duty rate where applicable.

Adding the cost over-run worked out above due to (a) inflation (b) changes in foreign exchange rate (c) additional taxes would give the extent of cost over-run due to uncontrollable factors.

ii. Controllable cost over-run: The auditors now deduct the figure of uncontrollable cost over-run worked out above from the total cost over-run. It gives the extent of cost over-run due to controllable factors for which the management should take responsibility.

(f) Analysis of Time Over-run: The auditors come across delays in implementation of projects and often the management has various types of explanations and excuses for the delays. Some of the causes of delay are, genuinely beyond d the control of the management. However, some of the delay can be traced back only to the lack of sense

of urgency, inefficiency and negligence of the management and its employees. The analysis of time over-run is done into (a) uncontrollable time over-run (b) controllable time over-run. The auditors proceed as follows:

i. Uncontrollable time over-run: The auditors get the explanation of the management about the time over-run. Whatever explanation they get, the auditors go deep into the factors enumerated by the management and try to quantify the extra time taken by the project due to these factors. For example, it could be that the execution was delayed as the approval of the government did not arrive on time as planned. The auditors work out the exact time taken by the government in giving approval and then work out the extent of delay due to this factor. However, they also see what other activities that did not require approval of the government but could be undertaken for being in a state of preparedness were not started or were started in a lukewarm manner. The management could, perhaps, save some time by showing a sense of urgency in such cases. Similarly, the auditors probe further deep into each factor and work out the exact delay and add up all these factors to determine the extent of uncontrollable time over-run.

ii. Controllable time over-run: The auditors then work out the extent of controllable time over-run. They do so by deducting the extent of uncontrollable time over-run from the total time over-run. They go a step further by recalculating the decrease in NPV and IRR of the project due to controllable delay, if it runs into years. This can be done by recasting the original cost and benefit estimates in time series by adding uncontrollable time over-run period into the planned time series. For example, if the planned time series of the project started from 2004 onward and the extent of uncontrollable delay was two years, the auditors can recast the planned cost and benefit data by building into it delay of two years. Once this is done, the base data would be with cost and benefits having an uncontrollable delay of two years. They can then work out the NPV and IRR for this data and use it as benchmark for comparison with the actual data and show how much decrease in NPV and IRR took place only due to controllable delay. This would be attributable to the inefficiency of the management.

There are other more sophisticated techniques of analysis such as program logic modeling, simulation, content analysis, work-flow and communication analysis, etc. They require specialized knowledge and are rarely used in practice by the auditors.

8. Developing audit observationsWith the analysis of data the auditors are ready to draw their conclusions. While finalizing their conclusions, they try to seek answers to such questions:

Is the deficiency isolated or systemic?

What is the cause?

Can the organization fix the problem?

What are the impacts?

Answers to such questions determine the audit conclusions. Based on the conclusions the auditors prepare a fact-sheet and discuss it with the auditee management. The discussion takes place both at the operational level with those managers and staff who deal with those

issues as well as with senior level managers (thought not the chief executive at this stage.) In light of these discussions, the auditors adjust their conclusions and then proceed to clear the report first with their own management and then with the chief executive of audit entity. The discussion with the chief executive is also known as ‘exit conference’ and is an indication of the completion of the audit. The auditors discuss at this stage the possible recommendations with the auditee management and try to assess their initial reaction if these recommendations are made. The objective is to frame the recommendations in a manner that would be acceptable to management and they would be willing to implement them. The recommendations which are resisted by the auditee management have a little utility.

9. Reporting results of performance audit(a) Auditors’ challenge

The performance audit report is a vehicle of accountability. In public sector, the ultimate audience of the report are legislators or members of governing boards of public enterprises. They are usually busy people and have limited interest in the audit report until its presentation is eye-catching. They would like to look for some 'horror stories' which they can use against their adversaries. A report which does not carry such material would not be very interesting for the politicians and members of the governing bodies. At the same time the auditors have to do a professional job. They would like to report the achievements and shortcomings of the auditee in a professional manner without any prejudice or bias. The real challenge in report writing starts from this misalignment of expectations and professional standards.

The auditors have a challenge, thus. They have to do a professional job but they have to attract the attention of the legislators or members of governing bodies as well. They need to make the presentation of the report attractive for them without compromising on their professional stance. How can they make the report effective? The following discussion attempts to answer the question.

(b) Conditions for an effective performance audit report

There are five conditions that make performance audit report effective:

Current and topical information

Unbiased and fair reporting carrying the point of view of the management as well

Plain language avoiding technical jargon as far as possible

Graphics, charts, photos, headings, sub-headings with text boxes for significant points

Actionable and practical recommendations

(c) Style of the performance audit report

An effective performance report has the following style.

It states the structure of the report in the beginning.

It begins with achievements of the auditee before it goes on to the shortcomings.

The chapters and paragraphs are logically connected.

Each audit observation begins with a topical sentence.

The language of the report avoids subjective judgments and expressions like ‘frauds, embezzlements, misappropriations, etc’ until there is an irrefutable evidence to that.

The language of the report uses active voice as far as possible.

The report avoids using nouns where verbs can be used. For example, use of “determine” instead of “determination”; “provide” instead of “provision” etc.

The table of contents is elaborate enough to give an outline of the report.

(d) Structure of audit findings

An effective performance report adopts the following generally accepted practice of presenting the audit findings. Each audit observation should have five segments, without using the sub-heading for each segment:

What were the audit criteria?

What were the conditions found?

What were the causes of the conditions?

What were the effects of the conditions?

What should be done now?

(e) Use of graphics in the audit report

There is a growing awareness that the audit report should use graphics and audio-visual aids to make a simple and quick reading. The following table illustrates the use of different types of graphics.

Table 2: Using Graphics in Performance Audit Reports

For Type of Graphic

Numerical information and comparison of figures

Tables

Trends over a period Line graph

Comparison of trends of two or more variables

Multiple-line graphs

Comparison of totals of one variable over the same period

Simple bar chart

Comparison between two or more variables over the same period

Multiple bar chart

Comparison between proportion of the total made up by two or more discrete items over the same period of time

Percentage bar chart

Division of a total between parts Pie chart

Procedure or instructions sequence Flow chart

Achievements/ deficiencies noted during physical verification/visit

Camera photos

Reference to geographical locations Maps

(f) Format of the performance audit report

The performance audit report consists of the following parts:

Executive summary (1-2 pages)

Findings and recommendations (12-20 pages)

Annexes

(g) Issues in performance audit reporting

Audit opinion: Should the performance audit report contain an over-all opinion? In financial auditing, the auditors express an over-all opinion on the truth and fairness of the financial statements. In fact, one of the primary objectives of financial auditing is to formulate such an opinion. But in performance auditing, there are no clear-cut guidelines or standards. The most common practice is that the auditors write a long-form audit report which comments on various aspects of performance of the organization, program or project. It is interesting to note that INTOSAI Auditing Standards (1989) also endorse the position that performance auditors need not express an over-all opinion. In some organizations, however, the performance audit reports contain a section titled “Overall Assessment”. In this section, the auditors try to distil the main message of the report. But, again, the language, size, style and format of this paragraph are not standardized.

Materiality: What guidelines should the auditors follow in performance auditing in deciding whether a matter is material or not? In financial auditing, the guidelines are fairly straightforward: the auditors should keep the materiality level followed by the audit office in view. In performance auditing, almost everyone agrees that the matters to be reported should be significant. But how to decide, whether a matter is significant or not, is only a judgment of the auditor. INTOSAI Auditing Standards (1989) paragraph 163 states:

“In the case of performance audits the judgment will be more subjective as the report does not relate so directly to financial or other statements. Consequently, the auditor may find that materiality by nature or by context are more important considerations than materiality by amount”.

In seems that in case of performance auditing, both quantitative and qualitative aspects will have to be kept in view.

Extent of probe: How far should auditors go in determining the causes of the observed conditions? In one sense the search for locating causes could be endless because everything is caused by something else. This can be an infinite process. The desirable approach is to pursue the matter until auditors can make recommendations that will make up the deficiency in performance or improve the situation for the future. Sometimes an event may be caused by multiple factors and the auditors may have missed some of them. To safeguard against this possibility, the auditors should discuss the matter with the concerned staff of the

auditee or at least at the time of last meeting with the head of the organization. Many hidden facts come to knowledge of the auditors at that time.

Nature of the recommendations: An important question is whether the auditors should make general or specific recommendations? Should they propose detailed steps to be taken by the auditee or indicate a direction for action, leaving the matter of details to the management? The general guideline is that the auditors should restrict themselves to general and broad suggestions and should not get into the specifics because of the following reasons:

o Auditors may not have the necessary expertise.

o It could be time consuming.

o There could be practical constraints.

o It could become controversial.

o It may not lack objectivity.

INTOSAI Auditing Standards (1989) also support the above approach, though with some qualification. Paragraph 159 of the Standards reads as follows:

“Performance audit reports should not concentrate solely on criticism of the past but should be constructive. The auditor’s conclusions and recommendations are an important aspect of the audit and, where appropriate, are written as a guide for action. Generally, these recommendations suggest what improvements are needed rather than how to achieve them, though circumstances sometimes arise which warrant a specific recommendation, for example to correct a defect in the law in order to bring about an administrative improvement”.

Feeding audit recommendations into future planning process : The real benefit of performance audit accrues if the lessons learnt are fed into the planning process of future projects and programs. But there is no consensus on how best to achieve this objective. One possible method is that the auditee organizations should set up small units for codifying the “lessons learnt”. These units should use a database to enter all lessons learnt, including those reported in the audit reports. The database should be accessible to all executive sections and departments.

Audit report and the press: Should the auditors send their reports to the press as a matter of routine? If so, at what stage? These questions nag the auditors without a clear-cut answer. By sending the reports to the press, there is always the risk of exposing the executing agency to public criticism that may not be totally true. The audit report may have misplaced its emphasis on trivial points or the executing agencies may have a valid explanation but the auditors had missed it. Such fears arise from the auditors who are not sure of the quality of their work or who may not be confident of their evidence. But once a report is done with due professional care, there does not seem to be any reason for the above worries. Because of this, some people plead sending audit report to the press at the same time it is laid before the legislature. The idea is that the public criticism would increase the extent of accountability of public managers and may also bring greater improvements in future. Press coverage is also a potential deterrent to fraud. The exposure in the press means that fraud will not be covered up “internally”. Another point is that a copy of the actual report forecloses the door for publishing heresy and un-authentic stories by the newsmen.

10.Issues in performance auditing(a) Effectiveness of performance auditing

Performance audits are an expensive enterprise. They take much longer and require extensive work by teams of professional staff. It is pertinent to ensure that the resources spent on performance audit do not go waste. They would go waste if the recommendations made by the performance auditors are discarded by the auditee to gather dust in the shelves. How can we ensure that the performance audit reports are implemented? Ideally, it would require an institutional mechanism to feed back the audit recommendations in the planning process of future development projects. However, not much thinking has gone into the subject. One suggestion, besides the idea of database discussed above, is that it should be mandatory for the project managers to specifically document in the project completion report the recommendations made by the audit on this project and the extent to which these recommendation have been implemented. Simultaneously, it should be mandatory for project managers of new projects to present before a team of senior management the profile of the new project and specifically mention the recommendations of past audit reports that have been kept in view in planning this project. No project should be approved until this requirement is met. This type of institutional arrangement can provide a mechanism to ensure that audit recommendations pay back in terms of better future project planning.

(b) Enabling environment for performance auditing

Despite all the tall talk in praise of performance auditing, it has not yet become a going concern in most of the Supreme Audit Institutions. Performance audits are still a sort of an ‘add-on’ type of activity. How can performance audit be made a going concern? Some of the prerequisites for effective performance auditing are as follows:

Public demand for an assurance that the best use is being made of their funds

An understanding that performance auditing is a learning process and is focused on improvement and not on finger-pointing

A code of ethics for performance auditors, which gives satisfaction to the auditee about the quality of the auditors’ work and their competence

An audit methodology that leads to an objective assessment

A research and development institution, which promotes excellence in performance auditing

An institutional arrangement for training of auditors, development of training materials and training of trainers

A comprehensive system of reward and incentives for auditors

(c) Institutional impediments to performance auditing

A related question is: what are the impediments in making performance audit a going concern? Following conditions impede the implementation of performance audit:

An exclusive preoccupation with the financial controls on the part of funding or implementing agency

Undue focus on the legal fulfillment of a contract rather than successful outcome of the project

Absence of a clear, objective and agreed audit criteria

Inadequate or non-comprehensive performance indicators

A management preoccupied with the enforcement of regulations rather than optimization of the resource use

Perception that performance auditing is an unnecessary nuisance or obstruction in the day-to-day operations

Inadequate documentation in the auditee office

A lax tone at the tope leading to a low priority to the accountability process

The benefits of performance auditing can be multiplied by removing these impediments. This will require top support, setting up institutions of excellence in performance auditing, promotional work by the auditors to create awareness and insistence of the legislature to get audit reports on performance rather than mere financial audit reports.

References

Australian National Audit Office.1986. Selected Addresses on Public Sector Auditing (Vol. II) , Sydney: ANAO.

Canadian Comprehensive Auditing Foundation. 1991. Comprehensive Auditing: Concepts, Components and Characteristics, Ottawa: CCAF.

Everard, P. & D. Wolter . 1989. Selection of Terms and Expressions Used in the External Audit of the Public Sector, Vienna: INTOSAI Secretariat. (Working Paper distributed during XIIIth INCOSAI at Berlin.)

General Accountability Office, USA. 2007. Government Auditing Standards. Washington, D.C.: GAO.

International Organization of Supreme Audit Institutions. 2004. Implementation Guidelines for Performance Auditing: Vienna: INTOSAI.

International Organization of Supreme Audit Institutions. 1989. INTOSAI Auditing Standards. Vienna: INTOSAI.

Khan, Muhammad Akram. 2005. A Practitioner’s Guide to Performance Auditing. California: Pleier Corporation.

Waring, Colleen G. & Stephen L. Morgan. 2007. “Public Sector Performance Auditing in Developing Countries.” In Performance Accountability and Combating Corruption, ed. Anwar Shah. Washington, D.C.: The World Bank.