nas audit evidence subgroup - wikispacesaudit-network.wikispaces.com/file/view/draft+audit... ·...

Version 1 rev.3 – 25-26/02/2015

REFERENCE DOCUMENT FOR NAS NETWORK

WORKING GROUP - AUDIT EVIDENCE

MEETING 25 AND 26 FEBRUARY 2015

FVO, GRANGE

NAS Audit Evidence subgroup - working document

Version 1 rev.3 – 25-26/02/2015

The National Audit Systems (NAS) Network

The NAS network is a network of officials (auditors) from national competent authorities, responsible for the performance of audits of official control systems as provided for by article 4(6) of Regulation (EC) No 882/20041. The networks meet regularly, under the chairmanship of, and facilitated by, the FVO to exchange experiences in implementing national audit systems on official control activities. During the course of these exchanges; discussions, workshops etc. good principles and practices are identified and agreed by the network.

To enable dissemination of information the network, working in plenary session and through sub-groups, facilitated by the FVO, consolidate agreed principles and good practices on specific topics into reference documents. These reference documents may be used as guidance documents, however, they do not constitute an audit standard and are not legally binding.

Audit Evidence

OBJECTIVES

The objective of this document is to guide and support Competent Authorities (CA) and audit bodies in managing audit evidence.

The aim is:

To provide principles and definitions regarding audit evidence

To identify characteristics, types, and sources of audit evidence

To discuss evidence collection planning

To give principles/discuss verification of audit evidence

This document is intended to assist in the implementation of Section 6 of the Annex to Commission Decision 2006/677/EC.

SCOPE AND INTENDED AUDIENCE

This guidance applies to planning of audits as required by Article 4(6) of Regulation (EC) No 882/2004.

It is intended for use by CAs / audit bodies that carry out audits on official control (systems) according to the requirements of Article 4(6) of Regulation (EC) No 882/2004.

It supports the development of good practice in audit evidence collection and verification in the area of official control activities e.g. feed, food, animal health and welfare and plant health.

1 OJ L 191, 28.5.2004


Version 1 rev.3 – 25-26/02/2015

I. BACKGROUND AND CONTEXT

{where does evidence fit in the audit cycle – evidence and evidence collection plan}

{The objective of collecting evidences : measure, compare, evaluate to meet the objective, in order to support audit conclusions}

{audit criteria and how it links with audit objectives}

The collection of audit evidence is a familiar but important step in the audit process. The quality of the evidence collected has a direct and significant effect on the audit findings and conclusions.

The audit team should, at the audit planning stage of an audit, consider what audit evidence should be required. During the audit process, the audit team should verify the audit evidence collected and ensure it is appropriate and sufficient to achieve the audit objectives. Audit evidence needs to be compared to the audit criteria and the audit objectives to allow the audit team produce audit findings and present persuasive audit conclusions.

Only audit evidence that is appropriate and sufficient will effectively support audit findings and conclusions which are capable of withstanding challenge and satisfy internal and external scrutiny.

{Audit objectives: (IIA) 2210.A1- Internal auditors must conduct a preliminary assessment of the risks relevant to the activity under review. Engagement objectives must reflect the results of this assessment}

{Audit criteria: means the set of policies, procedures or requirements used as a reference against which audit evidence is compared, i.e. the standard against which the auditee’s

activities are assessed.}

. (ISO) – (IIA) 2210.A3- Adequate criteria are needed to evaluate controls. Internal auditors must ascertain the extent to which management has established adequate criteria to

determine whether objectives and goals have been accomplished. If adequate, internal auditors must use such criteria in their evaluation. If inadequate, internal auditors must work

with management to develop appropriate evaluation criteria. For me the approach seems different, the criteria are linked for the IIA to the measure of the organisation’s goals, the

criteria used for the appreciation of the evidences come from the key internal control points of the procedures (in the risk matrix of the audit team)}

{Different terminology may be used by different MS for the same processes}

{reference to ISO to help with language versions}


Nagle, Kevin, 27/02/15,

Other than the map there was little to organise our thoughts in this section of the paper, so I have put in some text describing audit evidence in relation to the audit process map and the context.


Should the definitions for audit objectives and criteria be here or below in the definitions section?

HeusR, 06/03/15,

I do agree, that a preliminary risk assessment could be carried out. But this is not a definition of an “audit objective”.

Version 1 rev.3 – 25-26/02/2015

{Evaluation of Evidence: Is this within the scope of this document or does it belong to the next phase ? Possibly dealt with in a separate reference document on “Recommendations”}

[II.] DEFINITION(S)

This document should be read in conjunction with the definitions contained in Regulation (EC) No 882/2004 and Commission Decision 2006/677/EC bearing in mind that the definitions of those documents apply.

Audit Evidence: records, statements of fact or other information which are relevant to the audit criteria and verifiable. (ISO 19011:2011 from ISO 9000:2005)

Effectiveness: is the extent to which official controls produce an (intended) effect / achieve an objective2. In this particular context the objectives are those of Regulation (EC) 882/2004. Effectiveness is not to be confused with efficiency, which is normally used when we want to refer to input-output ratio i.e. cost and/or resources required to produce an output. (“Auditing effectiveness of official control systems” NAS network document)

Findings: results of the evaluation of the evidence collected during the audit against the applicable standard (e.g. legislation), described in an objective manner. (FVO SOP Audit Performance)

Conclusions: statements made by the audit team concerning the outcome of the audit which are based on and after consideration of all the findings and the audit objectives but which do not propose any course of action. (FVO SOP Audit Performance)

{Note: characteristics not mentioned in this section are better described in the next chapter, less prescriptive and explaining the concept}

II.[III.] AUDIT EVIDENCE

Include statement on usefulness of evidence, i.e. when it helps to reach goals of the audit "An effective audit has persuasive findings and conclusions. The quality of audit findings and conclusions relies on the judgements the auditor makes and these judgements are directly dependent on the quality of the audit evidence collected and the competence of the auditor collecting it”

Audit Evidence: information used by the auditor in arriving at the conclusion on which the auditor's opinion is based…."(international Standard on Auditing (UK and Ireland)

Audit Evidence: Audit evidence is the information internal auditors obtain through observing conditions, interviewing people, and examining records. Audit evidence should provide a factual basis for audit opinions, conclusions, and recommendations. (IIA - SAWYER) / Audit evidence is

the information that supports or refutes an audit objective (IIA – David O’Regan)”.

The nature of audit evidence in systems audits

{particularities of systems audits vs financial or compliance audits}

2 Objectives may be at a strategic or operational level.


TAVARES Sara (SANCO), 25/02/15,

Adapt these definitions to the NAS context.

Version 1 rev.3 – 25-26/02/2015

For financial or compliance audits, evidence only needs to be collected to demonstrate activities are being carried out to planned arrangements. For systems audits, evidence needs to be collected to verify the effective implementation of planned arrangements

Quantitative versus qualitative

A. Characteristics of audit evidence

{Some text?}

Description

PersuasiveThe persuasiveness of evidence is linked to its appropriateness (relevant and reliable) and sufficiency.

(Linked also to target audience and findings)

Appropriateness / Usefulness

The appropriateness of the evidence is the measure of the quality of the evidence determined by its reliability and relevance.

Sufficient

When there is enough evidence to persuade a reasonable person that the audit findings and conclusions are valid, and that the recommendations are appropriate. [IIA]

Amount of evidence considered enough: [Scoping paper]

i) for the auditor to form a reasonable opinion (sample size, representativeness)

ii) to convince interested parties/stakeholders of validity of auditors opinions (persuasive)

Relevant

When the evidence is clearly and logically related to the audit questions, audit criteria and audit findings. [IIA]

Extent to which the information bears a clear and logical relationship to the audit findings (and audit objectives). [Scoping paper]

Reliable

When evidence is obtained through the use of appropriate techniques. When the same findings arise when alternative techniques are used or when information is obtained from different sources.

The best obtainable information through the use of appropriate engagement techniques. [IIA]

The degree to which evidence can be considered trustworthy (accurate and credible), the likelihood of coming up with the same answers if audit test is repeated or information is obtained from a different source or test. [Scoping paper]

Continuity and integrity of evidence. e.g. in a laboratory the reliability of results could be in question if sample identification, documentation and/or security is suspect.

Verifiable MA

Objective MA

Representative Representative of the audit universe and … time… {UK’s example}

Logical/ Rational/

Linked to persuasiveness MA



JMTo clarify: The characteristic “Reliable” should include that evidence has not been, nor is likely to have been, interfered with/altered/amended in an unauthorised or un-endorsed manner. This is particularly important in the laboratory environment as suggested, but in other areas also for example in the context of the traceability of animals and animal products. This is what I meant by the “Continuity and integrity of evidence”.

Version 1 rev.3 – 25-26/02/2015

Description

Reasonable/ Sound

Reference to Annex I – Audit Evidence Mind Map

B.[A.] Types

Type Description Examples of evidence/Techniques Considerations

Observed (or Physical)

Information gathered by the auditor through personal observation of people, events and physical.

Examples:

Visual control

Photo?

Sample

Techniques:

Direct inspection or observation of people, property or events.

Listening, smelling?

On-site verification

Shadow inspection / Witness audit

Review audit3

Whilst usually the most persuasive evidence, the auditor must be aware that a risk exists that his/her presence may distort or prejudice what would normally occur, thus reducing the quality of the evidence.*

Ways to record this type of evidence – photo, notes? Cross reference with Section V. Verification of Evidence.

Documentary

Information prepared by others than the auditor. Documentary information can exist both in paper and electronic form.

Exemples :

Documents containing routines website information, etc.

Photos

Internal/external

Paper/electronic

Legal/work

ISO definition of “document”?

Techniques:

Review of documents, reports, manuals, literature, external and internal websites, postal or web-based surveys.

This evidence may be in electronic or hardcopy format. *

However, useful information may not always be documented, thus necessitating the use of other approaches also.*

Be sure to record the date on which the information was gathered as the information may change later on.

Oral / Inquiry

Information gathered from people through

Examples:

Oral / written

Oral evidence is generally important in performance audits, as information obtained in this manner is up-to-date

3 Audis of the FBO without the presence of the inspector



We have agreed that this is documentary evidence. It was given as an example in an international definition given by Maura.

Version 1 rev.3 – 25-26/02/2015

Type Description Examples of evidence/Techniques Considerations

interviews and focus groups. Such information may take the form of written or oral statements.

interview

Single / group

Techniques:

Interviews

Presentations

Questionnaires?

Knowledge/facts?

and may not be available elsewhere.*

However, information should be corroborated and statements confirmed if they are being used as evidence.*

Analytical

Indirect or derived evidence / information constructed by the auditor combining information from different sources and analysing that information to reach a conclusion.

Examples:

Comparison

Computation

Ratio

Crossing

Techniques

Analysis through reasoning, reclassification, computation and comparison

Such evidence is obtained by using professional judgement to evaluate physical, documentary and oral evidence. *

Be aware of importance of Audit experience and skills

Based on page 96 of “Internal Audit Practice”, chapter on “Gathering and analysing information”* based on page 60 of Court of Auditors’ “Performance Audit Manual”, Chapter 4 Examination Phase

Note: Types are not related to description of ways to record evidence as this aspect may be covered by internal procedures.

C.[B.] Sources

Source Type of Evidence Examples / Techniques Considerations

Obtained directly by the auditors

Observed (and Physical)

Oral / Inquiry

Analytical

Direct inspection,On-site verificationObservation

Interviews,Preparation of questionnaires

Previous audit reports from the audit bodies,Analysis

The auditors can determine the methods that will provide the best quality of evidence for the particular audit. However, their skills in designing and applying the methods will determine the quality of the evidence. *

Provided by the auditee

Documentary Information from databases, documents, activity statements and files (e.g. procedures, instructions, legal acts, inspection reports,

Auditors must determine the reliability of data that is significant to the audit questions by review and corroboration, and by testing the auditee's internal controls over information, including general and application controls over


Version 1 rev.3 – 25-26/02/2015

Source Type of Evidence Examples / Techniques Considerations

Oral / Inquiry

management reviews, organisational and planning documents, certifications).*

Answers to questionnairesOral replies during interviews

computer-processed data. *

Provided by third parties

Documentary

Oral / Inquiry

Information which may have been verified by others or whose quality is well known, e.g. national statistical data.*Information belonging to third parties (Business Operators, Customs, Stakeholder representatives, other CAs, etc.)Third parties audit reportsWebsites

Answers to questionnairesOral replies during interviews

The degree to which such information can be used as audit evidence depends on the extent to which its quality can be established and its significance in relation to the audit findings. *

* based on page 59 of Court of Auditors’ “Performance Audit Manual”, Chapter 4 Examination Phase

{Maybe we could add a point for the common deficiencies: failing to scrutinize important point, failing to maintain auditor independence, failing to supervise work}

III. EVIDENCE COLLECTION PLANNING

Why do we need it?

Main purpose is to allow a targeted evidence gathering to support the audit findings. This should focus on the audit objective and scope.

Reference to Annex II – Diagram of Audit Process

What is the benefit?

To gather enough evidence and not more than needed.

Plan the audit so that enough (sufficient) evidence can be obtained to be able to draw conclusions that have a bearing on the object of the audit.

(RdH - link evidence sufficiency to the audit objectives)

How do we do it?

Which methodologies are used? Is there a “good practice” that we can identify? Iterative approach? Use of external experts (e.g. in data analysis)?


HeusR, 25/02/15,

This is a rather challenging part of the document. I would say; it’s not only a matter of adequate planning. But this is about the process in which you jump to conclusions on the basis of audit findings compared to the audit objective.For example: audit objective is to assess the effectiveness of official controls. You do have some audit findings, some are positive, some are negative. But how and on what basis can you conclude about effectiveness?

[email protected], 22/02/15,

I suggest it would, if possible, be useful to consider some examples of best practice as regards the actual documentation and cross-referencing of audit evidence.

Version 1 rev.3 – 25-26/02/2015

Knowledge of, and information available to, internal auditors vs. external auditors. Bias of auditors?

Importance of on-site

Note: important to link with characteristics of evidence (how to ensure we get useful information)

Factors to consider when judging the quality and quantity of audit evidence:

the purpose for which the evidence will be used

a higher standard is required for evidence supporting audit findings than for background information provided in the audit report

the level of the significance of the audit finding

in general, the higher the level of significance, the higher the standard of evidence that is required

the degree of independence of the source of the evidence

greater reliance can be placed on evidence which emanates from independent sources

the cost (money or time) of obtaining additional evidence relative to likely benefits in terms of supporting findings and conclusions

at some point, the cost of obtaining more evidence will outweigh the improved persuasiveness of the total body of evidence

the risk involved in making incorrect findings or reaching invalid conclusions

the greater the risk of legal action, controversy or surprise from reporting an audit finding, the higher the standard of evidence needed

the care taken in collecting and analysing the data Including the extent of the auditors' skills in these areas

* based on page 58 of Court of Auditors’ “Performance Audit Manual”, Chapter 4 Examination Phase

Reference to Annex III – example of evidence matrix

When does it take place?

Create a timeline with audit steps and where evidence collection planning takes place:

Planning - Audit Objectives + scope - Phase 1 (desk study) – Risk analysis/desk study results – Phase 2 (test – on-site activities) – Audit report? Diagram?

Planning – preparation – execution – reporting? Diagram?

Evidence collection planning may take place at different stages, depending on the audit planning approach (Desk-based / on-site). Refined, adapted and developed along the audit process. On-site evidence collection is particularly important where the audit is being used to confirm/verify the effective implementation of planned arrangements.

Retention of Audit Evidence.

This would have particular significance in respect of independent scrutiny, evaluation of or challenge to an audit system and /or its findings.

Should be kept during a period described by the audit body or national rules.

Link to Section III, table “A. Types”, type “Observed (and Physical)”, ways to record this type of evidence.


Version 1 rev.3 – 25-26/02/2015

IV. VERIFICATION OF AUDIT EVIDENCE

A. Verification: (ISO 9000 definition?)

Is the evidence really “evidence” The information collected is not audit evidence until it has been verified.

(meeting its characteristics – described above in Section III.A)? Root-cause-analysis – link with the evidence, runs along with the collection and verification of evidence.4Reference to Annex IV – to be developed.

Who does it?

Auditors (and their managers?).

When to do it?

Along with evidence gathering. Importance of on-site (do we need to emphasise here ; need additional text, refer to the document on effectiveness)

How to verify?

Cross-checking; / Review of auditor’s work (own review or supervision);) / Quality checks;/ Peer review.

[in BTSF CB-D3-P04]

4 Reference to “Root-cause analysis of non-compliance – outcome of the workshop (MANCP WG-meeting 21-22/11/2012)”


Version 1 rev.3 – 25-26/02/2015

B. Validation: (ISO 9000 definition?)

Who validates?

Auditors (and their managers?).

Importance of competence and roles/responsibilities when validating evidence.

When to validate?

How to validate?

Supervision of auditor’s work; Peer review


Version 1 rev.3 – 25-26/02/2015

Annex I - Example of a mind map on Audit Evidence (to be adapted)


Version 1 rev.3 – 25-26/02/2015

Annex II – Diagram of Audit Process


Annex III - Example of an evidence matrix5:

1 - From the risks cartography of the competent authority showing a high level of criticality on the subject of “species substitution”, the audit team planned a mission

2 – The audit team analyses the process of the CA to deal with that subject. The audit team elaborates a risk matrix of the process to identify the key points and the criteria showing they are under control. The audit team also reduces the scope of the mission to the horse meat, because it appears to be, one of the easiest products to substitute with a cheaper one, hard to detect and enables quick and strong profits.

3 – The process of audit can be summarized with the following matrix

Audit Objective

Steps and criteria Audit evidence

Type of evidence Level of evidencedocument

aryobservat

iontestimonia

lAnalytical enough Too

weakToo

muchIs the CA efficient in horse meat species substitution controls?

Planning

All the country is covered, all year long

The annual control plan

The Y-1 synthesis

Interview of the meat board manager

Data registered in the information system related to the meat control plan

Yes

From the production to the distribution chain

Yes

the orders are efficiently transmitted to the agents

Message and instructions given to the agent through the country

Local interview of meat agents

Yes

Execution

The control plan is respected (quantity, quality, data recorded)

Local planning declining the national instructions

Interviews of management and agents

Local results vs local objectives in the information system

Interview of agents is useless

The agent knows how to make a sampling

Training records of agents on the subject

Agent evaluations

Interviews of agents

Need to add an on-site observation to conclude

5 This matrix is linked to a specific kind of audit and can be adapted to other cases.

NAS Audit Evidence subgroup - working document 14

Audit Objective

Steps and criteria Audit evidence

Type of evidence Level of evidencedocument

aryobservat

iontestimonia

lAnalytical enough Too

weakToo

muchThe agent knows the product, the law, the internal procedures,

Quality of local records

Yes

Analyse The labs used a the right equipment

The analysts have the competencies

The lab is referenced

Prosecution

The level correspond to the level of the fraud

The rate of prosecution is homogeneous on the territory

The rate of validation by the court is high

We can also add a column to the matrix to write the findings and another one for conclusions


Annex IV – Audit map and root-cause analysis

Alternative diagram



HUwe may put into „the planning to conclusion” scheme the cross check box too, because cross check can generate new audit questions, like the root cause analysis.