Statistical sampling or judgemental sampling

Andy Wynne – andywynne@lineone.net

Many audit textbooks and manuals suggest that auditors should adopt a statistical approach to audit sampling. If auditors use statistical sampling, the mathematical theory of statistics provides the link between their sample size and the certainty of the conclusions. Statistical sampling is very complex and mathematical, however, so few auditors are really able to understand it (although the use of computers may overcome such complexity). As a result, many auditors do not adopt statistical sampling in practice. This article suggests a structured approach to audit sampling for the public sector. A similar approach may also be adopted for the private sector.

Some form of judgemental sampling is almost always adopted as this:

can be used even when relatively large error rates are expected

can be used when the total population is not known precisely

is usually more cost effective, as the selection procedures and the evaluation of errors discovered is less time consuming than with a statistical approach.

Many auditors, who decide not to adopt statistical sampling because of its complexity (or for other reasons), adopt a standard size for their samples, for example:

one large audit firm uses sample sizes of 20 when undertaking substantive testing of the fundamental financial systems at public sector organisations

the office of the auditor-general of a large developed country generally uses a sample size of 10 when undertaking substantive testing

another large audit firm uses a sample size of five when auditing public sector organisations (in a developing country).

What public sector external auditors need if they are not using a statistical sampling approach is a practical approach to sampling which will still allow them to justify the particular sample sizes used for their audit tests. The sample sizes an auditor adopts should be justified and adjusted by reference to the materiality of the particular system being reviewed and the actual quality of its internal controls.

Systems-based audit

If the auditor’s initial review suggests that reliance can be placed on the systems of internal control, then a systems-based approach should be adopted. The size of the sample to be used for testing the reliability of the key internal controls identified will usually be varied according to an assessment of the materiality and significance of the system. The following may be used as a guide.

Fundamental systems 25

Non-fundamental 15

Minor systems 10

Substantive testing

Deciding on the size of the sample to be tested is more difficult when undertaking substantive testing. To be able to justify the sample size used, auditors should follow the following steps:

1. Assess the materiality of the specific system being audited by reference to the value of the transactions that are processed and the nature of the transactions.

2. Assess the soundness (or otherwise) in practice of the internal controls of the specific system that is subject to audit.

3. From the above assessments of materiality and the quality of internal control, determine the size of the sample of transactions to be tested.

1. Materiality of the system

Systems that process transactions with a value of more than 20% of the organisation’s annual payments (or receipts) may be considered to have a high materiality by value. Systems that process transactions with a value of less than 5% of the organisation’s total annual payments can be considered to have a low materiality by value.

The materiality of a system will also depend on the nature of its transactions. Thus new high profile systems with a high level of political significance and public interest may be considered to have high materiality even if their actual monetary value is relatively low. Some systems, for example, well established payments systems, may be considered to have a low significance or materiality by nature.

The overall materiality of a system will depend on a combination of the value of the transactions that it processes and the significance of the system. The following table may act as a guide to the resulting overall materiality.

Figure 1: Materiality of the System

Materiality by value

Materiality by nature

High Moderate Low

High – more than 20% of organisation’s annual payments (or receipts)

Fundamental Fundamental Fundamental

Moderate – between 5% and 20% of the organisation’s annual payments (or receipts)

Fundamental Non-fundamental

Non-fundamental

Low – less than 5% of the organisation’s annual payments (or receipts)

Fundamental Non-fundamental

Minor

2. Assess the quality of the particular internal control system

A systems-based approach should allow the auditor to make an assessment of the quality of the internal controls. The assessment should be undertaken of the particular system that is subject to audit before substantive testing is started. The following table may act as a guide to auditors when assessing the quality of internal control.

Figure 2: The quality of internal control

Adequacy of internal controls Internal control assessment

Controls satisfactory and always found to be working effectively. Existing internal controls are adequate and reliable and should provide full assurance that material risks to the achievement of the system’s objectives are being adequately managed.

High

Controls usually satisfactory with only minor control failures. Existing internal controls are mainly adequate and reliable and should provide substantial assurance that material risks to the achievement of the system’s objectives are being adequately managed.

Medium

Controls sometimes satisfactory, but with common control failures. Many existing internal controls are not adequate and reliable and so provide only limited assurance that material risks to the achievement of system’s objectives are being adequately managed.

Low

Controls unsatisfactory with widespread failure to apply required controls. Existing internal controls are not adequate and reliable and so provide little assurance that risks material to the achievement of system objectives are adequately managed.

Poor

3. Assessment of suitable sample size

Auditors should now be able to determine an appropriate sample size for substantive testing from their professional opinion guided by the materiality of the system and the quality of the internal controls. Larger samples should be used for more material systems or where the quality of the actual internal controls is not as good.

Thus the combination of the auditor’s assessment of the materiality of a particular system and his/her assessment of the quality of the actual internal controls in operation will guide the size of the sample which is to be tested during substantive testing. For example, if the auditor’s assessment of the materiality of the particular system is that it is high (see section 1. above) and the actual level of internal control is assessed as being low (see section 2. above) then a sample of 10% of the annual number of transactions may be tested.

The figures in the table are indicative and should be adjusted by each auditor-general. Two approaches are adopted. The absolute approach, for example, sample 100 transactions; or the percentage approach, for example, sample 5% of transactions.

Figure 3: Assessing suitable sample sizes

Auditor’s assessment of actual quality of internal controls:

High Medium Low Poor

Auditor’s assessment of materiality of system

Fundamental 5% or 100 10% 25% 50% of transactions

High 50 transactions 5% or 100 10% 25%

Medium 25 transactions 50 transactions 5% or 100 10%

Low 15 transactions 25 transactions 50 transactions 5% or 100

What do you think? Should public sector auditors adopt statistical sampling? Can judgemental sampling, guided by the above considerations, be considered acceptable? What is the effect of computers and especially computer-assisted audit techniques (CAATS)? How do you decide on the size of sample you will test?