do procedures matter when communicating assurance… · do procedures matter when communicating...
TRANSCRIPT
Do Procedures Matter When Communicating Assurance?
An Experiment Applying New IAASB Standards
Sandra Vera-Muñoz* University of Notre Dame
Notre Dame, Indiana [email protected]
Lisa Gaynor University of South Florida
Tampa, Florida [email protected]
Linda McDaniel University of Kentucky (emeritus)
Lexington, KY [email protected]
William Kinney University of Texas at Austin
Austin, Texas [email protected]
ISAR version - June 6, 2014 - Preliminary
*Corresponding author
The authors gratefully acknowledge the helpful comments of ISAE 3000, ISAE 3410, and ISRE 2400 Task Force members, as well as Roger Martin, Marcy Shepardson and her Indiana University PhD students, Roger Simnett, Elizabeth Carson, and an anonymous reviewer for the AAA Annual Meeting. We also thank Rachel Webster for excellent research assistance.
Do Procedures Matter When Communicating Assurance?
An Experiment Applying New IAASB Standards
ABSTRACT
Can assurance report users reliably assess incremental assurance achieved via an audit firm’s customized description of its work effort? The answer is important because international assurance standards now mandate such procedure descriptions to communicate achieved assurance for most assurance engagements. Also, auditing standards and U.S. assurance standards remain risk-focused and the change may impair comparative credibility assessment by intended users of assurance reports. We conduct a between subjects experiment using ordered alternatives to parse participants’ confidence judgments after they read actual procedure descriptions for greenhouse gas emissions assurance reports. We find that, given a report format, including or explicitly excluding a critical procedure necessary for “reasonable” assurance did not affect user perceptions. Further, varying report formats while holding constant description of “limited” assurance procedures produced median confidence judgments ranging from 60% to 81%. These results suggest that, other things equal, procedures descriptions do not allow users to assess relative assurance risk.
Key Words: Limited Assurance, Perceived Confidence, IAASB Assurance Standards
Do Procedures Matter When Communicating Assurance?
An Experiment Applying New IAASB Standards
1. Introduction
Traditionally, U.S. and international standards for audit firms’ independent assurance
reports have stated the risk of material misstatement remaining after applying commonly
understood verification procedures. 1 For example, higher credibility financial statement audits
apply typical audit procedures to achieve “low risk” of material financial misstatement (or
“reasonable assurance”), while lower credibility financial statement reviews apply analytical
procedures and inquiries to achieve “moderate risk” (or “moderate assurance”).2 The stated risk
approach has also been applied to other subject matters such as internal control quality,
greenhouse gas emissions, labor practices, and sustainability assertions.
The IAASB recently eliminated the stated risk approach for assurance engagements that
provide “limited assurance.” That is, “moderate assurance,” such as “about as likely as not,” is
replaced by variable assurance that could range from low to fairly high, depending on the
assurance procedures applied. The new standards allow an assurance practitioner to choose (and
even negotiate) verification procedures to be applied for a particular limited assurance
engagement and then to customize description of the procedures in the assurance report (ISAE
3410 [2011], ISRE 2400 [2012], and ISAE 3000 [2013]).
The IAASB assumes implicitly that limited assurance report users can determine the level
of risk of material misstatement from these customized procedures descriptions. This assumption
raises the fundamental question: Can typical assurance report readers reliably determine the
1 According to the IAASB’s Assurance Framework, an assurance engagement is “an engagement in which a
practitioner aims to obtain sufficient appropriate evidence in order to express a conclusion designed to enhance the degree of confidence of the intended users other than the responsible party about the outcome of the measurement or evaluation of an underlying subject matter against criteria.” (IAASB 2013.10).
2 For example, see ISRE 2400.09 (2006 version).
2
assurance achieved through the practitioner’s particularized descriptions of verification
procedures applied?
Answers are potentially important because if intended users cannot reliably differentiate
the credibility added by a particular assurance report, then lower quality/lower cost limited
assurance engagements will drive out audit firms’ higher quality but higher cost assurance
services. Further, differences between IAASB and U.S. standards would impair credibility
comparisons across jurisdictions as well as focus assurance practitioner attention on describing
procedures rather than achieving a stated assurance.
Historically, assurance standards setters around the world have adopted essentially
equivalent standards for audit firms’ attestations to information quality across potential subject
matters. The AICPA developed such assurance standards in the 1980s that were adopted
verbatim by the PCAOB in 2003 and were a template for the IAASB’s initial standards. U.S.
standards provide for examinations (e.g., financial statement audits) that achieve “high
assurance” (denoted “reasonable assurance”), and reviews that achieve “moderate” assurance.
Prescribed reasonable assurance examination reports for U.S. and international standards
are all based on “examining evidence and such procedures as we considered necessary in the
circumstances” to achieve reasonable assurance (e.g., AT101.66), and moderate assurance
reviews are based primarily on “applying analytical procedures to management’s financial data
and making inquiries of company management” that typically achieve moderate assurance (e.g.,
AR 90.73). Thus, by intent, procedures are limited relative to those needed for reasonable
assurance. As to expressing a conclusion, both U.S. and international standards require a
positive form for examinations -- “In our opinion, the [information] is properly prepared, in all
material respects, based on XYZ criteria” (e.g., ISAE 3000 A.178) and a negative form for a
3
review -- “. . . nothing has come to our attention that causes us to believe that the information is
not properly prepared, in material respects, based on XYZ criteria” (e.g., ISAE 3000 A.180).
With the IAASB’s 2011-13 changes, international and U.S. standards for limited
assurance now differ on the level(s) of assurance achieved and the evidential basis needed.
Rather than prescribing that the procedures are to produce “moderate” assurance, “limited
assurance” can now be any level over a wide range. Specifically, under new IAASB standards,
limited assurance can vary from
“. . . just above assurance that is likely to enhance the intended users’ confidence about the subject matter information to a degree that is clearly more than inconsequential to just below reasonable assurance” (ISAE 3000.A6, emphasis added).
So if reasonable assurance is 95%, then limited assurance might provide 10% to 90% assurance
depending on chosen procedure descriptions, rather than “moderate assurance” of, say 60%.
In developing its new guidance, the IAASB concluded that assurance professionals (i.e.,
audit firm personnel) are unable to make such quantitative judgments and that, at least in theory,
infinite variation in assurance is possible (ISAE 3000. 69(k), .A6, .A173-75). Thus, the IAASB
rejected any quantified characterizations of limited assurance and instead maintained that a
description of the chosen procedures allows each report user to individually assess the
confidence level achieved. As an alternative, the Board decided to require engagement-specific
assurance report descriptions of the procedures chosen for both reasonable and limited assurance
engagements with elaboration as necessary for limited assurance engagements. The IAASB did
not specify how intended users can assess incremental assurance from reading the audit firm’s
procedures description.
To address whether users can reliably assess assurance based on procedure descriptions,
we conduct an experiment that uses 210 undergraduate accounting majors as surrogates for
4
typical non-expert GHG emissions report users. We elicit their confidence judgments about the
credibility of an assured GHG emissions report applying new IAASB standards. We vary
engagement description (limited vs. reasonable assurance) and conclusion form (negative vs.
positive) as well as varying the procedures described. To enhance external validity, our
procedure descriptions use actual GHG emissions assurance reports for a global group that
include five assurance procedures. Some segments received limited assurance reports applying
four stated procedures. Other segments received reasonable assurance reports applying the same
four procedure descriptions, plus an especially important fifth procedure deemed necessary by
the audit firm for reasonable assurance.3
To isolate the effect of assurance procedures per our research question, we vary the
procedures (four vs. five) across three combinations of conditions of report wording (limited vs.
reasonable and negative vs. positive). The five resulting cells are ordered according to our
expectations of report users’ confidence assuming that the IAASB’s presumptions in writing the
new standards are descriptive.
Four cells list four procedures for limited assurance, but vary reference to the omitted
procedure (explicit vs. no mention), stated assurance (limited vs. reasonable), and conclusion
framing (positive vs. negative). A fifth cell includes the four procedures and adds the critical
fifth procedure judged necessary for reasonable assurance. The stated assurance and conclusion
framing manipulations allow us to establish “other things equal” conditions to (a) address our
primary research question about procedure description, and (b) to test the continued relevance of
traditional wording of prescribed assurance reports (Hasan et al. [2003]).
3 The assurance report is based on E&Y’s 2012 Independent Reasonable and Limited Assurance Reports on
Toronto-Dominion Bank’s GHG emissions, available at: http://www.tdcanadatrust.com/easyweb5/crr-2011/pdfs/E&Y%20Assurance.pdf.
5
Regarding our tests of procedure description effects, we find that participants are not able
to reliably assess confidence using customized procedure descriptions, other things equal. In
particular, participants did not differentiate between assurance via a positive form reasonable
assurance opinion with all five essential procedures applied (reasonable assurance in our real
world engagement) from an identical report with only four described procedures (which did not
achieve reasonable assurance). Also, participants receiving a limited assurance negative frame
conclusion report explicitly noting omission of the essential procedure did not assess
significantly lower confidence compared with the same report without noting the omission.
As to report format, participants did distinguish assurance from a limited assurance
engagement from a reasonable assurance engagement and a negative frame from a positive
frame. Thus, participants seemed to comprehend the warnings inherent in the IAASB’s
engagement descriptions as well as recognize the inherent difference between negative and
positive assurance. Further, our manipulations of engagement type and conclusion frame had
significant effects in our effort to achieve “other things equal” in order to isolate the role of
procedure descriptions.
Overall, these results suggest that customized procedure descriptions alone do not
reliably convey to users the level of assurance attained and that engagement characterizations
and conclusion format convey at least the direction of difference in assurance achieved.
In section 2, we review the assurance standards and the conceptual background for our
predictions of the effect of stated assurance, conclusion form, and procedures, while Section 3
describes our research design. Section 4 provides statistical analyses and robustness tests, while
Section 5 briefly concludes.
6
2. Background and Hypothesis Development
2.1 THE NEW IAASB ASSURANCE STANDARDS
Globally, there is growing demand for a broad range of independent assurance beyond
traditional audits of financial statements that stems from market factors affecting entities, such as
demands of customers, suppliers and workers, investors, as well as regulators and the general
public. 4 Subject matters include greenhouse gas emissions and climate-change risk (Matsumura,
Prakash, and Vera-Muñoz [2014]; Eccles et al. [2011]), sustainability, and integrated reporting,
plus companies’ corporate social responsibility (CSR) reports on their triple bottom lines of
environmental, social, and economic performance (Mock, Rao, and Srivastava [2013]); Brockett
and Rezaee [2012]; Simnett, Nugent, and Huggins [2009]; Simnett, Vanstraelen, and Chua
[2009]).
Most of these reports are widely available to the general public as well as stockholders,
with more than 3,000 business organizations worldwide voluntarily issuing stand-alone reports
(Brockett and Rezaee [2012], 27-35).5 Firms use CSR reports for college student recruiting
(Auger et al. [2013]), as marketing tools (Simnett, Nugent, and Huggins [2009, p. 351]), or for
reputation risk management (Borkowski, Welsh, and Wentzel [2010]).6 To the extent that
entities make public disclosures intended for an external non-expert audience, independent
assurance on these statements helps the entity mitigate stakeholders’ perception that such
4 See, e.g.,
http://www.aicpa.org/InterestAreas/BusinessIndustryAndGovernment/Resources/Sustainability/Pages/SustainabilityFAQs.aspx (accessed 12/9/13).
5 In addition to stand-alone sustainability reports, entities’ voluntary disclosures of GHG emissions information can take many other forms, including press releases, single GHG statements, the Carbon Disclosure Project, or MD&As in the10-K reports filed with the SEC.
6 For instance, firms use CSR reports to claim that their operations are, or are moving toward “carbon neutrality” or reducing their “carbon footprint” or to associate “green” attributes to their products in order to capitalize on the popular trend towards eco-conscience (Brockett and Rezaee [2012]).
7
statements or disclosures are merely “green-wash,” that is, an attempt to persuade stakeholders
of an entity’s sustainability credentials (Simnett, Nugent, and Huggins [2009, p. 351]).
Collectively, these recent developments provide incentives to publish non-financial
information and to consider whether an independent assurer can also add value by enhancing
user confidence about the information. If assurance is to be provided, then management,
contracting parties, and regulators must decide whether reasonable assurance or limited
assurance adds more value net of cost. The value of audit firms’ assurance report depends in part
on the assurance standards that are intended to promote reliable credibility enhancement
services.
In addition to these new demands for non-financial assurance, many countries outside the
U.S. have recently exempted some smaller entities from statutory financial audit mandates and
many of these entities have decided that audits are not cost-effective and thus reduced demand
from audit services. However, relatively little is known about the cost and value added by
assurance other than financial statement audits.
The IAASB responded to new demands for non-traditional assurance by developing
simultaneously three related and synthesized assurance standards. The results are (a) a revised
ISAE 3000 (issued 2013) to provide independent assurers updated and integrated guidance to
evaluate or measure a wide range of potential subject matter other than financial statements
against suitable criteria, (b) a revised ISRE 2400 (issued 2012) to improve financial statement
reviews (see Figure 1), and (c) a new standard, ISAE 3410 (issued 2011), for assurance on an
important and highly specialized subject matter—GHG emissions.
| Insert Figure 1 about here |
8
Each of the new IAASB standards defines limited assurance as less than reasonable
assurance by design, but without quantifying either in probability terms. 7 Regarding possible
quantification, the IAASB takes the position “The level of assurance the practitioner plans to
obtain is not ordinarily susceptible to quantification . . ..” (ISAE 3000.A4). In the 2013 version
of ISAE 3000, no reference is made to “moderate” risk or “less likely than not.” In fact, when
debating the 2013 revision to ISAE 3000, the IAASB explicitly rejected as impracticable, use of
other standard’s quantification of limited assurance as “moderate” and defining “likely” as “more
likely than not” (per IAASB Board meeting transcript, December 2012).
This presumed inability of audit firm assurance professionals to quantify risk even in
general terms is at least arguable, given that conventional definitions of audit “quality” are
expressed in terms of the probability that financial statements contain no material omissions or
misstatements (e.g., Palmrose [1988]). Also, U.S. and IAASB auditing standards require that
achieved audit risk that the auditor fails to detect financial statements that are materially
misstated be appropriately or acceptably “low” (AU 312.13, ISA 200.17]).
2.2 DETERMINANTS OF PERCEIVED ASSURANCE ATTAINED
While revising its assurance standards, the IAASB considered changes to several factors
suggested as value relevant in communicating to intended users the level of assurance achieved
by applying its new assurance standards, but without using quantitative terms.8 We consider
these factors to quantify their effects in order to assess the effect of procedure choices by limited
7 The original ISAE 3000, issued by the IAASB in 2003, replaced and extended ISAE 100 to provide guidance on
both reasonable (formerly high) and limited (formerly moderate) assurance. Similarly, the original ISRE 2400, issued by the IAASB in 2005, states “A review engagement provides a moderate level of assurance that the information subject to review is free of material misstatement . . ..” (para. 9) (emphasis added).
8 As examples, the IAASB considered the positive opinion form to express review conclusions, whether to require warnings about the higher risk of reviews, and whether noting omission of a procedure commonly applied in an examination was needed. The Board’s discussion was motivated by the need to balance proper warnings about review limitations while not denigrating reviews through cautions or negative framing.
9
assurance practitioners. Specifically, we consider the conceptual basis supporting relevance of
(a) practitioner description of assurance procedures, (b) explicit mention of omitted procedures,
(c) description of the engagement, and (d) the conclusion frame.
2.2.1 Assurer’s Description of Assurance Procedures
As discussed above, ISAE 3000, ISAE 3410, and ISRE 2400 allow the assurer to choose
the verification procedures to be applied to form the basis for assurance and to describe the
chosen procedures in the assurance report. The guidance for describing procedures in ISAE 3000
states that the assurance report must include:
. . . an informative summary of the work performed as the basis for the practitioner’s conclusion. In the case of a limited assurance engagement, an appreciation of the nature, timing, and extent of procedures performed is essential to understanding the practitioner’s conclusion (ISAE 3000.69(k)) (emphasis added).
Such a description is needed, in part, “[b]ecause the level of assurance obtained by the
practitioner in limited assurance engagements varies (ISAE 3000.A6).
In addition, for limited assurance the guidance suggests that the report “is ordinarily more
detailed” regarding procedures than a reasonable assurance report (ISAE 3000.A175). In
practice, this additional detail option could include, for example, dividing an integrated
procedure into a list of several component procedures. Also, as discussed in 2.2.4 below, ISAE
3000 offers optional guidance on reporting procedures “not” performed in a limited assurance
engagement.
Collectively, the guidance for extensive and detailed description of limited assurance
procedures may have the unintended effect on users because an elaborate work effort description
in a limited assurance report may appear to warrant more confidence than does a reasonable
assurance report that describes only a work effort overview.
10
2.2.2 Reporting “Typically Applied” Procedures Omitted
For a limited assurance engagement report, ISAE 3000 guidance for procedure or work
effort descriptions states:
It also may be appropriate to indicate in the summary of the work performed certain procedures that were not performed that would ordinarily be expected to be performed in a reasonable assurance engagement (ISAE 3000.A175).
This guidance could assist the user in seeing what “might not have come to the attention
of the assurer” in a limited assurance engagement and thus lower perceived confidence in a
negative assurance conclusion. In practical application, however, the guidance is problematic for
non-financial information because there is no available description of the procedures “ordinarily
expected to be performed in a reasonable assurance engagement” for, say, GHG emissions or
compliance with some non-financial statement measurement criteria.9 Thus, audit firms typically
rely on teams of GHG subject matter experts and assurance professionals to determine particular
procedures for GHG engagements, resulting in such procedures being unknown and unknowable.
There is, then, a conceptual and practical question of how assurers can determine “typical”
procedures for a reasonable assurance examination and how users can ascertain the relevance of
any procedures that may be explicitly stated as omitted.
Nonetheless, the IAASB decided instead to require the practitioner to list his or her
procedures and when unusual or exceptional verification procedures must be used, the
practitioner lists the alternative procedure. Thus, determining the confidence justified by the
listed procedures depends on the capabilities of the assurance report user.
Psychology research on salience effects shows that known to be salient stimuli (e.g.,
explicit mention of an omitted procedure) that triggers the perceiver’s attention have a
9 There is guidance on expected procedures for integrated audits of internal control effectiveness via the PCAOB’s
AS No. 5.
11
disproportionately large impact on judgment processes (Taylor, Crocker, Fiske, Sprinzen, and
Winkler [1979]; Taylor and Fiske [1978]). Similarly, according to rules-plus-exception theory
(Palmeri and Nosofsky [1995]; Nosofsky and Palmeri [1998]), people assign relatively more
weight to exceptions when evaluating similarities. These theories suggest that report users may
attend more to an explicit statement that singles out a procedure that has been omitted than to
procedures that are listed as performed.
Another body of psychology research provides an explanation for why explicit mention
of an omitted procedure may not lower report users’ confidence judgments (Richey, Koenigs,
Richey, and Fortin [1975]; Pratto and John [1991]; Mummendey et al. [2000]; see Mummendey
and Otten [1998] for a review). In a series of experiments, Richey et al. (1975) provide evidence
that predominantly negative information is weighted heavier when mixed with opposing positive
information, but predominantly positive information is not weighted any lower when combined
with varying amounts of negative information. If these results extrapolate to report users’
confidence assessments, then a list describing applied procedures followed by an explicit
statement of an omitted procedure will not lower report users’ confidence assessments.
2.2.3 Stated Level of Assurance
Cognitive psychology research demonstrates that information presentation and labels
influence how individuals classify information (Sherman, Mackie, and Driscoll [1990],
Kozminsky [1977]). Consistent with psychology research, accounting research documents that
“labeling” matters – examples include “comprehensive income items” vs. “balance sheet items”
(Maines and McDaniel [2000]), interpretation of fair value gains and losses (Gaynor, McDaniel,
and Yohn [2011]), resource allocation based on data labeled as historical accounting earnings-
based vs. future cash flows-based (Vera-Muñoz, Kinney, and Bonner [2001]).
12
The IAASB uses labels to describe the relative assurance provided by reasonable and
limited assurance engagements and offers the following explanation:
“Because the level of assurance obtained in a limited assurance engagement is lower than in a reasonable assurance engagement, the procedures the practitioner performs in a limited assurance engagement vary in nature and timing from, and are less in extent than for, a reasonable assurance engagement” (ISAE 3000.A3).
In practice, this suggests that the purposely less rigorous verification procedures for limited
assurance may not make the practitioner aware of all significant matters that might be identified
in a reasonable assurance engagement, thus reducing achieved assurance.
Cognitive psychology and accounting research discussed above (see also Libby,
Bloomfield, and Nelson [2002]) suggests that defining one service relative to another implies
that the referent is better known or understood. Therefore, communicating assurance depends in
part on user familiarity with the procedures that would typically be applied for a reasonable
assurance examination. This is, of course, problematic for new services such as providing
assurance on GHG emissions or compliance with a new law or process.
2.2.4 Assurer’s Conclusion Frame
Psychology research documents that decision makers respond differently to different, but
objectively equivalent, descriptions of the same problem (Levin et al. [1998]) and that how
information is framed can evoke different emotions or different mental accounts, and thus affect
decisions (Levin et al. [2002]; Levin and Gaeth [1988]). Accounting research documents that
framing affects professional skepticism (Nelson [2009]), audit risk assessments and audit effort
allocations (Fukukawa and Mock [2011]), and budget hours for auditing fair value estimates
(Maksymov, Nelson, and Kinney [2014]).
As to assurance conclusion frame, many commentators on the IAASB’s non-audit
assurance standards projects (particularly practicing auditors of small to medium-sized entities),
13
suggested that the negative form be eliminated for limited assurance to avoid the use of the
double negative (“nothing” and “not”) and the negative connotations of the negative form. These
commentators recommended that the review report be adapted to say “In our opinion, based on
the limited evidence collected, the [information] is properly prepared, in all material respects,
based on XYZ criteria.”
In this study, we vary conclusion frame and stated assurance to assess their incremental
effects on user perceptions of assurance and to isolate the effect of procedures descriptions, other
things equal. To our knowledge, with one notable exception (Hasan et al. [2003]), research has
not explored users’ ability to reliably assess assurance conveyed by assurance reports other than
financial statement audits and reviews. Using early guidance from the IFAC report [2002] and
the ISAE 100 [2001] assurance framework, Hasan et al. develop an experiment on alternative
report formats to communicate “moderate” assurance. Four report formats are labeled as
“moderate” assurance (consistent with ISAE 100 [2001]), and a fifth format, labeled “high
assurance,” is used as the control condition and procedures performed are held constant across
the five conditions. 10 Hasan et al. find significantly higher perceived assurance from the high
assurance report vs. three of the four “moderate” assurance reports and no difference between the
high assurance and the moderate assurance “opinion on procedures paragraph.”
2.3 EXPECTATIONS AND RESEARCH DESIGN
We use the IAASB’s assumptions in adopting ISAE 3000, ISAE 3410, and ISRE 2400 and
the differences between U.S. and international assurance standards discussed as well as
10 The key wordings in the four moderate assurance reports are, respectively: (1) “The procedures for gathering and
processing environmental information have, in our opinion, been planned and implemented in an appropriate way;” (2) “As a result of the procedures performed, nothing has come to our attention that causes us to believe…”; (3) “As a result of the procedures performed, it is our opinion that the Report fairly presents, in all material respects…”; (4) Same as (3), with the caveat “Given the …extent of the work performed on the engagement, we are not in a position to express a high level of assurance.”
14
psychology and accounting research above to assess how new IAASB standards will affect
users’ perceptions of achieved assurance. To be specific, the research and practical arguments
suggest that users’ confidence will be increased by (a) adding an important procedure to the
work effort description, (b) deleting explicit mention of an important procedure omitted, (c)
describing the engagement objective as providing “reasonable assurance,” and (d) presenting the
assurance conclusion in a positive frame (see Figure 2).
| Insert Figure 2 about here |
We test these predictions across five cells with cell 1 (lowest expected assurance) noting
explicitly that an important procedure has been omitted, expresses limited assurance, uses a
negative frame conclusion, and applies only four procedures. We end with cell 5 (highest
expected assurance) that expresses reasonable assurance, notes no omitted procedures, uses a
positive frame conclusion, and applies five procedures. We change one of the four
characteristics at a time across adjacent cells.
Comparing cell 1 with cell 2 and comparing cell 4 with cell 5 allows assessment of
whether procedures descriptions matter while holding stated assurance and conclusion frame
constant. Specifically, does deletion of explicit notice of omission of a procedure essential for
reasonable assurance and/or including the essential procedure increase users’ confidence
assessments, other things equal? The cell 4 to cell 5 comparison is particularly important
because addition of this procedure allowed valid expression of reasonable assurance and a
positive form opinion. Comparisons within cells 2 through 4 serve to isolate procedures effects
by holding constant wording of the assurance report.
15
3. Research Method
3.1 PARTICIPANTS AND ADMINISTRATION
Two hundred ten junior and senior students enrolled in an accountancy program at a
private university participated in our study and are representing the general third party assurance
report users who are appropriate for our research goals (Libby et al. [2002]). The participants
completed the study outside of class using their own computers to access an internet link that
was emailed by their instructors. We used the Qualtrics software platform to administer the
experiment. To encourage participants to attend to the study, their instructors announced that
they were eligible to receive extra credit points, with the credit magnitude contingent upon their
performance in post-experimental questions.
3.2 RESEARCH DESIGN AND EXPERIMENTAL INSTRUMENT
Participants were provided with a hypothetical global audit firm’s assurance report, based
on guidance provided by ISAE 3000, for a hypothetical company’s Schedule of GHG emissions.
As a measure of the confidence (denoted CONF) associated with the assurance report, we elicit
each participant’s assessment of the assurance (or confidence level) they perceived for the GHG
information based on the independent assurer’s investigation and report.
We vary combinations of four conditions (procedure exclusion noted, conclusion frame,
stated assurance, procedures completeness) across five cells ordered in expected CONF values
based on our discussion in section 2. For all five cells, we hold constant the following bullet-
point list of four assurance procedures descriptions based on E&Y’s reasonable and limited
assurance reports for segments of Toronto-Dominion Bank’s 2012 GHG emissions reports:
“To obtain reasonable (limited) assurance, our procedures included:
• Interviewing select personnel to understand the key corporate responsibility issues related to the data and processes for the reporting of the emissions
16
• Inquiring of management regarding key assumptions and the evidence to support the assumptions
• Validating, on a sample basis, the accuracy of calculations performed primarily through inquiry and analytical procedures
• Performing, where relevant, walkthroughs of systems and processes for data aggregation and reporting.”
Participants in cell 5 (the cell expected to yield the highest CONF) also saw a fifth
procedure description:
• Examining evidence supporting the information in the Schedule of Greenhouse Gas Emissions.
and those in cell 1 (the cell expected to yield the lowest CONF), instead, saw the notice:
“Our procedures did not include examining evidence supporting the information in the Schedule of Greenhouse Gas Emissions.”
Note that the nature of the four procedures common to both limited and reasonable
assurance (denoted LA and RA) engagements is directed to whether the information should be
correct by asking questions about processes and assumptions, and to see if “calculations” seem
reasonable, but there is no indication of verifying whether the particular information produced is
correct. That is, there is no testing of the details supporting the information produced.
Determining that processes could or should produce accurate output is not as persuasive as
evidence that the information produced is, in fact, accurate. Thus, the fifth procedure that E&Y
used for a RA engagement should be important in assessing CONF.
In contrast to those in cells 1 and 5, participants in cells 2, 3, and 4 saw no
mention of the fifth procedure. Thus, cell 1 participants would be explicitly aware of the
omission, while those in cells 2 through 4 could be aware implicitly—if they expected it
to be applied and also noted its omission from the work effort description. Therefore, the
procedure omission would be considered only if the participant expected application of
the procedure and noticed its omission.
17
Finally, following the IAASB’s discussion of alternatives leading to ISAE 3000, we
manipulate between-participants the conclusion paragraph frame as either negative or positive,
and the engagement description as LA or RA.
Dependent Variable
We construct the dependent variable, CONF, based on the participants’ judgments of
their level of assurance provided by an auditor’s report on GHG schedule. We elicited the
judgments as follows: “Given the report you read, how assured (or confident) do you feel that
DuLac Corporation’s Schedule of Greenhouse Gas Emissions is fairly presented in all material
respects?” Participants provided their responses on a 0−100 scale, where 0 = “Not at all assured
(No confidence),” and 100 = “Completely assured (Complete confidence).”
As noted in Figure 2, we predict/assume that CONF will be higher if (a) an important
omitted procedure is not explicitly noted, (b) RA is stated, (c) the conclusion frame is positive,
and (d) an important procedure is included and reported. These predictions are tested by
comparing adjacent cell medians. Factors (a) and (d) relate to procedures and factors (b) and (c)
relate to wording. Finally, if all predictions of the prior research and the IAASB’s presumption
underlying ISAE 3000, ISAE 3410, and ISRE 2400 obtain, then the means of our five cells will
increase with each cell as the cell number increases. Also, as shown in Figure 2, it should be
noted that cells 1 and 2 each reflect an externally valid combination of factors (a) – (d) for a LA
engagement under IAASB standards, and cell (5) is an externally valid combination for RA.
Cells 3 and 4 vary assurance description and frame for experimental purposes only to separate
the effects of factors (b) and (c) on CONF. In contrast, cells 3 and 4 are comprised of
18
hypothetical combinations of procedures, stated assurance, and conclusion format that we
use to assess the effect of alternative wording.11
3.3 MATERIALS
The instrument first introduces the company (DuLac Corporation), whose management
has prepared a Schedule of GHG Emissions and hired an independent assurer (Global Audit &
Assurance, LLP) to report on the Schedule’s compliance with stated GHG emissions criteria.
Next, participants are asked to assume the role of an interested outside reader of the assurance
report along with a brief assurance analogy based on a used automobile. To give participants a
better idea of their role, the case provides an analogy using the role of a prospective buyer for a
particular used car whose seller states is in good condition.12 The participants are then asked to
read Global Audit’s assurance report and asked for their evaluation (CONF) and personal
impressions. Post experimental questions conclude the exercise.
We refined our materials in two sequential steps. First, we pilot-tested our instruments
with a panel of seventeen experts, all of whom had experience with development of ISAE 3000
as a member of the IAASB, or as a Technical Advisor to a Board member, or as a member of the
ISAE 3000 Task Force. We also asked panel members for written comments and suggestions
(e.g., regarding external validity, clarity of exposition, format and content of the auditor’s
report). We revised our instruments as appropriate to incorporate their comments.
3.4 EXPERIMENTAL PROCEDURES
Because of the large number of participants required by our research design, we
staggered administration of our experiment with an administration near the beginning of a 4 ½ 11 Hasan et al. [2003] use the same approach to assessing the effects of alternative language, but without the
procedure description considerations. 12 In the analogy, the prospective buyer feels unsure about potential hidden defects in the car. To reduce the
buyer’s uncertainty, the seller hired MIDAS Auto Service™, an independent and reputable automotive technicians company, to conduct a set of tests. In a signed report, MIDAS Auto Service™ reports that the car passed all tests and appears to be in good condition.
19
month semester and the second near the end of the semester. As protection against possible
information leakage and hypothesis guessing with respect to procedures applied, students for the
earlier administration were randomly assigned using Qualtrics to only cells 1, 2, 3, and 4—all of
which received four procedure descriptions. All students in the later administration were
assigned to cell 5, which is our five procedure RA and positive frame cell and is consistent with
the real world RA report.13
The study is presented via six screens using the Qualtrics platform. Our primary cue
screen shows the full independent assurance report on DuLac Corporation’s Schedule of GHG
Emissions, signed by Global Audit and Assurance, LLP, followed by our request for the
assurance assessment (CONF).14 After entering their judgments, participants are presented with
five forced-choice attention/recall questions, and then four background questions.15
4. Results
4.1 RESULTS
Higher CONF is expected as the cell number increases per the IAASB’s maintained
assumption. Panel A of Figure 3 presents a plot of cell medians and means of our dependent
achieved assurance metric, CONF, and Panel B presents box-and-whisker plots of cell median
CONF. As shown in Panel A, both metrics follow an “S” shape with the first two cells and the
last two cells forming the relatively “horizontal” parts of the S-shape. Cells 2 through 4, which
differ only by their hypothetical combinations of stated assurance and conclusion frame, reflect
13 Twenty-seven of the later students had not participated in the earlier administration and we rely on these for our
primary analyses. Thirty-eight of the later students had also participated in the earlier administration and we do not include them in our primary analysis. However, we conduct limited test/retest analyses as well as robustness tests (discussed below in section 4.2) to assess whether there was apparent recall or information leakage.
14 Consistent with Hasan et al. [2003] we focus on the assurance report and do not present the GHG emissions statement.
15 Across all five experimental conditions, the average time taken to complete the experiment was 9.08 minutes (s.d. = 12.241).
20
an upward slope as the frame and stated assurance move toward the RA opinion. Panel A of
Table 1 shows that the median CONF for each cell ranges from 60% for cells 1 and 2 to 81% and
80% for cells 4 and 5, respectively, and Panel B shows a similar pattern of results using mean
CONF.
| Insert Figure 3 and Table 1 about here |
By construction, our five combinations of conditions (cells) are ordered a priori as to
expectation of report users’ perceptions of achieved assurance based on the IAASB’s maintained
assumptions as to which combinations would yield higher perceived CONF by report users.
Expectations for CONF are increasing from cell 1 through cell 5, but no prediction is made about
the magnitude of any increase. Given our a priori ordering of cells and the fact that differences
between cells is measured on an ordinal scale (and not an interval or ratio scale), we use the
Jonckheere-Terpstra (J-T) independent-samples test for ordered alternatives. The J-T test for
ordered alternatives is suitable when the alternative hypothesis for a k independent-samples
design specifies the rank-order of the k independent samples (Siegel and Castellan [1988]). We
use the J-T test for our primary analyses.
Table 2 presents pairwise J-T tests of difference in medians for our four cell difference
expectations shown in Figure 2. Note that cells 1 and 2 as well as cells 4 and 5 do not differ
significantly (p > 0.10, one-tailed), suggesting that our users did not distinguish achieved
assurance based on the differing procedure descriptions underlying these two pairs of cells.
These results obtain even though the procedure is an important one logically because it involves
the accuracy of the underlying information in the GHG report. And important professionally
because the differences are based on a real world assurance report prepared by a large global
network audit firm and the fifth procedure was essential to meeting the RA criteria regarding
21
material misstatement. The addition of this important/essential procedure does not add
significantly to the CONF for a “four procedure” report labeled as RA and with a positive frame
conclusion. Further, the explicit mention of this important procedure does not significantly
lower CONF for the four procedure LA condition with a negative frame report.
| Insert Table 2 about here |
Table 2 also shows that J-T tests of the median CONF for cells 2 and 3 (i.e., negative vs.
positive conclusion framing, respectively) differ significantly (p < 0.05, one-tailed), as well as
the median CONF for cells 3 and 4 (i.e., LA vs. RA, respectively) (p = 0.00), consistent with the
presumption of those wanting the IAASB to alter its guidance for reporting limited assurance
results. In particular, our results suggest that both stating a limited assurance report’s conclusion
in a positive opinion form (as opposed to negative) and characterizing the level of assurance as
reasonable (as opposed to limited with warnings) result in higher perceived achieved assurance
or CONF among users.
The box-and-whisker plots of median CONF across our five cells (Panel B of Figure 3)
show a wider spread in the lower and upper quartiles as well as the interquartile ranges for cells 1
and 2 than for cells 4 and 5. The overall standardized J-T statistic (6.575) is highly significant (p
= 0.00), which provides support for our expectation of median CONF increasing from cell 1
through cell 5.
As a final analysis of our CONF scores and ordered cells, we converted CONF scores to
ranks assigning 1 to the lowest and 210 for the highest (and substituting mean rank for ties). By
making this transformation, we can determine the expected median rank for each of the ordered
cells if the IAASB’s assumptions were perfectly described by each our users’ CONF judgments.
For example, the 48 participants in cell 1 would have the lowest 48 scores and their median rank
22
would be 24. The 46 participants in cell 2 would have the next lowest scores and their median
rank would be 70, and so on through cell 5. Figure 4 plots the expected median CONF rank
(E[MCR]) conditional on perfect application of the IAASB’s assumption and shows an
approximately straight upward sloping line. Figure 4 also plots the actual cell median CONF
ranks (MCR) and shows that cells 2 through 4 are very close to their theoretical expectation –
thus suggesting that the IAASB’s assumption that stated LA and RA wording and conclusion
frames are important in communicating assurance is correct. On the other hand, cells 1 (and 5)
are considerably above (and below) expectations.16
| Insert Figure 4 about here |
4.2 ATTENTION CHECKS AND ROBUSTNESS TESTS
We asked three attention check questions to determine whether participants attended to or
noted the independent variable cues as intended. Regarding the stated level of assurance (LA vs.
RA), we asked the forced choice question:
“In the assurance report you read, what was the stated level of assurance?”
with possible responses: (1) Reasonable Assurance; (2) Limited Assurance; (3) I do not recall.17 Regarding explicit vs. implicit excluded procedure manipulation we asked:
“In the assurance report you read, do you recall whether Global Audit and Assurance, LLP identified specific procedures that were not included in the work plan?”
with responses: “Yes – I recall such a statement;” “No – I do not recall such a statement.” Finally
to address the conclusion frame, we asked:
“At the end of the assurance report that you read, there was a stated conclusion. Indicate which of the following two wordings (A or B) was used in that conclusion: “A. “…nothing came to our attention that causes us to conclude that the Schedule…is not
16 We also regressed CONF against our rank-ordered cells and found significantly positive slopes with the slope for
the cell range restricted to cells 2 through 4 significantly steeper and with a higher R-square than for the full cell range 1 through 5.
17 We randomized the order of the first two choices in our three attention check questions.
23
fairly presented, in all material respects…”; B. “In our opinion, the Schedule…is fairly presented, in all material respects…,”
followed by the choices: “(1) The conclusion used the wording in A; (2) The conclusion used the
wording in B; and (3) I do not recall.”
For the first and third attention check questions, 91.9% and 92.7% of participants’
responses, respectively, were consistent with their condition (stated level of assurance and
conclusion frame conditions, respectively). We also noted no significant differences in these
response rates across conditions. For the second attention check question on implicit versus
explicit procedure exclusion, we find that 83.5% of the responses overall were consistent with
their respective condition. However, we also noted that the response rates varied across
conditions, specifically, in cell 1, where the procedure was explicitly excluded. We found that
while the majority (58.3%) of those participants in cell 1 gave responses that were consistent
with the explicit exclusion, a significant minority (41.7%) of the participants gave responses that
were not.18
To test the sensitivity of our full sample to potential lack of participant attention, we
excluded two groups of participants and reran our J-T tests of medians on our restricted samples.
We excluded (a) participants who failed to recall the stated assurance (question 1), and (b)
participants who failed to recall any of our three attention checks. Our results are shown in Table
3 where the two robustness tests are qualitatively similar to our full sample results.
Finally, recall that 38 students from cell 5 also participated in the earlier administration as
a participant in cell 1, 2, 3, or 4, and we excluded these participants from our primary analyses.
We conducted various robustness tests to assess whether there was apparent recall or information
18 Mummendy et al. (2000) suggest that initial categorization upon receiving positive information (e.g., procedures
that were performed) may affect the salience of negative information that follows (e.g., omitted procedure note). Hence, participants may not recall the statement that a procedure was not performed.
24
leakage. The results of our first robustness test are shown in Panel C of Table 3, which combines
these 38 participants with the 210 participants in our sample for our primary analyses. As shown
in Panel C, the results are qualitatively similar to our primary results that do not include these 38
participants. Finally, our second set of robustness tests (untabulated) shows that the cells 1 to 4
medians for these 38 students were very close to the medians of students who participated only
once and also very close to the medians for cell 5 who had not previously participated. We
conclude that there was little information leakage, information recall, or hypothesis guessing by
these participants.
| Insert Table 3 about here |
5. Conclusion
In this study we use key features simultaneously shared by the IAASB’s
new ISAE 3000, ISAE 3410, and ISRE 2400 to investigate whether practitioner-customized work
effort descriptions and assurance report words matter in that they affect users’ perceptions of
assurance achieved. We have two broad findings: (1) whether an important verification
procedure is performed (as in cell 5) or not performed and its omission noted in the report (as in
cell 1) did not significantly affect report users’ perceptions of achieved assurance, but (2)
changing traditional standardized report wording that characterizes assurance attained by
applying identical verification procedures did significantly change perceptions.
Results for our two tests regarding procedures-based standards do not support the
IAASB’s presumption that users can reliably determine or differentiate achieved assurance from
the practitioner’s customized descriptions of assurance procedures chosen and applied. Results
for our two tests regarding traditional report language characterizing assurance do support the
25
IAASB’s decision to retain traditional warnings and conclusion frame for limited assurance
engagements.
These results have implications for assurance report users, information preparers making
assurance purchase decisions, regulators deciding whether to mandate assurance, and audit firm
practitioners as well as assurance standards setters. Because, other than the stated assurance
claim and the conclusion frame, a report user has only the practitioner’s description of
procedures performed to assess confidence, he or she must be diligent regarding whether
descriptions reflect substantial verification or are merely descriptions that are “overstated or
embellished” (ISAE 3000.A177). Information preparers must evaluate whether reasonable
assurance examinations will raise user confidence sufficiently to justify their higher cost and
regulators must make similar evaluations for decisions to mandate particular assurance
reporting. Audit firms applying the new guidance may unwittingly join a “race to the bottom”
by competing with embellished limited assurance procedure descriptions rather than on service
quality necessary for reasonable or “moderate” assurance.
Regarding standards setting for assurance, a significant conceptual and practical
difference now exists between the risk-based standards of the AICPA and the PCAOB vs. the
stated procedures-based approach of the IAASB. An overarching question is whether multiple
limited assurance standards affect the value of and demand for quality-differentiated assurance
for other than financial statement audits. At present, few jurisdictions mandate particular
assurance standards and the largest (the U.S.) has a different conceptual basis for limited
assurance from IAASB standards that, in contrast to financial statement audits, have few
mandates at present. Implications of these differences warrant further study at many levels.
26
REFERENCES
AUGER, P., T.M. DEVINNEY, G.R. DOWLING, C. EXCKERT, and N. LIN. “How much does a company’s reputation matter in recruiting?” MIT Sloan Management Review 54 (2013): 78—88.
BORKOWSKI, S.C., M.J. WELSH, and K. WENTZEL. “Johnson & Johnson – A model for Sustainability Reporting.” Strategic Finance (2010): 29—37.
BROCKETT, A.M., and Z. REZAEE. “Corporate Sustainability—Integrating Performance and
Reporting” (2012) John Wiley & Sons, Inc. Hoboken, N.J. ECCLES, R. G.; M. P. KRZUS; and G. SERAFEIM. “Market Interest in Nonfinancial Information.”
Journal of Applied Corporate Finance 23 (2011): 113−127. FUKUKAWA, H. and T.J. MOCK. “Audit Risk Assessments Using Belief Versus Probability.”
Auditing: A Journal of Practice and Theory 30 (2011): 75−99. GAYNOR, L. M., L. MCDANIEL, AND T. L. YOHN. 2011. “Fair Value Accounting For Liabilities:
The Role of Disclosures in Unraveling the Counterintuitive Income Statement Effect From Credit Risk Changes.” Accounting, Organizations and Society 36 (2011): 125−134.
HASAN, M., P.J. ROEBUCK, AND R. SIMNETT. “An investigation of alternative report
formats for communicating moderate levels of assurance.” Auditing: A journal of Practice & Theory 22 (2003): 171-187.
INTERNATIONAL AUDITING AND ASSURANCE STANDARDS BOARD. “ISAE 3000 (Revised),
Assurance Engagements Other Than Audits or Reviews of Historical Financial Information” (2013). New York, NY: IAASB.
INTERNATIONAL AUDITING AND ASSURANCE STANDARDS BOARD. “ISAE 3410, Assurance
Engagements on Greenhouse Gas Statements.” (2011). New York, NY: IAASB. INTERNATIONAL AUDITING AND ASSURANCE STANDARDS BOARD. “ISRE 2400 (Revised),
Engagements to Review Historical Financial Statements” (2012). New York, NY: IAASB.
INTERNATIONAL FEDERATION OF ACCOUNTANTS (IFAC). The Determination and Communication
of Levels of Assurance other than High.” New York, NY: International Federation of Accountants (2002).
KOZMINSKY, E. “Altering Comprehension: The Effect of Biasing Titles on Text Comprehension.” Memory and Cognition 5 (1977): 482−490.
LEVIN, I.P., and G.J. GAETH. “Framing of Attribute Information Before And After Consuming
The Product.” Journal of Consumer Research 15 (1988): 374−378.
27
LEVIN, I.P., G.J. GAETH, J. SCHREIBER, and M. LAURIOLA. “A New Look at Framing Effects: Distribution of Effect Sizes, Individual Differences, and Independence of Types of Effects.” Organizational Behavior and Human Decision Processes 88 (2002): 411−429.
LEVIN, I.P., S.L. SCHNEIDER, and G.J. GAETH. “All Frames Are Not Created Equal: A Typology
and Critical Analysis of Framing Effects.” Organizational Behavior and Human Decision Processes 76 (1998): 149−188.
LIBBY, R., R. BLOOMFIELD, and M. W. NELSON. “Experimental Research in Financial
Accounting.” Accounting, Organizations and Society 27 (2002): 775−810. MAINES, L. A., and L. S. MCDANIEL. “Effects of Comprehensive-Income Characteristics on
Nonprofessional Investors’ Judgments: The Role of Financial-Statement Presentation Format.” The Accounting Review 75 (2000): 179−207.
MAKSYMOV, E., M.W. NELSON, and W.R., KINNEY, Jr. “Effects of Procedure Frame, Procedure
Verifiability, and Audit Efficiency Pressure on Planning Audits of Fair Values.” Working paper, Cornell University and University of Texas at Austin, 2014.
MATSUMURA, E.M.; R. PRAKASH; and S.C. VERA-MUÑOZ. “Firm-Value Effects of Carbon
Emissions and Carbon Disclosures.” The Accounting Review 89/2 (2014): 695-724. MOCK, T.J.; S.S. RAO; and R.P. SRIVASTAVA. “The Development of Worldwide Sustainability
Reporting Assurance.” Australian Accounting Review 23 (2013): 280-294. MUMMENDEY, A., AND S. OTTEN. “Positive-Negative Asymmetry in Social Discrimination.” In
W. Stroebe and M. Hewstone (Eds.), European Review of Social Psychology 9, pp. 107-143 (1998). Chichester, UK: Wiley.
MUMMENDEY, A., S. OTTEN, U. BERGER, AND T. KESSLER. “Positive-Negative Asymmetry in
Social Discrimination: Valence of Evaluation and Salience of Categorization.” Personality and Social Psychology Bulletin 26 (2000): 1258-1270.
NELSON, M.W. “A Model and Literature Review of Professional Skepticism in Auditing.”
Auditing: A Journal of Practice and Theory 28 (2009): 1−34. NOSOFSKY, R.M, AND T.J. PALMERI. “A Rule-Plus-Exception Model for Classifying Objects in
Continuous-Dimension Spaces. Psychonomic Bulletin & Review 5 (1998): 345−369. PALMERI, T.J., AND R.M. NOSOFSKY. “Recognition Memory for Exceptions to the Category
Rule.” Journal of Experimental Psychology: Learning, Memory, and Cognition 21 (1995): 548−568.
PALMROSE, Z-V. “An Analysis of Auditor Litigation and Audit Service Quality.” The Accounting
Review 63 (1988): 55−73.
28
PRATTO, F., AND O. JOHN. “Automatic vigilance: The attention-grabbing power of negative social information.” Journal of Personality and Social Psychology 61 (1991): 380−391.
RICHEY, M., R.J. KOENIGS, H.W. RICHEY, AND R. FORTIN. “Negative Salience in Impressions of
Character: Effects of Unequal Proportions of Positive and Negative Information.” Journal of Social Psychology 97 (1975): 233-241.
SHERMAN, S.J.; D.M. MACKIE; AND D.M. DRISCOLL. “Priming and the Differential Use of
Dimensions in Evaluation.” Personality and Social Psychology Bulletin 16 (1990): 405−418.
SIEGEL, S., AND N.J. CASTELLAN, JR. “The Case of k Independent Samples (Chapter 8).” In
Nonparametric Statistics for the Behavioral Sciences (2nd Edition), 216−221 (1988). Boston, MA: McGraw-Hill.
SIMNETT, R.; M. NUGENT; and A. L. HUGGINS. “Developing an International Assurance Standard
on Greenhouse Gas Statements.” Accounting Horizons 23 (2009): 347–363. SIMNETT, R.; A. VANSTRAELEN; and W. F. CHUA. “Assurance on Sustainability Reports: An
International Comparison.” The Accounting Review 84 (2009): 937−967. TAYLOR, S.E., J. CROCKER, S.T. FISKE, M. SPRINZEN, AND J.D. WINKLER. “The
Generalizability of Salience Effects.” Journal of Personality and Social Psychology 37 (1979): 357−368.
TAYLOR, S.E., AND S.T. FISKE. “Salience, Attention, and Attribution: Top of the Head
Phenomena.” In L. Berkowitz (Ed.), Advances in Experimental Psychology (Vol. 11). New York: Academic Press (1978).
VERA-MUÑOZ, S.C.; W.R. KINNEY, JR.; AND S.E. BONNER. “The Effects of Domain Experience
and Task Presentation Format on Accountants’ Information Relevance Assurance.” The Accounting Review 76 (2001): 405−429.
29
FIGURE 1 Assurance Level, Risk, Procedures across Standards, and Conclusion Frame
Panel A. Financial Statement Assertions
Reasonable Assurance (Audits/Examinations)
Limited Assurance (Reviews)
PCAOB (AU508)
IAASB (ISA700)
AICPA (AR90)
IAASB (ISRE 2400) (ISRE 2400 revised)
Stated assurance Reasonable Reasonable Limited Moderate (Appendix 3) Limited (> “inconsequential”)3
Target level risk Low Low Reasonable basis1 Moderate (para.9)2 Varies – User assessed Procedures Prescribed Prescribed Analytical proc. & Inq. Analytical proc. & Inq.
(26(d)(ii)) Analytical proc. & Inq.
Conclusion frame Positive Positive Negative Negative Negative Panel B. Other than Financial Statement Assertions
Reasonable Assurance (Audits/Examinations)
Limited Assurance (Reviews)
PCAOB (AT101)
IAASB (ISAE 3000 revised)
PCAOB (AT101)
IAASB (ISAE 3000) (ISAE 3000 revised)
Stated assurance Reasonable Reasonable Limited Limited (> “inconsequential”)
Limited (> “inconsequential”)
Target level risk Low Low Moderate Varies – User assessed Varies – User assessed Procedures Vary – Aud.
choice, broad summary
Vary – Aud. choice, broad summary
Analytical proc. & Inq.
Vary – Aud. choice, detailed summary
Vary – Aud. choice, detailed summary
Conclusion frame Positive Positive Negative Negative Negative 1 A “reasonable basis” is that “ordinarily attained by applying appropriately designed and implemented analytical procedures and inquiries of management.” 2 Practitioner is required to “obtain moderate assurance as to whether the financial statements are free of material misstatements.” (para. 9) 3 Limited assurance can vary from “just above assurance that is likely to enhance the intended users’ confidence about the subject matter information to a degree that is clearly
more than inconsequential to just below reasonable assurance. (ISAE 3000.A6, emphasis added)
30
FIGURE 2 Expectations and Research Design
Conditions: Cell 1 Cell 2 Cell 3 Cell 4 Cell 5 Excluded Procedure Noted Noted No Excluded Procedure Noted
Conclusion Frame
Negative
Positive Opinion
Stated Level of Assurance Limited Reasonable
Procedures Performed 4 Procedures 5 Procedures
Perceived Confidence
+
+ +
+ + +
+ + + +
+ + + + + Perceived Assurance (CONF) Increasing →→→→→
Description of Condition
EXCL / NEG / LA /
PROC4 /
NEX / NEG / LA /
PROC4 /
NEX / POS / LA /
PROC4
NEX / POS / RA /
PROC4
INC / POS / RA /
PROC5 Legend: Manipulated Variable: Excluded Procedure EXCL = Excluded procedure noted;
NEX = Excluded procedure not noted INC = Procedure Included and noted
Conclusion Frame NEG = Negative conclusion; POS = Positive Opinion Procedures Performed PROC4 = 4 Procedures;
PROC5 = 5 Procedures
Stated Level of Assurance LA = Limited Assurance; RA = Reasonable Assurance Predictions and tests:
Prediction Test Exclusion matters CONF (cell 1) < CONF (cell 2), i.e., CONF1 < CONF2 Conclusion frame matters CONF (cell 2) < CONF (cell3) Stated assurance matters CONF (cell 3) < CONF (cell4) Procedures matter CONF (cell 4) < CONF (cell5) All factors matter Correlation of CONF with cell rank order per the IAASB’s
assumptions and expected rank analysis
31
FIGURE 3 Plots of Median and Mean CONFIDENCE Assessments
(n = 210) Panel A. Mean and Median CONFIDENCE Assessments, by Rank-Ordered Cells
For expectations and legend see Figure 2.
32
FIGURE 3 Plots of Median and Mean CONFIDENCE Assessments
(n = 210) Panel B. Box-and-Whisker Plots of Median CONFIDENCE Assessments, by Rank-Ordered Cells
For expectations and legend see Figure 2.
33
FIGURE 4 Plots of Median CONF Ranks (MCR) and Expected Median CONF Ranks (E[MCR]) per IAASB
For expectations and legend see Figure 2.
34
TABLE 1 Median and Mean CONFIDENCE Assessments [n = 210]
Panel A. Median CONFIDENCE Assessments by Rank-Ordered Cells [# of participants]
(1)
EXC/LA/NEG/ PR4
(2) NEX/LA/NEG/
PR4
(3) NEX/LA/POS/
PR4
(4) NEX/RA/POS/
PR4
(5) INC/RA/POS/
PR5 60.00
[48] 60.00 [46]
70.00 [46]
81.00 [43]
80.00 [27]
Panel B. Mean CONFIDENCE1 Assessments by Rank-Ordered Cells2
(Standard Deviation) [# of participants]
(1) EXC/LA/NEG/
PR4
(2) NEX/LA/NEG/
PR4
(3) NEX/LA/POS/
PR4
(4) NEX/RA/POS/
PR4
(5) INC/RA/POS/
PR5 56.17
(24.12) [48]
58.83 (19.26)
[46]
64.20 (19.99)
[46]
78.56 (17.03)
[43]
78. 00 (13.61)
[27] 1 Participants were asked, “Given the report you read, how assured (or confident) do you feel that DuLac
Corporation’s Schedule of Greenhouse Gas Emissions is fairly presented in all material respects?” Participants provided their responses on a 0−100 scale, where 0 = “Not at all assured (No confidence),” and 100 = “Completely assured (Complete confidence).”
2 The five combinations of conditions (cells) are ranked-ordered as to expectation of report users’ perceptions of
achieved assurance, from cell 1 increasing to cell 5 representing the expected increasing amount of CONFIDENCE. See legend in Figure 2.
35
TABLE 2 Pairwise Independent-Samples Jonckheere-Terpstra Tests1 for Ordered Cell Medians – Full
Sample [N = 210]
Prediction (CONFi < CONFi+1) Cell Medians Std. J-T-statistic2
p-value (one-tailed)
CONF1 < CONF2 (Does EXCL vs. NEX matter?)
60.00 [48]
60.00 [46]
0.338 0.368
CONF2 < CONF3 (Does NEG vs. POS matter?)
60.00 [46]
70.00 [46]
1.658 0.049
CONF3 < CONF4 (Does LA vs. RA matter?)
70.00 [46]
81.00 [43]
4.291 0.000
CONF4 < CONF5 (Does NEX vs. INC. matter?)
81.00 [43]
80.00 [27]
-0.589 0.722
1 The Jonckheere-Terpstra (J-T) test for ordered alternatives is suitable when the alternative hypothesis for a k independent-samples design specifies the rank-order of the k independent samples (Siegel and Castellan [1988]). Our five combinations of conditions (cells) are ranked-ordered as to expectation of report users’ perceptions of achieved assurance, from cell 1 increasing to cell 5 representing the expected increasing amount of CONFIDENCE. See legend in Figure 2.
2 The overall standardized J-T statistic is 6.575 (p = 0.000), which allows to reject the null hypothesis of no trend in the values of the CONF ratings across our five experimental conditions.
36
TABLE 3 Pairwise Independent-Samples Jonckheere-Terpstra Tests1 for Ordered Cell Medians –
Robustness Tests Panel A. Robustness Test 1 – Excluding participants who failed the stated level of assurance
attention check [N = 190]
Prediction (CONFi < CONFi+1) Cell Medians Std. J-T-statistic1
p-value (one-tailed)
CONF1 < CONF2 (Does EXCL vs. NEX matter?)
60.00 [35]
60.00 [42]
1.196 0.116
CONF2 < CONF3 (Does NEG vs. POS matter?)
60.00 [42]
69.00 [43]
2.109 0.018
CONF3 < CONF4 (Does LA vs. RA matter?)
69.00 [43]
81.00 [43]
4.193 0.000
CONF4 < CONF5 (Does NEX vs. INC. matter?)
81.00 [43]
80.00 [27]
-0.589 0.722
Panel B. Robustness Test 2 – Excluding participants who failed any of the three attention checks
(stated level of assurance, statement of omitted procedure, conclusion paragraph format) [N = 150]
Prediction (CONFi < CONFi+1) Cell Medians Std. J-T-statistic2
p-value (one-tailed)
CONF1 < CONF2 (Does EXCL vs. NEX matter?)
60.00 [21]
60.00 [31]
0.291 0.386
CONF2 < CONF3 (Does NEG vs. POS matter?)
60.00 [31]
70.00 [35]
2.163 0.016
CONF3 < CONF4 (Does LA vs. RA matter?)
70.00 [35]
82.00 [38]
3.586 0.000
CONF4 < CONF5 (Does NEX vs. INC. matter?)
82.00 [38]
80.00 [25]
-0.297 0.617
Panel C. Robustness Test 3 – Including thirty-eight cell 5 participants who also participated in
the earlier administration of the study [N = 248]
Prediction (CONFi < CONFi+1) Cell Medians Std. J-T-statistic3
p-value (one-tailed)
CONF1 < CONF2 (Does EXCL vs. NEX matter?)
60.00 [48]
60.00 [46]
0.338 0.368
CONF2 < CONF3 (Does NEG vs. POS matter?)
60.00 [46]
69.50 [46]
1.658 0.049
CONF3 < CONF4 (Does LA vs. RA matter?)
69.50 [46]
81.00 [43]
4.291 0.000
CONF4 < CONF5 (Does NEX vs. INC. matter?)
81.00 [43]
80.00 [65]
-0.262 0.603
1 Overall standardized J-T statistic is 7.624 (p = 0.000). 2 Overall standardized J-T statistic is 6.559 (p = 0.000). 3 Overall standardized J-T statistic is 7.550 (p = 0.000).