Find Strength in Numbers: Sampling Techniques to Improve Financial Audit, Control, & Program

Performance

April 18, 2012AGA Training Audio Conference

Albert J. Lee, Ph.D., CEO, Summit Consulting LLCDenise Wu, CGFM, CPA, Partner, CliftonLarsonAllen LLP

1

About CliftonLarsonAllen

CliftonLarsonAllen (CLA) is the union of Clifton Gunderson and LarsonAllen

This merger brings over 100 years of combined experience

The nation’s newest top ten public accounting and consulting firm

More than 3,600 professionals operating from more than 90 offices

CLA’s Government niche (Federal, state and local governments) is a significant part of the company’s firm-wide public sector practice

2

About CliftonLarsonAllen (cont’d)

Assurance and related IT services include a sampling of the following: CFO and Non-CFO Act audits and advisory services State and Local Government Financial and Compliance

Audits Internal control/FMFIA reviews/other financial

management advisory services Statement on Auditing Standards SSAE-16 examinations Performance Audits Federal Information Security Management Act services Information system security reviews

3

About Summit Consulting LLC

Founded in 2003, specializes in cutting-edge quantitative techniques, including statistical sampling, financial modeling, and program evaluation

More than 60% of staff professionals have advanced degrees in economics, statistics, finance, and other quantitative disciplines

Summit has provided sophisticated consulting services for many federal agencies, including the Treasury, HUD, USDA, VA, DOT, DOE, and DOL

Affiliated experts and staff members testified at courts of law and other judicial settings for matters involving economics and statistics

4

Presentation Objectives1.Define audit sampling and distinguish attribute vs. variables sampling, and statistical vs. non-statistical sampling.2.Discuss relevant activities of a sampling project.3.Define the project team members’ roles and responsibilities.4.Provide real life case studies to highlight the use of sampling and sampling best practices.5.Provide sampling best practices for performing audit, internal control, and program evaluation sampling projects.

5

Presentation Outline

Why sampling? What comprises a sampling project? Who does what during a sampling project? Types of sampling designs Real life examples of sampling design What are some of sampling’s best

practices? How should sampling results be used?

6

Why Sampling?

Primary objective of sampling is to create a subset of elements from a population that is as similar to the population as possible to allow users to evaluate certain characteristics of the population

Benefits of sampling: Reduces costs associated with evaluation Increases efficiency of population evaluation Provides the ability to quantify variability (i.e.,

range of values) in the population Allows measurement of the sufficiency of the

evidence obtained Attribute sampling (test for deviations) vs.

variables sampling (test for misstatements)

7

Why Sampling? (cont’d)

Statistical (projectable) sampling vs. non-statistical (judgmental) sampling

Based on a statistical sample, important projections can be made about the population: Averages, totals, ratios Data validation and quality Trends identification Projection of errors and identification of

misstatements Rate of errors/deviations Sampling risk can be quantified

8

What Comprises a Sampling Project?

Planning ImplementationConclusion & Reporting

9

What Are the Planning Activities of a Sampling

Project? Sampling Frame Formation and Population

Analysis Designed to ensure the dataset used for

selecting the sample corresponds to the scope of the project and adequately addresses risk areas.

Clearly define the objective of the project and the population subject to testing.

Determine completeness of the dataset and validity of data.

Analyze the population characteristics and assign risks to the various characteristics/strata.

Deploy data checks to eliminate duplicates and reversals.

10

What Are the Planning Activities of a Sampling

Project? (cont’d) Sample Design

Dictated by the project’s objectives with consideration for the project’s constraints.

Two broad categories of statistical sampling: Attribute sampling – used primarily in tests of

controls Variable sampling – used to estimate/project a

quantity and often used to test the monetary value of account balances

Clearly define the sampling unit. Create subgroups of observations with similar

characteristics that correlate with audit risk.

11

What Are the Planning Activities of a Sampling

Project? (cont’d) Sample Size Determination

Predicated on the sample design, risk assessment, and precision requirements.

Estimate the expected misstatement/expected deviation rate to achieve a prescribed level of precision.

Tailor sample size to address emerging risks or issues.

Balance precision with team resources (i.e., time, people, and money).

12

What Are the Implementation Activities of a Sampling

Project? Sample Selection

Select the items from the sampling frame that will be tested.

Sample items should be selected in such a way that the sample is representative of the population.Sample items should have a fixed,

known and positive probability of selection.

13

What Are the Implementation Activities of a Sampling

Project? (cont’d) Sample Testing and Evaluation

Test the sampled items and evaluate sample results mathematically.

Determine whether unexpected test results could skew conclusions.

Determine whether initial assessments should be revised and the sample expanded.

14

What Are the Conclusion & Reporting Activities of a

Sampling Project? Extrapolation and Conclusion

Project the sample findings to the population. Conclude on the rate of error of attribute

sampling or dollar misstatement in variables sampling.

Reporting Document the sampling plan, including

evaluation considerations, calculation of the point estimates (best guess value) along with the confidence intervals, and conclusions reached regarding the population.

15

Who Does What During a Sampling Project?

Planning ImplementationConclusion& Reporting

16

Who Wants to be a Millionaire?

Background: Agency uses statistical sampling to collect improper payments. By law, the agency can only collect the lower bound of the improper payment estimate.

Objective: To collect as close to the true improper payment amount as possible.

Challenge: Historically, using a simple random sample design, the agency under collects due to the lack of precision. The point estimate (e.g., $15 million) is much

greater than the lower bound (e.g., $300K).

17

Who Wants to be a Millionaire? (cont’d)

Solution: Stratify the population according to risk characteristics. Using the same sample size as the simple random sample design, precision improves and the lower bound gets closer to the point estimate. Point estimate remains $15 million, but the

lower bound increases (e.g., to $3 million).

18

Frequent Flyer Miles, Anyone?

Background: Agency uses statistical sampling to identify assets that are misstated on the general ledger.

Objective: To determine the size and extent of misstatement of the agency’s recorded asset balance.

Challenge: Assets are located in all fifty states and six territories. Using a simple random sample it is likely to select as many locations as the sample size (i.e., a great deal of traveling).

19

Frequent Flyer Miles, Anyone? (cont’d)

Solution: Implement a multistage sampling design in which locations are selected in the first stage and assets to be tested are selected in the second stage within the selected locations. The sample controls the number of locations visited and the number of assets evaluated.

Benefit: Substantially reduce travel costs and significantly enhance audit efficiency and effectiveness.

20

Should I Stop or Should I Go

Background: Agency uses statistical sampling to determine the extent of misstatement in the UDO balance at year end.

Objective: To achieve precision by evaluating as small a sample size as possible.

Challenge: No prior history to estimate the sample size for a given precision. It is 9/30 and only two weeks remain to finish the audit.

21

Should I Stop or Should I Go (cont’d)

Solution: Prepare a sample with a randomized sort-order and evaluate sample batches sequentially to determine if the prescribed precision requirements have been achieved. If yes, STOP. If no, evaluate the next batch according to the random order.

Benefit: Performing the least amount of work to achieve the prescribed precision requirements.

22

Sampling Best Practices - Audit

1. Sampling frame corresponds to the target population.2. Every transaction in the frame is substantive (i.e.,

net out all the reversals and eliminate phantom transactions).

3. Use stratification to test all large transactions with a dollar amount above a certain threshold.

4. Define methodology for determining the point estimate: Mean, proportion, totals, ratios.

5. Create a unique sort, and set random seeds to ensure sample can be reproduced.

6. Careful documentation of sampling design, sample size, expected precisions, point estimates, and confidence intervals.

23

Sampling Best Practices – Program Performance

Evaluation1. Sampling frame corresponds to the target population

2. Every subject in the frame is substantive, i.e., eliminate phantom subjects

3. Select sample designs according to data structure and availability

4. Define performance metrics and all the components

5. Create a unique sort, set random seeds to ensure sample reproducibility

6. Careful documentation, sampling design, sample size, expected precisions, point estimates, and confident intervals

24

How Should Sampling Results be Used?

1. Point estimate from a statistical sample is the unbiased and best guess of a population’s true value

2. Precision and confidence interval quantify how point estimates are likely to change from one sample to the next Lower (upper) bound of a confidence interval is a

threshold value below (above) which a population value is unlikely

3. These statistics determine the likelihood of an estimated population value (e.g., population error rate)

4. Based on these likelihoods, certain population values could be rejected for being extremely unlikely

25

Contact Information

Albert J. Lee, Ph.D., CEO, Summit Consulting LLC

albert.lee@summitllc.us202-407-8300

Denise Wu, CGFM, CPA, Partner, CliftonLarsonAllen LLP

Denise.Wu@cliftonlarsonallen.com301-931-2050

26

Questions?27