10/22/2019

1

Artificial Intelligence & Internal Control TestingSeries 1 – October 23, 2019

• Participate in entire webinar• Answer polls when they are provided• If you are viewing this webinar in a group

Complete group attendance form with• Title & date of live webinar• Your company name• Your printed name, signature & email address

All group attendance sheets must be submitted to within 24 hours of live webinar Answer polls when they are provided

• If all eligibility requirements are met, each participant will be emailed their CPE certificates within 15 business days of live webinar

To Receive CPE Credit

10/22/2019

2

Shauna Woody-Coussens, CFEManaging Director, Forensics & Valuation Services

Lanny Morrow, EnCE®, CTFIManaging Consultant, Forensics & Valuation Services

Our Goals Today• Discuss the benefits of AI-assisted internal control testing• Discuss general accounting system requirements to be able to

implement internal control testing using artificial intelligence (AI)• Understanding AI & complex systems • Discuss the technology used for AI-assisted internal control

testing

10/22/2019

3

Traditional Internal Control Testing• Very manual in nature

Submit electronic internal control questionnaires to various organizational personnel

Read through organization’s policies, procedures & internal control documentation

Perform interviews & walk-throughs of random transactions

Data Analytic-Assisted Transaction Testing• Use data analytics to identify potentially higher risk transactions

for testing Identify transactions that appear to possibly violated required internal

controls Identify transactions with common red flags for risk Identify transactions that are anomalies for the organization

10/22/2019

4

The Past• Very manual & time-consuming process• Subjective & random sampling means some risky transactions

may not be tested• Internal control gaps may not be highlighted

The Future• Less involvement of organizational personnel in the process• Population testing of transactions against organizational & best

practices for internal controls• Heighted assurance that financial operations are operating

within acceptable risk tolerances

10/22/2019

5

What Is AI?• AI systems typically demonstrate some of the following behaviors

associated with human intelligence Planning Learning Reasoning Problem-solving Knowledge representation Perception Motion Manipulation Potentially some social intelligence & creativity

… & What AI Is Not …

10/22/2019

6

Artificial General Intelligence• Intelligence of a machine that has the capacity to understand or

learn any intellectual task that a human being can • It is a primary goal of some AI research & a common topic in

science fiction & future studies• NOT the focus of what we’re talking about today

The Right Kind of AI• We will focus on machine learning, deep learning & similar AI

methods• Machine learning – computer programs that can learn from

experience using statistical methods• Deep learning – more sophisticated form of machine learning• Computer vision – useful for video & scanned PDF analysis

10/22/2019

7

Types of Machine LearningSupervised Unsupervised

Give examples & answers, machine finds more like it

Give data, machine finds patterns & applies its

own rulesExamples

• Fraud detection• Known controls weaknesses

Examples• Knowledge discovery• Grouping like things

Supervised Machine Learning

10/22/2019

8

Why AI?

AI IN INTERNAL CONTROLS TESTING?

YOU’RE KILLING ME SMALLS!

Organizations are complex systems

“A complex system included a large number of interacting components (agents, processes, etc.) whose aggregate activity is nonlinear (not derivable from the summations of the activity of individual components) & typically exhibits hierarchical self-organization …”

Source: https://www.informatics.indiana.edu

All These Organizations Had Internal Controls – They Even Tested Them• Equifax, Target, Home Depot, SWIFT• NASA• Three Mile Island• Enron• Wells Fargo• WorldCom• Encompass Health (formerly HealthSouth) • & one more … yours

10/22/2019

9

Complexity & Internal Controls• Complex systems behavior is intrinsically

difficult to model due to the dependencies, relationships, or interactions between their parts or between a given system & its environment

• Most internal controls treat the organization like an engine (component view)

Why AI? The Key Idea• AI is capable of finding patterns, relationships & threats that are

Too subtle for humans to recognize efficiently Too deeply buried in massive amounts of data Too disparate (different systems) In a bad format (scanned PDFs) Emergent – never seen before Cascading – only happens when something else happens Concealed – intentional acts to suppress the pattern or observation

10/22/2019

10

Using AI in Internal Controls Testing• Natural language processing

Natural language processing is AI that can read textual content AI can read, summarize & classify internal control policy documents AI will then identify transactions, communications & activities that

violate those policies Email analytics Invoice review Contract review Potential uses in interview speech & tone recognition for fraud Computer vision may help with handwritten/scanned content

Using AI in Internal Controls Testing• Predictive analytics

AI can be taught what controls departures have looked like in the past, then interrogate new, never-before-seen data to find those patterns

Example 1 – spending patterns related to vendors offering bribes or kickbacks

Example 2 – training AI to spot emails containing evasive, vague or other emotional tone that may indicate a controls violation

Example 3 – full coverage journal entry testing, not just sampling Systems learn from experience & improve with each iteration

10/22/2019

11

Logistic Regression• Concise “yes/no” answer for each

transaction, journal entry or activity• Associated “certainty” score• Near-instantaneous feedback

Using AI in Internal Controls Testing• Prescriptive analytics

Predictive tells us what could happen Prescriptive tells us what should happen AI system preprogrammed with “best practices” internal controls,

exposed to an enterprisewide system Using “best practices” template, identifies shortcomings in the system

being evaluated Automation is key

10/22/2019

12

Using AI in Internal Controls Testing• Agent-based modeling

Only type of analysis capable of modeling outcomes in a complex, nonlinear system

Great at identifying “unknown unknowns” & black swans For a given internal control or policy, potential outcomes can be modeled

& tested (in some cases, before it’s ever implemented) Difference between actual & best practices can be modeled Great approach for IT controls testing Demonstration

Using AI in Internal Controls TestingAgent-based modeling IT controls example

Outcome under current policies Outcome under best practices

10/22/2019

13

Using AI in Internal Controls Testing• Robotic process automation

Not AI, but will certainly support automation & efficiency efforts Robotic process automation (RPA) is the term used for software tools

that partially or fully automate human activities that are manual, rule-based & repetitive

RPA will be the glue that ties various AI internal controls testing components together

Any computerized process currently manual (contract review, extracting data, reporting, computing) can be automated; even mouse clicks

Potential to replace up to 48 percent of current manual functions

Using AI in Internal Controls Testing• Generative adversarial networks – warring AIs

AI playing “war games” simulations, over & over & over & … AI that detects internal control violations pitted against an AI that tries to

break the rules Systems on both sides learns & improves before next battle Result – new weaknesses reported to human internal auditors, system

becomes improved Potential to discover previously unknown weaknesses Further down the road – self-learning & self-correcting systems

10/22/2019

14

Using AI in Internal Controls Testing• The role of data visualization

Humans still need to understand AI output Data visualization tools (Microsoft Power BI, Amazon QuickSight,

Tableau) can connect to technical output for visual analysis Human eye can still spot patterns & put them into proper context in a

manner that rivals AI New trend – data visualization tools incorporating AI to provide “insight

generation” as the human analyst reviews content

Using AI in Internal Controls Testing• Summary of concepts

Predictive analytics – spotting IC weaknesses Prescriptive analytics – best practices versus current state Natural language processing – including text content in IC testing Deep learning & computer vision – better ML & accessibility Simulations – stress testing internal controls in varying situations Adversarial models – better controls through warring AIs Robotic process automation – “driverless” internal controls testing Data visualization – the human window into the AI mind

10/22/2019

15

QUESTIONS?

Continuing Professional Education (CPE) Credits

The information in BKD webinars is presented by BKD professionals, but applying specific information to your situation requires careful consideration of facts & circumstances. Consult your BKD advisor before acting on any matters covered in these webinars

BKD, LLP is registered with the National Association of State Boards of Accountancy (NASBA) as a sponsor of continuing professional education on the National Registry of CPE Sponsors. State boards of accountancy have final authority on the acceptance of individual courses for CPE credit. Complaints regarding registered sponsors may be submitted to the National Registry of CPE Sponsors through its website: www.nasbaregistry.org

10/22/2019

16

• CPE credit may be awarded upon verification of participant attendance

• For questions, concerns or comments regarding CPE credit, please email the BKD Learning & Development Department at training@bkd.com

CPE Credit

THANK YOU!Lanny Morrow, EnCE®, CTFIManaging Consultant Forensics & Valuation ServicesKansas City | 816.221.6300 | lmorrow@bkd.com

Shauna Woody-Coussens, CFEManaging DirectorForensics & Valuation ServicesKansas City | 816.221.6300 | swoodycoussens@bkd.com