A SURVEY OF RANDOM FOREST USAGE FOR FRAUD DETECTION

AT LLOYDS BANKING GROUP

Adam Langron

AUGUST 2015

CONTENTS

Background

Current Account Fraud

Insider Fraud

Mortgage Broker Risk

Conclusions

2

BACKGROUND

3

BACKGROUND

4

2012 Masters project with Xin Huang suggested that random forests would provide improved discrimination over traditional logistic regression models.

This formed the basis of 2013 CRC conference paper by Kevin Barrett.

RANDOM FORESTSBRIEF INTRODUCTION

5

Bootstrap Sample

Split 1a

Split 2a Split 2b

Split 1b

Bootstrap Sample

Split 1a

Split 2a Split 2b

Split 1b

Bootstrap Sample

Split 1a

Split 2a Split 2b

Split 1b

Bootstrap Sample

Split 1a

Split 2a Split 2b

Split 1b

Bootstrap Sample

Split 1a

Split 2a Split 2b

Split 1b

Bootstrap Sample

Split 1a

Split 2a Split 2b

Split 1b

Hundreds of trees – a forest!

These leaves at the bottom of the trees ‘vote’

If > 50% of the training sample were good at this leaf, the vote is ‘good’

If > 50% of the training sample were bad at this leaf, the vote is ‘bad’

Each new observation will fall to one final leaf node in each tree

Votes are counted and a predicted outcome assigned

This is a ratio that can be interpreted as a score

RANDOM FORESTSPROS AND CONS

PROS CONS

IMPLEMENTATION

‘GREY BOX’

INTERPRETABILITY

MONITORING

OPTIMAL?

DISCOVERS INTERACTIONS

RAPID - NO BINNING

INCORPORATES MANY PREDICTORS

6

CURRENT ACCOUNT FRAUD

FIRST PARTY FRAUD DETECTION USING PERSONAL

CURRENT ACCOUNT TRANSATIONS

7

-7-6-5-4-3-2-10123456789

101112

01/08/2012

31/08/2012

30/09/2012

30/10/2012

29/11/2012

29/12/2012

28/01/2013

27/02/2013

29/03/2013

28/04/2013

28/05/2013

27/06/2013

27/07/2013

26/08/2013

25/09/2013

25/10/2013

24/11/2013

Bal

ance

/ Tr

ansa

ctio

n Va

lue

(£ T

hous

ands

)

Other Non-MonFalcon FlagsAccount FeatureIndicatorBalance EnquiryCredit Limit

CURRENT ACCOUNTS – THE PROBLEMWhat is the best way to use the vast amount of data that the bank holds to detect fraud and refer accounts at the right time to allow prevention?

8

Fraudulent Cheque Credit

Online Spend Charge-Off

Atypical Behaviour

Cheque Reversal

A Cheque Fraud Case Study

CHARACTERISTICSThe richness of transactional data allows over 100,000 possible predictors to be created

•Visa Debit authorisations•Accepts/Declines•Grouped by Merchant Category

Code

Point of Sale

•Branch transactions•Deposits and withdrawals

Counter

•Non–counter payments and transfers

• Faster Payments, DD•All channels

Transfers

•Non-monetary events•Balance enquiries•Account status checks•Grouped by channel

Enquiries

9

•Days from obs point•0 -10 days

Days

•Weeks from obs point•0 - 6 weeks

Weeks

•Volume•Value•Accept/Decline•Excess•Utilisation

Measure

Combining transaction type with time interval and measure yields a multitude of predictors

Using Merchant Category Code and Money Manger within the relevant transaction types enabled even more granularity

Infrastructure necessitated the reduction of these characteristics Information Value macros cannot cope with this number of chars Chars populated very sparsely were removed Chars with suspiciously high bad rates were removed if there

was evidence of fraud investigation activity Information Value and was used to select final candidate chars

X X

MODEL PERFORMANCEModel performance on the validation sample is strong

10

Performance is shown for the development validation data set Discrimination is very high: Gini is in excess of 80% Validation has been performed on three out of sample holdouts and one out of time

holdout and shows stable performance The model discriminates well for all three fraud categories: First Party Definitions, EUCs

and Mules Investigator feedback from a small sample verified that the model ranks effectively and

finds a wide variety of fraud typologies

0102030405060708090

100

0 10 20 30 40 50 60 70 80 90 100

Cum

ulat

ive

% B

ad

Cumulative % Good

Validation Sample Gini

0102030405060708090

100

0.00.10.20.30.40.50.60.70.80.91.0

Cum

ulat

ive

Pop

%

Trinity Score

Cumulative Score Distribution

BadDefsEUCMuleGood

The model was initially piloted in an Operational environment To confirm referrals were generated at the ‘right’ point That volumes were manageable That Trinity could ‘add value’ to the existing strategy suite

Pilot results were very positive c60% of reviewed cases were high risk c25% of the high risk referrals were unique to Trinity The model identified a good mix of fraud types

The model now implemented with fraud operations Scores are refreshed daily Trinity scores are crossed with a fraud propensity model to drive

referrals

IMPLEMENTATIONAccounts now being referred to Fraud Operations

11

StrategyMatrix of Trinity & Fraud Score

Low Risk Med or High Risk

Very High Risk

0 - 800 Not Referred

800 - 900

900 - 1000 Referred

Trin

ity S

core

High Risk Fraud Score

Credit Manipulation Bust-Out

Runaway Spend Refund Fraud

Fraud Typologies Identified

INSIDER FRAUD

DETECTING INSIDER FRAUD USING COLLEAGUE

INTERACTIONS

12

Working Pattern

1

Location

Branch / Telephone

Days worked

Insert

2

Account notifications

Transaction earmarks

Enquire

3

Standing order

Balance

Financial

4

Inter account transfers

Cheques

Counter withdrawals

Amend

5

Change of address

Change of telephone

number

MODEL DRIVERSThe model uses hundreds of event types as predictors, these are driven by colleagues and related to system log on identifiers

13

TransCategory

Rank

TransType

MODEL PERFORMANCEModel performance on the validation sample is strong

0%10%20%30%40%50%60%70%80%90%

100%

0% 20% 40% 60% 80% 100%

% o

f Bad

s

% of Goods

14

Performance is shown for the development validation data set Discrimination is very high: Gini c70% The model discriminates particularly well at the top of the distribution Initial use case will be to exclude all staff members from investigation where the model

shows sufficiently low risk

MORTGAGE BROKER RISK

ASSESSING THE RISK OF MORTGAGE BROKERS USING

APPLICATION MIX

15

Model developed to assess risk posed by Mortgage Broker Panel Over 55,000 characteristics assessed relating to application mix Random Forest and logistic regression models developed RF model quickly deteriorates in out of time sample

MORTGAGE BROKER RISKUNABLE TO VALIDATE RESULTS OUT OF TIME

16

Out of Time Sample

0%

20%

40%

60%

80%

100%

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov

GIN

I

Random ForestLogistic Regression

Desire to investigate further. Is the algorithm overfitting to the development sample? Lack of resource to investigate at this time

CONCLUSION

17

Positive Results• Very high discrimination• Relatively quick to build• Many predictors used to assess risk• Hard for fraudsters to game

CONCLUSIONRandom Forests can add value to fraud prevention

18

Ongoing Challenges• Validation issues. Are the models more likely to over fit to the development period?• Implementation challenges. Currently no decisioning platforms for deployment• Monitoring. What to monitor on monthly / quarterly basis?• Desire to develop best practice for development and ongoing monitoring and maintenance

19