guerrilla analytics - introduction and case study

#GuerrillaAnalytics http://guerrilla-analytics.net 1

Guerrilla AnalyticsIntroduction and Case StudyEnda Ridge, PhD

Copyright Enda Ridge 2015


What we are told about Data Science

“the sexy job in the next 10 years will be statisticians”

“Data Scientist: The Sexiest Job of the 21st Century”

“Information is the oil of the 21st century, and analytics is the combustion engine.”

http://www.gapminder.org/http://www.statistics.com/data-science-quotes/https://github.com/mbostock/d3/wiki/Gallery


http://www.gapminder.org/

http://www.statistics.com/data-science-quotes/

https://github.com/mbostock/d3/wiki/Gallery


Hi, we need an update on the insurance policy classification work. It’s going to the Head of Underwriting this afternoon.

Um. Which work? I think Jo did that butJo’s on holidays.

I’ll check my mailbox and send you my spreadsheet from last week. Err.....the population changed with

the extra system extract on Tuesday.

And we added a bunch of business rules to accommodate that....

so we can’t go back to the earlier numbers.

The Reality


#GuerrillaAnalytics http://guerrilla-analytics.net 4Copyright Enda Ridge 2015

Those were the droids I was looking for ...


My Journey to Guerrilla Analytics

Mechanical Engineer

PhD Computer

Science

Boutique Consultancy

Forensic Data Analytics

Senior Manager


Constraints Constraints+

DynamicReproducible

Constraints+

DynamicReproducible

+Tested

Constraints+

DynamicReproducible

+Tested

+Audit


Common format

Data Analytics Insight



Misconception

Shearer C., The CRISP-DM model: the new blueprint for data mining, J Data Warehousing (2000); 5:13—22



Reality is Guerrilla Analytics

Data• Extraction• Receipt• Loading

Analytics• Transform• Algorithms• Consolidate

Insight• Reporting• Work Products

Disruptions



Maintain Data Provenance


Maintaining Data Provenance mitigates disruptions

10

7 Principles of Guerrilla Analytics•S

pace is cheap, confusion is expensive

1

•Prefer simple, visual project structures

2

•Prefer automation with program code

3

•Link data on the file system, analytics environment, and work products

4

•Version control data and code

5

•Consolidate team knowledge in builds

6

•Prefer code that runs end to end

7Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net

~100 practice

tips


Guerrilla Analytics




Disruptions



Guerrilla Analytics Case Study


Client Retail Bank

Situation Error in credit card customer mailing processFailure to comply with regulations, potential fines

Mission • Understand system landscape & get the right data• Rebuild full customer history• Identify system errors and start of non-compliance• Quantify effected customers and cost to bank

Timeline 6-8 weeks


System Landscape

Customer Contact

Card System 1 Card System 2

Collections Manual Intervention



Data Receipt


Guerrilla Analytics Environment

• Lost Data• Multiple Copies of data• Limited supporting information• Local copies of data• Renamed data• ä~ delimited data


Data Receipt


Guerrilla Analytics Approach

• Have 1 Data location• Data Unique Identifiers• Data log• Supporting material near data• Never modify the data


Data Load

File System

Crazy-name spreadsheet 1Crazy-name spreadsheet 2Crazy-name spreadsheet 3

FNU810A

long_named_file_v0.2.1.pdf

Analytics Environment

Credit_Card_Samples

DBO.Accounts

Customer_Letters

Guerrilla Environment

• Renamed files• Scattered inconsistent

locations• Multiple versions of files• Replacements of files



Data Load

Data

Crazy-name spreadsheet 1

Crazy-name spreadsheet 2

FNU810A

long_named_file_v0.2.1.pdf


D010.Crazy-name spreadsheet 1

D026.Crazy-name spreadsheet 2

D040.FNU810A

D051.long_named_file_v0.2.1.pdf

Guerrilla Analytics Approach

• One-to-one mapping from files to datasets– Keep crazy names

• Minimize prep work• Put the Data Identifier in

the path


D010

D026

D040

D051


Guerrilla Analytics




Disruptions



Analytics: Guerrilla Analytics Environment


My Documents/Transactions

Accounts_Formatted.SQL

TransProf_FINAL.R

Trans_DO_NOT_USE.R

TransProf_v2.R

Sample_accounts.SQL

• Many code files/languages• Variety of output types• Data manipulation

– on file system– in analytics environment

• Combinations of tools• Many users• Many iterations


Analytics: Guerrilla Analytics Approach


• One folder for all team work products

• Give every work product an identifier

• Keep a work product log• Clear running order of files• No dead/orphaned files

Work_Products

Work_products.xls

WP_024• 010_Accounts_Cleaned.SQL• 030_Transaction_Profiles.R• 050_Sample_accounts.SQL

WP_96

WP_97




• Keep older versions in subfolder• Keep related information in a

subfolder WP_024010_Accounts_Cleaned.SQL030_Transaction_Profiles.R050_Sample_accounts.SQL

supporting

archive



File System

WP_024010_Accounts_Cleaned.SQL030_Transaction_Profiles.R050_Sample_accounts.SQL


WP_024.ACCOUNTS_CLEANED

WP_024.TRANSACTION_PROFILES

WP_024.SAMPLE_ACCOUNTS



Data Manipulation: Guerrilla EnvironmentAccount_ID Statement_ID Min_Payment Transaction_ID Amount Type

A 15 30.00 1 50.00 ExpenseA 15 30.00 2 25.00 ExpenseA 15 30.00 3 -75.00 PaymentA 15 30.00 4 20.00 Expense


ID Stmnt_ID Min_Payment Balance Min_Paym_Made

A 15 30.00 20.00 No

... ... ...


Data Manipulation: Guerrilla Analytics Approach


Account_ID Statement_ID Min_Payment

Transaction_ID

Amount Type RunningPayments

Min Paym Made

A 15 30.00 1 50.00 Expense 0.00 NoA 15 30.00 2 25.00 Expense 0.00 NoA 15 30.00 3 -75.00 Payment 75.00 YesA 15 30.00 4 20.00 Expense 75.00 Yes

Account_ID Statement_ID Min_Payment Transaction_ID Amount Type

A 15 30.00 1 50.00 ExpenseA 15 30.00 2 25.00 ExpenseA 15 30.00 3 -75.00 PaymentA 15 30.00 4 20.00 Expense


Guerrilla Analytics




Disruptions



Reporting – what is a report?



Reporting – Guerrilla Analytics Environment



Reporting – Guerrilla Analytics Environment


Select min/max of transaction_time

WP_030

•010_Late payments.SQL•030_Late payments.py

WP_042


Guerrilla Analytics




Disruptions



Why consolidate?

Raw

Duplicates

Customers Clean_Cust

Deduped New_dupes

Work Product



Why consolidate?

Raw

Duplicates

Customers Clean_Cust

Deduped New_dupes

Duplicates_02

Customers_02

Duplicates

Deduped Clean_cust New_dupes

Work Product



Guerrilla Analytics Approach: Builds

Deduped

Clean_cust

New_dupesDuplicates_02

Duplicates

Customers_02

Dupes_latest

Cust_Latest

Raw Latest Clean Rules Interface

Version Controlled Code and Data


WP_030

33

Summary

Copyright Enda Ridge 2015 #GuerrillaAnalytics http://guerrilla-analytics.net

A Realistic Workflow• Guerrilla Analytics Principles• Guerrilla Analytics Practice Tips

Case Study• Data receipt and load• Analytics• Reporting and work products• Consolidation with Builds

Why Data Science is Difficult• Disruptions, Constraints• These break Data Provenance

Those were the droids I was looking for ...


Keep in Touch!

@Enda_Ridge

http://guerrilla-analytics.net


Or contact me for 50% discount

https://twitter.com/enda_ridge

http://www.guerrilla-analytics.net/

http://www.guerrilla-analytics.net/

guerrilla analytics - introduction and case study

Data & Analytics

guerrillaanalytics http

phd copyright enda ridge

end7 copyright enda

netcopyright enda ridge

data science

case study enda ridge

data provenance

analytics environment