bigml fall 2015 release

Introducing Association Discovery

BigML 2015 Fall Release

BigMLInc Fall2015Release 2

Today’sWebinar• Speaker:

• PoulPetersen,CIO

• Moderator:

• AtakanCe>nsoy,VPPredic>veApplica>ons

• Enterques>onsintochatbox–we’llanswersomeviatext;othersattheendofthesession

• email:[email protected]

• TwiPer:@bigmlcom

mailto:[email protected]


Associa1onDiscovery

Algorithm“MagnumOpus”fromGeoffWebb

UnsupervisedLearning:unlabelleddata

LearningTask:Find“interes1ng”rela1onsbetweenvariables.

BigMLInc Fall2015Release

DecisionTreesBaggingDecisionForest

4

BigMLWorkflow

MODEL

DATASET

CLUSTER

ANOMALY

ASSOCIATION

SOURCE

K-MeansG-Means

Isola>onForest

MagnumOpus


date customer account auth class zip amountMon Bob 3421 pin clothes 46140 135Tue Bob 3421 sign food 46140 401Tue Alice 2456 pin food 12222 234Wed Sally 6788 pin gas 26339 94Wed Bob 3421 pin tech 21350 2459Wed Bob 3421 pin gas 46140 83The Sally 6788 sign food 26339 51

Clustering


AnomalyDetec1on

similar

unusual

UnsupervisedLearning



6

{customer = Bob, account = 3421} zip = 46140

Rules:

{class = gas} amount > 80

Associa1onRules



7

{customer = Bob, account = 3421} zip = 46140

Rules:

{class = gas} amount > 80

Antecedent Consequent

Associa1onRules


UseCases

• MarketBasketAnalysis

• WebusagepaPerns

• Intrusiondetec>on

• Frauddetec>on

• Bioinforma>cs

• Medicalriskfactors


MarketBasketAnalysis

• Datasetof9,834grocerycarttransac>ons

• Eachrowisalistofallitemsinacartatcheckout

GOAL:Discover“interes1ng”rulesaboutwhatstoreitemsaretypicallypurchasedtogether.


Associa1onMetrics

Instances

AC

Coverage

Percentageofinstanceswhichmatchantecedent“A”


Associa1onMetrics

Instances

AC

Support

Percentageofinstanceswhichmatchantecedent“A”andConsequent“C”


Confidence

Percentageofinstancesintheantecedentwhichalsocontaintheconsequent.

SupportCoverage

12

Associa1onMetrics

Instances

AC


CInstances

A C

A

Instances

C

Instances

A

13

Associa1onMetrics

Instances

AC

0% 100%

Instances

AC

Confidence

AneverimpliesC

Asome1mesimpliesC

AalwaysimpliesC


LiO

Ra>oofobservedsupporttosupportifAandCweresta>s>callyindependent.

Support==Confidencep(A)*p(C)p(C)

14

Associa1onMetrics

Independent

AC

C

Observed

A


C

Observed

A

15

Associa1onMetrics

Observed

AC

< 1 > 1

Independent

A C

Lift = 1

Nega>veCorrela>on NoAssocia>on Posi>ve

Correla>on

Independent

A C

Independent

A C

Observed

A C


Associa1onMetrics

Independent

AC

C

Observed

A

Leverage

DifferenceofobservedsupportandsupportifAandCweresta>s>callyindependent.

Support-[p(A)*p(C)]


C

Observed

A

17

Associa1onMetrics

Observed

AC

< 0 > 0

Independent

A C

Leverage = 0

Nega>veCorrela>on NoAssocia>on Posi>ve

Correla>on

Independent

A C

Independent

A C

Observed

A C

-1… …1


GOAL:Findgeneralrulesthatindicatediabetes.

• Datasetofdiagnos>cmeasurementsof768pa>ents.

• Eachpa>entlabelledTrue/Falsefordiabetes.

MedicalRisk


MedicalRiskAssocia1onRule

If plasma glucose > 146 then diabetes = TRUE

DecisionTree

If plasma glucose > 155 and bmi > 29.32 and diabetes pedigree > 0.32 and insulin <= 629 and age <= 44

then diabetes = TRUE


Par1alDependencePlots

VisualizeEnsembles


FlatlineEditor

hPps://github.com/bigmlcom/flatline

https://github.com/bigmlcom/flatline


DecisionTreesBaggingDecisionForest

22

BigMLWorkflow

MODEL

DATASET

CLUSTER

ANOMALY

ASSOCIATION

SOURCE

K-MeansG-Means

Isola>onForest

MagnumOpus

DATASET

FlatlineFlatlineEditor


Logis1cRegression

DATASET LOGISTIC REGRESSION

• Classifica>onalgorithm

• Categorical:one-hotencoded

• Text:mappedtotokenfreq

• Bindingssupportlocalmodel

• I1/I2regulariza>on

• CurrentlyAPIonly

hPps://bigml.com/developers/logis>cregressions

https://bigml.com/developers/logisticregressions


DecisionTreesBaggingDecisionForestLogis>cRegression

24

BigMLWorkflow

MODEL

DATASET

CLUSTER

ANOMALY

ASSOCIATION

SOURCE

K-MeansG-Means

Isola>onForest

MagnumOpus

DATASET



BigMLClassifiers

Advantages Disadvantages

SingleTree easytointerpretrobusttomissingdata overfiong

Ensemble topperformerrobusttomissingdata hardtointerpret

Logis1cRegression robusttonoiseoutputsprobability

nomissingdatahardtointerpret



26

BigMLWorkflow

MODEL

DATASET

CLUSTER

ANOMALY

ASSOCIATION

SOURCE

K-MeansG-Means

Isola>onForest

MagnumOpus

Sta>s>calTestsCorrela>ons

STATSDATASET



Correla1ons

DATASET CORRELATION

• PearsonCoefficient

• SpearmanCoefficient

• Chi-Square

• Cramér'sV

• Tschuprow'sT

• One-wayANOVA

hPps://bigml.com/developers/correla>ons

https://bigml.com/developers/correlations


Sta1s1calTests

DATASET STATISTICAL TESTS

• Benford’sLaw

• Anderson-Darling

• Jarque-Bera

• Z-score

• Grubbs

hPps://bigml.com/developers/sta>s>caltests

https://bigml.com/developers/statisticaltests



29

BigMLWorkflow

MODEL

DATASET

CLUSTER

ANOMALY

ASSOCIATION

SOURCE

K-MeansG-Means

Isola>onForest

MagnumOpus

Sta>s>calTestsCorrela>ons

STATSDATASET



Q&A

•Askques1onsandgetaFreeBigMLT-shirt!

•Alldemonstratedfeaturesareimmediatelyavailabletoallusersincluding:•Allsubscrip1onplans•VirtualPrivateCloud(VPC)customers•On-premiseimplementa1ons.

•Documenta1on@hRps://bigml.com/releases

https://bigml.com/releases


FEEDBACK

@bigmlcom TWITTER

[email protected]

GetStartedToday!

RESOURCES Join us for future webinars & hangouts

OFFICE HOURS

Every Wednesday 9:30am Pacific Time

mailto:[email protected]

bigml fall 2015 release

Data & Analytics