AI Modernization at AT&T and the application to Fraud with DatabricksMark Austin, VP Data Science, AT&TPrince Paulraj, AVP, Data Insights, AT&T

Agenda

§ AT&T’s History in AI

§ Fraud AI Application§ AI Modernization & Strategy

▪ Create AI ▪ Deploy & Serve AI▪ Monitor AI ▪ Govern AI

§ Conclusion and Opportunities

01

02

03

04

05

AT&T’s: History in Transformative AI/Data Science…

1950

1955“Artificial

Intelligence” term first coined, AT&T, IBM,

Harvard, Dartmouth

Shannon, AT&T Bell Labs “Programming a Computer to Play

Chess”

1970’sUnix, C,C++, and

Statistical Programming (S) which becomes

1980’s-90’sNeural Network

foundational work on Conv Neural Nets,

AT&T

2000’sAT&T Wins Netflix Recommender Competition

AI on Tech, Media, Telecom

AppliedAI/ML

3

Mobility Fraud: Organized Crime Stealing “iPhones”

4

Identity Theft:Identities stolen via social engineering or otherwise and used to obtain handset to resell overseas

Gaming Fraud:“Gaming Customer” has no intention to pay and uses their “credit” to sell new iPhone (and other devices) to fraud crime ring

Illegal Unlocks:Bribing or impersonating Call center employees to unlock phones that are under contract

Retail, Care, Digital

xxxx

_password

login

BILLION DOLLAR Mobility Fraud Industryaffecting all US Carriers

Combatting Fraud using Realtime ML/AI

5

Year 1: Fraud stops using rules only

Year 3: Fraud stops using AI/ML (+25 Models)

Deploying Real-Time AI/ML in addition to Rules is Effective

Year 2: Fraud stops using AI/ML (5 Models)

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Frau

d Ev

ents

NO

TSt

oppe

d

2018 2019 2020

Rules only

ML+Rules

Year 1 Year 2 Year 3

ML1

ML2

ML3

ML4 ML5ML6

ML7-8ML9

ML10ML11-12

ML13 -14

ML15-16

ML17-18

ML19-20

ML21-22

ML23-26

Combatting Fraud using Realtime ML/AI

The Technical Challenge:• Scoring >10M transactions/day• Scoring latency < 50ms• 100’s of real-time features• >4x as many batch features

6

DevelopFeatures

Productionize Model Pipeline

Discover Data Build Models Deploy

Model Integrate

Model Monitor Model

Can you put these features into production?

Can you put this model into production?

Can you integrate this model and app?

Can you setup monitoring?

Can you give me the access to the raw data?

Can I use this data for this model?

Create AI Deploy & Serve AI Monitor AI

AIaaS - AI Modernization using

Can we visualize model outcome & insights

Can you build the best model?

Govern AI

Can you get the data fast?

Can you notify for model retraining?

Create AIGetting the best features and models…

▪ Batch data pipeline

▪ Near real-time data pipeline

▪ Streaming feature pipeline

▪ Speed to market

▪ Code once and model many

▪ Merged or derived features sets across enterprise

▪ Collaboration

• Share/Reuse Features• Creating Features

▪ Model and hyperoptexperiment scorecard

▪ Model Repository

• Experiment & Catalog

▪ Transformative features (autoML)

• Best Model Enterprise

Atlantis The Enterprise Feature Store

Model Benchmarking by Individuals & Robots

Model Benchmarking Amongst “the Crowd”7

8

Pinnacle crowd sourcing Competitive/collaborative internal platform

Pinnacle– AI as a team sport yields ~29% improvement Pinnacle by the numbers….

219 Competitions

1,101 people

4 Automl bots

29% avg. improvement!

>4,750 models benchmarkedML

8

Pinnacle crowd sourcing Competitive/collaborative internal platform

Pinnacle– AI as a team sport yields ~29% improvement Pinnacle by the numbers….

219 Competitions

1,101 people

4 Automl bots

29% avg. improvement!

>4,750 models benchmarkedML

Deploy & Serve AIDeploying models to score at 10M+/day @50ms latency

▪ Model Lineage and Versioning

▪ A/B Model Framework

▪ Model Lifecycle management

▪ Higher throughput and scalability

▪ Feature versioning

▪ Time travel model evaluation

▪ Backfilling features

• Model Offline Training• Model Deployment

▪ Lightening fast

▪ High scalability

▪ Consistent features

▪ High availability

▪ Feature Time to live

• Model Online Scoring

▪ Metadata management

▪ Discoverability

▪ Access Control

▪ Feature Health & statistics

▪ Compliance & Legal

• Feature Governance

AtlantisThe Enterprise Feature Store

9

10

Atlantis – Multi-Pipeline Feature Store for Machine Learning

Databricks ML Pipeline

Snowflake ML Pipeline

Pinnacle ML Pipeline

H2O Driverless ML Pipeline

Feature Engineering Pipelines

Offline/Online Feature Store

Real-Time Data

Batch Data

Raw Data

Model Scoring (mS)

Model Training

Jupyter ML Pipeline

Palantir ML Pipeline

11

Feature Store Benefits: Ensures the “same features” are used in training and serving, preventing serving ML loss

Training Features Serving Features

Blue (serving predicted)

Offline/Static N/A

Green (serving actual of Blue )

Offline/Static Online

Red (serving actual when train/serve in sync (Feature Store)

Online Online

Using Feature Store For Train/Serve Improves Lift ~2X @first decile

Monitor AINeeds to cover data, model, infra, and process

▪ Detecting Data Quality issues

▪ Data drifting -Feature value is missing or invalid

▪ Data readiness -Out of SLA

▪ Model Drifting

▪ Performance Issues

▪ Production Model Governance

▪ Visualization of model health

• Model• Data

▪ Out of SLA response▪ CPU, RAM, I/O

usage▪ Application, mS or

VM goes down▪ Network and

connectivity issues▪ Correlation of

system and model performance

• Infrastructure

▪ Drag and drop to create custom rules

▪ Auto remediations like re-training or rollback model version

▪ Workflow of actions

▪ Predict the Root cause

• Process

AI Watchtower – An end-to-end Data, and Models Monitoring & Decision engine

12

13

Watchtower – An end-to-end Data, Features and Models Monitoring Engine

Cases created for Mitigation, Alerts, Notifications

Apply ML models & Business Rules

Setup Monitoring

Real Time Data pipelines

Decision Engine

Any data pipelines into Event HubIs available for monitoring

Take Auto/Manual Actions

Downstream Systems/Feedback

Action & Remediation

How?

Drifting

DataRulesModelActionsBYO

Doing it in a Self-Service Way

Multi-tenant subscription

API Integration to platforms

Self-service experience

On-Perm & Cloud availability

Model 3 – Percent Scored

Model 4 - Percent ScoredModel 5 - Percent Scored

Model 6 - Percent ScoredModel 7 - Percent Scored

Model 1 – Percent ScoredModel 2 – Percent Scored

SIFT

Learn Document Evaluate

Use caseML Model

MLDB

ML Project Metadata

Data Catalog AIaaSModel Catalog

Model Metadata

Disparate impact

Debiased Model

Feature importance

Drift assessment

Bias Detection

Bias Mitigation

Explainability

Data Drift

OpenOpen-Source Tools

Vendor Tools

Output

Business decision

Privacy (PRR)

ProcessLegal

PrivacyProcess Docs

Govern AIUsing AT&T’s System for Investigating Fairness and Transparency (SIFT)

14

Atlantis Feature Store

Feedback

Your feedback is important to us.

Don’t forget to rate and review the sessions.