how gpus power comcast's x1 voice remote and smart video...

How GPUs Power Comcast's X1 Voice Remote and Smart Video Analytics

Jan NeumannComcast Labs DCMay 10th, 2017

2

Comcast Applied Artificial Intelligence Lab

SmartHome

SmartTV

SmartInternet

Media&

VideoAnalytics

DeepLearning

DataScience

Recommendations

&Search

Voice&

NLP

3

Today: How Comcast Uses AI to Evolve and Reinvent the TV Experience

Smart Home

SmartTV

SmartInternet

Media&

VideoAnalytics

DeepLearning

DataScience

Recommendations

&Search

Voice&

NLP

4

Netflix

LIVETV

OnlineVideo

AI for Content Discovery –Voice Search

5

• Query: “HBO”

X1 Smart TV with Voice

AnswerSelector

Voice remote ASR

query

NLP modules

action

Set-top Box TV

6

Open NLP: Multiple Domains with Voice

TV

HOME

.

.

.

queryDomain

Selector

Answer

Selector

.

.

.

Answer

Selectorresponse

CUSTOMERCARE

NEWS

7

Open NLP: Multiple Domains with Voice

TV

HOMEturnontheheat

Domain

Selector

Answer

SelectorAnswer

Selector

CUSTOMERCARE

NEWS

response

0.80

0.15

0.02

0.03

Selected={TV,Home} Precision=100%

Applicable={TV,Home} Recall=100%

Threshold=0.10

8

Open NLP: Multiple Domains with Voice

TV

HOME

Showmemypassword

Domain

Selector

Answer

SelectorAnswer

Selector

CUSTOMERCARE

NEWS

response

0.03

0.04

0.03

0.90

Selected={CustomerCare} Precision=100%

Applicable={CustomerCare} Recall=100%

Threshold=0.10

9

• Cascade of Deep Learning Models of increasing complexity

Domain Selector in Practice

Entity Detection

Service“HBO”

SimpleModel

ComplexModel

SENDTODOMAIN

DONOTSENDTODOMAIN

YES

YES

NO YES

NO

NO

10

SENDTODOMAIN

NO

• Cascade of Deep Learning Models of increasing complexity

Domain Selector in Practice

Entity Detection

Service

“Show me funny

comedies”

SimpleModel

ComplexModel

DONOTSENDTODOMAIN

YES

YES

NO YES

NO

11

• Query: “who plays the oracle in matrix”

X1 Smart TV with Voice

Voice remote

query action

QA Answer (id or text)

Question(text)

ASR NLP modules Set-top Box TV

12

• Given:• Question in natural-language form q• Structured knowledge base that contains list of facts• [ subject – relation – (attribute) – object ]

• Return: • Answer to q

• Assuming:• q answerable by a single fact.• Source entity mentioned in q.• Answer is neighbor of source entity node.

First-order Question Answering

subject object

attribute

“Matrix” “Keanu Reeves”

“Neo”

“Tom Hanks” “9/1/1956”

13

Question Answering with Knowledge Graph

Predict

Relation

Question

Extract

Entities [e1,…,eN ]names/titles

Structured

Query

Subj=e1Obj=?Rel=r

Knowledge

Graph

Search

e1 |r |e2

relation

r

Generate

Answer

TextanswerTrain

subj |rel |obj

Howoldis

TomHanks?

14

Question Answering with Knowledge Graph

Predict

Relation

Question

Extract

Entities [e1,…,eN ]names/titles

Structured

Query

Subj=e1Obj=e2Rel=r

Knowledge

Graph

Search

e1 |r |e2

relation

r

Generate

Answer

TextanswerTrain

subj |rel |obj

[e1,…,eN ]names/titles

Subj=TomHanksRel=birthObj =?relation

r

TomHanks

is55years

old.

birth

Tom

Hanks TomHanks|birth|1956

TomHanks

is59yearsold

Howoldis

TomHanks?

15

EntityDetection [ e1, …, eN ]

names / titles

PredictRelation

relationr

subj=eobj=?attr=?rel=r

Question Answering with Knowledge Graph using Recurrent Neural Networks (RNNs)

StructuredQuery

Question

where Tom Hanks was

placeof birth

born

memory

where Tom Hanks was

NA Subj Subj NA NA

born

mem

ory Entity Detection ~ Tagging Relation Prediction ~ Classification

16

word

hidden

input

output

0.39 0.61

washingtonheights

0.89 0.11

memory

Recurrent Neural Networks

LOC PER PERLOC

17

Netflix

LIVETV

OnlineVideo

AI for Content Discovery – Automatic Content Analysis

18

Most metadata is at the asset level

• Genres• Credits• Synopsis• Keywords

19

Much more data exists within the asset

• Chapters• Moments• Annotations

MovieFrameShotScene

Chapter

20

Why is this useful?

Whoisinthis

scene?

Whatarethebest

momentsonTV?

In-game

highlight

navigation

Search&

Recommendations

21

How does Automatic Content Analysis work?

ComputerVision

Audio Analysis

NaturalLanguage

Processing

AI &Machine Learning

Chaptering

Scene-levelAnnotations

Video

Frame-levelAnnotations

22

Why is it possible now?

Large-scale Image recognition performance

Big

Data

Better

Algorithms

(Deeplearning)

Cloud/GPU

Computing

23

Super-human accuracy in speech and image recognition!

Large-scale Image recognition performance

Big

Data

Better

Algorithms

(Deeplearning)

Cloud/GPU

Computing

24

New experiences!

Big

Data

Better

Algorithms

(Deeplearning)

Cloud/GPU

Computing

25

• Place highlights over games recorded onto customers’ DVRs for football, baseball, hockey, basketball and soccer.

Example Application: In-Game Highlights

“I’llrecordasmanygamesasIcan.WhenIdon’twanttowatchthewholegame,it’sagreatwaytodoit.”– CustomerTestimonial

“In-GameHighlights”

FeatureforNFLhasbeen

releasedonComcastX1

lastfall

26

Netflix

LIVETV

OnlineVideo

AI for Content Discovery – Personalization

27

+

=

Personalized Entertainment Experiences

What is popular right now? What do you like?

PersonalizedRecommendations

28

Deep learning-based recommender system for Live TV - Training a joint embedding space to combine the scores- Channel- and Program-based recommendations

- Time-dependent recommendations

- Trending/popular and personal favorite channels, programs, sport teams

- Rich content descriptions from automatic content analysis

What should I watch right now?

Live TVRecommenderSystem

Favorite

Channels

Favorite

Programs

Collaborative

Filtering

Trending

Popularity

Content

Descriptions

29

Netflix

LIVETV

OnlineVideo

Deep Learning Infrastructure

30

• Deep Learning Frameworks – Keras, Tensorflow, Theano, PyTorch, Caffee (older models)

• All deployments using nvidia-docker– Thanks to Nvidia solutions team to help with best practices

• All deep learning training done on multi-GPU servers– NvidiaTesla (Production) and 8xTitan X (Dev) GPUs– Nvidia DGX-1 for large scale training – video and nlp

• Next steps– Container scheduler – Kubernetes and Hashicorp Nomad– Network compression/simplification for increased efficiency (TensorRT)

Deep Learning Infrastructure

31

Machine Learning Data Science

Big Data AI

Improving Customer

Experience Everywhere at

Comcast/NBCU

Deep Learning-based ML is applied everywhere at Comcast

HighSpeedInternet

Video

IPTelephony

HomeSecurity/

Automation

UniversalParks

MediaProperties

Formoreinfosee:

dclabs.comcast.com