cognitive graph in practice with their applications in ... · 10. anrl: attributed network...

30
Cognitive Graph in Practice with Their Applications in Ecommerce Recommendation Dr. Hongxia Yang For Discovery, A dventure, Momentum and Outlook Senior Staff Data Scientist

Upload: others

Post on 28-May-2020

11 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Cognitive Graph in Practice with Their Applications in

Ecommerce Recommendation

Dr. Hongxia Yang

For Discovery, Adventure, Momentum and Outlook

Senior Staff Data Scientist

Page 2: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Part 1

01 Motivation

Part 2

02 Graph Neural Network in Practice

CONTENTS

Part 3

03 Cognitive Graph

Page 3: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Part 1

01 Motivation

Page 4: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Cognitive Recommendation

• What is recommendation based on cognitive reasoning? Moisturizing vs Girls born after 90s need moisturizing if staying up late

• Why cognitive reasoning? • Supermarket (Initial) -> Personalized Shelf (Thousand People Thousand Faces) ->

Personalized Shelf with AI Shopping Guide (Cognitive Reasoning)

Content Display Common Sense Knowledge Graph +

GNN As Reasoning Engine

Fashion Icon Articles to connect KG and

common sense

Recommendation based on

Cognitive Reasoning

Content

Commercialization Generate recommended reasons and

recall related videos

Video and Articles product detection

Page 5: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

DEMO

Page 6: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Cognitive Intelligence Algorithm Platform

GNN Build Personalized Super Brain

Capable of inductive reasoning, explainable, better know consumers

than themselves

Alibaba Economy Big

Data Platform Lead Consumers Lives

Large Scale Graph Neural Network Oriented

Material, Entertainment and Health

Text, image and video influence the mind of

consumers

Page 7: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Part 2

02 Graph Neural Network in

Practice

Page 8: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Graph Embedding

Shallow Embedding Methods

Matrix Factorization Based Methods

• [Belkin et al. NIPS2002] • [Ahmed et al. WWW2013] • [Cao et al. KDD2015] • [Ou et al. KDD2016]

Random Walk Based Methods

• [Perozzi et al. KDD2014] • [Grover et al. KDD2016]

Deep Embedding Methods

Graph Autoencoder

• [Cao et al. AAAI2016] • [Wang et al. KDD2016]

Graph Convolutional Network

• [Bruna et al., ICLR2014] • [Duvenaud et al., NIPS2015] • [Kipf&Welling, ICLR2017]

Graph-Gated Recurrent Unit

• [Li et al. ICLR2018] • [Yan et al. AAAI’2018] • [Wu et al. IJCAI’2019] • [Guo et al. AAAI2019]

GE Family

No parameters sharing • Simply an embedding lookup based on arbitrary node id Fail to leverage node attributes • E.g., user profiles on a recommendation task Transductive model • The cold start problem

Difficult to deal with large scale graphs • The input dimension to the autoencoder is fixed

at |V| Cannot cope with unseen nodes • The structure and size of the autoencoder is fixed

Page 9: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Extremely Large Scale Attributed Heterogeneous Graphs with Billions of

Nodes and Trillions of Edges

Unified Graph Embeddings for Both Various Topological and

Attributed Spaces Practical Challenges and Our Current

Focuses

Sampling: Representative and

Negative Samples

Multiplex: Node and Edge Heterogeneities

Mixture Modes: Users with Different Latent Interest Categories

Hierarchical: Users’ community structures and

items’ categories

A GE Oriented Recommendation System in Practice

Page 10: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

GNN Framework Overview

Page 11: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

System Optimizations

Graph partitioning and clustering

Co-locate embedding variables with graph partitions

Reuse embedding if possible

Embedding h

Page 12: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

TensorFlow Heterogeneous Graph

Computing CPU GPU MPI

RAW DATA PROCESS DISPATCH &

LOAD TRAINING INFERENCE OUTPUTS FEEDBACK

WORKER

PARAMETERS GRADIENTS

Extremely

Large

Scale

Graph

Represent-

ation

Platform

PARALLEL LEARNING AND INFERNCE EXTREMELY LARGE SCALE GRAPHS

Billions of nodes, trillions of edges

GNN Platform Overview

Node

prediction

Edge

prediction Community

detection

Dynamic

graph

Data Sources

Load

Store Graphs

Built in

Algo

Sample

Aggregate Merge

Apply

Pattern

Recognition

WORKER

WORKER

PS

PS

PS 1. Rich algorithm warehouse with improved accuracy measures (5%-90%) for practical

challenges;

2. Performs an order of magnitude faster in terms of graph building, e.g., 492.90 million

vertices, 6.82 billion edges and rich attributes, 2 minutes vs hours by other state-of-

the-arts;

3. 40%-50% faster with the novel caching strategy and demonstrates around 12 times

speed up with the improved runtime.

AliGraph: A Comprehensive Graph Neural Network Platform. 45th International Conference on Very Large Data Bases (VLDB), 2019.

Page 13: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Various GE/GNN Methods

Page 14: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Multiplex Representation Learning for Attributed Multiplex Heterogeneous Network, KDD 2019

Page 15: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

• Transductive Model:

• Inductive Model:

Page 16: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection
Page 17: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Mixture GNN Is a Single Vector Enough? Exploring Node Polysemy for Network Embedding, KDD 2019

Page 18: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Hierarchical GNN Hierarchical Representation Learning for Bipartite Graphs, IJCAI 2019

Page 19: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Evolving GNN Large Scale Evolving Graphs with Burst Detection, IJCAI 2019

Page 20: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Bayesian GNN

• BEM is proposed to bridge KG and BG seamlessly, with the consideration of the behavior-specific bias. This framework provides a new perspective of making a reasoning mechanism (cognitive graph).

• As a method, BEM is generic and flexible in that it can use any KG embeddings to correct any BG embeddings. On the contrary, it is potentially able to help KG embeddings acquire novel knowledge from the BG embeddings that does not exist in the knowledge graph.

Refining Representation by Integrating Knowledge Graphs and Behavior-specific Networks, CIKM 2019

Page 21: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

1. Representation Learning for Attributed Multiplex Heterogeneous Network. 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD),

2019.

2. Is a Single Vector Enough? Exploring Node Polysemy for Network Embedding. 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD),

2019.

3. Towards Knowledge-Based Personalized Product Description Generation in E-commerce. 25th ACM SIGKDD Conference on Knowledge Discovery and Data

Mining (KDD), 2019.

4. Sequential Scenario-Specific Meta Learner for Online Recommendation. 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD),

2019.

5. AliGraph: A Comprehensive Graph Neural Network Platform. 45th International Conference on Very Large Data Bases (VLDB), 2019.

6. Large Scale Evolving Graphs with Burst Detection. 28th International Joint Conference on Artificial Intelligence (IJCAI), 2019.

7. Hierarchical Representation Learning for Bipartite Graphs. 28th International Joint Conference on Artificial Intelligence (IJCAI), 2019.

8. Cognitive Graph for Multi-Hop Reading Comprehension at Scale. 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019.

9. Subgraph-augmented Path Embedding for Semantic User Search on Heterogeneous Social Network, WWW, 2018.

10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018.

11. Adversarial Detection with Model Interpretation. KDD, 2018.

12. SPARC: Self-Paced Network Representation for Few-Shot Rare Category Characterization. KDD, 2018.

13. Mobile access record resolution on large-scale identifier-linkage graphs. KDD, 2018.

14. Interactive Paths Embedding for Semantic Proximity Search on Heterogeneous Graphs. KDD, 2018.

15. PRRE: Personalized Relation Ranking Embedding for Attributed Network. 27th ACM International Conference on Information and Knowledge Management

(CIKM), 2018.

17. Heterogeneous Embedding Propagation for Large-scale E-Commerce User Alignment, 2018 IEEE International Conference on Data Mining (ICDM), 2018

18. Local Algorithm for User Action Prediction Towards Display Ads. 23rd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2017.

19. Hybrid Framework for Text Modeling with Convolutional RNN. 23rd ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2017.

20. Bayesian Heteroscedastic Matrix Factorization for Conversion Rate Prediction. 26th ACM International Conference on Information and Knowledge

Management (CIKM), 2017.

21. Will Triadic Closure Strengthen Ties in Social Networks?, ACM Transactions on Knowledge Discovery from Data (TKDD), 2017.

Selected Publications

Page 22: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Part 3

03 Cognitive Graph

Page 23: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Cognitive GNN Towards Knowledge-Based Personalized Product Description Generation in E-commerce, KDD 2019

Page 24: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Cognitive GNN Cognitive Graph for Multi-Hop Reading Comprehension at Scale, ACL 2019

Page 25: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Causality/GNN/Disentanglement

Causality

GNN Disentangle Disentangled RepLearn on Graphs

Page 26: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

• Representation learning (RepLearn):

• To infer 𝒁 that explains 𝑿, usually via a generative model 𝑃 𝑿 ∣ 𝒁 .

• Causal discovery:

• To infer the causal mechanism 𝑃 𝑿 ∣ 𝑑𝑜 𝒁 , when given 𝒁 and 𝑿.

• Disentangled RepLearn[Definition from Suter et al., 2019]:

• To infer (1) 𝒁 that causes 𝑿, and (2) the causal mechanism 𝑃 𝑿 ∣ 𝑑𝑜 𝒁 .

• Moreover, (3) 𝒁 should contain independent, direct causes of 𝑿.

RepLearn

Generative RepLearn

Causal Discovery

Disentangled RepLearn

Robustly Disentangled Causal Mechanisms. Suter et al., ICML 2019.

Causality vs Disentanglement

Page 27: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

• Anti-causal learning: To infer the cause given the effect.

• E.g., to predict a user’s preference (cause) given his/her behavior (effect).

• E.g., to predict lung cancer (cause) given a lung image (effect).

On Causal and Anticausal Learning. Schölkopf et al., ICML 2012. Generalization in anti-causal learning. Kilbertus et al., arXiv 1812.00524.

𝒁 𝒁

𝑿

𝒇 𝒁 = 𝑿

Causality vs RepLearn

Page 28: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

• All these representations can achieve the maximum likelihood.

• Disentangled representation I: 𝑃𝜃𝐴(𝐴) and 𝑃𝜃𝐵∣𝐴(𝐵 ∣ 𝐴) .

• Disentangled representation II: 𝑃𝜃𝐵(𝐵) and 𝑃𝜃𝐴∣𝐵(𝐴 ∣ 𝐵) .

• But if 𝐴 → 𝐵 is true, then 𝑃𝜃𝐴 𝐴 ⋅ 𝑃𝜃𝐵∣𝐴(𝐵 ∣ 𝐴) needs much fewer new samples to adapt to the new distribution once 𝑃(𝐴, 𝐵) shifts.

• Because we need to update only 𝜃𝐵|𝐴.

It adapts faster if correctly disentangled.

This is a new approach to infer the causal direction!

Causality vs RepLearn

Page 29: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection

Thanks

Page 30: Cognitive Graph in Practice with Their Applications in ... · 10. ANRL: Attributed Network Representation Learning via Deep Neural Networks, IJCAI, 2018. 11. Adversarial Detection