1 irene li , alexander fabbri , swapnil hingmire and
TRANSCRIPT
![Page 1: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/1.jpg)
R-VGAE: Relational-variational Graph Autoencoder for Unsupervised Prerequisite Chain Learning
Irene Li1, Alexander Fabbri1, Swapnil Hingmire2 and Dragomir Radev1
1LILY Lab, Yale University, USA2Tata Consultancy Services Limited (TCS), India
COLING'2020
![Page 2: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/2.jpg)
Motivation
With the increasing amount of information available online, there is a rising need for structuring how one should process that information and acquire knowledge efficiently in a reasonable order.
Concept prerequisite chain learning: automatically determining the existence of prerequisite relationships among concept pairs.
![Page 3: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/3.jpg)
Motivation: Concept Graph
GraphConvolutional
NetworksVariational
Autoencoders
VariationalGraph
Encoders
Autoencoders
Laplacian Matrix
Convolutional Networks
Concept Graph as a directed graph.
Concept is a prerequisite topic of concept
Concept Variational Autoencoders is a prerequisite of the concept Variational Graph Autoencoders.
![Page 4: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/4.jpg)
Related WorkPrerequisite Chain Learning
Feature engineering and classifiers to measure the relation of a given concept pair: (Liang et al., 2018; Liu et al., 2016; Liang et al., 2017)
Materials including university course descriptions and materials as well as online educational data: (Liu et al., 2016; Liang et al., 2017)
Our previous work Li et al. (2019): a dataset containing 1,352 English lecture files collected from university-level lectures as well as 208 manually-labeled prerequisite relation topics (LectureBank1.0). We applied GAE and VGAE (Kipf and Welling, 2017) to predict relations in a concept graph.
Deep Models for Graph-structured DataNode representations: Deepwalk (Perozzi et al., 2014), Node2vec (Grover and Leskovec,
2016) GCN models: Graph Convolutional Networks (Kipf and Welling, 2017), TextGCN (Yao et al.,
2018): text classification using GCN.
![Page 5: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/5.jpg)
Method: A Resource-Concept Graph for Link Prediction
Two node X types:Concept Node (c)Resource Node (r)
Three edge types:✔Concept-Resource✔Resource-Resource? Concept-Concept
Slide resource: https://slazebni.cs.illinois.edu/spring17/lec12_vae.pdf
![Page 6: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/6.jpg)
PreliminariesGCNs Graph Convolutional Networks (Kipf and Welling, 2017)
A two-layer GCN is defined as:
R-GCNs Relational Data with Graph Convolutional Networks (Schlichtkrull, Michael, et al. 2018)
![Page 7: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/7.jpg)
Preliminaries V-GAEs Variation Graph Autoencoders (Kipf, Thomas N., and Max Welling, 2016)
VAE to graph-structured data; unsupervised learning via reconstructing the adjacency matrix.
Generate adjacency Matrix A
GCN as Encoder: input X, A
Hidden Layer: Z
Reconstructed A
dot -product
~ N(0,1)
~ N(0,1)
![Page 8: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/8.jpg)
Proposed Model: R-VGAEEncode nodes using R-GAE:
Training - DistMult (Yang et al., 2014)Take the last layer node representation:
![Page 9: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/9.jpg)
Proposed Model: R-VGAENode Feature (X)
Sparse: TFIDFDense: Phrase2vec (Artetxe et al., 2018) and BERT (Devlin et al., 2018)
Adjacency Matrix (A) Cosine similarity based on TF-IDF features;Predict : 1 if there is a prerequisite relation; 0 otherwise.
Semi-supervised and unsupervised for learning prerequisite chains: Unsupervised if no concept-concept relations are given; semi-supervised
if a part of them are given in the training stage.
![Page 10: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/10.jpg)
Lecturebank 2.0: A collection of English slides in NLP-related classes
Compared with Lecturebank 1.0: added more than 500 slide files and expanded the number of topics from 208 to 322, including the prerequisite annotations.
![Page 11: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/11.jpg)
Experimental Results
Accuracy, Macro-F1, MAP and AUC
Classifier: Support Vector Machines, Logistic Regression, Naive Bayes and Random Forest (report the best)
Compared with mainly graph-based models: node embedding and GCN-based methods.
Our method is able to train in both a semi-supervised and unsupervised way.
![Page 12: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/12.jpg)
Analysis: case study
![Page 13: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/13.jpg)
Conclusion● We propose an unsupervised learning method which makes use of advances
in graph-based deep learning algorithms. Experimental results demonstrate that our model performs well in an unsupervised setting and is able to further benefit when labeled data is available.
● We introduce an expanded dataset for prerequisite chain learning with additional an 5,00 lecture slides, totaling 1,717 files. We also provide prerequisite relation annotations for each concept pair among 322 concepts.
● In the future we plan to perform a more comprehensive model comparison and evaluation by bringing other possible variations of graph-based models to learn a concept graph.
![Page 14: 1 Irene Li , Alexander Fabbri , Swapnil Hingmire and](https://reader034.vdocument.in/reader034/viewer/2022051517/627f224a379d854dc875a174/html5/thumbnails/14.jpg)
ThanksQ&A