modern history for text summarization and beyondpfliu.com/talk/summarization.pdfsummarization and...

58
Modern History for Text Summarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com

Upload: others

Post on 28-May-2020

6 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Modern History for Text Summarization and Beyond

Pengfei LiuPostdoc at LTI of CMUpfliu.com

Page 2: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Outline

Overview of Modern History

Some Highlighted Topics

Our Recent Work

Page 3: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Outline

Overview of Modern History

Some Highlighted Topics

Our Recent Work

Page 4: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Preparation: Research Papers

Summarization Papers:

• Year: 2013-Now

• Conference: ACL / EMNLP / NAACL / ICML / ICLR / AAAI / IJCAI / NeurIPS

Page 5: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Preparation: Research Concepts

rich of task settings!

Page 6: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Holistic Analysis

0

5

10

15

20

25

2013 2014 2015 2016 2017 2018 2019

# Pa

persACL EMNLP NAACL

2013-2015: increasing!

2015-2016: trough!

2016-2019: rapid increasing

Page 7: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Fine-grained Analysis: Overview (2013-19)

Page 8: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

2013-2014 Task-setting Generation way

1) No NNs!2) First RL Paper3) Multimodal

Page 9: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Top10 Cited Papers

Page 10: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

2015 Neural Arch.Task-setting Generation way

1) Finally, NNs arrived!

2) New dataset for NN!

Page 11: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Top10 Cited Papers

Page 12: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

2016 ArchitectureTask-setting Generation way

1) Relatively more NNs2) Few papers

3) Important Techniques!

Page 13: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

2016 ArchitectureTask-setting Generation way

1) Relatively more NNs2) Few papers

3) Important Techniques!0

10

20

2013 2014 2015 2016 2017 2018 2019

# Pa

pers

ACL EMNLP NAACL

Page 14: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Top10 Cited Papers

CNNDM

Copy

Coverage

Copy

Neural Ext

Page 15: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Neural Arch.2017

2) VAE arrived!3) More NNs but not booming … lag behind …

1) GNNs arrived!

0

10

20

2013 2014 2015 2016 2017 2018 2019

# Pa

pers

ACL EMNLP NAACL

Page 16: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Top10 Cited Papers

Page 17: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

2018: booming!

1) Booming of NNs2) Booming of RL!3) Booming of Abs!

Page 18: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Top10 Cited Papers

Page 19: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

2019

1) Booming of Pre-training2) Booming of GNNs3) Booming of Evaluation

Page 20: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Top10 Cited Papers

Page 21: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Lessons from History

• The development of deep NNs lags behind other tasks

• Summarization tasks requires some customized techniques (e.g. copy)

• Only technique-ready is not enough … dataset also matters!

• A good match between “techniques” and “datasets”

Page 22: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Outline

Overview of Modern History

Some Highlighted Topics

Our Recent Work

Page 23: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Outline

Overview of Modern History

Some Highlighted Topics

Our Recent Work

FactualitySemantic equivalencePre-trained models

Page 24: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Evaluation Metric: Factuality

• Event factuality prediction• the degree to which an event mentioned in a sentence has happened

• Social Media• fake news detection

• Dialog• consistency

• Machine Translation• semantic divergence

• Summarization

Page 25: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Factuality in Text Summarization

• Motivation

Whether facts in the generated summaries can be covered

• GoalGiven a document , and generated summary: the purpose is to learn a function fact checker:

d 1 2g { , , , }

ms s s L

Fact(d,g)

Born in Honolulu, Hawaii, Obama is a US Citizen

Obama is American

Document Sentence?

Page 26: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

A Case Study on “FactCC” (Kryscinski et al.)

cnndm xsum arxiv pubmed bigpatent_b tifu_long0.413981 0.198641 0.438764 0.475539 0.637429 0.413868

Setting:

• Evaluating “FactCC” on (document, references) (Ideally, 100%)• Not good at predicting positive pairs.

Page 27: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

A Case Study on “FactCC” (Kryscinski et al.) Setting:

• Cross-dataset• Extractive model: higher factuality score (but not 100%)• Abstractive model: using a in-domain training set is not a

guarantee.

Page 28: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Factuality in Text Summarization

• Challenges

• How to define a fact? (sentence? triple?)

• How to evaluate the effectiveness of your proposed

factuality checker?

• Source documents are too long!!!

• Negative predictive power

Page 29: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

More Recent Progress …

• Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (Wang et al. 2020, ACL)

Page 30: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

More Recent Progress …

• Asking and Answering Questions to Evaluate the Factual Consistency of Summaries (Wang et al. 2020, ACL)

Page 31: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Evaluation Metric: Semantic equivalence

• Growing Trends:

• BLEURT: Learning Robust Metrics for Text Generation (Sellam et al. 2020, ACL)

• BERTScore: Evaluating Text Generation with BERT(Zhang et al. 2019)

• MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (Zhao et al. 2019)

• Pre-trained (+ fine-tuned) • Semantic Matching

• Learnable

• Examples:

• Fine-grained

Page 32: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Growing Trend: Fine-grained Evaluation

• Fine-grained Meta-evaluation

• Evaluation metrics behave similarly (average-scoring range) strongly disagree in the higher-scoring range (Peyrard et al.2019)

• Calculating correlation in different top-K systems (Ma et al.2019)

Page 33: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Unsupervised Pre-trained Models

Extractive

Abstractive

Summarization

NLU (BERT-family)

NLG (BART)

Pre-trained

Page 34: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Unsupervised Pre-trained Models• Extractive

• Extractive -> Abstractive

• Exploring Task-specific Loss

Page 35: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Outline

Overview of Modern History

Some Highlighted Topics

Our Recent Work

Page 36: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Current Work

Interpretable Analysis

Model Refining

(1) Explicitly modelling inter-sentence interaction MATTERS!

(2) Summary-level optimization is rewarding

(1) Heterogeneous Graph Neural Networks for ExtractiveDocument Summarization (ACL 2020)

(2) Extractive Summarization as Text Matching (ACL 2020)

Searching for Effective Neural Extractive Summarization: What Works and What’s Next (ACL 2019)

Page 37: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Interpretable Analysis: Why?

65

70

75

80

85

90

1 2 3 4 5 6 7 8 9 10 11

Change trend of scores on GLUE

38.539

39.540

40.541

41.542

Nal lapat i

et …

Narayan e

t…

Chen et…

Zhou et…

Xu et a

l .2019

Change trend of R-1 on a Text Summarization Dataset

The performances of many NLP tasks have begun to plateau

gradually flattening.

Page 38: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Interpretable Analysis: Why?

• Superior performance but low interpretability

• If the pros and cons are unknown, how could we:

• make suitable choices under different scenarios

• design more powerful methods

Page 39: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Interpretable Analysis: How?

• Explaining the prediction behavior

• Understanding the functionality of a neural component

• Revealing the darkness of pre-trained models

Page 40: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Methodology for Understanding NLP-oriented Models

Training-Testing environment:

• Training environment: different models are first generated with different specifications

• Testing environment : a model should be evaluated with different observed testbeds

Page 41: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Bring it into Text Summarization

• Training environment• Testing environment

Searching for Effective Neural Extractive Summarization: What Works and What’s Next (ACL 2019)

Page 42: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Bring it into Text Summarization

Page 43: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Bring it into Text Summarization

• Training environment

• Testing environment

Page 44: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Bring it into Text Summarization

Besides the “Rouge”

• Other metrics:• Positional bias• Repetition• Sentence length

• Evaluation testbeds:• Cross-domain evaluation (eight domains)• Sentence shuffling

observe from different aspects

Page 45: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Explicitly modelling inter-sentence interaction MATTERS!

The gap between datasets heavily influences the cross-datasets generalization

Summary-level optimization is rewarding

Takeaways

Page 46: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Explicitly modelling inter-sentence interaction MATTERS!

The gap between datasets heavily influences the cross-datasets generalization

Summary-level optimization is rewarding

Takeaways

Page 47: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Heterogeneous Graph Summarizer

Heterogeneous Graph Neural Networks for ExtractiveDocument Summarization (ACL 2020)

words

sentences

Sequential order

Page 48: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Heterogeneous Graph Summarizer

Heterogeneous Graph Neural Networks for ExtractiveDocument Summarization (ACL 2020)

Graph-structure

Page 49: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Heterogeneous Graph Summarizer

Heterogeneous Graph Neural Networks for ExtractiveDocument Summarization (ACL 2020)

Graph:• Node: word, sentence, document• Edge: tf-idf

Page 50: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Advantages

1) Aware of overlapping information

2) Both words and sentences keep themselves updated

3) Flexibly extended• Relay node: entity• Satellite node: document

Page 51: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Results

Page 52: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Explicitly modelling inter-sentence interaction MATTERS!

The gap between datasets heavily influences the cross-datasets generalization

Summary-level optimization is rewarding

Takeaways

Page 53: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Matching-based Summarization

Extractive Summarization as Text Matching (ACL 2020)

Extracting sentences as a sequence labeling problem

Page 54: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Matching Summarization

Extractive Summarization as Text Matching (ACL 2020)

1) paradigm shift with regard to the way we build neural extractive summarization systems

3) bypasses the difficulty of summary-level optimization (e.g. RL) by contrastive learning

2) a good summary should be more semantically similar to the source document than the unqualified summaries

Page 55: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Optimization Principles

Page 56: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Experiment

Page 57: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Beyond a SOTA result

Theoretical Analysis

• On what types of datasets, the expected gain of summary-level approach is large over sentence-level approach?

• And how to characterize the expected gain?

Outlook

• The power of matching framework has not been fully exploited

• Build the connections with learnable evaluation metrics:• BERTScore• MoverScore• BLEURT

Page 58: Modern History for Text Summarization and Beyondpfliu.com/talk/summarization.pdfSummarization and Beyond Pengfei Liu Postdoc at LTI of CMU pfliu.com. Outline Overview of Modern History

Thank you