learning to transform, combine, and reason in open-domain ... · source: lorem ipsum dolor sit...
TRANSCRIPT
![Page 1: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/1.jpg)
Proprietary + Confidential
Learning to Transform, Combine, and Reason
in Open-Domain Question Answering
Mostafa Dehghani, Hosein Azarbonyad, Jaap Kamps, and Maarten de Rijke
University of Amsterdam
1
![Page 2: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/2.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Open Domain Question Answering
● Users seek direct answers to complex questions from
large open-domain knowledge sources like the Web.
● Most open-domain question answering systems: ○ Use a search engine to retrieve a set of candidate documents.
○ Select one or a few of them as the context.
○ Apply reading comprehension models to extract answers.
2
![Page 3: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/3.jpg)
Example
3
![Page 4: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/4.jpg)
Example
3
![Page 5: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/5.jpg)
Example
3
![Page 6: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/6.jpg)
Example
3
![Page 7: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/7.jpg)
Example
3
![Page 8: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/8.jpg)
Some questions, however, require taking a broader
context into account, e.g., by considering low-ranked
documents that are not immediately relevant,
combining information from multiple documents, and
reasoning over multiple facts from these documents
to infer the answer.
4
![Page 9: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/9.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Need for considering a larger set of documents?
● This can potentially come at the cost of:
○ Efficiency, as we need to process a larger input.
○ Effectiveness, as there will be more noisy and non-relevant
documents when we go down the ranked list of candidate
documents.
5
![Page 10: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/10.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
● TraCRNet: Transform, Combine, and Reason
○ Pronounced “Tracker Net”!
○ Is based on Transformer architecture.
○ Efficiently operates over a larger set of candidate documents.
○ Combining the evidence from all documents during multiple steps of reasoning.
○ Robust against noise from low-ranked non-relevant documents.
6
TraCRNet!
![Page 11: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/11.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Recurrent Models
7
![Page 12: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/12.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Transformer (general idea!)
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is
all you need. In proceedings of Advances in Neural Information Processing Systems 2017.
8
![Page 13: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/13.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Transformer (general idea!)
Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is
all you need. In proceedings of Advances in Neural Information Processing Systems 2017.
8
![Page 14: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/14.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Universal Transformer (general idea!)
Dehghani M, Gouws S, Vinyals O, Uszkoreit J, Kaiser Ł. Universal transformers, In proceedings
of Seventh International Conference on Learning Representations 2019.
9
![Page 15: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/15.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
TraCRNet Architecture
document level encoding
token level encoding
10
![Page 16: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/16.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
TraCRNet Architecture
11
![Page 17: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/17.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
On the Architectural Choices
● Transformer-based (No recurrence!) ○ Easily parallelizable → FAST!
● Transformer Encoder for Input Encoding ○ Global receptive field for dealing with long documents (token level encoding)
○ Memorizing the meaning of infrequent words
12
![Page 18: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/18.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
On the Architectural Choices
● Universal Transformer for Multi-hop Reasoning ○ Recurrence in depth → a strong inductive bias that enables the model to learn
iterative or recursive transformations → chain of reasoning!
● Transformer + Universal Transformer ○ Encoding local information + Combining global information
○ Enough memorization + Ability of generalization.
13
![Page 19: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/19.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Experiments Datasets
● Two datasets
○ SearchQA ■ 140k question-answer pairs crawled from J!Archive, and augmented with text snippets retrieved
using the Google search engine.
○ Quasar-T ■ 43k open-domain trivia questions and their answers obtained from various internet sources. The set
of candidate documents for each question is retrieved using Lucene from the ClueWeb09 corpus as
the background corpus.
14
![Page 20: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/20.jpg)
Experiments Main Results
● TraCRNet has the chance of using
“full'' use of the information of “all''
the candidate documents.
● TraCRNet generates the answer
instead of extracting it.
15
![Page 21: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/21.jpg)
Experiments Effect of multihop reasoning
○ The decoder has access to
document-level representations from
the encoder.
○ The pooling operation is removed and
the decoder has access to word-level
representations from the encoder.
16
![Page 22: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/22.jpg)
Back to our Example
17
![Page 23: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/23.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Multihop Reasoning
18
![Page 24: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/24.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Multihop Reasoning
18
![Page 25: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/25.jpg)
Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem
Multihop Reasoning
18
![Page 26: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/26.jpg)
Experiments Impact of number of documents
● Train and evaluate models using different
numbers of candidate documents associated
with each question.
● This shows that, for some questions, the
correct answer can only be extracted when
considering information from low-ranked
documents that are not immediately relevant
to the question.
19
![Page 27: Learning to Transform, Combine, and Reason in Open-Domain ... · Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem TraCRNet: Transform, Combine, and](https://reader034.vdocument.in/reader034/viewer/2022050415/5f8be24b69c8643dbe06094e/html5/thumbnails/27.jpg)
THANK YOU
20