![Page 1: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/1.jpg)
Learning with Latent LanguageJacob Andreas, Dan Klein, Sergey Levine
CS330 Student Presentation
![Page 2: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/2.jpg)
MotivationThe structure of natural language reflects the structure of the world.
The authors propose to use language as a latent parameter space for few-shot learning.
Experiment with tasks including classification, transduction and policy search.
They aim to show that this linguistic parameterization produces models that are both more accurate and more interpretable than direct approaches to few-shot learning
![Page 3: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/3.jpg)
MethodMethods for training is 2 fold:
1. Encoder - Decoder model for learning language representations2. Classic few shot meta learning models
Authors import the relevant structure for problem solving from the first stage and utilize that in the second.
They achieve this in 3 steps:
1. Model Pre-Training / Language Learning Phase2. Concept Learning Phase3. Evaluation Phase
![Page 4: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/4.jpg)
Method - PreTraining/ Language Learning1. Involves pre-training a language
model on specific subtasks using natural language parameters “w”
2. A language interpretation model is also learned to turn a description w into a function from inputs to outputs.
3. These natural language parameters are only observed at language-learning time.
![Page 5: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/5.jpg)
Method - Concept Learning1. The pretrained model is adapted to
fit data for a specific new task2. Model generates natural language
strings ‘wc’3. These are sampled from the model
as approximations to the distribution of descriptions, given the task data.
4. By sampling from the pre-trained model, candidate descriptions are likely to obtain small loss.
![Page 6: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/6.jpg)
Method - Evaluation1. At evaluation time the hypothesis
‘wc’ that hopefully obtains the lowest loss is selected, and applied to a new task, i.e new input x to predict y.
![Page 7: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/7.jpg)
Experiments1. Few-Shot image classification2. Programming by demonstration3. Policy search
![Page 8: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/8.jpg)
Experiments: few-shot image classification
![Page 9: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/9.jpg)
Experiments: few-shot image classificationf performs task conditioned on task representationq generates task representation as English sentence
![Page 10: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/10.jpg)
Experiments: few-shot image classification
q
![Page 11: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/11.jpg)
Experiments: few-shot image classification
f
![Page 12: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/12.jpg)
Experiments: few-shot image classification
qw
![Page 13: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/13.jpg)
Experiments: few-shot image classification
f
w
prediction
![Page 14: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/14.jpg)
Experiments: few-shot image classification
![Page 15: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/15.jpg)
Experiments: programming by demonstration
![Page 16: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/16.jpg)
Experiments: programming by demonstration
![Page 17: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/17.jpg)
Experiments: policy search- Use latent language for structured exploration- Imitation learning with expert trajectories
![Page 18: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/18.jpg)
Experiments: policy search- Unconditioned q!
![Page 19: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/19.jpg)
Experiments: policy search- Concept learning:
- sample w from q to get exploration strategies- roll out policies conditioned on w
- Fine tuning:- policy gradient on best policy found in
concept learning
Conceptlearning
Finetuning
![Page 20: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/20.jpg)
TakeawaysPresent an approach for optimizing models by using natural language as latent representations.
Approach outperformed some baselines on classification, structured prediction and reinforcement learning tasks.
Language encourages/allows for better compositional generalization - Few Shot
Language helps simplify structured exploration - RL
![Page 21: CS 330 Deep Multi-Task and Meta Learning - CS330 Student Presentation …cs330.stanford.edu/presentations/presentation-11.4-4.pdf · 2019-11-05 · CS330 Student Presentation. Motivation](https://reader033.vdocument.in/reader033/viewer/2022042407/5f22aa14f6017c4649243cf8/html5/thumbnails/21.jpg)
Discussion / Strengths and Weaknesses- Not really clear what distinction is between concept learning and evaluation- Good baselines for backing up their “philosophical” goal- Limitation: need task-specific human language annotations
- Challenge to move beyond toy examples
- Could this method be streamlined with an end-to-end approach?- Take cues from SeqGAN?