rapid learning or feature reuse? towards understanding the ... · samy bengio, oriol vinyals. few...
TRANSCRIPT
![Page 1: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/1.jpg)
Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness
of MAML
Aniruddh Raghu*, Maithra Raghu*, Samy Bengio, Oriol Vinyals
![Page 2: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/2.jpg)
Few Shot LearningMany tasks, little data for each task
Task 2Chair/Lion
Task 1Dog/Cat
Task 3Plane/Tree
![Page 3: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/3.jpg)
Few Shot Learning(Optimization-based) Meta Learning Algorithms● Model Agnostic Meta Learning, (Finn et al), ICML 2017
![Page 4: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/4.jpg)
Few Shot Learning(Optimization-based) Meta Learning Algorithms● Model Agnostic Meta Learning, (Finn et al), ICML 2017
Outer Loop: meta-initialization; Inner Loop: adaptation
Outer Loop
Inner Loop
![Page 5: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/5.jpg)
Rapid Learning or Feature Reuse?Outer Loop: meta-initialization; Inner Loop: adaptation
![Page 6: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/6.jpg)
Rapid Learning or Feature Reuse?
How do hidden representations behave
(during inner loop)?
![Page 7: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/7.jpg)
Rapid Learning or Feature Reuse?
How do hidden representations behave
(during inner loop)?
SVCCA
https://github.com/google/svcca
Measure Representation Similarity
![Page 8: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/8.jpg)
Rapid Learning or Feature Reuse?
How do hidden representations behave
(during inner loop)?
SVCCA
Measure Representation Similarity
Before inner loop
After inner loop
![Page 9: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/9.jpg)
Rapid Learning or Feature Reuse?SVCCA Before/After Inner Loop
SV
CC
A S
imila
rity
Layer
![Page 10: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/10.jpg)
Rapid Learning or Feature Reuse?SVCCA Before/After Inner Loop
SV
CC
A S
imila
rity
Layer
Very high similarity!
![Page 11: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/11.jpg)
Rapid Learning or Feature Reuse?SVCCA Before/After Inner Loop
SV
CC
A S
imila
rity
Layer
Very high similarity!
![Page 12: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/12.jpg)
ANIL: Almost No Inner Loop Algorithm
![Page 13: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/13.jpg)
ANIL: Almost No Inner Loop Algorithm● Removes inner loop for all but
head of network
● Much more computationally efficient, same performance
● Insights into meta learning and few shot learning
![Page 14: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/14.jpg)
ANIL: Performance ResultsMatches performance of MAML in few-shot classification and RL
![Page 15: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/15.jpg)
ANIL and NIL (No Inner Loop)NIL: No Inner Loop (at test time), performs equally well
![Page 16: Rapid Learning or Feature Reuse? Towards Understanding the ... · Samy Bengio, Oriol Vinyals. Few Shot Learning Many tasks, little data for each task Task 2 Chair/Lion Task 1 Dog/Cat](https://reader033.vdocument.in/reader033/viewer/2022042806/5f6cfe8cfe4f3f3f1f697ac2/html5/thumbnails/16.jpg)
Thanks and Future Directionshttp://learn2learn.net/tutorials/anil_tutorial/ANIL_tutorial/
● Exploring Medium Shot Learning?● Meta-Learning as Pretraining?● Learning Regimes for Interpolating
between Rapid Learning and Feature Reuse?
Theoretical Analysis: Few-Shot Learning via Learning the Representation, Provably, (Du, Hu, Kakade, Lee, Lei)
Analyzing Feature Reuse: Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need?, (Tian, Wang, Krishnan, Tenenbaum, Isola)