lifelong few-shot learning - artificial intelligence › ~cbfinn › _files ›...

Chelsea Finn, Pieter Abbeel, Sergey Levine

Lifelong Few-Shot Learning

Lifelong Learning Goal: learn many tasks, typically in sequence

Key Elements: - avoid forgetting previously learned tasks - reuse prior experience for fast learning of new tasks

Chelsea Finn, UC Berkeley

Lifelong Learning Goal: learn many tasks, typically in sequence

Key Elements: - avoid forgetting previously learned tasks - reuse prior experience for fast learning of new tasks

Typical Approach: Run optimization algorithm (e.g. policy gradient)a.k.a. continual fine-tuning


Fine-tuning for lifelong learning

Suppose, we have trained a model on N tasks.

Now, we want to learn a new task with little data

Why is fine-tuning not effective?

None of these are satisfactory :(


…1 2 3 N

Option 0: collect a lot more data Option 1: Finetune on (N+1)’th task Option 2: Replay old task data, including new data


Lifelong Meta-Learningcontinuously learn to learn sequence of tasks

[D1 for learning, D2 for meta-learning]

non-stationary task distribution

Sample a task

Sample N datapoints from

Result: Can reuse prior learning experience to quickly learn new tasks

meta-learning

, split into datasets D1 and D2

Update model so that learning using D1 enables effective performance on D2

How can we do better? Learn how to learn efficiently

Key idea: Train over many tasks, to learn parameter vector θ that transfers

Fine-tuning:task

pretrained parameters

Our method:

[test-time]

Background: Model-Agnostic Meta-Learning fine-tuning from ImageNet-trained features (Deng et al. ’09, Donahue et al. ’14)

+ simple, works well, same learning rule- no ImageNet for non-vision domains, won’t extend to few-shot setting


One slice of lifelong learning:

Preliminary Investigation into Lifelong Meta-Learning


- given experience with N tasks - evaluate ability to efficiently learn (N+1)’th

task from different distribution

Illustrative Regression Example

Adaptation in Reinforcement Learning

RegressionQuickly learn a new real-valued function

meta-learning

training data evaluation data

… …


Tasks: sine function with varied amplitude & frequency

x y

new task

Comparison: task-conditioning


x

y

Condition model on task information

Train with standard supervised learning on training tasks. Fine-tune on new task.

In-distribution task performance


10 datapoints used for all fine-tuning steps

Out-of-distribution task performance

Chelsea Finn, UC Berkeley10 datapoints used for all

fine-tuning steps

Task representation out-of-distribution


10 datapoints used for all fine-tuning steps

the importance of task representation

Meta-Learning: Half-Cheetah


Training task distribution:

Quickly adapt behavior to run at a goal velocity

sa

Comparison to task-conditioning

Evaluate on:

Half-Cheetah in-distribution performance


20 trajectories used for each fine-tuning step

Half-Cheetah out-of-distribution performance


20 trajectories used for each fine-tuning step

Conclusion:- lifelong meta-learning enables faster learning of new tasks


CollaboratorsSergey Levine Pieter Abbeel

[email protected]

mailto:[email protected]

lifelong few-shot learning - artificial intelligence › ~cbfinn › _files ›...

Documents