Upload File

worksheets · $ cl cat exp2/stdout manage runs: $ cl kill exp2; cl rm exp2 run an entire pipeline...

  • Home
  • Documents
  • Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar
92
Worksheets Percy Liang STATS 385 — Oct 21, 2019

Upload: others

Post on 18-Mar-2020

7 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Worksheets

Percy Liang

STATS 385 — Oct 21, 2019

Page 2: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

The current research process

1

Page 3: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Problem 1: reproducibility

Previous method New method

Dataset 1 88% accuracy 92% accuracy

2

Page 4: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Problem 1: reproducibility

Previous method New method

Dataset 1 88% accuracy 92% accuracy

Dataset 2 72% accuracy 77% accuracy

2

Page 5: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Problem 1: reproducibility

Previous method New method

Dataset 1 88% accuracy 92% accuracy

Dataset 2 72% accuracy 77% accuracy

Dataset 3 ? ?

2

Page 6: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Problem 1: reproducibility

Previous method New method

Dataset 1 88% accuracy 92% accuracy

Dataset 2 72% accuracy 77% accuracy

Dataset 3 ? ?

Dataset 4 ? ?

... ... ...

2

Page 7: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Problem 2: efficiency

Step 1: come up with a good idea

3

Page 8: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Problem 2: efficiency

Step 1: come up with a good idea

Step 2: execute on it

• Obtain data, clean it, convert between formats

3

Page 9: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Problem 2: efficiency

Step 1: come up with a good idea

Step 2: execute on it

• Obtain data, clean it, convert between formats

• Try to reproduce results from previous work, email authors

3

Page 10: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Problem 2: efficiency

Step 1: come up with a good idea

Step 2: execute on it

• Obtain data, clean it, convert between formats

• Try to reproduce results from previous work, email authors

• Run experiments with different versions, keep track of provenance

3

Page 11: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Problem 2: efficiency

Step 1: come up with a good idea

Step 2: execute on it

• Obtain data, clean it, convert between formats

• Try to reproduce results from previous work, email authors

• Run experiments with different versions, keep track of provenance

3

Page 12: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Tradeoff?

efficiency reproducibility

Folk wisdom: reproducibility slows down research.

4

Page 13: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Tradeoff?

efficiency reproducibility

Folk wisdom: reproducibility slows down research.

Our claim: reproducibility accelerates research (with the right tool).

4

Page 14: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

MLcomp.org (2008)

5

Page 15: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

MLcomp paradigm

dataset algorithm

6

Page 16: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

MLcomp paradigm

dataset algorithm

accuracy metrics

6

Page 17: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

MLcomp paradigm

dataset algorithm

accuracy metrics

Problem: too rigid, doesn’t help with the efficiency problem

6

Page 18: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

CodaLab Worksheets (2013-present)

7

Page 19: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Bundles

Worksheets

8

Page 20: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Bundles

Bundle: an arbitrary file/directory (code or data or results)

0x191aad8fa0ae4741b3123b15a8d59efa

9

Page 21: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Bundles

Uploaded by user (code or data):

10

Page 22: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Bundles

Uploaded by user (code or data):

Derived by running an arbitrary command:

10

Page 23: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Bundles

cnn.py(0x45d17c)

#!/usr/bin/python

import numpy as np

...

mnist(0x1ba223)

- train.dat

- test.dat

exp2(0x2d4192)

- stdout

- stderr

- stats.json

...

cnn.pydata

exp

11

Page 24: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Bundles

cnn.py(0x45d17c)

#!/usr/bin/python

import numpy as np

...

mnist(0x1ba223)

- train.dat

- test.dat

exp2(0x2d4192)

- stdout

- stderr

- stats.json

...

cnn.pydata

exp

- data/train.dat

- data/test.dat

- cnn.py

- stdout

- stderr

- stats.json

python cnn.py data/train.dat data/test.dat

11

Page 25: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Command-line Interface (CLI)

Search for existing code and data:

$ cl search mnist

12

Page 26: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Command-line Interface (CLI)

Search for existing code and data:

$ cl search mnist

Upload new code or data:

$ cl upload cnn.py

12

Page 27: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Command-line Interface (CLI)

Search for existing code and data:

$ cl search mnist

Upload new code or data:

$ cl upload cnn.py

Run experiments with arbitrary commands:

$ cl run :cnn.py data:mnist "python cnn.py data/train.dat data/test.dat"

12

Page 28: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Command-line Interface (CLI)

Search for existing code and data:

$ cl search mnist

Upload new code or data:

$ cl upload cnn.py

Run experiments with arbitrary commands:

$ cl run :cnn.py data:mnist "python cnn.py data/train.dat data/test.dat"

Look at output of runs:

$ cl cat exp2/stdout

12

Page 29: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Command-line Interface (CLI)

Search for existing code and data:

$ cl search mnist

Upload new code or data:

$ cl upload cnn.py

Run experiments with arbitrary commands:

$ cl run :cnn.py data:mnist "python cnn.py data/train.dat data/test.dat"

Look at output of runs:

$ cl cat exp2/stdout

Manage runs:

$ cl kill exp2; cl rm exp2

12

Page 30: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Command-line Interface (CLI)

Search for existing code and data:

$ cl search mnist

Upload new code or data:

$ cl upload cnn.py

Run experiments with arbitrary commands:

$ cl run :cnn.py data:mnist "python cnn.py data/train.dat data/test.dat"

Look at output of runs:

$ cl cat exp2/stdout

Manage runs:

$ cl kill exp2; cl rm exp2

Run an entire pipeline with a different dataset or newer version of your code:

$ cl mimic mnist exp2 cifar -n exp3

12

Page 31: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Command-line Interface (CLI)

Search for existing code and data:

$ cl search mnist

Upload new code or data:

$ cl upload cnn.py

Run experiments with arbitrary commands:

$ cl run :cnn.py data:mnist "python cnn.py data/train.dat data/test.dat"

Look at output of runs:

$ cl cat exp2/stdout

Manage runs:

$ cl kill exp2; cl rm exp2

Run an entire pipeline with a different dataset or newer version of your code:

$ cl mimic mnist exp2 cifar -n exp3

Copy from one CodaLab instance to another:

$ cl add bundle mnist stanford::pliang-demo main::pliang-demo12

Page 32: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Modularity

Real-world problems require efforts of entire community

13

Page 33: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Modularity

Real-world problems require efforts of entire community

People specialize, contribute in decentralized way

13

Page 34: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Modularity

Real-world problems require efforts of entire community

People specialize, contribute in decentralized way

13

Page 35: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Modularity

Real-world problems require efforts of entire community

People specialize, contribute in decentralized way

13

Page 36: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Modularity

Real-world problems require efforts of entire community

People specialize, contribute in decentralized way

13

Page 37: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Modularity

Real-world problems require efforts of entire community

People specialize, contribute in decentralized way

13

Page 38: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Modularity

Real-world problems require efforts of entire community

People specialize, contribute in decentralized way

13

Page 39: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Modularity

Real-world problems require efforts of entire community

People specialize, contribute in decentralized way

13

Page 40: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Modularity

Real-world problems require efforts of entire community

People specialize, contribute in decentralized way

13

Page 41: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Modularity

Real-world problems require efforts of entire community

People specialize, contribute in decentralized way

13

Page 42: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Modularity

Real-world problems require efforts of entire community

People specialize, contribute in decentralized way

13

Page 43: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Modularity

Real-world problems require efforts of entire community

People specialize, contribute in decentralized way

13

Page 44: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Modularity

Real-world problems require efforts of entire community

People specialize, contribute in decentralized way

13

Page 45: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Intermediate tasks

• Old way: use intermediate metrics, rhetoric

14

Page 46: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Intermediate tasks

• Old way: use intermediate metrics, rhetoric

• New way: plug in and see ramifications automatically

14

Page 47: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Intermediate tasks

• Old way: use intermediate metrics, rhetoric

• New way: plug in and see ramifications automatically

14

Page 48: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Intermediate tasks

• Old way: use intermediate metrics, rhetoric

• New way: plug in and see ramifications automatically

14

Page 49: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Intermediate tasks

• Old way: use intermediate metrics, rhetoric

• New way: plug in and see ramifications automatically

14

Page 50: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Immutability

Inspiration: Git version control system

15

Page 51: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Immutability

Inspiration: Git version control system

• All programs/datasets/runs are write-once

• Enable collaboration without chaos

• Capture the research process in a reproducible way

15

Page 52: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Bundles

Worksheets

16

Page 53: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Literacy

Bundle graphs are about truth; what about interpretation?

17

Page 54: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Literacy

Bundle graphs are about truth; what about interpretation?

Worksheet: an arbitrary document with embedded bundles

description

description

description

17

Page 55: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Literacy

Bundle graphs are about truth; what about interpretation?

Worksheet: an arbitrary document with embedded bundles

description

description

description

Inspiration: Mathematica notebook, Jupyter notebook

17

Page 56: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

A worksheet

We now train the classifier with more data.

18

Page 57: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

A worksheet

We now train the classifier with more data.

Program : SVMlight

Arguments : -n 2000

Dataset : thyroid

Error : 2.6%

Time : 1 second

18

Page 58: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

A worksheet

We now train the classifier with more data.

Program : SVMlight

Arguments : -n 2000

Dataset : thyroid

Error : 2.6%

Time : 1 second

Notice that the error remains the same, suggesting that we’ve saturated our model.

18

Page 59: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

19

Page 60: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

nanc-1m.txt(0xc19b66)

Two New Orleans...

run1(0xad3d69)

- stdout

415

run2(0x992ced)

- stdout

872

run-count(0xd4815b)

- stdout

1 1

2 4

3 9

data data

19

Page 61: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

nanc-1m.txt(0xc19b66)

Two New Orleans...

run1(0xad3d69)

- stdout

415

run2(0x992ced)

- stdout

872

run-count(0xd4815b)

- stdout

1 1

2 4

3 9

data data

## Heading

You can type in **any** markdown with any $LATEX$.

[dataset nanc-1m.txt]{0xc19b6600afe74e91a441e6d13e823ead}

% display contents / maxlines=2

[dataset nanc-1m.txt]{0xc19b6600afe74e91a441e6d13e823ead}

% schema mySchema

% add query command "s/.*grep / | s/...wc.*/"

% add count /stdout

% display table mySchema

[run data:nanc-1m.txt : cat data | grep Montreal | wc -l]{0xad3d69e373eb4702ab89dc4991aa0f82}[run data:nanc-1m.txt : cat data | grep Toronto | wc -l]{0x992ced33e6e848aa8cfb8988c12bb221}

% display graph /stdout xlabel=time ylabel=accuracy maxlines=30

[run : for x in {1..50}; do echo -e "$x�$((x*x))"; done]{0xd4815bf677bc4ab492a4c28744224c87}

Largest bundles:

% display table uuid:uuid:[0:8] name summary data size

% search size=.sort- .limit=3

embed bundles

render bundle contents

customize table schema

graph points in a TSV file

embed search results

19

Page 62: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Use case: executable papers

20

Page 63: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Use case: benchmarking results

21

Page 64: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Use case: software tutorials

22

Page 65: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Use case: research development environment

23

Page 66: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

[demo]

24

Page 67: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

System architecture

website

bundle service

worker

worker

worker

Note: workers can be run by the user

25

Page 68: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Running your own CodaLab server

Check out the repo:

$ git clone https://github.com/codalab/codalab-worksheets

Start the full stack:

$ cd codalab-worksheets; ./codalab service.py start

Try it out:

$ open http://localhost

26

Page 69: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

A case study...

27

Page 70: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

SQuAD dataset for reading comprehension[Hirschman+ 1999; Richardson+ 2013; Rajpurkar+ 2016]

28

Page 71: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

SQuAD dataset for reading comprehension

Must submit model on CodaLab to evaluate on test set

[Hirschman+ 1999; Richardson+ 2013; Rajpurkar+ 2016]

28

Page 72: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Evaluation using ”mimic”

29

Page 73: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

30

Page 74: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Adversarial evaluation

Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls. He is also the oldestquarterback ever to play in a Super Bowl at age 39. The past record was held by John Elway, who led the Broncos to victory inSuper Bowl XXXIII at age 38 and is currently Denver’s Executive Vice President of Football Operations and General Manager.

What is the name of the quarterback who was 38 in Super Bowl XXXIII?

BiDAF

John Elway

[with Robin Jia 2017; outstanding paper award]

31

Page 75: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Adversarial evaluation

Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls. He is also the oldestquarterback ever to play in a Super Bowl at age 39. The past record was held by John Elway, who led the Broncos to victory inSuper Bowl XXXIII at age 38 and is currently Denver’s Executive Vice President of Football Operations and General Manager.Jeff Dean is the name of the quarterback who was 37 in Champ Bowl XXXIV.

What is the name of the quarterback who was 38 in Super Bowl XXXIII?

BiDAF

[with Robin Jia 2017; outstanding paper award]

31

Page 76: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Adversarial evaluation

Peyton Manning became the first quarterback ever to lead two different teams to multiple Super Bowls. He is also the oldestquarterback ever to play in a Super Bowl at age 39. The past record was held by John Elway, who led the Broncos to victory inSuper Bowl XXXIII at age 38 and is currently Denver’s Executive Vice President of Football Operations and General Manager.Jeff Dean is the name of the quarterback who was 37 in Champ Bowl XXXIV.

What is the name of the quarterback who was 38 in Super Bowl XXXIII?

BiDAF

Jeff Dean

[with Robin Jia 2017; outstanding paper award]

31

Page 77: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Results on public models on CodaLab

Model Original F1 Adversarial F1

ReasoNet-E 81.1 49.8

SEDT-E 80.1 46.5

BiDAF-E 80.0 46.9

Mnemonic-E 79.1 55.3

Ruminating 78.8 47.7

jNet 78.6 47.0

Mnemonic-S 78.5 56.0

ReasoNet-S 78.2 50.3

MPCM-S 77.0 50.0

RaSOR 76.2 49.5

BiDAF-S 75.5 45.7

32

Page 78: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Results on public models on CodaLab

Model Original F1 Adversarial F1

ReasoNet-E 81.1 49.8

SEDT-E 80.1 46.5

BiDAF-E 80.0 46.9

Mnemonic-E 79.1 55.3

Ruminating 78.8 47.7

jNet 78.6 47.0

Mnemonic-S 78.5 56.0

ReasoNet-S 78.2 50.3

MPCM-S 77.0 50.0

RaSOR 76.2 49.5

BiDAF-S 75.5 45.7

Humans 92.6 89.2

New research enabled by CodaLab32

Page 79: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Other competitions on CodaLab

Note: separate from CodaLab Competitions

33

Page 80: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Final remarks

34

Page 81: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Q: What programming language can I use?

A: Anything: Python, C++, Java, Julia, etc.

We run arbitrary Unix commands in a docker container.

35

Page 82: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Q: What programming language can I use?

A: Anything: Python, C++, Java, Julia, etc.

We run arbitrary Unix commands in a docker container.

Q: What computing resources does CodaLab provide?

A: worksheets.codalab.org uses Microsoft Azure.

You can connect your own worker or setup a local installation.

35

Page 83: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Q: What programming language can I use?

A: Anything: Python, C++, Java, Julia, etc.

We run arbitrary Unix commands in a docker container.

Q: What computing resources does CodaLab provide?

A: worksheets.codalab.org uses Microsoft Azure.

You can connect your own worker or setup a local installation.

Q: How is CodaLab different from Jupyter notebook?

A: Jupyter building blocks are notebooks (like worksheets) and are mutable.

CodaLab building blocks are bundles and are immutable.

35

Page 84: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Q: What programming language can I use?

A: Anything: Python, C++, Java, Julia, etc.

We run arbitrary Unix commands in a docker container.

Q: What computing resources does CodaLab provide?

A: worksheets.codalab.org uses Microsoft Azure.

You can connect your own worker or setup a local installation.

Q: How is CodaLab different from Jupyter notebook?

A: Jupyter building blocks are notebooks (like worksheets) and are mutable.

CodaLab building blocks are bundles and are immutable.

Q: How is CodaLab different from releasing a VM?

A: VMs are monolithic black boxes.

CodaLab bundles are immutable data/code modules that can be composed.

35

Page 85: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Q: What programming language can I use?

A: Anything: Python, C++, Java, Julia, etc.

We run arbitrary Unix commands in a docker container.

Q: What computing resources does CodaLab provide?

A: worksheets.codalab.org uses Microsoft Azure.

You can connect your own worker or setup a local installation.

Q: How is CodaLab different from Jupyter notebook?

A: Jupyter building blocks are notebooks (like worksheets) and are mutable.

CodaLab building blocks are bundles and are immutable.

Q: How is CodaLab different from releasing a VM?

A: VMs are monolithic black boxes.

CodaLab bundles are immutable data/code modules that can be composed.

Q: Why can’t I just release my code on GitHub?

A: Releasing code is a big step forward, but code has unspecified dependencies.

CodaLab encapsulates these.

35

Page 86: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Q: What programming language can I use?

A: Anything: Python, C++, Java, Julia, etc.

We run arbitrary Unix commands in a docker container.

Q: What computing resources does CodaLab provide?

A: worksheets.codalab.org uses Microsoft Azure.

You can connect your own worker or setup a local installation.

Q: How is CodaLab different from Jupyter notebook?

A: Jupyter building blocks are notebooks (like worksheets) and are mutable.

CodaLab building blocks are bundles and are immutable.

Q: How is CodaLab different from releasing a VM?

A: VMs are monolithic black boxes.

CodaLab bundles are immutable data/code modules that can be composed.

Q: Why can’t I just release my code on GitHub?

A: Releasing code is a big step forward, but code has unspecified dependencies.

CodaLab encapsulates these.

Q: What’s the relationship to CodaLab Competitions?

A: It’s a sister project led by Isabelle Guyon.

Competitions brings people together and bundles/worksheets provides a rich foundation.35

Page 87: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Open challenges

Reproducibility (community):

What’s the incentive to upload an executable paper?

How do we encourage creation of reusable modules?

How do we build a community?

36

Page 88: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Open challenges

Reproducibility (community):

What’s the incentive to upload an executable paper?

How do we encourage creation of reusable modules?

How do we build a community?

Productivity (individual):

Is there enough flexibility to support interactive development?

Can we scale to really large-scale experiments?

36

Page 89: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Tradeoff?

efficiency reproducibility

Folk wisdom: reproducibility slows down research.

37

Page 90: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Tradeoff?

efficiency reproducibility

Folk wisdom: reproducibility slows down research.

Our claim: reproducibility accelerates research (with the right tool).

37

Page 91: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

CodaLab contributors

Shaunak Kishore Christophe Poulain Eric Carmichael

Francis Cleary Justin Carden Scott Kovach

Pujun Bhatnagar Stephen Koo Eric Li

Konstantin Lopyrev Max Wang Fabian Chan

Kerem Goksel Dennis Jeong Nikhil Bhattasali

Hao Wu Jane Ge Ashwin Ramaswami

Levi Lian Yipeng He

We’re looking for strong developers!

38

Page 92: Worksheets · $ cl cat exp2/stdout Manage runs: $ cl kill exp2; cl rm exp2 Run an entire pipeline with a di erent dataset or newer version of your code: $ cl mimic mnist exp2 cifar

Worksheets

Supported by:

Thank you!

39


Aecs Lab Exp2

Ccna Exp2 - Chapter01 - Smith.N Studio

Exp2 passive band pass and band-stop filter

Radio Exp2

Ccna Exp2 - Chaabacpter09 - Eigrp

Module 6 – Redirections, Pipes and Power Tools.. STDin 0 STDout 1 STDerr 2 Redirections

Exp2 Magnetic Amplifier

Introduction to UNIX command-line II · Sol Genomics Network Text handling commands command > file saves STDOUT in a file command >> file appends STDOUT in a file cat concatenate

Experiment 2: Amplitude Modulation and Demodulationfernando/ELE635/Exp2/Exp2 Amplitude... · The Hilbert transform block is described by the ... (FG1): The settings are: [Waveform:

Knockdown of the translocon protein EXP2, reduces growth ... · Although EXP2 lacks predicted transmembrane spanning domains typical of membrane pores, it is the most membrane associated

Biochem34.1 Exp2 Report

CCNA Exp2 - Chapter01 - Introduction to Routing and Packet Forwarding

MA 306 EXP2

EXP2 - Photolithography for Implanted Piezoresistorslouisville.edu/.../EXP2PhotolithographyforImplantedPiezoresistors.pdf · Experiment 2 - Photolithography for Implanted Piezoresistors

Exp2 Report

CCNA Exp2 - Chapter08 - The Routing Table_A Closer Look

Gt Exp2 Exams Cyl

CCNA Exp2 - Chapter04 - Distance Vector Routing Protocols

CCNA Exp2 - Chapter02 - Static Routing

CCNA Exp2 - Chapter11 - OSPF

Report Exp2

Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na

Knockdown of the translocon protein EXP2 in …...Upon EXP2 knockdown, growth arrest was probably due to the inhibi-tion of export of essential effector proteins since the export of

12 VOLT SERIES CORONA LANDSCAPE ... · CL-801-D CL-802-D CL-803-D CL-806-D CL-807-D T-ST300-TP CL-650B CL-719B CL-722B CL-730B CL-740C CL-639 CL-363B CL-364B CL-365B CL-635 CL-647C

CCNA Exp2 - Chapter05 - RIP Version 1

winprotocoldoc.blob.core.windows.net · Web viewFor example, if the client only needs "stdout" during the session, it can list "stdout" as the sole stream of interest. If a stream

Exp2 Avogadro's Number

Full Report Exp2

Assignment2---------------keddiyan.com/files/DSA/Answers.pdf · Exp2: Let T (n) denote the time complexity of Exp2(a, n). Then, T (n) < T(n — 1) c, for some constant c. Thus, T(n)

Temperature and solids retention time control microbial ... · (Exp1 and Exp2; Exp1 for 34 days and Exp2 for 90 days) to assess if Table 1 | Overviewof the residual cellulose and

Pupil dilation reflects perceptual selection and predicts ...exp2 rivalry exp2 replay p

By Joshua Lande SASS January 21A few basic programs echo – sends the command line arguments to stdout cat – reads file(s) as command line arguments and sends the lines to stdout

Computing at DESY Zeuthen · > ls > list.txt redirects stdout of ls into file list.txt errors are still printed to the terminal > ls > list.txt 2>&1 redirects stderr (2) to stdout

CCNA Exp2 - Chapter03 - Introduction to Dynamic Routing Protocols

Init 10/28/2010 by Daniel R. Barnes Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na + Cl - Na