❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤
❆r♥❛✉❞ ▲❡❣r❛♥❞
❏♦✉r♥é❡ ❚r❛❝❡s✱ ●r❡♥♦❜❧❡▼❛r❝❤ ✷✹✱ ✷✵✶✺
❆r♥❛✉❞ ▲❡❣r❛♥❞ ✶ ✴ ✶✼
❆ ❋❡✇ ❊❞✐❢②✐♥❣ ❊①❛♠♣❧❡s
◆❛✐❝❦❡♥✱ ❙t❡♣❤❡♥ ❡t ❆❧✳✱ ❚♦✇❛r❞s ❨❡t ❆♥♦t❤❡r P❡❡r✲t♦✲P❡❡r❙✐♠✉❧❛t♦r✱ ❍❊❚✲◆❊❚s✬✵✻✳
From 141 P2P sim.papers, 30% use a custom tool,50% don’t report used tool
25
50
75
0/100
Type
Chord−(SFS)
Custom
NS−2
Other
Unspecified
Simulator usage [Naicken06]
❈♦❧❧❜❡r❣✱ ❈❤r✐st✐❛♥ ❡t ❆❧✳✱ ▼❡❛s✉r✐♥❣ ❘❡♣r♦❞✉❝✐❜✐❧✐t② ✐♥ ❈♦♠♣✉t❡r ❙②st❡♠s ❘❡s❡❛r❝❤✱ ❤tt♣✿✴✴r❡♣r♦❞✉❝✐❜✐❧✐t②✳❝s✳❛r✐③♦♥❛✳❡❞✉✴
601
NC63
HW30
508
Article85
Web54
EMyes
87
EX106
226
OK≤30
130OK
>30
64OK
A u
23
Buildfails9
176
EMno
146EM
∅
30
8 ACM conferences (❆❙P▲❖❙✬✶✷✱❈❈❙✬✶✷✱ ❖❖P❙▲❆✬✶✷✱ ❖❙❉■✬✶✷✱ P▲❉■✬✶✷✱
❙■●▼❖❉✬✶✷✱ ❙❖❙P✬✶✶✱ ❱▲❉❇✬✶✷) and 5journals
EM♥♦= the code cannot be provided
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✷ ✴ ✶✼
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
Thanks for your interest in the implementation of our paper. The goodnews is that I was able to find some code. I am just hoping that it isa stable working version of the code, and matches the implementationwe finally used for the paper. Unfortunately, I have lost some data whenmy laptop was stolen last year. The bad news is that the code is notcommented and/or clean.
Attached is the 〈system〉 source code of our algorithm. I’m not very surewhether it is the final version of the code used in our paper, but it shouldbe at least 99% close. Hope it will help.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
Unfortunately, the server in which my implementation was stored had adisk crash in April and three disks crashed simultaneously. While the helpdesk made significant effort to save the data, my entire implementationfor this paper was not found.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
Unfortunately the current system is not mature enough at the moment,so it’s not yet publicly available. We are actively working on a numberof extensions and things are somewhat volatile. However, once thingsstabilize we plan to release it to outside users. At that point, we wouldbe happy to send you a copy.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
I am afraid that the source code was never released. The code was neverintended to be released so is not in any shape for general use.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
〈STUDENT〉 was a graduate student in our program but he left a whileback so I am responding instead. For the paper we used a prototypethat included many moving pieces that only 〈STUDENT〉 knew how tooperate and we did not have the time to integrate them in a ready-to-share implementation before he left. Still, I hope you can build on theideas/technique of the paper.
Unfortunately, the author who has done most of the coding for this paperhas passed away and the code is no longer maintained.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
Since this work has been done at 〈COMPANY〉 we don’t open-sourcecode unless there is a compelling business reason to do so. So unfortu-nately I don’t think we’ll be able to share it with you.
The code owned by 〈COMPANY〉, and AFAIK the code is not open-source. Your best bet is to reimplement :( Sorry.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
Unfortunately, the 〈SYSTEM〉 sources are not meant to be opensource(the code is partially property of 〈UNIVERSITY 1〉, 〈UNIVERSITY 2〉and 〈UNIVERSITY 3〉.)
If this will change I will let you know, albeit I do not think there is anintention to make the 〈SYSTEM〉 sources opensource in the near future.
If you’re interested in obtaining the code, we only ask for a descriptionof the research project that the code will be used in (which may leadto some joint research), and we also have a software license agreementthat the University would need to sign.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
❚❤❡ ❉♦❣ ❆t❡ ♠② ❍♦♠❡✇♦r❦ ✦✦✦
❱❡rs✐♦♥♥✐♥❣ Pr♦❜❧❡♠s
❇❛❞ ❇❛❝❦✉♣ Pr❛❝t✐❝❡s
❈♦❞❡ ❲✐❧❧ ❜❡ ❆✈❛✐❧❛❜❧❡ ❙♦♦♥
◆♦ ■♥t❡♥t✐♦♥ t♦ ❘❡❧❡❛s❡
Pr♦❣r❛♠♠❡r ▲❡❢t
❈♦♠♠❡r❝✐❛❧ ❈♦❞❡
Pr♦♣r✐❡t❛r② ❆❝❛❞❡♠✐❝ ❈♦❞❡
❘❡s❡❛r❝❤ ✈s✳ ❙❤❛r✐♥❣
✳✳✳
✳✳✳
In the past when we attempted to share it, we found ourselves spendingmore time getting outsiders up to speed than on our own research. SoI finally had to establish the policy that we will not provide the sourcecode outside the group.
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❋✉♥ ❊①❛♠♣❧❡s ✸ ✴ ✶✼
2 Reproducible Research ‘11 Juliana Freire UBC, Vancouver
Science Today: Data Intensive
Obtain
Data
Analyze/
Visualize
User studies
Sensors
Web
Databases
Simulations
Publish/
Share
Sequencing
machines
Particle
colliders
❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✹ ✴ ✶✼
3 Reproducible Research ‘11 Juliana Freire UBC, Vancouver
Science Today: Data + Computing Intensive
Obtain
Data
Analyze/
Visualize
User studies
Sensors
Web
Databases
Simulations
Publish/
Share
Sequencing
machines
Particle
colliders
AVS
Taverna
VisTrails
❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✺ ✴ ✶✼
4 Reproducible Research ‘11 Juliana Freire UBC, Vancouver
Science Today: Data + Computing Intensive
Obtain
Data
Analyze/
Visualize
User studies
Sensors
Web
Databases
Simulations
Publish/
Share
Sequencing
machines
Particle
colliders
❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✻ ✴ ✶✼
5 Reproducible Research ‘11 Juliana Freire UBC, Vancouver
Science Today: Data + Computing Intensive
Obtain
Data
Analyze/
Visualize
User studies
Sensors
Web
Databases
Simulations
Publish/
Share
Sequencing
machines
Particle
colliders
❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✼ ✴ ✶✼
6 Reproducible Research ‘11 Juliana Freire UBC, Vancouver
Science Today: Incomplete Publications
Publications are just the tip of the iceberg
- Scientific record is incomplete---
to large to fit in a paper
- Large volumes of data
- Complex processes
Can’t (easily) reproduce results
❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✽ ✴ ✶✼
7 Reproducible Research ‘11 Juliana Freire UBC, Vancouver
Science Today: Incomplete Publications
Publications are just the tip of the iceberg
- Scientific record is incomplete---
to large to fit in a paper
- Large volumes of data
- Complex processes
Can’t (easily) reproduce results
“It’s impossible to verify most of the results that computational scientists present at conference
and in papers.” [Donoho et al., 2009]
“Scientific and mathematical journals are filled
with pretty pictures of computational experiments
that the reader has no hope of repeating.” [LeVeque, 2009]
“Published documents are merely the
advertisement of scholarship whereas the
computer programs, input data, parameter
values, etc. embody the scholarship itself.” [Schwab et al., 2007]
❈♦✉rt❡s② ♦❢ ❏✉❧✐❛♥❛ ❋r❡✐r❡ ✭❆▼P ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✮
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✾ ✴ ✶✼
❊✈✐❞❡♥❝❡ ❢♦r ❛ ▲❛❝❦ ♦❢ ❘❡♣r♦❞✉❝✐❜✐❧✐t②
❙t✉❞✐❡s s❤♦✇✐♥❣ t❤❛t s❝✐❡♥t✐✜❝ ♣❛♣❡rs ❝♦♠♠♦♥❧② ❧❡❛✈❡ ♦✉t ❡①♣❡r✐♠❡♥t❛❧❞❡t❛✐❧s ❡ss❡♥t✐❛❧ ❢♦r r❡♣r♦❞✉❝t✐♦♥ ❛♥❞ s❤♦✇✐♥❣ ❞✐✣❝✉❧t✐❡s ✇✐t❤ r❡♣❧✐❝❛t✐♥❣♣✉❜❧✐s❤❡❞ ❡①♣❡r✐♠❡♥t❛❧ r❡s✉❧ts✿
J.P. Ioannidis. Why Most Published Research Findings Are False PLoSMed. 2005 August; 2(8)
❍✐❣❤ ♥✉♠❜❡r ♦❢ ❢❛✐❧✐♥❣ ❝❧✐♥✐❝❛❧ tr✐❛❧s✳
Do We Really Know What Makes Us Healthy?, New-York Times —September 16, 2007Lies, Damned Lies, and Medical Science, The Atlantic. 2010, Nov.
■♥❝r❡❛s❡ ✐♥ r❡tr❛❝t❡❞ ♣❛♣❡rs✿
Steen RG, Retractions in the scientific literature: is the incidence ofresearch fraud increasing? J Med Ethics 37: 249–253.
❈♦✉rt❡s② ♦❢ ❆❞❛♠ ❏✳ ❘✐❝❤❛r❞s❆r♥❛✉❞ ▲❡❣r❛♥❞ ❍♦✇ ❞♦❡s ✐t ✇♦r❦ ✐♥ ♦t❤❡r s❝✐❡♥❝❡s❄ ✶✵ ✴ ✶✼
❊①♣❡r✐♠❡♥t✐♥❣ ✐♥ t❤❡ ❲✐❧❞
❊①♣❡r✐♠❡♥ts ✐♥ ❍P❈✿ ❊✈❡♥ ❲♦rs❡✦
❘❡❧② ♦♥ ❧❛r❣❡✱ ❞✐str✐❜✉t❡❞✱ ❤②❜r✐❞✱ ♣r♦t♦t②♣❡ ❤❛r❞✇❛r❡✴s♦❢t✇❛r❡
▼❡❛s✉r❡ ❡①❡❝✉t✐♦♥ t✐♠❡s ✭♠❛❦❡s♣❛♥s✱ tr❛❝❡s✱ ✳ ✳ ✳ ✮
▼❛♥② ♣❛r❛♠❡t❡rs✱ ✈❡r② ❝♦st❧② ❛♥❞ ❤❛r❞ t♦ r❡♣r♦❞✉❝❡
❆r♥❛✉❞ ▲❡❣r❛♥❞ ❲❤❛t ❛❜♦✉t ✉s ❄ ✶✶ ✴ ✶✼
❘❡♣r♦❞✉❝✐♥❣ ✐♥ t❤❡ ❲✐❧❞
■♥ ♠② ❈♦♠♠✉♥✐t②
❘❡♣r♦❞✉❝✐♥❣ r❡s✉❧ts ♦❢ ♦t❤❡rs ✐s ♥♦t ✐♥ t❤❡ ❝✉❧t✉r❡
❘❡✇❛r❞s ❢♦r ♣♦s✐t✐✈❡ r❡s✉❧ts ❛♥❞ ♥♦✈❡❧t② r❛t❤❡r t❤❛♥ ❝♦♥s♦❧✐❞❛t✐♥❣ ❡①✐st✲✐♥❣ r❡s✉❧ts
❊①❝✉s❡ ★✶✿ ❍❛r❞✇❛r❡ ❛♥❞ s♦❢t✇❛r❡ ❡✈♦❧✈❡ ✈❡r② q✉✐❝❦❧②
❊①❝✉s❡ ★✷✿ ❘❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤ ✐s ♥♦t ♠❛♥❞❛t♦r②
❲❤② ✐s ✐t s♦ ❍❛r❞ ✐♥ ●❡♥❡r❛❧❄
❈♦♣②r✐❣❤t✴❝♦♠♣❡t✐t✐♦♥ ✐ss✉❡ ✭✇♦✉❧❞ ❣✐✈❡ ❛❞✈❛♥t❛❣❡ t♦ ✧♦t❤❡rs✧✮
P✉❜❧✐❝❛t✐♦♥ ❜✐❛s ✭♦♥❧② t❤❡ ✐❞❡❛ ♠❛tt❡rs✱ ♥♦t t❤❡ ❣♦r② ❞❡t❛✐❧s✮
❊①♣❡r✐♠❡♥t❡r ❜✐❛s
Pr♦❣r❛♠♠✐♥❣ ❡rr♦rs ♦r ❞❛t❛ ♠❛♥✐♣✉❧❛t✐♦♥✴❛♥❛❧②s✐s ♠✐st❛❦❡s
▲❛❝❦ ♦❢ ❡❛s②✲t♦✲✉s❡ t♦♦❧s
❲❤② ❜♦t❤❡r❄❆r♥❛✉❞ ▲❡❣r❛♥❞ ❲❤❛t ❛❜♦✉t ✉s ❄ ✶✷ ✴ ✶✼
❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚r②✐♥❣ t♦ ❇r✐❞❣❡ t❤❡ ●❛♣
❘❡❛❞❡r
❆✉t❤♦r
✭❉❡s✐❣♥ ♦❢ ❊①♣❡r✐♠❡♥ts✮
Pr♦t♦❝♦❧
❙❝✐❡♥t✐✜❝
◗✉❡st✐♦♥
P✉❜❧✐s❤❡❞
❆rt✐❝❧❡
◆❛t✉r❡✴❙②st❡♠✴✳✳✳
■♥s♣✐r❡❞ ❜② ❘♦❣❡r ❉✳ P❡♥❣✬s ❧❡❝t✉r❡ ♦♥ r❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✱ ▼❛② ✷✵✶✹❆r♥❛✉❞ ▲❡❣r❛♥❞ ❇r✐❞❣✐♥❣ ❚❤❡ ●❛♣ ✶✸ ✴ ✶✼
❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚r②✐♥❣ t♦ ❇r✐❞❣❡ t❤❡ ●❛♣
❆♥❛❧②t✐❝
❉❛t❛
❈♦♠♣✉t❛t✐♦♥❛❧
❘❡s✉❧ts
▼❡❛s✉r❡❞
❉❛t❛
◆✉♠❡r✐❝❛❧
❙✉♠♠❛r✐❡s
❋✐❣✉r❡s
❚❛❜❧❡s
❚❡①t
❘❡❛❞❡r
❆✉t❤♦r
✭❉❡s✐❣♥ ♦❢ ❊①♣❡r✐♠❡♥ts✮
Pr♦t♦❝♦❧
❙❝✐❡♥t✐✜❝
◗✉❡st✐♦♥
P✉❜❧✐s❤❡❞
❆rt✐❝❧❡
◆❛t✉r❡✴❙②st❡♠✴✳✳✳
■♥s♣✐r❡❞ ❜② ❘♦❣❡r ❉✳ P❡♥❣✬s ❧❡❝t✉r❡ ♦♥ r❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✱ ▼❛② ✷✵✶✹❆r♥❛✉❞ ▲❡❣r❛♥❞ ❇r✐❞❣✐♥❣ ❚❤❡ ●❛♣ ✶✸ ✴ ✶✼
❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚r②✐♥❣ t♦ ❇r✐❞❣❡ t❤❡ ●❛♣
❚r② t♦ ❦❡❡♣ tr❛❝❦ ♦❢ t❤❡ ✇❤♦❧❡ ❝❤❛✐♥
❊①♣❡r✐♠❡♥t ❈♦❞❡
✭✇♦r❦❧♦❛❞ ✐♥❥❡❝t♦r✱ ❱▼ r❡❝✐♣❡s✱ ✳✳✳✮
Pr♦❝❡ss✐♥❣
❈♦❞❡
❆♥❛❧②s✐s
❈♦❞❡
Pr❡s❡♥t❛t✐♦♥
❈♦❞❡
❆♥❛❧②t✐❝
❉❛t❛
❈♦♠♣✉t❛t✐♦♥❛❧
❘❡s✉❧ts
▼❡❛s✉r❡❞
❉❛t❛
◆✉♠❡r✐❝❛❧
❙✉♠♠❛r✐❡s
❋✐❣✉r❡s
❚❛❜❧❡s
❚❡①t
❘❡❛❞❡r
❆✉t❤♦r
✭❉❡s✐❣♥ ♦❢ ❊①♣❡r✐♠❡♥ts✮
Pr♦t♦❝♦❧
❙❝✐❡♥t✐✜❝
◗✉❡st✐♦♥
P✉❜❧✐s❤❡❞
❆rt✐❝❧❡
◆❛t✉r❡✴❙②st❡♠✴✳✳✳
■♥s♣✐r❡❞ ❜② ❘♦❣❡r ❉✳ P❡♥❣✬s ❧❡❝t✉r❡ ♦♥ r❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✱ ▼❛② ✷✵✶✹❆r♥❛✉❞ ▲❡❣r❛♥❞ ❇r✐❞❣✐♥❣ ❚❤❡ ●❛♣ ✶✸ ✴ ✶✼
▼②t❤❜✉st❡rs✿ ❙❝✐❡♥❝❡ ✈s✳ ❙❝r❡✇✐♥❣ ❆r♦✉♥❞
❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚r②✐♥❣ t♦ ❇r✐❞❣❡ t❤❡ ●❛♣✧❚r✐❝❦②✧♣❛rt
✧❊❛s②✧ ♣❛rt
✧❚r✐❝❦②✧ ❛♥❞ ✧❊❛s②✧ r❡❢❡r t♦
♣❛r❛❧❧❡❧ ❝♦♠♣✉t❡r s❝✐❡♥t✐st ✉s❡ ❝❛s❡s
❊①♣❡r✐♠❡♥t ❈♦❞❡
✭✇♦r❦❧♦❛❞ ✐♥❥❡❝t♦r✱ ❱▼ r❡❝✐♣❡s✱ ✳✳✳✮
Pr♦❝❡ss✐♥❣
❈♦❞❡
❆♥❛❧②s✐s
❈♦❞❡
Pr❡s❡♥t❛t✐♦♥
❈♦❞❡
❆♥❛❧②t✐❝
❉❛t❛
❈♦♠♣✉t❛t✐♦♥❛❧
❘❡s✉❧ts
▼❡❛s✉r❡❞
❉❛t❛
◆✉♠❡r✐❝❛❧
❙✉♠♠❛r✐❡s
❋✐❣✉r❡s
❚❛❜❧❡s
❚❡①t
❘❡❛❞❡r
❆✉t❤♦r
✭❉❡s✐❣♥ ♦❢ ❊①♣❡r✐♠❡♥ts✮
Pr♦t♦❝♦❧
❙❝✐❡♥t✐✜❝
◗✉❡st✐♦♥
P✉❜❧✐s❤❡❞
❆rt✐❝❧❡
◆❛t✉r❡✴❙②st❡♠✴✳✳✳
■♥s♣✐r❡❞ ❜② ❘♦❣❡r ❉✳ P❡♥❣✬s ❧❡❝t✉r❡ ♦♥ r❡♣r♦❞✉❝✐❜❧❡ r❡s❡❛r❝❤✱ ▼❛② ✷✵✶✹❆r♥❛✉❞ ▲❡❣r❛♥❞ ❇r✐❞❣✐♥❣ ❚❤❡ ●❛♣ ✶✺ ✴ ✶✼
❙♦ ▼❛♥② ◆❡✇ ❚♦♦❧s
New Tools for Computational Reproducibility
• Dissemination Platforms:
• Workflow Tracking and Research Environments:
• Embedded Publishing:
VisTrails Kepler CDE
Galaxy GenePattern Synapse
Sumatra Taverna Pegasus
Verifiable Computational Research Sweave knitR
Collage Authoring Environment SHARE
ResearchCompendia.org IPOL Madagascar
MLOSS.org thedatahub.org nanoHUB.orgOpen Science Framework The DataVerse Network RunMyCode.org
❈♦✉rt❡s② ♦❢ ❱✐❝t♦r✐❛ ❙t♦❞❞❡♥ ✭❯❈ ❉❛✈✐s✱ ❋❡❜ ✶✸✱ ✷✵✶✹✮
❆♥❞ ❛❧s♦✿ ❋✐❣s❤❛r❡✴❩❡♥♦❞♦✱ ❆❝t✐✈❡P❛♣❡rs✱ ❊❧s❡✈✐❡r ❡①❡❝✉t❛❜❧❡ ♣❛♣❡r✱✳ ✳ ✳
❆r♥❛✉❞ ▲❡❣r❛♥❞ ▼❛♥② ❆❧t❡r♥❛t✐✈❡s ❢♦r ❘❡♣r♦❞✉❝✐❜❧❡ ❆♥❛❧②s✐s ✶✻ ✴ ✶✼
❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ t❤❡ ◆❡✇ ❇✉③③✇♦r❞❄
❍✷✵✷✵✲❊■◆❋❘❆✲✷✵✶✹✲✷✵✶✺
❆ ❦❡② ❡❧❡♠❡♥t ✇✐❧❧ ❜❡ ❝❛♣❛❝✐t② ❜✉✐❧❞✐♥❣ t♦ ❧✐♥❦ ❧✐t❡r❛t✉r❡ ❛♥❞ ❞❛t❛
✐♥ ♦r❞❡r t♦ ❡♥❛❜❧❡ ❛ ♠♦r❡ tr❛♥s♣❛r❡♥t ❡✈❛❧✉❛t✐♦♥ ♦❢ r❡s❡❛r❝❤ ❛♥❞
r❡♣r♦❞✉❝✐❜✐❧✐t② ♦❢ r❡s✉❧ts✳
▼♦r❡ ❛♥❞ ♠♦r❡ ✇♦r❦s❤♦♣s
❲♦r❦s❤♦♣ ♦♥ ❉✉♣❧✐❝❛t✐♥❣✱ ❉❡❝♦♥str✉❝t✐♥❣ ❛♥❞ ❉❡❜✉♥❦✐♥❣ ✭❲❉❉❉✮ ✭✷✵✶✹ ❡❞✐t✐♦♥✮
❘❡♣r♦❞✉❝✐❜❧❡ ❘❡s❡❛r❝❤✿ ❚♦♦❧s ❛♥❞ ❙tr❛t❡❣✐❡s ❢♦r ❙❝✐❡♥t✐✜❝ ❈♦♠♣✉t✐♥❣✭✷✵✶✶✮
❲♦r❦✐♥❣ t♦✇❛r❞s ❙✉st❛✐♥❛❜❧❡ ❙♦❢t✇❛r❡ ❢♦r ❙❝✐❡♥❝❡✿ Pr❛❝t✐❝❡ ❛♥❞ ❊①♣❡r✐❡♥❝❡s ✭✷✵✶✸✮
❘❡♣r♦❞✉❝✐❜✐❧✐t②❅❳❙❊❉❊✿ ❆♥ ❳❙❊❉❊✶✹ ❲♦r❦s❤♦♣
❚❘❯❙❚ ✷✵✶✹
❘❊PP❆❘✬✶✺✿ ✷♥❞ ■♥t❡r♥❛t✐♦♥❛❧ ❲♦r❦s❤♦♣ ♦♥ ❘❡♣r♦❞✉❝✐❜✐❧✐t② ✐♥ P❛r❛❧❧❡❧ ❈♦♠♣✉t✐♥❣
❘❡♣r♦❞✉❝✐❜✐❧✐t② ♦❢ ❈♦♠♣✉t❛t✐♦♥ ❇❛s❡❞ ❘❡s❡❛r❝❤✴◆❚▼❙✬✶✺
❙❤♦✉❧❞ ❜❡ s❡❡♥ ❛s ♦♣♣♦rt✉♥✐t✐❡s t♦ s❤❛r❡ ❡①♣❡r✐❡♥❝❡✳
r❡❝❤❡r❝❤❡✲r❡♣r♦❞✉❝t✐❜❧❡❅❧✐st❡s✳✉♥✐✈✲♦r❧❡❛♥s✳❢r
❆r♥❛✉❞ ▲❡❣r❛♥❞ ▼❛♥② ❆❧t❡r♥❛t✐✈❡s ❢♦r ❘❡♣r♦❞✉❝✐❜❧❡ ❆♥❛❧②s✐s ✶✼ ✴ ✶✼