unsw reproducibility workshop

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

A new era of large-scale replication and collaboration?

Alex.Holcombe@sydney.edu.au

@ceptional

http://www.slideshare.net/holcombea/

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

False positives zone Look for areas wherein victory declared whenever null hypothesis is rejected, where size of effect, plausibility considered unimportant.

Where science is not self-correcting

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

"This one's Psych Science. 1998. Put your money on ‘will replicate’?”

"What, you think I was born yesterday?”

"Fine," muttered the bookie.

#AcademicNoir

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Well, Mr. P. Hacker, your luck is about to run out… down at the journals, they’re running a new game. They call it ‘preregistration’.

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

5

https://osf.io/tvyxz/wiki/home/

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

6

https://osf.io/tvyxz/wiki/home/

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

7

https://osf.io/tvyxz/wiki/home/

100% P-hacking FreeHere, check our numbers.

Here’s how you can replicate our result.

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

"This one's Psych Science. 1998. Put your money on ‘will replicate’?”

"What, you think I was born yesterday?”

"Fine," muttered the bookie.

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php 9

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php 10

Failed to replicate my result?

Have you considered

the unknown moderators?

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php 11

Any one of us alone, he’ll say we messed up. But if we all run the replication together, we

can show the big guy we’re right.

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Meta-analysis forest plot

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Meta-analysis forest plot

Failed to replicate my result?

Have you considered the unknown moderators?

You messed up!

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

PreregistrationLarge-scale replicationOpen dataOpen protocol and code

A new era of large-scale replication

and collaboration?

Can we do this bottom-up?

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php

Reproducibility workshop 26 Nov http://bizlab.unsw.edu.au/website/conference-2015.php