research into open research data

Open research data

Heather PiwowarDataONE postdoc with Dryad and NESCent, UBC

@researchremix

OA week 2010University of British Columbia

It matters

http://www.metmuseum.org/toah/ho/09/euwf/ho_24.45.1.htm

http://www.flickr.com/photos/jsmjr/62443357/

http://www.flickr.com/photos/camilleharrington/3587294608/

http://www.flickr.com/photos/rkuhnau/3318245976/

http://www.flickr.com/photos/conformpdx/1796399674/

http://www.flickr.com/photos/rkuhnau/3317418699/

http://www.flickr.com/photos/zemlinki/261617721/

http://www.flickr.com/photos/tracenmatt/3020786491/

http://www.flickr.com/photos/the-o/2078239333/

http://www.flickr.com/photos/75166820@N00/5318468/

Wayfinding + progress

http://www.flickr.com/photos/paulhami/1020538523//

Which data?

Where?

With whom?

Under what terms?

FindOrganizeDocumentDeidentifyFormatAskSubmit

Answer questionsWorry about mistakes being foundWorry about data being misinterpretedWorry about being scoopedForgo money and IP and prestige???

not very motivating.

http://www.flickr.com/photos/tonivc/2283676770/

http://www.flickr.com/photos/johnnyvulkan/381941233/

a) policies + expectations

- NSF- Joint Data Archiving Policy- BioMed Central- PLoS

b) repositories

- datatype-based- institution-based- discipline-based- journal-based

c) standards

- data licenses- data citation - IDs for datasets, people, entities

d) part of something bigger

- open government data- citizen science- supplemental materials- dataset-based usage metrics- awards, recognition

Is it working?

http://www.genome.jp/en/db_growth.html

lots of data sharing!

but how much isn’t shared?

what isn’t shared?

who isn’t sharing it?why not?

what can we do about it?

how much does it matter?

you can not manage what you do not measure

quote: Lord Kelvinhttp://www.flickr.com/photos/archeon/2941655917/

http://www.flickr.com/photos/ryanr/142455033/

Why is it important?Are we sure?

Errors.

Gore et al 1977, Kantoer and Taylor 1994, McGuigan 1995, Hurlbert and White 1993

More than half of all papers contain errors

5‐10% contain errors that change the conclusions

Ok, let’s share on request.

Doesn’t work

self-reported denying a request in last 3 years

trainees self-reported denying a request

been denied access to data, materials, code

authors “not able to retrieve raw data”

not willing to release data

0% 10% 20% 30% 40%

Campbell et al. JAMA. 2002.Kyzas et al. J Natl Cancer Inst. 2005.

Vogeli et al. Acad Med. 2006.Reidpath et al. Bioethics 2001.

Don’t get the email

Evangelou et al. FASEB J. 2006.Wren. Bioinformatics 2008.Wren et al. EMBO Rep 2006.

Say no

Hedstrom. Society of Am Archivists Ann Meeting. 2008.

want to publish more papers first

want exclusive use

ensure data confidentiality

control

avoid cost of preparation0% 10% 20% 30% 40% 50%

Ask why

Reidpath et al. Bioethics 2001.

`Before I send you the data could I ask what you want it for?'

`Can you be more explicit, please, about the analyses you have in mind and what you plan to do with them?'

`We'll have to discuss your request with the other coauthors. Before we do that, I'd like to know your proposed analysis plan.'

`We are not finished using the data, but when we are finished with it, we would be open to requests for the data.'

`Any use of the data other than for the specific purpose laid down in the contract of collaboration is effectively ruled out.'

Not efficient.

Not efficient. Not fair.

Campbell et all 2000

Not random:

‐ young

‐ productive

Has real costs.Survey of doctoral students and postdocs:

28-50% reported withholding negative effects:• hurt progress of their research, • hurt rate of discovery in their lab/research group, • hurt quality of their relationships with academic

scientists, • hurt quality of their education, • hurt level of communication in their lab/research

group.

Vogeli et al. Acad Med. 2006 Feb; 81(2):128-36

Ok, then on a website?No. Urls stop working.

Evangelou et al. FASEB J. 2006.Wren. Bioinformatics 2008.Wren et al. EMBO Rep 2006.

Ok, in a repository?

lots of data sharing!

http://www.genome.jp/en/db_growth.html

http://www.flickr.com/photos/g_kat26/4255119413/

http://www.flickr.com/photos/jima/606588905/

Combined, these full-text portals reach 85% of the articles available through U of Pittsburgh library subscriptions.

microarray data

http://en.wikipedia.org/wiki/DNA_microarray

http://en.wikipedia.org/wiki/Image:Heatmap.png

http://commons.wikimedia.org/wiki/File:DNA_double_helix_vertikal.PNG

11,603 studies that created

gene expression microarray data

Is research data shared after publication?

Funder Journal Investigator Institution Study

funded by NIH?

size of grant

sharing plan req’d?

funded by non-NIH?

impact factor

strength of policy

open access?

number of microarray studies published

years since first paper

# pubs

# citations

previously shared?

previously reused?

gender

sector

impact rank

country

humans?

plants?

cancer?

clinical trial?

number of authors

Funder Journal Investigator Institution Study

“An inherent principle of publication is that others should be able to replicate and build upon the authors' published claims. Therefore, a condition of publication in a Nature journal is that authors are required to make materials, data and associated protocols available in a publicly accessible database …”

http://www.nature.com/authors/editorial_policies/availability.html

http://www.nature.com/nature/journal/v453/n7197/index.html

journal data sharing policy

journal rank

institution rank

Yu et al. BMC medical informatics and decision making (2007) vol. 7 pp. 17

funding level

PubMed grant lists + NIH grant details

study type

author gender

and so on...

124 variables

11,603 studies

25% had links from datasets in databases

Year article published

2000 2001 2002 2003 2004 2005 2006 2007 2008 2009

Proportion of articles with shared datasets, by year

Across time

What can we do about it?

Funder policies.

Piwowar and Chapman. Journal of Informetrics 2010

What can we do about it? Journal policies.

We looked at data sharing policies within Instruction to Author statements of 70 journals, as they apply to gene expression microarray data.

Piwowar and Chapman. ELPUB 2008

No applicable policy (43%)

Weak policy (24%)

should, recommend, requestmust, but without requiring database accession number

Strong policy (33%)

must, required, condition of publicationrequires database accession number

strength of data sharing policies

High-impact journals

tend to have

a strong data-sharing

policy

Articles published in journals with a strong data-sharing policy are more likely to have publicly

available datasets

What can we do about it?

• Learn from those who do it well• Focus on places that need it

Microbiology

Diabetes

Genomics

dCarcinogenesis

Endocrinology

Oncogene

f data

1.0 Journals(Physiological Genomics)

rsity o

f Illin

rsity o

ritish

rsity o

rsity M

rsity o

rsity in

rsity o

Institutions(Stanford)

Institutionrank

Odds Ratio

0.25 0.50 1.00 2.00 4.00 8.00

Has journal policy0.95Count of R01 & other NIH grants

Authors prev GEOAE sharing & OA & microarray creation

NO K funding or P funding

Institution high citations & collaboration

Journal impact

Journal policy consequences & long halflife

NOT animals or mice

Instititution is government & NOT higher ed

Last author num prev pubs & first year pub

Large NIH grant

Humans & cancer

NO geo reuse + YES high institution output

First author num prev pubs & first year pub

Multivariate nonlinear regressions with interactions

Odds Ratio

0.25 0.50 1.00 2.00 4.00 8.00

Journal impact

NOT animals or mice

Large NIH grant

Humans & cancer

Odds Ratio

0.25 0.50 1.00 2.00 4.00 8.00

Journal impact

NOT animals or mice

Large NIH grant

Humans & cancer

Odds Ratio

0.25 0.50 1.00 2.00 4.00 8.00

Journal impact

NOT animals or mice

Large NIH grant

Humans & cancer

Odds Ratio

0.25 0.50 1.00 2.00 4.00

OA journal & previous GEO-AE sharing

0.95Amount of NIH funding

Journal impact factor and policy

Higher Ed in USA

Cancer & humans

Multivariate nonlinear regression with interactions

Odds Ratio

0.25 0.50 1.00 2.00 4.00

OA journal & previous GEO-AE sharing

0.95Amount of NIH funding

Journal impact factor and policy

Higher Ed in USA

Cancer & humans

Multivariate nonlinear regression with interactions

Carrot?

http://www.flickr.com/photos/sunrise/35819369/

currency of value?

Citations.

currency of value?

Citations.

Diamond,Arthur M. What is a Citation Worth?. The Journal of Human Resources (1986) vol. 21 (2) pp. 200-215

dataset85 cancer microarray trials published in 1999-2003, as identified by Ntzani and Ioannidis (2003)

citationsISI Web of Science Citation index, citations from 2004-2005

data sharing locationsPublisher and lab websites, microarray databases, WayBack Internet Archive, Oncomine

statisticsMultivariate linear regression

Note:log scale

http://www.flickr.com/photos/gatewaystreets/3838452287/

Impact of JDAP

Abadie et al. Journal of the American Statistical Association 2010

Reuse.

http://www.flickr.com/photos/boitabulle/3668162701/

http://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/Gamma_distribution_pdf.svg/500px-Gamma_distribution_pdf.svg.png

We are the culture.Let’s do it.

http://www.flickr.com/photos/joellevand/279468607/

http://www.flickr.com/photos/huzzahvintage/4577075021/

a) in our communities

- strengthening policies:- journal, conference, institutional

- decision-makers- role-models and educators

b) in our tools

- measure opinions- measure use- be transparent!

c) with our data

- share it.- ugly? incomplete? strange?

“Flawed, but out there” is a million times better than “perfect, but unattainable”

http://sciblogs.co.nz/seeing-data/2010/10/12/the-zen-of-open-data/

“Does anyone want your data?

That’s hard to predict […] After all, no one ever knocked on your door asking to buy those figurines collecting dust in your cabinet before you listed them on eBay.

Your data, too, may simply be awaiting an effective matchmaker.”

Got data? Nature Neuroscience (2007)

I post my data, code, and statistical scripts: http://researchremix.org

Share yours too!

http://www.flickr.com/photos/myklroventine/892446624/

More info?

• OATP oa.data tag on Connotea, Twi1er

• FriendFeed• Mendeley “data sharing” group

• @researchremix piwowar@zoology.ubc.ca

thank youTodd Vision,

Michael Whitlock, Wendy Chapman

The open science online community and those who release their articles, datasets and photos openly

http://www.flickr.com/photos/youraddresshere/6649228/

research into open research data

multivariate

multivariate

geo reuse

ae sharing

high institution

policy higher

usa cancer

higher ed

Technology

research article investigations into deformation...

community mozilla's analytics - github pages · •...

translating research into action: an international study of...

open research online an investigation into the use of...

insights into the transcriptional and post-transcriptional...

research open access two antarctic penguin genomes...

research article open access an investigation into ige...

metatranscriptomic and comparative genomic insights into...

welcome to lsbu open research : lsbu open research · web...

open research into open education: the role of mapping and...

research article open access genomic and ......research...

research article open access insights into the molecular

open research online · 2020-06-12 · open research online...

gates open research · gates open research conclusion:...

research article open access diversification of...

research open access vpu-dependent block to incorporation of...

teachers’ perception of open educational resources:...

research article open access an investigation into ige...

research open access insights into archaeal evolution and

open education open research open innovation open ... ·...