piwowar amia 2008: identifying data sharing in biomedical literature

Identifying data sharingin the biomedical literature

Heather Piwowar and Wendy Chapman

Department of Biomedical Informatics, U of Pittsburgh

Visualized as a “Wordle” (font size ~ word frequency, location and orientation are random)

Our full paper:

Created at IBM’s data sharing and visualization site Many Eyes

Our aim:

Identify research articles for which the authors have shared their datasets

For this research:

sharing = submitted to centralized databases

Links between article and dataare important

The data provides detail for the results of the article

The article provides detail for the data

Specialized searching methods help us find articles OR data...

but what about when we want articles WITH data?

How can we find articles that have shared their datasets?

Sometimes the links are easy to discover

1. Through database citations:

When authors upload data to a database, they have the opportunity to cite the paper that describes the data collection

Text

Unfortunately, the citation is often left blankbecause the data is submitted before

the paper is published

2. Through hyperlink urls in the text

Authors often reference their datasets within their paper with a website url

But the meaning of the hyperlinks is ambiguous. Sometimes they point to datasets that have been

accessed, rather than submitted.

But the meaning of the hyperlinks is ambiguous. Sometimes they point to datasets that have been

accessed, rather than submitted.

And often the text contains no hyperlinks at all:

3. Through text mining

What if we could extract phrases like

“data of the experiment can be accessed at”

full-text phrases containing “... accessed”

“can be accessed” suggests data is shared

BUT “was/were accessed” suggests data reuse!

full-text phrases containing “... downloaded”

“was/were downloaded” suggests data reuse

while “can be downloaded” suggests data sharing

Our aim:

Identify research articles for which the authors have shared their raw datasets.

Proposed approach:

Develop a system to identify statements of shared data from an article’s full text.

Materials:

Full text from a subset of the open access literature

Database submission citations from five databases:

• Genbank

• Protein Data Bank

• Gene Expression Omnibus

• ArrayExpress

• Stanford Microarray Database

Our Gold standard:

An article was considered to have a “shared dataset” if the article was cited within the primary submission field of a database entry

(+ a small amount of manual screening to find additional positives based on full text)

Approach:

For those articles that mention database names,

• Extract a 300-character window around every mention of a database name

• Apply various mining algorithms to decide if there is evidence that the authors deposited data from this study in the database

Results:

• queried 24 000 articles across 27 journals

• 25% of all open access articles mentioned one of the database names (50% Genbank)

• development set of 4434 articlestraining set of 2000test set of 1028

True positives:

23% of the articles that mentioned a database were cited from within a database submission field

= evidence that article shared its data!

Three simple methods for identifying sharing

Does the excerpt surrounding the database name contain:

1. the word “accession”

2. an accession number

3. a URL

Two complex methods:

4. A manually-derived regular expression to match lexical cues that suggest sharing

5. An automatically-derived bag of words decision tree

Snippet of manually-developed regular expression

wehavehasisarewaswerebebeen

+

accessionedaddedarchivedassigneddepositedenteredimportedincludedinsertedloadedlodgedplacedpostedprovidedregisteredreported tostoredsubmitteduploaded to

How accurately were these methods able to identify papers with evidence of public database submissions?

Recall: % of papers cited in database submission fields that were found by our methods

Recall: % of papers cited in database submission fields that were found by our methods

Best method for

recall depends on

database

Recall: % of papers cited in database submission fields that were found by our methods

“accession”good for

some, <url> for others

Recall: % of papers cited in database submission fields that were found by our methods

lexical regular

expressions do well overall

Precision: % of papers found by our methods that were cited in database submissions fields

Precision: % of papers found by our methods that were cited in database submissions fields

lexical regular

expressions do well overall,

bag-of-words doeseven better

Precision: % of papers found by our methods that were cited in database submissions fields

Precision of simple

patterns depends on

database

Precision: % of papers found by our methods that were cited in database submissions fields

Simple patterns do poorly on the most popular

databases (those with the most

statements of reuse?)

Precision vs. Recall plot of all methods for each database.

Diverse!

<url>

bag of words

“accession”

<accession>

<lexical patterns>

Relative strength of methods for this taskacross databases

Limitations:

• bias due to manual screening of negatives

• database-centric classifier

• approach requires computational access to literature full text!

Impact:

• A recent version that runs in PubMed Central:

• could increase GEO article links by 2.6%

• by 5.5% annually when all NIH in PMC

• double the recall (to 80%), double these estimates

• 40 links already added by GEO staff!

Ongoing work:

1. Continue focusing on methods that use existing full-text query interfaces, like PubMed Central

2. Use this tool to evaluate the patterns and prevalence ofbiomedical research data sharing and reuse

Thanks to

the Dept of Biomedical Informatics at the U of Pittsburgh,

the NLM for funding through training grant 5 T15 LM007059-22,

and everyone who publishes “gold” open access, thereby facilitates reuse of article full text for studies like this.

My shared data: www.dbmi.pitt.edu/piwowarShare your research data too!

Our manual filter for additional positive classifications identified more cases in some databases than others: we

reclassified 19% of [article,database] cases from ArrayExpress as positive despite an omitted literature

link, compared to 11%, 7%, 2%, and 1% for GEO, Genbank, PDB, and SMD respectively (see Table 2 for raw number of cases). The most common situations included: the

database entry listed a citation for another paper by the same authors, the entry listed an erroneous PubMed ID,

the entry included a citation without a PubMed ID, or the entry had a blank citation field.

Usage?

• scientists looking for datasets for reuse

• curators looking for primary citations

• researchers studying data sharing behaviour

Regular expression

• Precise one +

• "(\b(accession.{0,20}(for|at).{0,100}(is|are)))",

• r"(\b(raw|original|our|complete|detailed).{0,20}data)",

• r"(\b(we|have|is|was|were|is|are|be|have|has|been).(exported|gave|given|listed|provided|reported))"

• ]) + ")"

Precise Regular expression

• wehavehasisarewaswerebebeen

accessioned|added|archived|assigned|deposited|entered|imported|included|inserted|loaded|lodged|placed|posted|provided|registered|reported.to|stored|submitted|uploaded.to))",

is|are|will.be|made).{0,20}(available|accessible)

(be).(accessed|browsed|downloaded|found|obtained|queried|retrieved|searched|viewed)

(through|under|as).{0,20}accession

(given)|new|received|assigned).{0,20}(accession)

(data.{0,20}availability|for public distribution|for.{0,20}release upon publication|for the.{0,20}data.{0,20}generated|from this study have.{0,20}accession|data.{0,10}from this study|access to.{0,20}data.

Stopwords are important!

Recall

Precision

• queried 24 000 articles across 27 journals

• 25% mentioned one of the database names

• development set of 4434 training set of 2000test set of 1028

Evaluation

Research data

http://upload.wikimedia.org/wikipedia/commons/7/76/PeptideMSMS.jpg; http://en.wikipedia.org/wiki/Image:Helices.png; http://en.wikipedia.org/wiki/Image:Heatmap.png; http://en.wikipedia.org/wiki/Image:Microarray2.gif;

http://zellig.cpmc.columbia.edu/medlee/demo/; htp://www.plosone.org/article/fetchArticle.action?articleURI=info:doi/10.1371/journal.pone.0000441

Research data

http://upload.wikimedia.org/wikipedia/commons/7/76/PeptideMSMS.jpg; http://en.wikipedia.org/wiki/Image:Helices.png; http://en.wikipedia.org/wiki/Image:Heatmap.png; http://en.wikipedia.org/wiki/Image:Microarray2.gif;

http://zellig.cpmc.columbia.edu/medlee/demo/; htp://www.plosone.org/article/fetchArticle.action?articleURI=info:doi/10.1371/journal.pone.0000441

PAST MEDICAL HISTORY:Past medical history showed she had superficial phlebitis times two in the past, had non-insulin

dependent diabetes mellitus for four years.She had been hypothyroid for three years.

HISTORY OF PRESENT ILLNESS:The patient is a 58-year-old female, …

Research data

http://upload.wikimedia.org/wikipedia/commons/7/76/PeptideMSMS.jpg; http://en.wikipedia.org/wiki/Image:Helices.png; http://en.wikipedia.org/wiki/Image:Heatmap.png; http://en.wikipedia.org/wiki/Image:Microarray2.gif;

http://zellig.cpmc.columbia.edu/medlee/demo/; htp://www.plosone.org/article/fetchArticle.action?articleURI=info:doi/10.1371/journal.pone.0000441

PAST MEDICAL HISTORY:Past medical history showed she had superficial phlebitis times two in the past, had non-insulin

dependent diabetes mellitus for four years.She had been hypothyroid for three years.

HISTORY OF PRESENT ILLNESS:The patient is a 58-year-old female, …

Research data

http://upload.wikimedia.org/wikipedia/commons/7/76/PeptideMSMS.jpg; http://en.wikipedia.org/wiki/Image:Helices.png; http://en.wikipedia.org/wiki/Image:Heatmap.png; http://en.wikipedia.org/wiki/Image:Microarray2.gif;

http://zellig.cpmc.columbia.edu/medlee/demo/; htp://www.plosone.org/article/fetchArticle.action?articleURI=info:doi/10.1371/journal.pone.0000441

PAST MEDICAL HISTORY:Past medical history showed she had superficial phlebitis times two in the past, had non-insulin

dependent diabetes mellitus for four years.She had been hypothyroid for three years.

HISTORY OF PRESENT ILLNESS:The patient is a 58-year-old female, …

Research data

http://upload.wikimedia.org/wikipedia/commons/7/76/PeptideMSMS.jpg; http://en.wikipedia.org/wiki/Image:Helices.png; http://en.wikipedia.org/wiki/Image:Heatmap.png; http://en.wikipedia.org/wiki/Image:Microarray2.gif;

http://zellig.cpmc.columbia.edu/medlee/demo/; htp://www.plosone.org/article/fetchArticle.action?articleURI=info:doi/10.1371/journal.pone.0000441

PAST MEDICAL HISTORY:Past medical history showed she had superficial phlebitis times two in the past, had non-insulin

dependent diabetes mellitus for four years.She had been hypothyroid for three years.

HISTORY OF PRESENT ILLNESS:The patient is a 58-year-old female, …

Research data

http://upload.wikimedia.org/wikipedia/commons/7/76/PeptideMSMS.jpg; http://en.wikipedia.org/wiki/Image:Helices.png; http://en.wikipedia.org/wiki/Image:Heatmap.png; http://en.wikipedia.org/wiki/Image:Microarray2.gif;

http://zellig.cpmc.columbia.edu/medlee/demo/; htp://www.plosone.org/article/fetchArticle.action?articleURI=info:doi/10.1371/journal.pone.0000441