the evaluation of ontologies editorial review vs. democratic ranking
DESCRIPTION
The Evaluation of Ontologies Editorial Review vs. Democratic Ranking. Barry Smith. The need for ontology evaluation. Ontologies are expensive. $8 million have been invested in the Gene Ontology thus far Has this investment been worthwhile? Are some ontologies more useful than others? - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/1.jpg)
The Evaluation of Ontologies
Editorial Review vs. Democratic Ranking
Barry Smith
1
![Page 2: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/2.jpg)
The need for ontology evaluation
Ontologies are expensive.
$8 million have been invested in the Gene Ontology thus far
Has this investment been worthwhile?
Are some ontologies more useful than others?
What are ontologies useful for ?2
![Page 3: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/3.jpg)
Uses of ‘ontology’ in PubMed abstracts
3
![Page 4: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/4.jpg)
In the olden days
people measured lengths using inches, ulnas, perches, king’s feet, Swiss feet, kanejaku, shaku, whale shaku, etc., etc.
5
![Page 5: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/5.jpg)
then, on June 22, 1799,everything changed
![Page 6: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/6.jpg)
we now have the International System of Units
![Page 7: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/7.jpg)
Through the SI System
science becomes a cumulative, distributed endeavormy measuring equipment can be callibrated against your measuring equipmentmy hypotheses can be checked against your data
![Page 8: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/8.jpg)
When should a new unit should be included in the SI system?
The work of the
CIPM International Committee for Weights and Measures
rests on an editorial process
9
![Page 9: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/9.jpg)
Obvious benefits of a peer review process for scientific work
Creating an environment which rewards better work
Helping people to find better work
...
10
![Page 10: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/10.jpg)
Proposal: Evaluate ontologies via an editorial process
of the sort used for scientific journals and scientific research projects
a process of peer review by human experts
11
![Page 11: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/11.jpg)
Peer review process
• appropriate when evaluating science
• (not, e.g., when evaluating poetry, or fairy tales, or Chinese mythology ...)
12
![Page 12: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/12.jpg)
Peer review process
appropriate in the domain of biomedical ontology
– ontology evaluation here can be of particularly acute concern
13
![Page 13: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/13.jpg)
14
RELATION TO TIME
GRANULARITY
CONTINUANT OCCURRENT
INDEPENDENT DEPENDENT
ORGAN ANDORGANISM
Organism(NCBI
Taxonomy)
Anatomical Entity(FMA, CARO)
OrganFunction
(FMP, CPRO) Phenotypic
Quality(PaTO)
Biological Process
(GO)CELL AND CELLULAR
COMPONENT
Cell(CL)
Cellular Compone
nt(FMA, GO)
Cellular Function
(GO)
MOLECULEMolecule
(ChEBI, SO,RnaO, PrO)
Molecular Function(GO)
Molecular Process
(GO)
The OBO Foundry
![Page 14: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/14.jpg)
THE OBO FOUNDRY
15
a suite of reference ontologies in the biomedical domain satisfying certain basic criteria and subject to an on-going process of peer review
![Page 15: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/15.jpg)
OBO FOUNDRY CRITERIA
16
The ontology is open and available to be used by all.
The ontology is in, or can be instantiated in, a common formal language.
The developers of the ontology agree in advance to collaborate with developers of other OBO Foundry ontology where domains overlap.
![Page 16: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/16.jpg)
17
CRITERIA UPDATE: The developers of each ontology
commit to its maintenance in light of scientific advance, and to soliciting community feedback for its improvement.
ORTHOGONALITY: They commit to working with other Foundry members to ensure that, for any particular domain, there is community convergence on a single controlled vocabulary.
![Page 17: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/17.jpg)
18
CRITERIA
IDENTIFIERS: The ontology possesses a unique identifier space within OBO.
VERSIONING: The ontology provider has procedures for identifying distinct successive versions.
The ontology includes textual definitions for all terms.
CRITERIA
![Page 18: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/18.jpg)
19
CLEARLY BOUNDED: The ontology has a clearly specified and clearly delineated content.
DOCUMENTATION: The ontology is well-documented.
USERS: The ontology has a plurality of independent users.
CRITERIA
![Page 19: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/19.jpg)
20
COMMON ARCHITECTURE: The ontology uses relations which are unambiguously defined following the pattern of definitions laid down in the OBO Relation Ontology
CRITERIA
![Page 20: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/20.jpg)
The OBO Foundry provides guidelines (traffic laws) to new groups of ontology developers in ways which can
• help to ensure interoperability through prospective synchronization
• counteract dispersion of effort • prevent some common types of
nonsense21
![Page 21: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/21.jpg)
22
Example: the Foundry seeks orthogonality
This brings division of labor and other benefits
Foundry editors adjudicate in areas of overlap
How the editorial process works
![Page 22: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/22.jpg)
23
Foundry editors balance– the flexibility that is indispensable to
scientific advance
– the institution of principles that is indispensable to successful coordination
How the editorial process works
![Page 23: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/23.jpg)
24
is a top down approach, relying on authority
Peer review
![Page 24: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/24.jpg)
25
democratic ranking
An alternative, bottom up approach
![Page 25: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/25.jpg)
26
If we build it, will they come? Social engineering of new technology to
disseminate biomedical ontologies
Mark A. Musen and the BioPortal Team
Stanford University
![Page 26: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/26.jpg)
27
If we build it, will they come? Social engineering of new technology to
disseminate biomedical ontologies
presentation to Ontolog Forum
July 6, 2007
![Page 27: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/27.jpg)
28
With thanks to
Mark Musen and Natasha Noy
![Page 28: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/28.jpg)
29
![Page 29: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/29.jpg)
30
In biology, lots of ontology developers are almost hobbyists
• Nearly always, ontologies are created to address pressing practical needs
• Biologists ... may have little appreciation for metaphysics, principles of knowledge representation, or computational logic
• There simply aren’t enough good ontologists to go around
![Page 30: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/30.jpg)
31
Issues in assuring ontology quality• Unlike the case with journal submissions, it makes no
sense for ontologies to be peer-reviewed by just a handful of experts
• Open, community-based review of ontologies may be haphazard and chaotic
• Top–down solutions may offer rigid review critieria at the expense of scalability
• There is a pressing need for empirical evaluation of methods for ontology evaluation
![Page 31: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/31.jpg)
32
OBO Foundry must address lots of questions
• Can the top-down approach scale? How many ontologies can be managed by a small panel of curators?
• Who gets to reject an ontology on the basis of form or content? What is the appeals process? How do we know whom to believe?
• Who will curate the curators?
![Page 32: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/32.jpg)
33
NCBO will offer
• Technology for uploading, browsing, and using biomedical ontologies
• Methods to make the online “publication” of ontologies more like that of journal articles
• Tools to enable the biomedical community to put ontologies to work on a daily basis
![Page 33: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/33.jpg)
34
http://bioportal.bioontology.org
![Page 34: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/34.jpg)
35Local Neighborhood view
Browsing/Visualizing Ontologies
![Page 35: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/35.jpg)
36
Hierarchy-to-root view
![Page 36: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/36.jpg)
38
Goals for BioPortal
• Web accessible repository of ontologies for the biomedical community– Archived locally– Anywhere in cyberspace
• Support for ontology– Peer review– Annotation (marginalia)– Versioning– Alignment– Search
![Page 37: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/37.jpg)
39
Ontologies are not like journal articles
• It is difficult to judge methodological soundness simply by inspection
• We may wish to use an ontology even though some portions – Are not well designed– Make distinctions that are different from
those that we might want
![Page 38: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/38.jpg)
40
Ontologies are not like journal articles
• The utility of ontologies– Depends on the task– May be highly subjective
• The expertise and biases of reviewers may vary widely with respect to different portions of an ontology
• Users should want the opinions of more than 2–3 hand-selected reviewers
• Peer review needs to scale to the entire user community
![Page 39: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/39.jpg)
41
Community-Based Annotation
• Makes ontology evaluation a democratic process
• Assumes users’ applications of ontologies will lead to insights not achievable by inspection alone
• Assumes end-users will be motivated to comment on and engage in dialog about ontologies in the repository
![Page 40: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/40.jpg)
![Page 41: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/41.jpg)
![Page 42: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/42.jpg)
Solution Snapshot
![Page 43: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/43.jpg)
45
Open ratings for ontologies• Any user can
– rate an ontology– add a “marginal note”
• Ontology evaluation becomes a community-based initiative
• A web of trust can enable users to filter comments or ratings to avoid “noise”
![Page 44: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/44.jpg)
Arguments in favor of the top-down approach in the scientific domain
• marginalia will contain a great deal of irrelevantalia
• scientists need ontologies, but are normally not experts in ontology; they are looking for authoritative guidelines
• the Foundry process is yielding guidelines on how to build ontologies compatible with those which already exist
46
![Page 45: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/45.jpg)
Arguments against the top-down approach
• ontologies are not like journal articles, and it is difficult to judge methodological soundness simply by inspection.
• the evaluation process does not yield a quantifiable result.
– but scientific journals face exactly similar problems, yet peer review, there, works well
47
![Page 46: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/46.jpg)
In defense of democratic rankings
• ranking by large numbers of users will tend to counteract such biases (but will the ranking service in fact attract users?)
• ranking by large numbers of users has a greater opportunity to scale up when ontologies proliferate
48
![Page 47: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/47.jpg)
We have common goals
• Both approaches seek quality assurance to support ontology selection.
• Both approaches need to address the fact that the expertise and biases of reviewers may vary widely with respect to different ontologies or to different portions of an ontology.
49
![Page 48: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/48.jpg)
One big difference
For Musen et al. there are no restrictions on entry
The bottom-up approach seeks community-based annotation of ontologies, with no difference being made between experts and non-experts
50
![Page 49: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/49.jpg)
One big difference
In the OBO Foundry reviews are created precisely by the peers of the ontology authors themselves—by persons with established and recognized expertise and with a demonstrated willingness to invest due diligence in ontology development, use, and evaluation.
51
![Page 50: The Evaluation of Ontologies Editorial Review vs. Democratic Ranking](https://reader036.vdocument.in/reader036/viewer/2022062500/56814f63550346895dbd18af/html5/thumbnails/50.jpg)
Both are needed
• in domains such as refrigerators
• but in science?
52