Download - Human Assessment of Ontologies
The Third International Conference on Information Systems and Technologies
ICIST 2013March 22 – March 24, 2013 - Tangier, Morocco
Position Paper: A New Approach for
Human Assessment of Ontologies
Authors:Leila ZEMMOUCHI-GHOMARI, [email protected]
UMBB, M’hamed Bouguerra University Boumèrdes, www.umbb.dzBoumèrdes, ALGERIA
&Abdessamed Réda GHOMARI, [email protected]
LMCS LaboratoryESI, national Superior School of Computer Science, www.esi.dz
Algiers, ALGERIA
2
OUTLINE
ONTOLOGY EVALUATION
RELATED WORK
PROPOSED APPROACH
CASE STUDY
3
Why ontology evaluation ?
Involved in Selection of an ontology with regard to Objectives of use or reuse Several ontologies: suitable ontology Single ontology: Quality of ontology (good or
bad quality ontology) Involved in an ontology engineering process
Ontology evaluation is a crucial step in this process (at the end or through the whole process)
4
ONTOLOGY EVALUATION TYPES[Gomez-Perez, 2004]
ONTOLOGY VERIFICATIONDeals with building the ontology correctly
ONTOLOGY VALIDATIONDeals with the correspondence between the semantics of the model and the real world for which the ontology was designed
5
ONTOLOGY VALIDATION APPROACHES
Comparison with a gold standard or a reference ontology
Comparison with a source of data Application based-ontology assessment Human assessment
ontology developer
end-user
domain expert
6
RELATED WORK
Human assessment of ontologies fits into
ontology verification area. It is intended to detect mistakes and inconsistencies that occur with human modeling.
For example: in [Ceusters and Smith, 04, 05]:
NCI (National Cancer Institute thesaurus)
SNOMED (Systematized Nomenclature of Medicine)
7
RELATED WORK
Human assessment of ontologies fits into
ontology verification area. It is intended to detect mistakes and inconsistencies that occur with human modeling.
For example: in [Ceusters and Smith, 04, 05]:
NCI (National Cancer Institute thesaurus)
SNOMED (Systematized Nomenclature of Medicine)
missing or inappropriately allocated informal and formal
definitions shifts in terms meaning and redundancy in concepts.
8
Scope of this presentation
ontology validation area which relates the degree of correspondence of the ontology to that part of reality which it is designed to represent from the point of view of domain experts.
9
Why is ontology human assessment difficult?
Some quality attributes judged by domain experts, such as clarity, relevance and accuracy can be difficult to evaluate as they may not be easily quantifiable
10
PROPOSED APPROACH
11
STEP 1: DERIVATION OF QUESTIONNAIRE FROM ONTOLOGY
Ontology expressed in a web language (RDF, OWL)
Questionnaire expressed in natural language composed of four parts:
Hierarchical ontology levels (ontology depth) Axioms Relations between concepts Descriptive attributes of concepts
12
We propose possible answers to questions, we rely on principles mandatory for good quality ontologies:
Clarity: ontology is easily understood by the users so that it can be consistently applied and interpreted across the domain of interest
Lawfulness : knowledge described by the ontology is encoded
with meaningful terms. This is achieved by checking out that the words used by the ontology are appropriate
Accuracy: claims an ontology makes are right or wrong
Relevance: ontology satisfies ontology requirements or not
13
Criterion Possible Answers Answers’ Location
Clarity Not Clear Classes validation Axioms validation Relations validation Attributes validation
Lawfulness
Right but another term would be more appropriate Relations validation
Relevant but used terms are not appropriate
Attributes validation
Accuracy
Right Wrong
Classes validation Relations validation
Always Sometimes Never
Axioms validation
Relevance Relevant Not really relevant Not relevant at all
Attributes validation
14
THE OTHER STEPS
STEP 2 (Aggregation of questionnaire results) is performed automatically by web form module (like drupal webform)
STEP 3 & STEP 4 (Analysis and Synthesis of obtained results & questionnaire update):
Delphi method [Dalker & Helmer, 1963]: its purpose is to achieve convergence of opinions of experts concerning a specific topic using questionnaire.
Generally, 3 iterations of updated questionnaire are sufficient to reach a consensus
15
CASE STUDY we built an ontology called HERO ontology which
stands for “Higher Education Reference Ontology” we derived a questionnaire (100 questions) from
ontology elements and proposed MCQ as possible answers according to ontology quality criteria
16
17
18
19
QUESTIONAIRE AGGREGATED RESULTS
20
ONTOLOGY UPDATE
21
CONCLUSION
The purpose of this proposal is to define a methodological baseline for human assessment of ontologies and to carry out a practical case study for its applicability
Limitations Much more experiences are needed about the
practical usage of proposed guidelines Semi-automatic support of the approach is
required
22
REFERENCES
A. Gomez-Pérez, Ontology Evaluation, Handbook on Ontologies, pp 251-274, 2004.
J. Brank, M. Grobelnik, and D. Mladenic, “A survey of ontology evaluation techniques”, Proceedings of Data Mining and Data Warehouses (SiKDD), 2005.
N.C Dalkey, and O. Helmer, “An experimental application of the Delphi method to the use of experts”, Management Science, 9 (3), pp 458-467, 1963.
More references are included in the paper