it's our presentation during the third international conference of information systems and technologies ICIST 2013 held at Tangier, Morocco in which we propose a new approach for human assessment of ontologies using an online questionnaire.

1. The Third International Conference on
Information Systems and Technologies
ICIST 2013
March 22 – March 24, 2013 - Tangier, Morocco
Position Paper:
A New Approach for
Human Assessment of Ontologies
Authors:
Leila ZEMMOUCHI-GHOMARI, l_ghomari@umbb.dz
UMBB, M’hamed Bouguerra University Boumèrdes, www.umbb.dz
Boumèrdes, ALGERIA
&
Abdessamed Réda GHOMARI, a_ghomari@esi.dz
LMCS Laboratory
ESI, national Superior School of Computer Science, www.esi.dz
Algiers, ALGERIA

2.  ONTOLOGY EVALUATION
 RELATED WORK
 PROPOSED APPROACH
 CASE STUDY
2

3.  Involved in Selection of an ontology with
regard to Objectives of use or reuse
 Several ontologies: suitable ontology
 Single ontology: Quality of ontology (good or
bad quality ontology)
 Involved in an ontology engineering process
Ontology evaluation is a crucial step in this
process (at the end or through the whole process)
3

4.  ONTOLOGY VERIFICATION
Deals with building the ontology correctly
 ONTOLOGY VALIDATION
Deals with the correspondence between the
semantics of the model and the real world for which
the ontology was designed
4

5.  Comparison with a gold standard or a reference
ontology
 Comparison with a source of data
 Application based-ontology assessment
 Human assessment
ontology developer
end-user
domain expert
5

6. Human assessment of ontologies fits into
ontology verification area. It is intended to detect
mistakes and inconsistencies that occur with human
modeling.
For example: in [Ceusters and Smith, 04, 05]:
 NCI (National Cancer Institute thesaurus)
 SNOMED (Systematized Nomenclature of Medicine)
6

7. Human assessment of ontologies fits into
ontology verification area. It is intended to detect
mistakes and inconsistencies that occur with human
modeling.
missing or inappropriately
allocated informal and formal
For example: in [Ceusters and Smith, 04, 05]:
definitions
shifts in terms meaning and
 NCI redundancy in concepts. thesaurus)
(National Cancer Institute
 SNOMED (Systematized Nomenclature of Medicine)
7

8. Scope of this presentation
ontology validation area which relates the
degree of correspondence of the ontology to
that part of reality which it is designed to
represent from the point of view of domain
experts.
8

9. Some quality attributes judged by domain
experts, such as clarity, relevance and
accuracy can be difficult to evaluate as they
may not be easily quantifiable
9

10. 10

11. Ontology expressed in a web language (RDF, OWL)
Questionnaire expressed in natural language
composed of four parts:
 Hierarchical ontology levels (ontology depth)
 Axioms
 Relations between concepts
 Descriptive attributes of concepts
11

12. We propose possible answers to questions, we rely on principles
mandatory for good quality ontologies:
 Clarity: ontology is easily understood by the users so that it
can be consistently applied and interpreted across the domain
of interest
 Lawfulness : knowledge described by the ontology is encoded
with meaningful terms. This is achieved by checking out that
the words used by the ontology are appropriate
 Accuracy: claims an ontology makes are right or wrong
 Relevance: ontology satisfies ontology requirements or not
12

13. Criterion Possible Answers Answers’ Location
Classes validation
Clarity Not Clear Axioms validation
Relations validation
Attributes validation
Right but another term would
be more appropriate Relations validation
Relevant but used terms are Attributes validation
Lawfulness
not appropriate
Right Classes validation
Wrong Relations validation
Always Axioms validation
Accuracy
Sometimes
Never
Relevance Relevant
Not really relevant Attributes validation
Not relevant at all
13

14.  STEP 2 (Aggregation of questionnaire results) is
performed automatically by web form module (like
drupal webform)
 STEP 3 & STEP 4 (Analysis and Synthesis of
obtained results & questionnaire update):
Delphi method [Dalker & Helmer, 1963]: its purpose
is to achieve convergence of opinions of experts
concerning a specific topic using questionnaire.
Generally, 3 iterations of updated questionnaire are
sufficient to reach a consensus
14

15.  we built an ontology called HERO ontology which
stands for “Higher Education Reference Ontology”
 we derived a questionnaire (100 questions) from
ontology elements and proposed MCQ as possible
answers according to ontology quality criteria
15

16. 16

17. 17

18. 18

19. 19

20. 20

21. The purpose of this proposal is to define a
methodological baseline for human
assessment of ontologies and to carry out a
practical case study for its applicability
Limitations
 Much more experiences are needed about the
practical usage of proposed guidelines
 Semi-automatic support of the approach is
required
21

22.  A. Gomez-Pérez, Ontology Evaluation, Handbook on
Ontologies, pp 251-274, 2004.
 J. Brank, M. Grobelnik, and D. Mladenic, “A survey of
ontology evaluation techniques”, Proceedings of Data
Mining and Data Warehouses (SiKDD), 2005.
 N.C Dalkey, and O. Helmer, “An experimental
application of the Delphi method to the use of experts”,
Management Science, 9 (3), pp 458-467, 1963.
 More references are included in the paper
22