My presentation to the Canadian Semantic Web Workshop, Kelowna BC, June 2009. The slideshow describes a portion of Dr. Benjamin Good's PhD thesis work in which he examines the quality of annotation done through an open tagging process when the tags are constrained by a controlled vocabulary. The target for annotations were a set of BioMoby Semantic Web Services.

1. Open Semantic Annotation an experiment with BioMoby Web Services Benjamin Good, Paul Lu, Edward Kawas, Mark Wilkinson University of British Columbia Heart + Lung Research Institute St. Paul’s Hospital

2. The Web contains lots of things

3. But the Web doesn’t know what they ARE text/html video/mpeg image/jpg audio/aiff

4. The Semantic Web It’s A Duck

5. Semantic Web Reasoning Logically… It’s A Duck Defining the world by its properties helps me find the KINDS of things I am looking for Add properties to the things we are describing Walks Like a Duck Quacks Like a Duck Looks Like a Duck

6. Asserted vs. Reasoned Semantic Web Catalog/ ID Selected Logical Constraints (disjointness, inverse, …) Terms/ glossary Thesauri “ narrower term” relation Formal is-a Frames (Properties) Informal is-a Formal instance Value Restrs. General Logical constraints Originally from AAAI 1999- Ontologies Panel by Gruninger, Lehmann, McGuinness, Uschold, Welty; – updated by McGuinness. Description in: www.ksl.stanford.edu/people/dlm/papers/ontologies-come-of-age-abstract.html

8. When we say “Web” we mean “Scale”

13. Example: Image Annotation

16. BUSTED! I just pulled a bunch of Semantics out of my Seioubo!

17. BUSTED! This is a picture of Japanese traditional wagashi sweets called “seioubo” which is modeled after a peach This is a totally sweet picture of peaches grown in the city of Seioubo, in the Wagashi region of Japan

18. So tagging isn’t enough… We need properties, but the properties need to be semantically-grounded in order to enable reasoning (and this ain’t gonna happen through NLP because there is even less context in tags!)

19. Social Semantic Tagging Q1: Can we design interfaces that assist “the masses” to derive their tags from controlled vocabularies (ontologies)? Q2: How well do “the masses” do when faced with such an interface? Can this data be used “rigorously” for e.g. logical reasoning? Q3: “The masses” seem to be good at tagging things like pictures… no brainer! How do they do at tagging more complex things like bioinformatics Web Services?

20. Context: BioMoby Web Services BioMoby is a Semantic Web Services framework in which the data-objects consumed/produced by BioMoby service providers are explicitly grounded (semantically and syntactically) in an ontology A second ontology describes the analytical functions that a Web Service can perform

21. Context: BioMoby Web Services BioMoby ontologies suffer from being semantically VERY shallow… thus it is VERY difficult to discover the Web Service that you REALLY want at any given moment… Can we improve discovery by improving the semantic annotation of the services?

23. BioMoby Annotator Information extracted from Moby Central Web Service Registry Tagging areas

24. Tagging Type-ahead tag suggestions drawn from myGrid Web Service Ontology & from Freebase

25. Tagging New simple tags can also be created, as per normal tagging

32. Inter-Annotator Agreement Type N pairs mean median min max stand. dev. coefficient of variation Free, Object 1658 0.09 0.00 0.00 1.00 0.25 2.79 Semantic, Object 3482 0.44 0.40 0.00 1.00 0.43 0.98 Free, Operation 210 0.13 0.00 0.00 1.00 0.33 2.49 Semantic, Operation 2599 0.54 0.67 0.00 1.00 0.32 0.58

33. Agreement to “Gold” Standard Subject Type measure mean median min max stand. dev. coefficient of variation Data-types (input & output) PSA 0.52 0.51 0.32 0.71 0.11 0.22 Precision 0.54 0.53 0.33 0.74 0.13 0.24 Recall 0.54 0.54 0.30 0.71 0.12 0.21 Web Service Operations PSA 0.59 0.60 0.36 0.75 0.10 0.18 Precision 0.81 0.79 0.52 1.0 0.13 0.16 Recall 0.53 0.50 0.26 0.77 0.15 0.28

34. Consensus & Correctness: Datatypes

35. Consensus and Correctness: Operations

36. Open Annotations are Different

38. IT’S ALL ABOUT CONTEXT!! We can get REALLY good semantic annotations IF we provide context!!