Towards a collaborative metadata management system for sensor networks using SMW. Steps taken and lessons learned

1. Effective
metadata management
of sensor networks
using SMW
Jan Willem Noteboom
29 October 2013
SWMCon Fall 2013 | 29 October 2

2. Outline
Need for effective metadata management
Towards a collaborative environment
Experiences with Semantic MediaWiki (SMW)
2

3. Need for effective metadata management
Situation
• sensor networks more popular, bigger and complex
• access to many metadata sources needed for interpretation
• consistency of metadata sources is demanding
Needs
• integrated access to all sensor network metadata
• linked sources of sensor network metadata
• collaborative management of metadata
• interfaces for scientists, field inspectors and applications
• using international standards and open technology
3

4. Need for effective metadata management
What metadata
Collection & Discovery metadata
Measurement Data info
Metadata
in
scope
Interpretation
metadata
Measurement System info
Provenance & Lineage metadata
4
Measurement Process info
Platform & instrument metadata
Phenomena
properties
(dynamic
nature)
Measuring
characteristics
(“static”
nature)

5. Need for effective metadata management
What services
Discovery, View and Retrieval of sensor network information
Discovery
Annotation of sensor network information
Notification about sensor network information updates
Update of sensor network information
Administration of users, information and components
Users &
Applications
Content
managers
Service
managers
Update
Administration
Services
Discovery
View
Retrieval
Annotation
Notification
(linked) Sensor Network Metadata
(processes, systems)
5

6. Towards a collaborative environment
Phase I: Inventory & Analysis
collaborative data model for sensor network metadata
collaborative platform/architectuur
Phase II: Definition & PoC & Migration
definition of requirements, architecture and data model
proof of concept (PoC) using SMW
data migration toolkit
Phase III: Realization, Cleansing & Operationalization
Realization of version 1.0
Migration and cleansing of migrated data (automated and manually)
Implementation in the operational environment
6

7. Towards a collaborative environment
(meta)Data model
Property-oriented (linked data)
Linked data with time validation
and status
KNMI terminology and taxonomy
“Based on” international
standards (ISO 19115,
SensorML)
Systems and processes
Annotations (commentary)
Events of several types
Links to media (docs, pictures)
and people/organisations
7
Example: Element T(temperature) is measured
according to measurement process P using instrument I
that is deployed at location L in field F of station S
(managed by person P) and part of network N

8. Towards a collaborative environment
Collaborative platform
Discovery
View
Notification
Retrieval
Update
Annotation
Administration
Repository
Linked Sensor Data
(systems, processes)
Migration
of
metadata
Metadata
sources
8
Internal
sources
External
sources
Existing Services
Other collaborative
environments
MetaData
layer
Services
External
MetaData
(Geo)Portal
Service
layer
Appl layer
Applications, Users
Data managers

9. Experiences with Semantic MediaWiki
Context:
MW 1.21.2, SMW 1.8, 30 extensions, no SPARQL endpoint, 23410 pages,
25(50) namespaces, 190 properties (611,171 property values), 150
templates, 25 forms, 12 categories (classes)….
Experiences:
In general
Data modelling or structuring
Data discovery (searching/browsing)
Data view & retrieval (displaying/encoding/querying)
Data update (creating/editing/transferring)
Data annotation (commenting)
Data notification (watching)
Administration (user management, access control)
9

10. Experiences with Semantic MediaWiki
General
Well-known environment to most users
Powerful collaborative environment
Steep learning curve
Many useful extensions
Documentation is getting better
(hint: “Working with Mediawiki” Yaron Korn)
Note
Differences in the quality of extensions
(error messaging, documentation, governance)
10

11. Experiences with Semantic MediaWiki
Data structure
Easy implementation of the data model using
properties, templates, categories and pages
Usage of several namespaces
Usage of semantic internal objects to add time validation to
links (property qualifier)
Note
Import of ontologies (foaf, skos, ssn, dc, etc) considered but
not applied.
11

12. Experiences with Semantic MediaWiki
Data discovery
Searching/Browsing:
Query forms via site-bar menu
very flexible filtering and preloadable
tabs for standard vs advance filters
no linkage of filters…
Standard search/browse
Semantic maps & timelines
Note
No use of drilldown for
performance reasons
12

13. Experiences with Semantic MediaWiki
Data View & Retrieval
Displaying/encoding/query:
Pages with Infoboxes
Several Result Formats and
custom templates
Collapsible sections/infoboxes
Custom templates to construct
“complex” ask queries
(increased $smwgQMaxDepth,
$maxRecursionDepth)
Concepts to ease retrieval
Notes
Restricting “view” access of page
parts with Userfunctions (weak).
13

14. Experiences with Semantic MediaWiki
Data update & edit
Creating/editing/transferring:
Namespaces and Usergroups to
restrict (create/edit) access
Forms with Headertabs
Formlink buttons for creation
Several form input types
(e.g. combobox, two listboxes)
• Data transfer for data migration
(e.g. xml import)
Note
Forms not well suited to handle of
many instances (>50) of a multiple
instance template
(e.g. internal objects/subobjects)
14

15. Experiences with Semantic MediaWiki
Data Annotation
Page commenting
Notes
Events
permanent
temporary (period)
planned/unplanned
Attachments
Talk (discussion)
Note
Clear policy needed for
event registration
15

16. Experiences with Semantic MediaWiki
Data Notification
Watchlist (pages)
Notification (Echo)
Note
Need for summary reports of
changes for dedicated user
groups….
16

17. Experiences with Semantic MediaWiki
Data Administration
Special page utilities
User management
Data housekeeping
Statistics
Maintenance scripts
Daily backup
Rebuilding MW
Refreshing SMW
17

18. To summarize
Semantic Mediawiki (SWM) is a powerful collaborative environment to
manage sensor network metadata effectively.
Notes/Needs
Extensions differs in quality (error handling, documentation,
governance).
Documentation has improved
Need for forms that can handle many instances (>50) of a multiple
instance template
Need for summary reporting of changes (dedicated for user groups)
18

19. Many thanks for your attention!
19