Validating JSON, XML and CSV data with SHACL-like constraints (DINI-KIM 2022)

Validating
JSON, XML and CSV data
with SHACL-like constraints
Péter Király, GWDG (Göttingen)
pkiraly@gwdg.de
Deutsche Initiative für Netzwerkinformation e.V.
Kompetenzzentrum Interoperable Metadaten (KIM) Workshop
2022-05-02
https://github.com/pkiraly/metadata-qa-api

Shapes Constraint Language (SHACL)
a language for validating RDF graphs against a set of conditions (expressed as
RDF graphs)
ex:PersonShape
a sh:NodeShape ;
sh:targetClass ex:Person ; # checks persons
sh:property [
sh:path ex:ssn ; # checks social
security nr.
sh:maxCount 1 ;
sh:datatype xsd:string ;
sh:pattern "^d{3}-d{2}-d{4}$" ;
] ;
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

Metadata Quality Assessment Framework (MQAF) API
★ an open source software for metadata quality assessment
★ quality dimensions: completeness, multilinguality, uniqueness, etc.
★ extensions: Europeana, MARC, Deutsche Digitale Bibliothek
★ Java API + command line interface (in progress)
★ reads XML, JSON, CSV, MARC
★ highly configurable
★ adaptable to different metadata schemas
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

RDF agnostic SHACL tests*
Cardinality minCount <number>, maxCount <number>
Value Range minExclusive <number>, minInclusive <number>, maxExclusive <number>, maxInclusive
<number>
String minLength <number>, maxLength <number>, hasValue <String>, in [String1, ...,
StringN], pattern <regular expression>, minWords <number>, maxWords <number>
Comparision of
properties
equals <field label>, disjoint <field label>, lessThan <field label>, lessThanOrEquals
<field label>
Logical operators and [<rule1>, ..., <ruleN>], or [<rule1>, ..., <ruleN>], not [<rule1>, ..., <ruleN>]
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints
* a subset of SHACL

MQAF API’s SHACL tests
Cardinality minCount <number>, maxCount <number>
Value Range minExclusive <number>, minInclusive <number>, maxExclusive <number>, maxInclusive
<number>
String minLength <number>, maxLength <number>, hasValue <String>, in [String1, ...,
StringN], pattern <regular expression>, minWords <number>, maxWords <number>
Comparision of
properties
equals <field label>, disjoint <field label>, lessThan <field label>, lessThanOrEquals
<field label>
Logical operators and [<rule1>, ..., <ruleN>], or [<rule1>, ..., <ruleN>], not [<rule1>, ..., <ruleN>]
extras contentType [type1, ..., typeN], unique <boolean>, dependencies [id1, id2, ..., idN],
dimension [criteria...] (min/max + Width/Height/Shortside/Longside)
properties id, description, failureScore, successScore, hidden, skip
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

abstracting the address of data element
XML
JSON
CSV
MARC21
have addressable data
elements (branches)
XPath
JSONPath
column
names
MARCSpec
addressing
languages
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

schema
definition
abstracting data element retrieval
XML
JSON
CSV
MARC21
data element
selector
uniform data
structure
May I
get the
title?
Title’s address
is //head/title
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

schema definition
Schema schema = new BaseSchema()
.setFormat(Format.CSV)
.addField(
new JsonBranch("title", "title")
.setRule(
new Rule()
.withDisjoint("description")))
.addField(
new JsonBranch("url", "url")
.setExtractable(true)
.setRule(
new Rule()
.withMinCount(1)
.withMaxCount(1)
.withPattern("^https?://.*$")))
format: csv
fields:
- name: title
rules:
disjoint: description
- name: url
extractable: true
rules:
minCount: 1
maxCount: 1
pattern: ^https?://.*$
Java API YAML configuration file
{
“format”: “csv”,
“fields”: [
{
“name”: “title”,
“rules”: [
{“disjoint”: “description”}
]
},
{
“name”: “url”,
“extractable”: true,
“rules”: [
{
“minCount”: 1,
“maxCount”: 1,
“pattern”: “^https?://.*$”}]}
JSON configuration file
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

one and only one data element instance
- name: about
path: $.['about']
rules:
- minCount: 1
- maxCount: 1
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

numeric value constraints
- name: price
path: $.['price']
rules:
- and:
- minInclusive: 1.0
- maxInclusive: 2.0
- name: price
path: $.['price']
rules:
- and:
- minExclusive: 1.0
- maxExclusive: 2.0
1.0 <= price <= 2.0 1.0 < price < 2.0
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

string constraints / length
- name: about
path: $.['about']
rules:
- minLength: 1
- name: about
path: $.['about']
rules:
- and:
- minLength: 3
- maxLength: 5
lenght(about) >= 1 5 >= lenght(about) >= 3
- name: status
path: $.['status']
rules:
- hasValue: published
status == “published”
- name: type
path: $.['type']
rules:
- in: [dataverse, dataset, file]
type == “dataverse” or
type == “dataset” or
type == “file”
- name: thumbnail
path: oai:record/dc:identifier[@type='binary']
rules:
- pattern: ^https?://.*.(jpe?g||png|tiff?|gif)$
thumbnail is an image or PDF file
- name: about
path: $.['about']
rules:
- minWords: 1
nr_words(about) >= 2
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

string constraints / fixed values
- name: about
path: $.['about']
rules:
- minLength: 1
- name: about
path: $.['about']
rules:
- and:
- minLength: 3
- maxLength: 5
- name: status
path: $.['status']
rules:
- name: type
path: $.['type']
rules:
type == “file”
- name: thumbnail
rules:
- name: about
path: $.['about']
rules:
- minWords: 1
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

string constraints / pattern
- name: about
path: $.['about']
rules:
- minLength: 1
- name: about
path: $.['about']
rules:
- and:
- minLength: 3
- maxLength: 5
- name: status
path: $.['status']
rules:
- name: type
path: $.['type']
rules:
type == “file”
- name: thumbnail
rules:
- name: about
path: $.['about']
rules:
- minWords: 1
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

string constraints / number or words
- name: about
path: $.['about']
rules:
- minLength: 1
- name: about
path: $.['about']
rules:
- and:
- minLength: 3
- maxLength: 5
- name: status
path: $.['status']
rules:
- name: type
path: $.['type']
rules:
type == “file”
- name: thumbnail
rules:
- name: about
path: $.['about']
rules:
- minWords: 2
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

comparisions of data elements
fields:
- name: id
path: $.['id']
rules:
- equals: isbn
- name: isbn
path: $.['isbn']
fields:
- name: title
path: $.['title']
rules:
- disjoint: description
- name: description
path: $.['description']
- name: startingPage
path: startingPage
rules:
- lessThanOrEquals: endingPage
id == isbn title != description startingPage <= endingPage
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

logical operations
- name: id
path: oai:record/dc:identifier
rules:
- and:
- minCount: 1
- maxCount: 1
- minLength: 1
- name: thumbnail
rules:
- or:
- pattern: ^.*.(jpe?g|png|)$
- contentType:
- image/jpeg
- image/png
- name: title
path: $.['title']
rules:
- not:
- equals: description
and or not
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

extras
- name: thumbnail
rules:
- contentType: [image/jpeg, image/png, …]
content type
- name: id
rules:
- unique: true
- name: url
path: oai:record/dc:identifier[@type='URL']
rules:
- id: Q-4.4
description: Both a media file and a link to an
object are referenced in context.
dependencies: [Q-3.0, Q-4.0]
- name: thumbnail
rules:
- id: 3.1
dimension:
minWidth: 200
minHeight: 200
only if other test has been passed image dimensions
unique value
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

other properties
id identifier, used in output, and in internal references
description explain what the rule checks
failureScore a numerical score assigned if the test fails
successScore a numerical score assigned if the test passes
hidden run the test, but hides from the output
skip do not run the test now (for debugging reason)
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

raw output
★ for each tests:
○ status: PASSED, FAILED, NA (if the data element is not available)
○ score: the output of successScore (if passed), failureScore (if failed) or 0
★ total score
The output could be CSV, JSON or Java objects (configurable)
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

visualization for metadata managers / single record
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

aggregation
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

status and scores
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

workflow 1. ingest
2. measure records
3. aggregate
4. report
5. evaluate with experts
catalogue
improve records
quality assessment tool
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

research partners
early adopters and contributors
★ Miel Vander Sande (meemoo, Belgium)
★ Richard Palmer (Victoria and Albert Museum, Great Britain)
Deutsche Digitale Bibliothek
★ Francesca Schulze
★ Cosmina Berta
★ Stefanie Rühle
★ Claudia Effenberger
★ Letitia-Venetia Mölck
special thanks
★ Juliane Stiller
Validating
JSON,
XML
and
CSV
data
with
SHACL-like
constraints

Validating JSON, XML and CSV data with SHACL-like constraints (DINI-KIM 2022)

Recommandé

Recommandé

Contenu connexe

Similaire à Validating JSON, XML and CSV data with SHACL-like constraints (DINI-KIM 2022)

Similaire à Validating JSON, XML and CSV data with SHACL-like constraints (DINI-KIM 2022) (20)

Plus de Péter Király

Plus de Péter Király (20)

Dernier

Dernier (20)

Validating JSON, XML and CSV data with SHACL-like constraints (DINI-KIM 2022)