SHEBANQ project (half-way) as a use case in querying language resources. The corpus is the text of the Hebrew Bible with linguistic features, packaged in de special text database and converted to LAF

1. Data Archiving and Networked Services!
SHEBANQ!
Dirk Roorda - researcher @ DANS,TLA!
System for HEBrew Text: ANnotations
for Queries and Markup!
TEI pre-conference workshop: Query!
Roma – 2013-10-01!

2. Overview
1. Context: text, data, research in Hebrew
Bible
2. MdF database model, MQL query
language
3. Sharing the research process
4. CLARIN-NL project: SHEBANQ
5. Towards new tools

3. 1 (of 5) Context
Text, data and research in the Hebrew Bible

4. VU Amsterdam
Eep Talstra Centre for Bible and Computer
text + linguistic features => database
database + research questions => publications
4!

5. 2 (of 5) MdF and MQL
• MdF database model
• MQL query language

6. Monad Object Feature
1977-now: Eep Talstra et al. ECA, WIVU.
Print reference (Google Books)
1988-1994 Crist-Jan Doedens: Text
Databases – One Database Model and
Several Retrieval Languages (google
books reference)
2004: Ulrik Petersen. Emdros - a text
database engine for analyzed or
annotated text. COLING

7. word objects
standard
edition
text
monads
(atomic chunks
of text)
lexeme_utf8= ‫ר‬‫א‬‫ׁש‬‫י‬‫ת‬
old_lexeme_utf8= ‫ר‬‫א‬‫ׁש‬‫י‬‫ת‬
vocalized_lexeme_utf8= ‫ֵר‬‫א‬‫ׁש‬ִ‫י‬‫ת‬
surface_consonants_utf8= ‫ר‬‫א‬‫ׁש‬‫י‬‫ת‬
graphical_lexeme_utf8= ‫ֵר‬‫א‬‫ׁש‬ִ֖‫י‬
‫בּ‬ְ‫ר‬ֵ‫א‬‫שׁ‬ִ֖‫י‬‫ת‬‫בּ‬ָ‫ר‬ָ֣‫א‬‫א‬ֱ.‫ה‬ִ֑‫י‬‫ם‬‫א‬ֵ֥‫ת‬‫ה‬ַ‫שּׁ‬ָ‫מ‬ַ֖‫י‬ִ‫ם‬‫ְו‬‫א‬ֵ֥‫ת‬‫ה‬ָ‫א‬ָֽ‫ר‬ֶ‫ץ‬‫׃‬
1234567891011
23456789101112
84383
59559
34680
7763777638
40770
7 .. 511 .. 9
11 .. 5
11 .. 5
11 .. 1
11 .. 1
clause_atom_number=1
clause_atom_relation=0
clause_atom_relation_daughter_tense=unknown
clause_atom_relation_kind=No_relation
clause_atom_relation_mother_tense=unknown
clause_atom_relation_preposition_class=none
clause_atom_type=xQtl
indentation=0
phrase objects
Monad-Object-Feature
subphrase objects
phrase_atom
objects
clause_atom
objects
sentence objects

8. MQL query language
topographic, i.e:
query expression =~= query results w.r.t.
• sequence
• embedding

9. Example
SELECT ALL OBJECTS!
WHERE!
[Clause!
[Phrase!
[Word FOCUS !
" " "part_of_speech = verb AND !
" " "lexeme = "FJM["]!
]!
..!
[Phrase FOCUS!
" "phrase_function = Objc OR!
" "phrase_function = IrpO!
]!
..!
[Phrase FOCUS!
" "phrase_function = Objc OR!
" "phrase_function = IrpO!
]!
]!

10. 3 (of 5) Sharing
Problem: how to share (intermediate)
results of analysis
Solution: saving queries as annotations

11. Lock - in
scholarly-bibles.com!
Stuttgart Electronic Study Bible
⇒ massive dissemination
But
⇒ not the right dynamics for
tool development

12. Leiden: international workshop
biblical scholarship
Desiderata:
new tool development
text transmission (variants)
linguistic analysis (features)
even combined!
a short history: 2012
leiden lorentz!

13. Hebrew Text in the Archive
urn:nbn:nl:ui:13-ikjj-ek!

14. Hebrew Text in the Archive
urn:nbn:nl:ui:13-ikjj-ek!
how can the
people annotate
our work?!

15. Research Data Cycle

16. Research Data Cycle
Text transmission,
tradition, editorial
processes
Free University,
theology faculty,
server department,
WIVU project
!
NWO projects!NWO projects
religious
communities
theol.
scholars
theol.
scholars
enlightened lay
people
scholarly-
ibles.com!

17. Research Data Cycle
Text transmission,
tradition, editorial
processes
Free University,
theology faculty,
server department,
WIVU project
!
NWO projects!NWO projects
religious
communities
theol.
scholars
theol.
scholars
CLARIN
SHEBANQ
linguists
Wider public:
Annotation,
Query Saving,
via Linked Data
dig. hum
comp. hum
enlightened lay
people
scholarly-
ibles.com!
Research Data
Archiving
DANS

18. 3 (of 5) Sharing (c’t’d)
Solution: Queries As Annotations

19. queries-as-annotations
model! query! example!
body! query instruction!
SELECT ALL OBJECTS WHERE [Word
FOCUS part_of_speech = verb AND
lexeme = "‫!]"שים‬
targets!
query results in
context!
‫ׁר‬ֶ‫ש‬ֲ‫א‬ ֙‫ן‬ֶ‫ב‬ֶ֨‫א‬ ָ‫ה‬ ‫ֶת‬‫א‬ ‫֤ח‬ַּ‫ק‬ִּ‫י‬ ַ‫ו‬ ‫ֶר‬‫ק‬ֹּ֗‫ב‬ ַּ‫ב‬ ‫֜ב‬ֹ‫ק‬ֲ‫ע‬ַ‫י‬ ‫֨ם‬ֵּ‫כ‬ְׁ‫ש‬ַּ‫י‬ ַ‫ו‬
‫ֶן‬‫מ‬ֶׁ֖‫ש‬ ‫֥ק‬ֹ‫צ‬ִּ‫י‬ ַ‫ו‬ ‫֑ה‬ָ‫ב‬ֵּ‫צ‬ַ‫מ‬ ּ‫ה‬ָ֖‫ת‬ֹ‫א‬ ‫ׂם‬ֶ‫ש‬ָּ֥‫י‬ ַ‫ו‬ ‫֔יו‬ָ‫ת‬ֹׁ‫ש‬ֲ‫א‬ַֽ‫ר‬ְ‫מ‬ ‫֣ם‬ָׂ‫ש‬
ּ‫ה‬ָֽׁ‫ש‬‫ֹא‬‫ר‬ ‫ַל‬‫ע‬
annotation! published query! qu123 (just an identifier)!
metadata!
researcher, date
created, date last
run, research
question!
Janet Dyk 2004-02-16 2012-01-27
Can the verb ‫ים‬ִׂ‫ש‬ have a double
object? - article in Foundations
for Syriac Lexicography!

20. OpenAnnotation openannotation.org!

21. provenance

22. motivation

23. demonstrator datanetworkservice.nl/qaa!

24. demonstrator datanetworkservice.nl/qaa!

25. demonstrator datanetworkservice.nl/qaa!

26. demonstrator datanetworkservice.nl/qaa!

27. demonstrator

28. demonstrator

29. demonstrator

30. demonstrator
still missing:
saving queries
not semantic-web-enabled
sustainability

31. 4 (of 5) Project
CLARIN-NL: SHEBANQ:
(A) Curation
(B) Demonstrator

32. SHEBANQ
System for Hebrew Text:
ANnotations for Queries
CLARIN-NL project
data curation: LAF
demonstrator: query saver
#!/etc bc
s/g$/q/!

33. Linguistic Annotation Framework
ISO 24612:2012
Nancy Ide, Laurent Romary

38. feature definitions

39. feature definitions

40. TEI ISO-FS schema

41. dcr:datcat on <fDecl> versus <f>
26,225,966 <f>s!
!
2.5 GB redundant
attribute
material !
!

42. 5 (of 5) Project
CLARIN-NL: SHEBANQ: (B) Demonstrator

43. select all objects where
[clause
[phrase phrase_function = Objc
[word FOCUS tense = infinitive_absolute]
]
]
Execute
Query executed
Passage
‫ּב‬ְ‫ֵר‬‫א‬‫ׁש‬ִ֖‫י‬‫ת‬‫ּב‬ָ‫ָר‬֣‫א‬‫א‬ֱ‫ֹל‬‫ה‬ִ֑‫י‬‫ם‬‫א‬ֵ֥‫ת‬‫ה‬ַ‫ּׁש‬ָ‫מ‬ַ֖‫י‬ִ‫ם‬‫ו‬ְ‫א‬ֵ֥‫ת‬
‫ה‬ָ‫א‬ָֽ‫ֶר‬‫ץ‬‫׃‬
‫ו‬ַ‫ּי‬ֹ֥‫א‬‫מ‬ֶ‫ר‬‫ח‬ִ‫ז‬ְ‫ִק‬‫ּי‬ָ֖‫ה‬‫ּו‬‫מ‬ָ֣‫ה‬‫א‬ֹ֑‫ו‬‫ת‬‫ּכ‬ִ֥‫י‬‫א‬ֶ‫ע‬ֱ‫ל‬ֶ֖‫ה‬‫ּב‬ֵ֥‫י‬‫ת‬
‫י‬ְ‫ה‬‫ו‬ָֽ‫ה‬‫׃‬
Controls
‫ו‬ַ‫ּי‬ֹ֥‫א‬‫מ‬ֶ‫ר‬‫ח‬ִ‫ז‬ְ‫ִק‬‫ּי‬ָ֖‫ה‬‫ּו‬‫מ‬ָ֣‫ה‬‫א‬ֹ֑‫ו‬‫ת‬‫ּכ‬ִ֥‫י‬‫א‬ֶ‫ע‬ֱ‫ל‬ֶ֖‫ה‬‫ּב‬ֵ֥‫י‬‫ת‬
‫י‬ְ‫ה‬‫ו‬ָֽ‫ה‬‫׃‬
Gen 1:1
2Chron 3:4
Gen 1:1
‫ּב‬ְ‫ֵר‬‫א‬‫ׁש‬ִ֖‫י‬‫ת‬‫ּב‬ָ‫ָר‬֣‫א‬‫א‬ֱ‫ֹל‬‫ה‬ִ֑‫י‬‫ם‬‫א‬ֵ֥‫ת‬‫ה‬ַ‫ּׁש‬ָ‫מ‬ַ֖‫י‬ִ‫ם‬‫ו‬ְ‫א‬ֵ֥‫ת‬
‫ה‬ָ‫א‬ָֽ‫ֶר‬‫ץ‬‫׃‬
‫ו‬ַ‫ּי‬ֹ֥‫א‬‫מ‬ֶ‫ר‬‫ח‬ִ‫ז‬ְ‫ִק‬‫ּי‬ָ֖‫ה‬‫ּו‬‫מ‬ָ֣‫ה‬‫א‬ֹ֑‫ו‬‫ת‬‫ּכ‬ִ֥‫י‬‫א‬ֶ‫ע‬ֱ‫ל‬ֶ֖‫ה‬‫ּב‬ֵ֥‫י‬‫ת‬
‫י‬ְ‫ה‬‫ו‬ָֽ‫ה‬‫׃‬
Text
1Sam 12:4
Ex 23:2
Query results
Prev 2 3 65 ... 2241 Next21 313 results
Executing query ...
view in context
Save this query
Researcher Oliver Glanz
Date created 2013-08-25
Date last run 2013-08-25
Project Data and Tradition
Institute VU/Eep Talstra Centre for Bible and Computing
Reason irregular valency of ‫ּב‬ָ‫ָר‬֣‫א‬
Comments
needs to be combined with query on ‫א‬ֱ‫ֹל‬‫ה‬ִ֑‫י‬‫ם‬
Save PublishCancel
Name valency ‫ּב‬ָ‫ָר‬֣‫א‬
Edit Query

44. Passage
‫ּב‬ְ‫ֵר‬‫א‬‫ׁש‬ִ֖‫י‬‫ת‬‫ּב‬ָ‫ָר‬֣‫א‬‫א‬ֱ‫ֹל‬‫ה‬ִ֑‫י‬‫ם‬‫א‬ֵ֥‫ת‬‫ה‬ַ‫ּׁש‬ָ‫מ‬ַ֖‫י‬ִ‫ם‬‫ו‬ְ‫א‬ֵ֥‫ת‬
‫ה‬ָ‫א‬ָֽ‫ֶר‬‫ץ‬‫׃‬
‫ו‬ַ‫ּי‬ֹ֥‫א‬‫מ‬ֶ‫ר‬‫ח‬ִ‫ז‬ְ‫ִק‬‫ּי‬ָ֖‫ה‬‫ּו‬‫מ‬ָ֣‫ה‬‫א‬ֹ֑‫ו‬‫ת‬‫ּכ‬ִ֥‫י‬‫א‬ֶ‫ע‬ֱ‫ל‬ֶ֖‫ה‬‫ּב‬ֵ֥‫י‬‫ת‬
‫י‬ְ‫ה‬‫ו‬ָֽ‫ה‬‫׃‬
Controls
‫ו‬ַ‫ּי‬ֹ֥‫א‬‫מ‬ֶ‫ר‬‫ח‬ִ‫ז‬ְ‫ִק‬‫ּי‬ָ֖‫ה‬‫ּו‬‫מ‬ָ֣‫ה‬‫א‬ֹ֑‫ו‬‫ת‬‫ּכ‬ִ֥‫י‬‫א‬ֶ‫ע‬ֱ‫ל‬ֶ֖‫ה‬‫ּב‬ֵ֥‫י‬‫ת‬
‫י‬ְ‫ה‬‫ו‬ָֽ‫ה‬‫׃‬
Gen 1:1
2Chron 3:4
Gen 1:1
‫ּב‬ְ‫ֵר‬‫א‬‫ׁש‬ִ֖‫י‬‫ת‬‫ּב‬ָ‫ָר‬֣‫א‬‫א‬ֱ‫ֹל‬‫ה‬ִ֑‫י‬‫ם‬‫א‬ֵ֥‫ת‬‫ה‬ַ‫ּׁש‬ָ‫מ‬ַ֖‫י‬ִ‫ם‬‫ו‬ְ‫א‬ֵ֥‫ת‬
‫ה‬ָ‫א‬ָֽ‫ֶר‬‫ץ‬‫׃‬
‫ו‬ַ‫ּי‬ֹ֥‫א‬‫מ‬ֶ‫ר‬‫ח‬ִ‫ז‬ְ‫ִק‬‫ּי‬ָ֖‫ה‬‫ּו‬‫מ‬ָ֣‫ה‬‫א‬ֹ֑‫ו‬‫ת‬‫ּכ‬ִ֥‫י‬‫א‬ֶ‫ע‬ֱ‫ל‬ֶ֖‫ה‬‫ּב‬ֵ֥‫י‬‫ת‬
‫י‬ְ‫ה‬‫ו‬ָֽ‫ה‬‫׃‬
Text
1Sam 12:4
Ex 23:2
Saved Query Results
Prev 2 3 65 ... 2241 Next21 313 results
view in context
Information on this query
Researcher Oliver Glanz
Date created 2013-08-25
Date last run 2013-08-25
Project
Institute
Reason
Comments
Name
Query Info
select all objects where
[clause
[phrase phrase_function = Objc
[word FOCUS tense = infinitive_absolute]
]
]
MQL query text Persistent Identifier urn:nbn:nl:ui:13-scpm-ji
http://www.persistent-identifier.nl/?identifier=urn...
valency ‫ּב‬ָ‫ָר‬֣‫א‬
Data and Tradition
VU/Eep Talstra Centre for Bible and Computing
irregular valency of ‫ּב‬ָ‫ָר‬֣‫א‬
needs to be combined with query on ‫א‬ֱ‫ֹל‬‫ה‬ִ֑‫י‬‫ם‬

45. datanetworkservice.nl/qaa!

46. SHEBANQ: implementing Q-a-A

47. 5 (of 5) Towards new tools
• LAF tools
• or generic graph algorithms
• Emdros tools
• or generic database technology
• Linked Data tools
• or generic SPARQL queries

48. Side conditions
• development close to the researchers
• preferably in their own institutions
• decent performance
• within the scale of a laptop
• usable to researchers
• that is: non-programmers
• persistence in mind
• new results will be archived and re-
enter the data cycle

49. thank you
dirk.roorda@dans.knaw.nl
slideshare.net/dirkroorda/
s/g$/q/!
#!/etc bc
Eep Talstra Centre for Bible and Computer!