This document provides an overview of the SHEBANQ project, which provides tools for querying annotated Hebrew text data. It describes the data sources and contributors that have built up the underlying text corpus over many years. It also outlines the steps taken to make this data and related tools more accessible, including developing a website, depositing data in archives, running demonstration projects, and integrating the data and tools into broader research environments through additional projects and publications. The goal has been to facilitate wider use of this linguistic resource and foster more digital humanities and data science work based on its contents.

1. Data management
what's the use for Research?
dirk.roorda@dans.knaw.nl
2016-04-21 Den Haag
a case study based on the Hebrew Bible

2. Scholarly letters 16th century

4. download as pdf
Welcome to
SHEBANQ
Wido van Peursen, leader of
ETCBC. Initiator and strategic
leader.
Oliver Glanz, Andrews
University. ETCBC data expert,
contributing numerous queries
for teaching.
Dirk Roorda, DANS. Author of
most of the code.
Eep Talstra, founder of
ETCBC. Still computing
(Pascal): participant
data in the making.
Constantijn Sikkel,
data designer for
ETCBC. Inventor of
efficient data creation
work flows.
Janet Dyk, linguist at
ETCBC. Long-time data
contributor, specialized
in verbal valence and
language variation.
Reinoud Oosting, data
designer for Leiden
University. Contributed
ETCBC data, now key
user.
Ulrik Sandborg-
Petersen, creator of
Emdros. Without it,
SHEBANQ would not
exist!
Henk van den Berg,
DANS. Programmed the
first versions.
Heleen van de Schraaf,
then DANS.
Programmed the first
user interface.
SHEBANQ relies on data
and tools created by
contributors in the past
User Guide
System for HEBrew Text:
ANnotations for Queries and Markup
funded by
CLARIN-NL, The Language Archive

5. Text. What is it?
bᵊrēšˈîṯ bārˈā ʔᵉlōhˈîm ʔˌēṯ haššāmˈayim wᵊʔˌēṯ hāʔˈāreṣ .
Genesis 1:1
In the beginning God created the heavens and the earth.

6. A string of words ...
bᵊrēšˈîṯ bārˈā ʔᵉlōhˈîm ʔˌēṯ haššāmˈayim wᵊʔˌēṯ hāʔˈāreṣ .

7. ... separated by spaces?
bᵊrēšˈîṯ bārˈā ʔᵉlōhˈîm ʔˌēṯ haššāmˈayim wᵊʔˌēṯ hāʔˈāreṣ .
bᵊrēšˈîṯ bārˈā ʔᵉlōhˈîm ʔˌēṯ haššāmˈayim wᵊʔˌēṯ hāʔˈāreṣ .

8. A string of letters ...
... in which alefbet?
bᵊrēšˈîṯ bārˈā ʔᵉlōhˈîm
ʔˌēṯ haššāmˈayim wᵊʔ
ˌēṯ hāʔˈāreṣ .
phonetic

9. hebrew with vowels and accents
‫֣א‬ ָ‫ר‬ ָ‫בּ‬ ‫ית‬ ֖ ִ‫אשׁ‬ ֵ‫ר‬ ְ‫בּ‬
‫ם‬ִ‫֖י‬ ַ‫מ‬ָּ‫שׁ‬ ַ‫ה‬ ‫֥ת‬ ֵ‫א‬ ‫֑ים‬ ִ‫ֱֹלה‬‫א‬
‫ץ׃‬ ֶ‫ר‬ ָֽ‫א‬ ָ‫ה‬ ‫֥ת‬ ֵ‫א‬ ְ‫ו‬

10. hebrew consonantal
‫ברא‬ ‫בראשית‬
‫השמים‬ ‫את‬ ‫אלהים‬
‫הארץ׃‬ ‫ואת‬

11. etcbc transcription (full)
B.:- R;>CI73JT
B.@R@74>
>:ELOHI92JM >;71T
HA- C.@MA73JIM W:-
>;71T H@- >@75REY00

12. etcbc transcription (consonantal)
B R>CJT BR> >LHJM
>T H CMJM W >T H
>RY

13. 1. The Text itself (representations)
2. Linguistics (feature structures)
3. "Manual" (really manual or software-generated)
4. Queries (exegetical search)
layers of annotation

14. words with
highlighted
occurrences
queries with
highlighted hits
click name to
toggle preview
of query
click author to
goto query and
all hits
click entry to
goto word and
all occurrences
click gloss to
toggle preview
of word
click any word to
toggle its highlight
Context items for this
chapter
enlarge preview
of query in a
pop-up

16. Data and tradition

17. text+
linguistics=>
data
+
research
=>
Wido van Peursen

18. What do we find?
wivu wivu hebrew

19. What do we find?

20. What do we find?

21. Observations?
The first hits are from archives, infrastructures
Researchers and their institutes follow later
The hits are mainly books, i.e. publications

22. What's missing?
metadata: descriptions, manuals, code books
analyses: what use have other researchers
made of this data?
instruments: tools to handle this kind of data
the very data!

23. Explanations?
These researchers started before the internet
they have developed a sphisticated data
workflow in their institute
the ETCBC has grown a thick cell membrane

24. research data cycle ?

25. research data cycle ?
religious
communities
theol.
scholars
theol.
scholars
enlightened lay
people

26. research data cycle !
religious
communities
theol.
scholars
theol.
scholars
enlightened lay
people
linguists
comp. hum
Research Data
Archiving
DANS
CLARIN
SHEBANQ
LAF-Fabric

27. More visibility

28. step 1: website (2008)
wivu.dans.knaw.nl

29. step 2: demo (2012)

30. step 3: deposit
(2012)

31. what has been deposited?

32. step 4: project (2013)
SHEBANQ
System for Hebrew Text: ANnotations for
Queries
CLARIN-NL project
data curation: LAF
demonstrator: query saver
#!/etc bc

33. LAF? Yes, ISO
Linguistic Annotation Framework
ISO 24612:2012
Nancy Ide, Laurent Romary

34. This is LAF
<node xml:id="n_88917">
<link targets="r1 r2 r3 r4 r5 r6 r7 r8 r9 r10 r11"/>
</node>
<edge xml:id="e1" from="n88917" to="n84383"/>
<a xml:id="ae1" label="parents" ref="e1" as="link"/>
<region xml:id="r_2" anchors="6 23"/>
<node xml:id="n_3"><link targets="r_2"/></node>
<a xml:id="a_3" label="word" ref="n_3" as="monads"/>labeled
edges
nodes
annotations
(features)
annotations
(empty)
primary data
regions
lexeme_utf8= ‫ר‬‫א‬‫שׁ‬‫י‬‫ת‬
surface_consonants_utf8= ‫ר‬‫א‬‫שׁ‬‫י‬‫ת‬
‫בּ‬ְ‫ר‬ֵ‫א‬‫שׁ‬ִ֖‫י‬‫ת‬‫בּ‬ָ‫ר‬ָ֣‫א‬‫א‬ֱ.‫ה‬ִ֑‫י‬‫ם‬‫א‬ֵ֥‫ת‬‫ה‬ַ‫שּׁ‬ָ‫מ‬ַ֖‫י‬ִ‫ם‬‫ְו‬‫א‬ֵ֥‫ת‬‫ה‬ָ‫א‬ָֽ‫ר‬ֶ‫ץ‬‫׃‬
0-56-2392 72-91r9r10r11
n2n3
word
sentence
phrase
determination=determined
phrase_function=Objc
phrase_type=PP
parents
mother
subphrase
clause
r11 r10 r9
clause_atom_number=1
clause_atom_relation=0
clause_atom_type=xQtl
indentation=0
<a xml:id="af22" label="ft" ref="n3" as="utf8"><fs>
<f name="lexeme_utf8" value=" ‫ר‬‫א‬‫ׁש‬‫י‬‫ת‬ "/>
<f name="surface_consonants_utf8" value=" ‫ר‬‫א‬‫ׁש‬‫י‬‫ת‬ "/>
</fs></a>
link to
regions
Linguistic Annotation Framework

36. IPython notebook

37. excursion: data and tools
data is not available separately
there is always the need for a tool: software
inspect
transport
transform

38. data science at the command line
http://datascienceatthecommandline.com
http://datasciencetoolbox.org
The Data Science Toolbox is a virtual
environment based on Ubuntu Linux that
is specifically suited for doing data
science. Its purpose is to get you started
in a matter of minutes. You can run the
Data Science Toolbox either locally
(using VirtualBox and Vagrant) or in the
cloud (using Amazon Web Services).

39. inspect
dirk:~/Dropbox/laf-fabric-data/etcbc4/laf > ls
etcbc4.hdr etcbc4_lingo.c.xml etcbc4_lingo.sp.xml etcbc4_regions.xml
etcbc4.lst etcbc4_lingo.p.xml etcbc4_lingo.xml etcbc4_sections.xml
etcbc4.txt etcbc4_lingo.pa.xml etcbc4_monads.lex.xml
etcbc4.txt.hdr etcbc4_lingo.s.xml etcbc4_monads.xml
dirk:~/Dropbox/laf-fabric-data/etcbc4/laf > du -h .
1.5G .
dirk:~/Dropbox/laf-fabric-data/etcbc4/laf > fgrep -l 'BR&gt;' *.xml
etcbc4_monads.lex.xml
BR> = ‫ברא‬ = maken
dirk:~/Dropbox/laf-fabric-data/etcbc4/laf > fgrep -c 'BR&gt;' etcbc4_monads.lex.xml
113

40. inspect
dirk:~/Dropbox/laf-fabric-data/etcbc4/laf > ls -lh *.txt
-rw-r--r-- 1 dirk staff 5.1M Jul 23 10:58 etcbc4.txt

41. inspect

42. inspect (xml)
dirk:~/Dropbox/laf-fabric-data/etcbc4/laf > ls -lh *.xml
-rw-r--r-- 1 dirk staff 104M Jul 23 11:00 etcbc4_lingo.c.xml
-rw-r--r-- 1 dirk staff 107M Jul 23 11:00 etcbc4_lingo.p.xml
-rw-r--r-- 1 dirk staff 148M Jul 23 11:00 etcbc4_lingo.pa.xml
-rw-r--r-- 1 dirk staff 22M Jul 23 11:00 etcbc4_lingo.s.xml
-rw-r--r-- 1 dirk staff 23M Jul 23 11:00 etcbc4_lingo.sp.xml
-rw-r--r-- 1 dirk staff 299M Jul 23 11:00 etcbc4_lingo.xml
-rw-r--r-- 1 dirk staff 642M Jul 23 10:58 etcbc4_monads.lex.xml
-rw-r--r-- 1 dirk staff 125M Jul 23 10:58 etcbc4_monads.xml
-rw-r--r-- 1 dirk staff 37M Jul 23 10:58 etcbc4_regions.xml
-rw-r--r-- 1 dirk staff 36M Jul 23 10:58 etcbc4_sections.xml
dirk:~/Dropbox/laf-fabric-data/etcbc4/laf > time xmllint --nonet --noout
--stream --schema /Users/dirk/Dropbox/laf-fabric-data/etcbc4/decl/graf-
standoff.xsd etcbc4_monads.lex.xml
etcbc4_monads.lex.xml validates
real 2m26.029s
user 2m20.308s
sys 0m2.376s

43. inspect (xml)
.hdr => .xml

44. transport

45. transform
The shortest path to having the computer work for me
scripting
shell, python
scientific programming
software as instrument
hourly cycle
by and for researchers
programming
C, C++, Java
software engineering
applications as product
weekly cycle
by ICT dev for
researcher

46. what do scholars want
they are not software developers
but they do program
they explore data, knead, massage
their products are not software
but analyses, visualizations, publications

47. scientific computing
more than (i)Python
more than an interface
more than an ecosystem
a culture:

48. culture
fragments from a
video of Fernando Perez
4:19 researchers and computing - 7:37
17:00 tools and the data life cycle - 20:26
42:09 data and publishing - 44:20 / 49:22

49. trees for Data Oriented Parsing

50. step 6: harvest (2014-2015)
Rens Bod:
ling/dighum
Data Oriented Parsing
Bible Online Learner
Nicolai Winther-Nielsen
EuroPlot, University of Aalborg
Martijn Naaijer
Linguistic Variation:
statistics with R

51. step 7: better versions (2013-2015)

52. step 7: better versions (2013-2015)

53. step 7: better versions (2013-2015)

54. step 8: website SHEBANQ (2013)
hey, Martijn is after something!
inform your followers with 1 click
just browsing Genesis 4

55. step 9: mature SHEBANQ (2015-2016)

56. Queries

57. visuals

58. step 10: more (2016-2020)
more projects (digging into data?)
more disciplines (linguistics, data science, archaeology)
more data sources (syriac, qumran)
more users
> 250 people
systems (Bible Online Learner, Tiberias)
institutes (VU University, Andrews University, Aalborg University)
more output (articles, derived data)
more training (workshops, master students, Ph.D students)
better position in the competition

59. turn-turn-turn
religious
communities
theol.
scholars
theol.
scholars
enlightened lay
people
linguists
comp. hum
Research Data
Archiving
DANS

60. research environment
function medium infra
data LAF in dataset DANS EASY
web site web2py
DANS=>KNAW,
Leaseweb, Cloud
tools
LAF-Fabric,
Shebanq, Emdros
Github,
Sourceforge
publishing
IPython notebooks,
Restructured Text
Github,
Readthedocs
products
apps, notebooks,
articles
Github, Science
Clouds, Journals

61. is this a success story?
there is certainly a degree of success ...
it took 6 years to get a feeling of acceleration
grab opportunities eagerly
persuade liberally
embrace technology
and combine it with affinity with sources and scholarship
make up-front investments (time, relationships)

62. why is it not going faster?
the team is efficiently organised already
new ways of work have not proved themselves
yet
technical support is a rare and expensive
commodity for small teams in the humanities

63. contributing factors
personnel mutations
new projects
new requirements from funders (open access)
competition and collaboration across disciplines
the digital world is increasingly penetrating
people's lives

64. yes, if they realize the importance of re-use
yes, if they find the path to archiving
yes, if archives go out of their way to be
relevant for researchers
yes, if archives use ICT proactively
dirk.roorda@dans.knaw.nl
Data management
useful for researchers? ...

65. yes, if they realize the importance of re-use
yes, if they find the path to archiving
yes, if archives go out of their way to be
relevant for researchers
yes, if archives use ICT proactively
none of these are
straightforward
dirk.roorda@dans.knaw.nl
Data management
useful for researchers? ...