pg loader

1. OAK Code Days: RDF loader into Postgres
OAK Code Days: RDF loader into Postgres
owner: Stamatis Zampetakis
presenter: Raphaël Bonaque
September 16th 2014
1 / 8

2. OAK Code Days: RDF loader into Postgres
What does it do ?
What
Create PostgreSQL representation from plain RDF triple files
Input:
.nt files containing a RDF triple (3 strings) per line
some parameters (for indexing, encoding, statistics)
Output:
a table containing the triples
(possibly) the same table with dictionary encoding
(possibly) indexes on the columns of the table
(possibly) a table counting the number of triples each URI is in
2 / 8

3. OAK Code Days: RDF loader into Postgres
What does it do ?
The code
Two similar versions
The oldest one in JAVA:
usable by other projects in JAVA
slower
location: part of Estimator: https:
//scm.gforge.inria.fr/svn/distriples/Estimator
size: approx. 8 classes and 730 lines of code
The newest one in bash:
much faster
location: https://scm.gforge.inria.fr/svn/
distriples/Estimator/scripts/estimatorSetup.sh
size: 219 lines
Some other versions by Alexandra and Damian might exist
3 / 8

4. OAK Code Days: RDF loader into Postgres
Who is dealing with the code ?
Who
Wrote it:
Java version : Damian and Stamatis
Bash version: Stamatis
Is using it:
None actively
(The statistics are used in Amanda and CliqueSquare)
4 / 8

5. OAK Code Days: RDF loader into Postgres
How does it work ?
General overview
How: overview
Java version:
DBPopulator.java: entry point, create the tables and call other
classes to fill them
DBCleaner.java: cleans the tables
N3TripleExtractor.java: extract triples from .nt files
TripleExtractor.java: interface for N3TripleExtractor
PostgreSQLInsertStringConvertor.java and its interface
TriplePatternConvertor.java: insert the triples in the database
Estimator.java: create statistics on the triples
Bash version, one file goes through every step:
connecting to the db
creating the tables
rewrite the nt file to dump it into postgres
potentially creates the encoded version
potentially creates the indexes
potentially create the statistics
5 / 8

6. OAK Code Days: RDF loader into Postgres
How does it work ?
Libraries
How: libraries and tools
PostgreSQL
log4j for logging in the Java version
Bash
6 / 8

7. OAK Code Days: RDF loader into Postgres
Next things to do
TODOs & bugs
Cannot handle very large URI (more than 2655 characters)
Potentially rewrite the bash version in Java
7 / 8

8. OAK Code Days: RDF loader into Postgres
Conclusion
Questions ?
Thank you for listening !
8 / 8