The document discusses combining big data processing frameworks like Spark and Flink with semantic technologies. It presents SANSA, a system that loads RDF data into Spark and enables distributed semantic processing including querying, reasoning, and analytics. SANSA allows expressing ontologies and rules as RDDs to leverage Spark's scalability while supporting W3C standards like RDF and OWL. The document provides examples of using SANSA to load and partition RDF graphs, perform reasoning using rules, and write inferred data back to disk.

1. September 2017

2. Source: LOD-Cloud (http://lod-cloud.net/ )
◎
→
◎

3. •
•
•
Disk
In-memory
Iteration 1 Iteration 2 Iteration n
Intermediate
Dataset
(in cluster
memory)
Intermediate
Dataset
(in cluster
memory)
Output

4. •
•
•
•
•
•
•

6. “Big Data” Processing (Spark/Flink) Semantic Technology Stack
Data Integration Manual pre-processing Partially automated,
standardised
Modelling Simple (often flat feature vectors) Expressive
Support for data
exchange
Limited (heterogeneous formats
with limited schema information)
Yes (RDF & OWL W3C
Standards)
Business value Direct Indirect
Horizontally
scalable
Yes No
Idea: combine advantages of both worlds

9. •
•
•
val graph: TripleRDD = NTripleReader.load(spark, uri)
graph.find(ANY, URI("http://dbpedia.org/ontology/influenced"), ANY)
val rdf_stats_prop_dist = PropertyUsage(graph, spark).PostProc()

10. •

11. •
•
•
•
•
val rdd = ManchesterSyntaxOWLAxiomsRDDBuilder.build(spark, "file.owl")
// get all subclass-of axioms
val sco = rdd.filter(_.isInstanceOf[OWLSubClassOfAxiom])

13. val graphRdd = NTripleReader.load(spark,input)
val partitions = RdfPartitionUtilsSpark.partitionGraph(graphRdd)
val rewriter = SparqlifyUtils.createSparqlSqlRewriter(spark, partitions)
val qef = new QueryExecutionFactorySparqlifySpark(spark, rewriter)
SANSA Engine
RDF Layer
Data Ingestion
Partitioning
Query Layer
Sparqlifying
Distributed Data
Structures
ResultsViews Views

15. •
•
•
→
→

16. val graph = RDFGraphLoader.loadFromDisk(spark, uri)
val reasoner = new ForwardRuleReasonerOWLHorst(spark.sparkContext)
val inferredGraph = reasoner.apply(graph)
RDFGraphWriter.writeToDisk(inferredGraph, output)
RDFS rule
dependency graph
(simplified)

18. •
•
•
•
•
•
•
•
•
•

19. •
•

20. Visit our demo at 6pm!

21. •
•
•
•
•
•

22. •
•
•
•
•
•
•

23. Web: http://sansa-stack.net
Twitter: @SANSA_Stack
Github: https://github.com/SANSA-Stack
Mail: sansa-stack@googlemail.com

24. •
•
•

26. •
•
•
•
•
•
•
•
•

27. •
•
•
•
•
•
•

28. •
•
•
•
•
•

29. •
•
•
•
•