My presentation on scalable incremental graph queries on the PhD Minisymposium, based on joint work with my Master's student.

1. Budapest University of Technology and Economics
Department of Measurement and Information Systems
Budapest University of Technology and Economics
Fault Tolerant Systems Research Group
Sharded Joins for Scalable
Incremental Graph Queries
János Maginecz, Gábor Szárnyas

2. Agile Model-Driven Development
Modeling

3. Agile Model-Driven Development
Modeling
Early validations
Transformations

4. Agile Model-Driven Development
Modeling
Code
generation
Early validations
Transformations

5. Agile Model-Driven Development
Modeling
Code
generation
Testing
Early validations
Transformations

6. Agile Model-Driven Development
Modeling
Code
generation
Testing
Early validations
Transformations

7. Agile Model-Driven Development
Modeling
Code
generation
Testing
Early validations
Transformations
Scalability
challenges

8. Performance
issues
Agile Model-Driven Development
Modeling
Code
generation
Testing
Early validations
Transformations
Scalability
challenges

9. Performance
issues
Agile Model-Driven Development
Modeling
Code
generation
Testing
Early validations
Transformations
Scalability
challenges

10. Scalability
Incrementality

11. Scalability
Incrementality
Storing partial
results

12. Scalability
Incrementality
Storing partial
results
Tracking
changes

13. MDD
Scalability
Incrementality
Storing partial
results
Tracking
changes

14. MDD
Scalability
Incrementality
Incremental
queries
Storing partial
results
Tracking
changes

15. MDD
Scalability
Incrementality
Incremental
queries
Incremental
transformation
Storing partial
results
Tracking
changes

16. MDD
Scalability
Incrementality
Incremental
queries
Storing partial
results
Tracking
changes

17. Motivating Example
 Error pattern for an AUTOSAR validation constraint
Communication
channel
Logical signal Mapping Physical signal

18. Motivating Example
 Error pattern for an AUTOSAR validation constraint
Communication
channel
Logical signal Mapping Physical signal
 Validation

19. Motivating Example
 Error pattern for an AUTOSAR validation constraint
Communication
channel
Logical signal Mapping Physical signal
Invalid submodel
 Validation

20. Motivating Example
 Error pattern for an AUTOSAR validation constraint
Communication
channel
Logical signal Mapping Physical signal
Invalid submodel
 Validation

21. Motivating Example
 Error pattern for an AUTOSAR validation constraint
Communication
channel
Logical signal Mapping Physical signal
Invalid submodel
 Validation
Valid submodel

22. Antijoin
Join
Join
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

23. Antijoin
Join
Join
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

24. Antijoin
Join
Join
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

25. Antijoin
Join
Join
Fill indexer nodes
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

26. Antijoin
Join
Join
Fill indexer nodes
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

27. Antijoin
Join
Join
Fill indexer nodes
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

28. Antijoin
Join
Join
Fill indexer nodes
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

29. Antijoin
Join
Join
Fill indexer nodes
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

30. Antijoin
Join
Join
Fill indexer nodesStore interim results
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

31. Antijoin
Join
Join
Fill indexer nodesStore interim results
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

32. Antijoin
Join
Join
Fill indexer nodesStore interim results
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

33. Antijoin
Join
Join
Fill indexer nodesStore interim results
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

34. Antijoin
Join
Join
Fill indexer nodesStore interim results
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

35. Antijoin
Join
Join
Fill indexer nodesStore interim results
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

36. Antijoin
Join
Join
Fill indexer nodesStore interim results
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

37. Antijoin
Join
Join
Fill indexer nodesStore interim resultsRead result set
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal
Result set

38. Antijoin
Join
Join
Fill indexer nodesStore interim resultsRead result set
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

39. Antijoin
Join
Join
Fill indexer nodesStore interim resultsRead result setEdit model
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

40. Antijoin
Join
Join
Fill indexer nodesStore interim resultsRead result setEdit modelPropagating changes
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

41. Antijoin
Join
Join
Fill indexer nodesStore interim resultsRead result setEdit modelPropagating changes
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

42. Antijoin
Join
Join
Fill indexer nodesStore interim resultsRead result setEdit modelPropagating changesRead result set
Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal
Result set

43. Single Workstation Rete Implementation
 Rete-based incremental graph query engine
 Open-source Eclipse project
 Java Virtual Machine limitations
o Cannot handle 15+ GB heap memory efficiently
 Proposed solution
o Horizontal scaling: distributed system

44. IncQuery-D Architecture
Transaction
In-memory
EMF model
Rete net
Indexer
layer
EMF-INCQUERY

45. IncQuery-D Architecture
Transaction
In-memory
EMF model
Rete net
Indexer
layer
EMF-INCQUERY
In-memory storage

46. IncQuery-D Architecture
Transaction
In-memory
EMF model
Rete net
Indexer
layer
EMF-INCQUERY
Indexing
In-memory storage

47. IncQuery-D Architecture
Transaction
In-memory
EMF model
Rete net
Indexer
layer
EMF-INCQUERY
Indexing
In-memory storage
Production network
• Stores intermediate query results
• Propagates changes

48. IncQuery-D Architecture
Transaction
In-memory
EMF model
Rete net
Indexer
layer
EMF-INCQUERY

49. IncQuery-D Architecture
Server 1
Database
shard 1
Server 2
Database
shard 2
Server 3
Database
shard 3
Transaction
Database
shard 0
Server 0
Rete net
Indexer
layer
INCQUERY-D

50. IncQuery-D Architecture
Server 1
Database
shard 1
Server 2
Database
shard 2
Server 3
Database
shard 3
Transaction
Database
shard 0
Server 0
Rete net
INCQUERY-D

51. IncQuery-D Architecture
Server 1
Database
shard 1
Server 2
Database
shard 2
Server 3
Database
shard 3
Transaction
Database
shard 0
Server 0
Rete net
INCQUERY-D
Distributed indexer Model access adapter

52. IncQuery-D Architecture
Server 1
Database
shard 1
Server 2
Database
shard 2
Server 3
Database
shard 3
Transaction
Database
shard 0
Server 0
INCQUERY-D
Distributed query evaluation network
Distributed indexer Model access adapter

53. IncQuery-D Architecture
Server 1
Database
shard 1
Server 2
Database
shard 2
Server 3
Database
shard 3
Transaction
Database
shard 0
Server 0
INCQUERY-D
Distributed query evaluation network
Distributed indexer Model access adapter
Distributed persistent
storage

54. IncQuery-D Architecture
Server 1
Database
shard 1
Server 2
Database
shard 2
Server 3
Database
shard 3
Transaction
Database
shard 0
Server 0
INCQUERY-D
Distributed query evaluation network
Distributed indexer Model access adapter
Distributed indexing,
notification
Distributed persistent
storage

55. IncQuery-D Architecture
Server 1
Database
shard 1
Server 2
Database
shard 2
Server 3
Database
shard 3
Transaction
Database
shard 0
Server 0
INCQUERY-D
Distributed query evaluation network
Distributed indexer Model access adapter
Distributed indexing,
notification
Distributed persistent
storage
Distributed production network
• Each intermediate node can be allocated
to a different host
• Remote internode communication

56. IncQuery-D Architecture
Server 1
Database
shard 1
Server 2
Database
shard 2
Server 3
Database
shard 3
Transaction
Database
shard 0
Server 0
INCQUERY-D
Distributed query evaluation network
Distributed indexer Model access adapter

57. IncQuery-D Architecture
Server 1
Database
shard 1
Server 2
Database
shard 2
Server 3
Database
shard 3
Transaction
Database
shard 0
Server 0
INCQUERY-D
Distributed query evaluation network
Indexer Indexer Indexer Indexer
Join
Join
Antijoin

58. Working around Memory Limits
Host-2
Host-1
Input
Node A
Node B
Distributed
Output
Host-1
Input
Node A
Node B
Local
Output
Solution 1
Simple and efficient
Memory of the machine is an upper
bound for the network
Nodes run on different computers
The memory of each node is
limited to the assigned machine
+
−
+
−

59. Working around Memory Limits
Host-2
Host-1
Input
Node A
Node B
Distributed
Output
Host-1
Input
Node A
Node B
Local
Output
Solution 1
EMF-IncQuery IncQuery-D
Simple and efficient
Memory of the machine is an upper
bound for the network
Nodes run on different computers
The memory of each node is
limited to the assigned machine
+
−
+
−

60. Host-3Host-1
Host-2
Working around Memory Limits
Distributed
+
Sharded
Input
Node A
Node B
Output
Solution 2
Host-2
Host-1
Input
Node A
Node B
Distributed
Output
Nodes may be allocated on
more than 1 computer
Network overhead
+
−
Nodes run on different computers
The memory of each node is limited to the assigned
machine
+
−

61. Host-3Host-1
Host-2
Working around Memory Limits
Distributed
+
Sharded
Input
Node A
Node B
Output
Solution 2
IncQuery-DS
Host-2
Host-1
Input
Node A
Node B
Distributed
Output
IncQuery-D
Nodes may be allocated on
more than 1 computer
Network overhead
+
−
Nodes run on different computers
The memory of each node is limited to the assigned
machine
+
−

62. Í
Join
Antijoin
Join / Shard 2Join / Shard 1
Sharded Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

63. Í
Join
Antijoin
Join / Shard 2Join / Shard 1
Sharded Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

64. Í
Join
Antijoin
Join / Shard 2Join / Shard 1
Sharded Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

65. Í
Join
Antijoin
Join / Shard 2Join / Shard 1
Sharded Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

66. Í
Join
Antijoin
Join / Shard 2Join / Shard 1
Sharded Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal

67. Í
Join
Antijoin
Join / Shard 2Join / Shard 1
Sharded Rete Algorithm
Communication
channel
Logical signal Mapping Physical signal
IncQuery-DS
Distributed and Sharded

68. Validation of Critical Systems
 Model validation for large models
 Well-formedness contraints with complex graph
patterns
 Train Benchmark
o Open-source performance measurement framework
o Presented yesterday

69. Train Benchmark
 Phases
o Initial read and validation
o Small changes and revalidation
• Simulating modifications from a user
 Goal: Measure response times
Execution timeExecution time
Read Transformation RevalidationValidation
× 10× 3

70. Sharding Results

71. Sharding Results

72. Sharding Results

73. Join Optimization
 Hash join
o Using hash maps
 Sort merge join
o Using red-black trees
 Collection frameworks
o Standard library in Scala
o Goldman Sachs Collections

74. Join Optimization Results – Execution Time

75. Join Optimization Results – Execution Time
Does not scale
well for updates

76. Summary
 Designed a sharded Rete engine
 Evaluated its scalability
 Analysis of join algorithms and collection
frameworks
 Future work
o Domains with similar challenges

77. Ω