In this talk Tareq will discuss graph solutions based on his experiences building a varied mix of graph-based systems. He will be sharing techniques and approaches that he has learned and will focus on a number of concepts that may be applied to a wider context.

1. Neo4j
Theory and Practice
Tareq Abedrabbo
Graph Connect - 19/11/2013

2. About me
•
CTO/Principal Consultant at OpenCredo
•
Working with Neo4j for (almost) 3 years on a
number of different projects
•
Co-author of Neo4j in Action (Manning)

3. What is this talk
about?

4. It’s for developers
designing and building
applications with Neo4j

5. It’s not a collection of war
stories but I will refer to
real-world examples

6. It is about sharing
thoughts and lessons
learnt in a useful way

7. “If I'm to believe Twitter, half of the
earth's population are importing
Wikipedia into Neo4j, for very
obscure reasons.”

8. Agenda

9. •
What is Neo4j?
•
Approaching graph-based applications
•
Design
•
Implementation
•
Test
•
Use cases
•
Lessons learnt

10. What really is Neo4j?

11. A graph model

12. A query engine

13. A database

14. Neo4j is a solid foundation
on which to build graphbased applications

15. How should I approach
graph-based
applications?

16. Is there a useful way to
categorise graph-based
applications?

17. Domain-centric
applications

18. Data-centric
applications

19. Domain-Centric
•
Well-defined data model
•
Data changes through user interactions
•
Flexible but predictable data structure(s)
•
Recommendation engines, social networks, etc…
•
Top-down design

20. Data-Centric
•
Complex connected data that typically models real
world networks
•
Integrated from a variety of different sources
•
Data can be unpredictable
•
Telco networks, utility networks, etc…
•
bottom-up design

21. Typically applications fall
somewhere between
these 2 types

22. How can I use the
information available in
my graph?

23. •
Search and pattern-matching
•
•
Graph algorithms
•
•
Find a recommendation based on behaviour
Shortest path, disconnected components
Optimisation
•
Maximise oil flow while minimising water

24. Graphs are naturally
data-driven

25. Use case 1:
Network Impact Analysis

27. Requirement: Identify the
impact of failing
components

28. Requirement: Identify
interesting patterns, such
as single points of failure

29. Labelled property graph
is a natural fit for the
model

30. Additional “dimensions” can be
added to capture abstract concepts:
network redundancy, load-balancing

31. Cypher queries are a
natural solution to delivering
the different requirements

32. Use case 2:
Oil flow optimisation

34. Requirement: Identify
candidate configurations
to maximise flow

35. Requirement: Identify the
most practical and valuable
adjustments to the network

36. Simply connected graph
with complex
components

37. Interlude: Genetic
Algorithms

38. •
Start from an initial population of candidate solutions
(individuals or phenotypes), ideally random
•
Attribute a score each solution using a fitness function
•
•
The only place with specific business knowledge
Apply genetic operators to create a new generation
•
•
•
Cross-breeding to retain best characteristics from each
parent
Mutation to maintain diversity and to avoid converging to a
local optima too quickly
Stop when you want!

40. Is this even a use
case for Neo4j?

41. Persist and share
calculated solutions

42. Inspect intermediary
steps

43. Use Cypher queries to
interrogate solutions

44. Lessons learnt

45. Understand your
domain

46. •
Don’t follow “best practices” blindly
•
For domain-centric applications you can use a
mapping framework, such as Spring Data Neo4j
•
For data-centric applications, you should stay as
close as possible to the graph model
•
In any case, don’t try to hide the graph!

47. Use Cypher

48. !
•
Expressive
•
Readable
•
Maintainable
•
Performant
•
Cypher + the web console is the quickest way to
experiment and to prototype solutions

49. Manage complexity
with domain knowledge

50. •
Graph algorithms are typically complex
•
Knowledge of the domain can simplify queries and
traversals
•
Make Cypher queries as specific as possible
•
Take “shortcuts” when you know the domain

51. Write robust and
flexible code

52. •
Break down problems into a small queries. Return
graph resources (or ids) to chain queries.
•
Robustness principal: “Be conservative in what you
do, be liberal in what you accept from others”
•
Use assertions as preconditions
•
Assertions document intent
•
Fail fast if data doesn’t match

53. Start with a
representative dataset

54. •
Create a small data sets to capture the initial use
cases
•
Write simple unit tests using these datasets to
support design and implementation
•
These tests tend to become less useful when
requirements are better understood
•
Throw them away!

55. Move to a realistic
dataset as soon as
possible

56. •
A realistic data set
•
Should capture the complexity of the real data
•
Should be sufficiently large
•
Ideally based on production data
•
Write functional and integration tests against this
dataset

57. Test non-functional
aspects

58. •
Graph data is inherently flexible and evolving
•
Queries need to be correct and sufficiently performant
•
Existing queries’s performance can degrade as the
underlying model changes
•
Assertions on timeouts should be part of the test suite
to detect loops and poor performance
•
JUnit’s @Test(timeout=5)
•
Spring’s @Timeout(value=5)

59. Links
•
Twitter: @tareq_abedrabbo
•
Blog: http://www.terminalstate.net
•
OpenCredo: http://www.opencredo.com
Thank you!