Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
SAN 
FRANCISCO 
| 
10.22.2014 
Scaling 
Neo4j 
Applica0ons 
@iansrobinson
The 
Burden 
of 
Success 
• More 
users 
• Larger 
datasets 
• More 
concurrent 
requests 
• More 
complex 
queries
Scaling 
is 
a 
Feature 
• It 
doesn’t 
come 
for 
free 
• Condi0ons 
of 
success: 
– Understand 
current 
needs 
• Design...
Overview 
• Scaling 
Reads 
– Latency 
– Throughput 
• Scaling 
Writes 
• Hardware
Scaling 
Reads 
-­‐ 
Latency
Query 
Latency 
latency = f(search_area)
Query 
Latency 
latency = f(search_area)
Query 
Latency 
latency = f(search_area)
Query 
Latency 
latency = f(search_area)
Query 
Latency 
latency = f(search_area)
Query 
Latency 
latency = f(search_area)
Search 
Area 
search_area = f(domain_invariants)
Search 
Area 
search_area = f(domain_invariants) 
Absolute 
Every 
user 
has 
50 
friends
Search 
Area 
search_area = f(domain_invariants) 
Absolute 
Every 
user 
has 
50 
friends
Search 
Area 
search_area = f(domain_invariants) 
Absolute 
Every 
user 
has 
50 
friends 
Rela,ve 
Every 
user 
is 
frien...
Search 
Area 
search_area = f(domain_invariants) 
Absolute 
Every 
user 
has 
50 
friends 
Rela,ve 
Every 
user 
is 
frien...
Reducing 
Read 
Latency 
• The 
Blackadder 
solu0on
Reducing 
Read 
Latency 
• The 
Blackadder 
solu0on 
• Improve 
the 
Cypher 
query 
• Change 
the 
model 
• Use 
an 
Unman...
Improve 
Cypher 
Query 
• Small 
queries, 
separated 
by 
WITH 
• Start 
from 
low-­‐cardinality 
nodes 
hp://thought-­‐by...
Change 
the 
Model 
Goal 
Do 
less 
work 
(in 
the 
query) 
– By 
exploring 
less 
of 
the 
graph 
How? 
Iden0fy 
inferred...
Change 
the 
Model 
-­‐ 
From 
MATCH (:Person{username:'ben'}) 
-[:WORKED_ON]->(:Project)<-[:WORKED_ON]- 
(colleague:Perso...
Change 
the 
Model 
-­‐ 
From 
MATCH (:Person{username:'ben'}) 
-[:WORKED_ON]->(:Project)<-[:WORKED_ON]- 
(colleague:Perso...
Change 
the 
Model 
-­‐ 
To 
MATCH (:Person{username:'ben'}) 
-[:WORKED_WITH]- 
(colleague:Person)
Tradeoff 
More 
expensive 
writes 
More 
data 
Cheaper 
reads 
When 
to 
add 
the 
new 
rela0onship? 
• With 
tx 
• Queue ...
Refactor 
Exis0ng 
Data 
MATCH (p1:Person) 
-[:WORKED_ON]->(:Project)<-[:WORKED_ON]- 
(p2:Person) 
WHERE NOT ((p1)-[:WORKE...
Select 
Batch 
MATCH (p1:Person) 
-[:WORKED_ON]->(:Project)<-[:WORKED_ON]- 
(p2:Person) 
WHERE NOT ((p1)-[:WORKED_WITH]-(p...
Add 
New 
Rela0onship 
MATCH (p1:Person) 
-[:WORKED_ON]->(:Project)<-[:WORKED_ON]- 
(p2:Person) 
WHERE NOT ((p1)-[:WORKED_...
Con0nue 
While 
count(r) 
> 
0 
MATCH (p1:Person) 
-[:WORKED_ON]->(:Project)<-[:WORKED_ON]- 
(p2:Person) 
WHERE NOT ((p1)-...
Use 
Unmanaged 
Extensions 
/db/data/cypher /my-extension/service 
REST 
API 
Extensions
RESTful 
Resource 
@Path("/similar-skills") 
public class ColleagueFinderExtension { 
private static final ObjectMapper MA...
JAX-­‐RS 
Annota0ons 
@Path("/similar-skills") 
public class ColleagueFinderExtension { 
private static final ObjectMapper...
Inject 
Database/Cypher 
Execu0on 
Engine 
@Path("/similar-skills") 
public class ColleagueFinderExtension { 
private stat...
1. 
Get 
Close 
to 
the 
Data 
Applica0on 
MATCH 
MATCH 
CREATE 
DELETE 
MERGE 
MATCH 
Single 
request, 
many 
opera0ons 
...
2. 
Mul0ple 
Implementa0on 
Op0ons 
REST 
API 
Extensions 
Cypher 
Traversal 
Framework 
Graph 
Algo 
Package 
Core 
API
3. 
Control 
Request/Response 
Format 
{ 
users: [ 
{ id: 1234}, 
{ id: 9876} 
] 
} 
JSON, 
CSV, 
protobuf, 
etc 
1a 03 08...
4. 
Control 
HTTP 
Headers 
GET /my-extension/service/top-10 
Applica0on 
Reverse 
Proxy 
HTTP/1.1 200 OK 
Cache-Control: ...
5. 
Integrate 
with 
Backend 
Systems 
Applica0on 
REST 
API 
Extensions 
RDBMS 
LDAP
Migra0ng 
to 
Extensions 
• Re-­‐implement 
original 
query 
inside 
extension 
• Modify 
request/response 
formats 
and 
...
Scaling 
Reads 
-­‐ 
Throughput
Scale 
Horizontally 
For 
High 
Read 
Throughput 
Applica0on
Scale 
Horizontally 
For 
High 
Read 
Throughput 
Applica0on 
Load 
Balancer 
Master 
Slave 
Slave
Scale 
Horizontally 
For 
High 
Read 
Throughput 
Applica0on 
Read 
Load 
Balancer 
Write 
Load 
Balancer 
Master 
Slave 
...
Configure 
HAProxy 
as 
Read 
Load 
Balancer 
global 
daemon 
maxconn 256 
defaults 
mode http 
timeout connect 5000ms 
ti...
Configure 
HAProxy 
as 
Read 
Load 
Balancer 
global 
daemon 
maxconn 256 
defaults 
mode http 
timeout connect 5000ms 
ti...
This 
Isn’t 
The 
Throughput 
You 
Were 
Looking 
For 
Applica0on 
MATCH (c:Country{name:'NZAaoumrsbwtairaya'l}i)a.'.}.) ....
Cache 
Sharding 
Using 
Consistent 
Rou0ng 
Applica0on 
Load 
Balancer 
1 
2 
3 
NZAaoumrsbwtairaya'l}i)a.'.}.) ... 
A-­‐I...
Configure 
HAProxy 
for 
Cache 
Sharding 
global 
daemon 
maxconn 256 
defaults 
mode http 
timeout connect 5000ms 
timeou...
Configure 
HAProxy 
for 
Cache 
Sharding 
global 
daemon 
maxconn 256 
defaults 
mode http 
timeout connect 5000ms 
timeou...
Scaling 
Writes 
-­‐ 
Throughput
Factors 
Impac0ng 
Write 
Performance 
• Managing 
transac0onal 
state 
– Crea0ng 
and 
commilng 
are 
expensive 
opera0on...
Improving 
Write 
Throughput 
• Delay 
taking 
expensive 
locks 
• Batch/queue 
writes
Delay 
Expensive 
Locks 
• Iden0fy 
contended 
nodes 
• Involve 
them 
as 
late 
as 
possible 
in 
a 
transac0on
Add 
Linked 
List 
Item 
+ 
Update 
Pointers
Add 
Linked 
List 
Item 
+ 
Update 
Pointers 
Locked
Add 
Linked 
List 
Item 
+ 
Update 
Pointers 
Locked
Add 
Linked 
List 
Item 
+ 
Update 
Pointers 
Locked
Add 
Linked 
List 
Item
Add 
Linked 
List
Add 
Linked 
List
Add 
Linked 
List
Add 
Pointers 
Locked
Batch 
Writes 
• Mul0ple 
CREATE/MERGE 
statements 
per 
request 
– Good 
for 
integra0on 
with 
backend 
systems 
• Queue...
Single-­‐Threaded 
Queue 
Write 
Write 
Write 
Queue 
Single 
Thread 
Batch
Queue 
Loca0on 
Op0ons 
Applica0on 
Applica0on
Benefits 
of 
Batched 
Writes 
• Less 
transac0onal 
state 
management 
– Create/commit 
per 
batch 
rather 
than 
per 
wr...
Query 
Consolida0on 
MATCH sam 
MATCH jenny 
CREATE sam-[:KNOWS]-jenny 
MATCH sam 
MATCH sarah 
CREATE sam-[:KNOWS]-sarah ...
Eliminate 
Duplicate 
Lookups 
MATCH sam 
MATCH jenny 
CREATE sam-[:KNOWS]-jenny 
MATCH sam 
MATCH sarah 
CREATE sam-[:KNO...
Eliminate 
Duplicate 
Lookups 
MATCH sam 
MATCH jenny 
CREATE sam-[:KNOWS]-jenny 
MATCH sam 
MATCH sarah 
CREATE sam-[:KNO...
Eliminate 
Duplicate 
Lookups 
MATCH sam 
MATCH jenny 
CREATE sam-[:KNOWS]-jenny 
MATCH sarah 
CREATE sam-[:KNOWS]-sarah 
...
Eliminate 
Duplicate 
Lookups 
MATCH sam 
MATCH jenny 
CREATE sam-[:KNOWS]-jenny 
MATCH sarah 
CREATE sam-[:KNOWS]-sarah 
...
Eliminate 
Unnecessary 
Writes 
MATCH sam 
MATCH jenny 
CREATE sam-[:KNOWS]-jenny 
MATCH sarah 
CREATE sam-[:KNOWS]-sarah ...
Eliminate 
Unnecessary 
Writes 
MATCH sam 
MATCH jenny 
CREATE sam-[:KNOWS]-jenny 
MATCH sarah 
CREATE sam-[:KNOWS]-sarah ...
Eliminate 
Unnecessary 
Writes 
MATCH sam 
MATCH jenny 
CREATE sam-[:KNOWS]-jenny 
MATCH sarah 
CREATE sam-[:KNOWS]-sarah ...
Tradeoff 
Latency 
Higher 
throughput 
In-­‐memory 
or 
durable 
queues? 
• Lost 
writes 
in 
event 
of 
crash 
• Transac0...
Further 
Reading 
hp://maxdemarzi.com/2013/09/05/scaling-­‐writes/ 
hp://maxdemarzi.com/2014/07/01/scaling-­‐concurrent-­‐...
Hardware
Memory 
• SLC 
(single-­‐level 
cell) 
SSD 
w/SATA 
• Lots 
of 
RAM 
– 8-­‐12G 
heap 
– Explicitly 
memory-­‐map 
store 
f...
Object 
Cache 
• 2G 
for 
12G 
heap 
• No 
object 
cache 
– consistent 
throughput 
at 
expense 
of 
latency
AWS 
• HVM 
(hardware 
virtual 
machine) 
over 
PV 
(paravirtual) 
• EBS-­‐op0mized 
instances 
• Provisioned 
IOPS
Prochain SlideShare
Chargement dans…5
×

GraphConnect 2014 SF: From Zero to Graph in 120: Scale

679 vues

Publié le

GraphConnect 2014 SF: From Zero to Graph in 120: Scale

Publié dans : Logiciels
  • Soyez le premier à commenter

GraphConnect 2014 SF: From Zero to Graph in 120: Scale

  1. 1. SAN FRANCISCO | 10.22.2014 Scaling Neo4j Applica0ons @iansrobinson
  2. 2. The Burden of Success • More users • Larger datasets • More concurrent requests • More complex queries
  3. 3. Scaling is a Feature • It doesn’t come for free • Condi0ons of success: – Understand current needs • Design for an order of magnitude growth – Itera0ve and incremental development – Unit tests • Bedrock of asserted behaviour – Performance tests
  4. 4. Overview • Scaling Reads – Latency – Throughput • Scaling Writes • Hardware
  5. 5. Scaling Reads -­‐ Latency
  6. 6. Query Latency latency = f(search_area)
  7. 7. Query Latency latency = f(search_area)
  8. 8. Query Latency latency = f(search_area)
  9. 9. Query Latency latency = f(search_area)
  10. 10. Query Latency latency = f(search_area)
  11. 11. Query Latency latency = f(search_area)
  12. 12. Search Area search_area = f(domain_invariants)
  13. 13. Search Area search_area = f(domain_invariants) Absolute Every user has 50 friends
  14. 14. Search Area search_area = f(domain_invariants) Absolute Every user has 50 friends
  15. 15. Search Area search_area = f(domain_invariants) Absolute Every user has 50 friends Rela,ve Every user is friends with 10% of the user base
  16. 16. Search Area search_area = f(domain_invariants) Absolute Every user has 50 friends Rela,ve Every user is friends with 10% of the user base
  17. 17. Reducing Read Latency • The Blackadder solu0on
  18. 18. Reducing Read Latency • The Blackadder solu0on • Improve the Cypher query • Change the model • Use an Unmanaged Extension
  19. 19. Improve Cypher Query • Small queries, separated by WITH • Start from low-­‐cardinality nodes hp://thought-­‐bytes.blogspot.co.uk/2013/01/op0mizing-­‐neo4j-­‐cypher-­‐queries.html hp://wes.skeweredrook.com/pragma0c-­‐cypher-­‐op0miza0on-­‐2-­‐0-­‐m06/
  20. 20. Change the Model Goal Do less work (in the query) – By exploring less of the graph How? Iden0fy inferred rela-onships – Replace with use-­‐case specific shortcuts
  21. 21. Change the Model -­‐ From MATCH (:Person{username:'ben'}) -[:WORKED_ON]->(:Project)<-[:WORKED_ON]- (colleague:Person)
  22. 22. Change the Model -­‐ From MATCH (:Person{username:'ben'}) -[:WORKED_ON]->(:Project)<-[:WORKED_ON]- (colleague:Person)
  23. 23. Change the Model -­‐ To MATCH (:Person{username:'ben'}) -[:WORKED_WITH]- (colleague:Person)
  24. 24. Tradeoff More expensive writes More data Cheaper reads When to add the new rela0onship? • With tx • Queue for subsequent tx • Periodic/batch
  25. 25. Refactor Exis0ng Data MATCH (p1:Person) -[:WORKED_ON]->(:Project)<-[:WORKED_ON]- (p2:Person) WHERE NOT ((p1)-[:WORKED_WITH]-(p2)) WITH DISTINCT p1, p2 LIMIT 10 MERGE (p1)-[r:WORKED_WITH]-(p2) RETURN count(r)
  26. 26. Select Batch MATCH (p1:Person) -[:WORKED_ON]->(:Project)<-[:WORKED_ON]- (p2:Person) WHERE NOT ((p1)-[:WORKED_WITH]-(p2)) WITH DISTINCT p1, p2 LIMIT 10 MERGE (p1)-[r:WORKED_WITH]-(p2) RETURN count(r) Batch size
  27. 27. Add New Rela0onship MATCH (p1:Person) -[:WORKED_ON]->(:Project)<-[:WORKED_ON]- (p2:Person) WHERE NOT ((p1)-[:WORKED_WITH]-(p2)) WITH DISTINCT p1, p2 LIMIT 10 MERGE (p1)-[r:WORKED_WITH]-(p2) RETURN count(r)
  28. 28. Con0nue While count(r) > 0 MATCH (p1:Person) -[:WORKED_ON]->(:Project)<-[:WORKED_ON]- (p2:Person) WHERE NOT ((p1)-[:WORKED_WITH]-(p2)) WITH DISTINCT p1, p2 LIMIT 10 MERGE (p1)-[r:WORKED_WITH]-(p2) RETURN count(r)
  29. 29. Use Unmanaged Extensions /db/data/cypher /my-extension/service REST API Extensions
  30. 30. RESTful Resource @Path("/similar-skills") public class ColleagueFinderExtension { private static final ObjectMapper MAPPER = new ObjectMapper(); private final ColleagueFinder colleagueFinder; public ColleagueFinderExtension( @Context CypherExecutor cypherExecutor ) { this.colleagueFinder = new ColleagueFinder( cypherExecutor.getExecutionEngine() ); } @GET @Produces(MediaType.APPLICATION_JSON) @Path("/{name}") public Response getColleagues( @PathParam("name") String name ) throws IOException { String json = MAPPER .writeValueAsString( colleagueFinder.findColleaguesFor( name ) ); return Response.ok().entity( json ).build(); } }
  31. 31. JAX-­‐RS Annota0ons @Path("/similar-skills") public class ColleagueFinderExtension { private static final ObjectMapper MAPPER = new ObjectMapper(); private final ColleagueFinder colleagueFinder; public ColleagueFinderExtension( @Context CypherExecutor cypherExecutor ) { this.colleagueFinder = new ColleagueFinder( cypherExecutor.getExecutionEngine() ); } @GET @Produces(MediaType.APPLICATION_JSON) @Path("/{name}") public Response getColleagues( @PathParam("name") String name ) throws IOException { String json = MAPPER .writeValueAsString( colleagueFinder.findColleaguesFor( name ) ); return Response.ok().entity( json ).build(); } }
  32. 32. Inject Database/Cypher Execu0on Engine @Path("/similar-skills") public class ColleagueFinderExtension { private static final ObjectMapper MAPPER = new ObjectMapper(); private final ColleagueFinder colleagueFinder; public ColleagueFinderExtension( @Context CypherExecutor cypherExecutor ) { this.colleagueFinder = new ColleagueFinder( cypherExecutor.getExecutionEngine() ); } @GET @Produces(MediaType.APPLICATION_JSON) @Path("/{name}") public Response getColleagues( @PathParam("name") String name ) throws IOException { String json = MAPPER .writeValueAsString( colleagueFinder.findColleaguesFor( name ) ); return Response.ok().entity( json ).build(); } }
  33. 33. 1. Get Close to the Data Applica0on MATCH MATCH CREATE DELETE MERGE MATCH Single request, many opera0ons – Reduce network latencies
  34. 34. 2. Mul0ple Implementa0on Op0ons REST API Extensions Cypher Traversal Framework Graph Algo Package Core API
  35. 35. 3. Control Request/Response Format { users: [ { id: 1234}, { id: 9876} ] } JSON, CSV, protobuf, etc 1a 03 08 96 01 Domain-­‐specific representa0ons – Compact – Conserve bandwidth
  36. 36. 4. Control HTTP Headers GET /my-extension/service/top-10 Applica0on Reverse Proxy HTTP/1.1 200 OK Cache-Control: max-age=60
  37. 37. 5. Integrate with Backend Systems Applica0on REST API Extensions RDBMS LDAP
  38. 38. Migra0ng to Extensions • Re-­‐implement original query inside extension • Modify request/response formats and headers • Refactor implementa0on to use lower parts of the stack where necessary • Measure, measure, measure
  39. 39. Scaling Reads -­‐ Throughput
  40. 40. Scale Horizontally For High Read Throughput Applica0on
  41. 41. Scale Horizontally For High Read Throughput Applica0on Load Balancer Master Slave Slave
  42. 42. Scale Horizontally For High Read Throughput Applica0on Read Load Balancer Write Load Balancer Master Slave Slave
  43. 43. Configure HAProxy as Read Load Balancer global daemon maxconn 256 defaults mode http timeout connect 5000ms timeout client 50000ms timeout server 50000ms frontend http-in bind *:80 default_backend neo4j-slaves backend neo4j-slaves option httpchk GET /db/manage/server/ha/slave server s1 10.0.1.10:7474 maxconn 32 check server s2 10.0.1.11:7474 maxconn 32 check server s3 10.0.1.12:7474 maxconn 32 check listen admin bind *:8080 stats enable
  44. 44. Configure HAProxy as Read Load Balancer global daemon maxconn 256 defaults mode http timeout connect 5000ms timeout client 50000ms timeout server 50000ms frontend http-in bind *:80 default_backend neo4j-slaves backend neo4j-slaves option httpchk GET /db/manage/server/ha/slave server s1 10.0.1.10:7474 maxconn 32 check server s2 10.0.1.11:7474 maxconn 32 check server s3 10.0.1.12:7474 maxconn 32 check listen admin bind *:8080 stats enable Master 404 Not Found false Slave 200 OK true 404 Not Found UNKNOWN Unknown
  45. 45. This Isn’t The Throughput You Were Looking For Applica0on MATCH (c:Country{name:'NZAaoumrsbwtairaya'l}i)a.'.}.) ... Load Balancer 1 2 3
  46. 46. Cache Sharding Using Consistent Rou0ng Applica0on Load Balancer 1 2 3 NZAaoumrsbwtairaya'l}i)a.'.}.) ... A-­‐I 1 J-­‐R 2 S-­‐Z 3 MATCH (c:Country{name:'BJZraiapmzabinal'b'}w})e)'..}..).. ..
  47. 47. Configure HAProxy for Cache Sharding global daemon maxconn 256 defaults mode http timeout connect 5000ms timeout client 50000ms timeout server 50000ms frontend http-in bind *:80 default_backend neo4j-slaves backend neo4j-slaves balance url_param country_code server s1 10.0.1.10:7474 maxconn 32 server s2 10.0.1.11:7474 maxconn 32 server s3 10.0.1.12:7474 maxconn 32 listen admin bind *:8080 stats enable
  48. 48. Configure HAProxy for Cache Sharding global daemon maxconn 256 defaults mode http timeout connect 5000ms timeout client 50000ms timeout server 50000ms frontend http-in bind *:80 default_backend neo4j-slaves backend neo4j-slaves balance url_param country_code server s1 10.0.1.10:7474 maxconn 32 server s2 10.0.1.11:7474 maxconn 32 server s3 10.0.1.12:7474 maxconn 32 listen admin bind *:8080 stats enable
  49. 49. Scaling Writes -­‐ Throughput
  50. 50. Factors Impac0ng Write Performance • Managing transac0onal state – Crea0ng and commilng are expensive opera0ons • Contending for locks – Nodes and rela0onships
  51. 51. Improving Write Throughput • Delay taking expensive locks • Batch/queue writes
  52. 52. Delay Expensive Locks • Iden0fy contended nodes • Involve them as late as possible in a transac0on
  53. 53. Add Linked List Item + Update Pointers
  54. 54. Add Linked List Item + Update Pointers Locked
  55. 55. Add Linked List Item + Update Pointers Locked
  56. 56. Add Linked List Item + Update Pointers Locked
  57. 57. Add Linked List Item
  58. 58. Add Linked List
  59. 59. Add Linked List
  60. 60. Add Linked List
  61. 61. Add Pointers Locked
  62. 62. Batch Writes • Mul0ple CREATE/MERGE statements per request – Good for integra0on with backend systems • Queue – Good for small, online transac0ons
  63. 63. Single-­‐Threaded Queue Write Write Write Queue Single Thread Batch
  64. 64. Queue Loca0on Op0ons Applica0on Applica0on
  65. 65. Benefits of Batched Writes • Less transac0onal state management – Create/commit per batch rather than per write • No conten0on for locks – No deadlocks • Query consolida0on – Reduce the amount of work inside the database
  66. 66. Query Consolida0on MATCH sam MATCH jenny CREATE sam-[:KNOWS]-jenny MATCH sam MATCH sarah CREATE sam-[:KNOWS]-sarah CREATE address1 CREATE address2 DELETE address1 MATCH sam CREATE sam-[:LIVES_AT]-address2
  67. 67. Eliminate Duplicate Lookups MATCH sam MATCH jenny CREATE sam-[:KNOWS]-jenny MATCH sam MATCH sarah CREATE sam-[:KNOWS]-sarah CREATE address1 CREATE address2 DELETE address1 MATCH sam CREATE sam-[:LIVES_AT]-address2
  68. 68. Eliminate Duplicate Lookups MATCH sam MATCH jenny CREATE sam-[:KNOWS]-jenny MATCH sam MATCH sarah CREATE sam-[:KNOWS]-sarah CREATE address1 CREATE address2 DELETE address1 MATCH sam CREATE sam-[:LIVES_AT]-address2
  69. 69. Eliminate Duplicate Lookups MATCH sam MATCH jenny CREATE sam-[:KNOWS]-jenny MATCH sarah CREATE sam-[:KNOWS]-sarah CREATE address1 CREATE address2 DELETE address1 CREATE sam-[:LIVES_AT]-address2
  70. 70. Eliminate Duplicate Lookups MATCH sam MATCH jenny CREATE sam-[:KNOWS]-jenny MATCH sarah CREATE sam-[:KNOWS]-sarah CREATE address1 CREATE address2 DELETE address1 CREATE sam-[:LIVES_AT]-address2
  71. 71. Eliminate Unnecessary Writes MATCH sam MATCH jenny CREATE sam-[:KNOWS]-jenny MATCH sarah CREATE sam-[:KNOWS]-sarah CREATE address1 CREATE address2 DELETE address1 CREATE sam-[:LIVES_AT]-address2
  72. 72. Eliminate Unnecessary Writes MATCH sam MATCH jenny CREATE sam-[:KNOWS]-jenny MATCH sarah CREATE sam-[:KNOWS]-sarah CREATE address1 CREATE address2 DELETE address1 CREATE sam-[:LIVES_AT]-address2
  73. 73. Eliminate Unnecessary Writes MATCH sam MATCH jenny CREATE sam-[:KNOWS]-jenny MATCH sarah CREATE sam-[:KNOWS]-sarah CREATE address2 CREATE sam-[:LIVES_AT]-address2
  74. 74. Tradeoff Latency Higher throughput In-­‐memory or durable queues? • Lost writes in event of crash • Transac0onal dequeue?
  75. 75. Further Reading hp://maxdemarzi.com/2013/09/05/scaling-­‐writes/ hp://maxdemarzi.com/2014/07/01/scaling-­‐concurrent-­‐writes-­‐in-­‐neo4j/
  76. 76. Hardware
  77. 77. Memory • SLC (single-­‐level cell) SSD w/SATA • Lots of RAM – 8-­‐12G heap – Explicitly memory-­‐map store files
  78. 78. Object Cache • 2G for 12G heap • No object cache – consistent throughput at expense of latency
  79. 79. AWS • HVM (hardware virtual machine) over PV (paravirtual) • EBS-­‐op0mized instances • Provisioned IOPS

×