Slides from presentation at Graph Day Texas discussing some of the problems we faced and what we did to fix them to keep our customer facing response times low and our data ingestion pipeline humming.

1. Addressing Performance
Issues in Titan+Cassandra

2. Introduction
● Nakul Jeirath
● Senior security engineer at WellAware (wellaware.us)
● WellAware: Oil & gas startup building a SaaS monitoring & analytics platform

3. Transitioned ~2 years ago

4. Titan+Cassandra Performance Factors
● Titan deployment methodology
● Cassandra tuning
● Titan JVM tuning
● Data modeling
● Indexing
○ Property indices
○ Vertex centric indices
● Query structure
● Caching
○ Transaction cache
○ Database level cache
● Titan options

5. Titan+Cassandra Performance Factors
● Titan deployment methodology
● Cassandra tuning
● Titan JVM tuning
● Data modeling
● Indexing
○ Property indices
○ Vertex centric indices
● Query structure
● Caching
○ Transaction cache
○ Database level cache
● Titan options
Ted Wilmes - Cassandra Summit 2015:
Slides: http://www.slideshare.net/twilmes/modeling-the-iot-with-titandb-and-cassandra
Video: https://vimeopro.com/user35188327/cassandra-summit-2015/video/143695770
This talk
Our focus will be reads, check out Ted's
talk for write optimization

6. A Toy Example
http://coachesbythenumbers.com/sportsource-college-football-data-packages/
2005 College Football Data
● Team names & conferences
● Game record with dates and scores
● Interesting questions:
○ Records for all teams in conference X
○ Top 25 ranking using record + strength of opponents
○ Three team loop (A beat B beat C beat A)

7. Toy Model
Label: team
name: Purdue
conf: Big 10
Label: team
name: IU
conf: Big 10
label: beat
date: 11/19/05
score: 41-14
gremlin> g.V().count()
==>239
gremlin> g.E().count()
==>718

8. Test Bench
Shut down
Titan
Clear
Titan DB
Start
Titan
Load test dataset
Source code:
https://github.com/njeirath/titan-perf-tester

9. Test Runner
public class PerfTestRunner {
public static DescriptiveStatistics test(final TitanGraph graph, int iterations, PerfOperation op) {
DescriptiveStatistics stats = new DescriptiveStatistics();
for (int i = 0; i < iterations; i++) {
TitanTransaction tx = graph.newTransaction();
Date start = new Date();
op.run(tx);
Date end = new Date();
stats.addValue(end.getTime() - start.getTime());
tx.rollback();
}
return stats;
}
}
Pass in test query
as LambdaStart new transaction
Run test query
Record time
Rollback transaction

10. Anatomy of Gremin Queries
● Simplest form of OLTP query
○ picks an entry point(s) to graph
○ traverses from initial vertices
Initial graph entry
selection
Edge traversal
Example:
How many games did a Big 10 team win?
g.V().has('conference', 'Big Ten Conference').outE('beat').count()

11. Selecting the Entry Point
Typically won't have vertex ID(s) to select directly
Will select based on one or more vertex property
Feasible to scan all vertices in small graphs
Becomes prohibitively expensive on large graphs
Start from these

12. Property Index
Test query: g.V().has('conference', 'Big Ten Conference').toList()
Output:
07:45:24 WARN com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx -
Query requires iterating over all vertices [(conference = Big Ten Conference)]. For
better performance, use indexes
Titan is nice enough to warn us of this issue

13. Creating Index on "Conference" Property
mgmt = graph.openManagement()
conf = mgmt.getPropertyKey('conference')
mgmt.buildIndex('byConference',
Vertex.class).addKey(conf).buildCompositeIndex()
mgmt.commit()
mgmt.awaitGraphIndexStatus(graph, 'byConference').call()
mgmt = graph.openManagement()
mgmt.updateIndex(mgmt.getGraphIndex("byConference"), SchemaAction.
REINDEX).get()
mgmt.commit()
Access graph management,
create composite index, and
commit
Wait for key to be
available and reindex
Reference: http://s3.thinkaurelius.com/docs/titan/1.0.0/indexes.html

14. "Conference" Index Timing Comparison
Without Index:
n: 10
min: 127.0
max: 203.0
mean: 159.9
std dev: 29.598986469134378
median: 151.0
With Index:
n: 10
min: 2.0
max: 7.0
mean: 3.2
std dev: 1.6865480854231356
median: 2.5
Represents 49.96875x increase

15. Property Indices in Titan
● Composite Index
○ Supports equality comparison only
○ Can handle combinations of properties but must be pre-defined (Ex: Name and Age)
● Mixed Index
○ Greater conditionality support
○ Can handle lookups on arbitrary combinations of indexed keys
● Titan also has support for other external indexing backend
● Reference documentation: http://s3.thinkaurelius.com/docs/titan/1.0.0
/indexes.html

16. High Order Vertices
Don't always want to traverse all edges incident on a vertex
Filtering based on some edge properties is desirable
Similar to vertices: feasible to inspect each edge for low order
vertices
Prohibitive on high order vertices
Traverse these
edges

17. Vertex Centric Index
Example query:
dateFilter = lt(20051000)
g.V().has('conference', 'Big Ten Conference').as('team', 'wins', 'losses')
.select('team', 'wins', 'losses')
.by('name')
.by(__.outE().has('date', dateFilter).count())
.by(__.inE().has('date', dateFilter).count())
Gets Big 10 team records for games played before October 2005

18. Notes on Vertex Centric Indices
From Titan 1.0.0 documentation:
Reference documentation: http://s3.thinkaurelius.com/docs/titan/1.0.0/indexes.html#vertex-indexes
Titan 0.4.4 does not automatically create vertex-centric indices
No need to create one for our example
May be necessary if a composite key query is being performed
Ex: Get Big 10 team records for games played before October 2005 and won by more than 14 points

19. Query Structuring
Order of steps in query can make a difference
Consider:
g.V().order().by(__.outE().count(), decr).has('conference', 'Big Ten Conference').values('name')
vs
g.V().has('conference', 'Big Ten Conference').order().by(__.outE().count(), decr).values('name')
Mean times: 1032.8 ms vs 42.8 ms respectively

20. Titan Caching
Support for database and transaction level caching
Storage
Backend
Titan
DB Cache
TX Cache
TX Cache
TX Cache
Client
Client
Client

21. Transaction Cache
Transaction starts on graph access and ends on commit or rollback
Useful for workloads accessing same data repeatedly
A
B
C
D
Rank of team A
is a count of
these "beat"
edges
Ex: Team Rankings
g.V().order().by(__.out().out().count(), decr).as('team', 'score', 'wins', 'losses').select('team', 'score', 'wins',
'losses').by('name').by(__.out().out().count()).by(__.outE().count()).by(__.inE().count()).limit(25)
With TX cache: 3361 ms, without TX cache: 5206 ms

22. /r/mildlyinteresting/
1. Texas
2. USC
3. Penn State
4. Ohio State
5. Virginia Tech
6. TCU
7. West Virginia
8. Lousianna State
9. Alabama
10. Oregon
11. Louisville
12. Georgia
13. UCLA
14. Miami (FL)
1. Texas
2. USC
3. Penn State
4. Virginia Tech
5. LSU
6. Ohio State
7. Georgia
8. TCU
9. West Virginia
10. Alabama
11. Boston College
12. Oklahoma
13. Florida
14. UCLA
http://www.collegefootballpoll.com/2005_archive_computer_rankings.html
2005 End of
Season
Computer
Rankings
Our Query
Results

23. Transaction Caching Gotchas
Cache Thrashing
Symptom: Queries suddenly & significantly slow
down as data size increases
Solve this by tuning transaction cache size
● Globally by setting cache.tx-cache-size
● Per transaction using TransactionBuilder
Memory Leak
Transactions automatically started and are
thread aware
With read only access in separate threads,
transaction caches can leak
Solved by calling g.rollback() at the end of the
thread execution (releases the TX cache)

24. Transaction Cache Settings
Transaction cache can be setup in properties files
Settings can be overridden when creating transaction using TransactionBuilder:
Example:
tx=graph.buildTransaction().vertexCacheSize(50000).start()
Other transaction settings can be found here: http://s3.thinkaurelius.
com/docs/titan/1.0.0/tx.html#tx-config

25. Database Level Caching
Database caching helps performance across transactions:
gremlin> stats2.getValues()
==>[1016.0, 41.0, 27.0, 26.0, 24.0, 23.0, 24.0, 21.0, 18.0, 18.0]
Trades consistency for speed in clusters
Node 1
Node 2
Node n
Titan 1
Titan 2
Titan n
Cold
cache
Warm cache
1. Read
2. Write
3. Read

26. Titan Options
query.batch - Whether traversal queries should be batched when executed
against the storage backend. This can lead to significant performance
improvement if there is a non-trivial latency to the backend.
query.fast-property - Whether to pre-fetch all properties on first singular vertex
property access. This can eliminate backend calls on subsequent property access
for the same vertex at the expense of retrieving all properties at once. This can be
expensive for vertices with many properties
http://s3.thinkaurelius.com/docs/titan/1.0.0/titan-config-ref.html

27. Using "query.fast-property"
Test query: g.V().group().by('conference').by('name')
query.fast-property = false:
n: 10
min: 243.0
max: 262.0
mean: 250.4
std dev: 5.125101625008685
median: 250.0
query.fast-property = true:
n: 10
min: 127.0
max: 151.0
mean: 138.1
std dev: 7.233410137841088
median: 139.5

28. Summary
● Titan indices
○ Property indices - vertex/edge lookups
○ Vertex centric indices - edge traversals
● Generally limiting elements early in traversal is a good thing
● Caching
○ Database level - improve speed while potentially increasing likelihood of stale data
○ Transaction level - helps when repeatedly visiting elements within a transaction
● Various other options available for specific tuning needs

29. Thanks For Watching
Questions
Nakul Jeirath
@njeirath
Senior Security Engineer - WellAware