This presentation covers IoT modeling in Titan along with a discussion of time series storage and performance analysis.

1. Modeling the IoT with TitanDB and Cassandra

2. Intro
• Ted Wilmes
• Data warehouse engineer at WellAware - wellaware.us
• Building a SaaS oil and gas production monitoring and
analytics platform
• Collect production O&G data from the field via cellular,
satellite, and other means and deliver to our customers via
mobile and browser clients
© 2015. All Rights Reserved. 2

3. © 2015. All Rights Reserved.
1 The property graph model and TitanDB
2 Modeling IoT
3 Time series and performance

4. Property graph model
© 2015. All Rights Reserved. 4
person
name: Ted
person
name: George
knows
metOn: June 1,2012

5. Querying with Gremlin
© 2015. All Rights Reserved. 5
person
name: Ted
person
name: George
knows
metOn: June 1,2012
g.V().hasLabel(“person”).has(“name”, “Ted”).out(“knows”).values(“name”)
> George

6. TitanDB
• Graph database that supports pluggable storage layers
• Designed from the ground up to provide OLTP
performance against large graphs with a particular
focus on supporting high degree vertices (vertices with
many edges)
• Implements Apache TinkerPop 3 APIs
• Cassandra acts as solid foundation providing high
availability, performance, and ease of operation
© 2015. All Rights Reserved. 6

7. Our Internet of Things
© 2015. All Rights Reserved. 7
Things
People
Organizations
Places
Time

8. A hypothetical use case: IoT…
© 2015. All Rights Reserved. 8
in SPACE

9. © 2015. All Rights Reserved. 9
Spaceship
Mars
Base
Space
Station
Rocket
Satellite

10. © 2015. All Rights Reserved. 10

11. Many dimensions
© 2015. All Rights Reserved. 11
Rocket
Starfleet
Acme
Rockets
Delta
Booster
operates
builds
isModel
Major Tom
pilots Joyce
maintains

12. Many times, a “thing” is a system of systems
© 2015. All Rights Reserved. 12
http://stardust.jpl.nasa.gov/mission/delta2.html
Rocket
1st Stage 2nd Stage
3rd Stage
Interstage
Fuel Tank
Oxidizer
Guidance
Electronics
CPU Memory

13. © 2015. All Rights Reserved. 13
Guidance
Electronics
CPU
Memory JVM
Heap
Usage
Thread
Count
Continuing to zoom in

14. Heap
Usage
© 2015. All Rights Reserved. 14
JVM
Thread
Count
Alarm
Alarm
Condition
Joyce
triggers
notifies
monitors

15. © 2015. All Rights Reserved. 15
Major Tom
Alarm
Alarm
Condition
Joyce
triggers
notifies
reports to
Starfleet
employs
employs

16. IoT modeling in summary
• Things can be interconnecting systems of other
things
• High fidelity model of ‘reality’ supports wide
variety of use cases vs. a disconnected set of
entities
• IoT app is really only one part about things,
don’t forget to include everything else! (social,
organizational, etc.)
© 2015. All Rights Reserved. 16

17. Time series & Performance
© 2015. All Rights Reserved. 17

18. © 2015. All Rights Reserved. 18
Guidance
Electronics
CPU
Memory JVM
Heap
Usage
Thread
Count

19. © 2015. All Rights Reserved. 19
Time series in Titan
Heap
Usage
JVM
?

20. © 2015. All Rights Reserved. 20
Our basic time series requirements
• Support a large volume of low latency
writes
• Low latency retrieval on primarily the most
recent data

21. © 2015. All Rights Reserved. 21
A selection of factors affecting Titan performance
• Titan deployment topology and configuration
• All your usual Cassandra tuning tips and tricks
• Titan JVM tuning
• selection of appropriate garbage collector
• GC parameters
• like Cassandra, worthwhile to adjust NewSize
• Data modeling
• Indexing
• Global graph indices (native Titan vs. external)
• Vertex centric indices
• Titan different caches - transaction cache & the database-level cache

22. © 2015. All Rights Reserved. 22
A selection of factors affecting Titan performance
• Titan deployment topology and configuration
• All your usual Cassandra tuning tips and tricks
• Titan JVM tuning
• selection of appropriate garbage collector
• GC parameters
• like Cassandra, worthwhile to adjust NewSize
• Data modeling
• Indexing
• Global graph indices
• Vertex centric indices
• Titan different caches - transaction cache & the database-level cache

23. Deployment options
© 2015. All Rights Reserved. 23
mars-north-1
Local
Embedded
Remote

24. © 2015. All Rights Reserved. 24

25. © 2015. All Rights Reserved. 25

26. © 2015. All Rights Reserved. 26
But first, time series with CQL
* Brady Gentile - https://academy.datastax.com/demos/getting-started-time-series-data-modeling

27. © 2015. All Rights Reserved. 27
But first, time series with CQL
* Brady Gentile - https://academy.datastax.com/demos/getting-started-time-series-data-modeling
CQL

28. © 2015. All Rights Reserved. 28
First approach
Heap
Usage
Chunk
chunkStart: 1442880000000
chunkEnd: 1442966400000
Chunk
chunkStart: 1442966400000
chunkEnd: 1442966400000
Observation Observation
tstamp: 1442880000001 tstamp: 1442880000002
• Intuitive and easy to query
• You can imagine adding further
levels to the hierarchy following
a year->month->day format
• Individual observations can be
associated with other pieces of
data
• Observations can be filtered by
timestamp with edge filter but
you still have to retrieve a large
number of disparate vertices

29. © 2015. All Rights Reserved. 29
A further refinement
Heap
Usage
Chunk
chunkStart: 1442880000000
chunkEnd: 1442966400000
Chunk
chunkStart: 1442966400000
chunkEnd: 1442966400000
• How do we reduce the number
of vertices (think Cassandra
partitions) that we need to
retrieve?

30. © 2015. All Rights Reserved. 30
timestamp value
1. Move all properties to the edge
2. Make the edge “undirected”
or, a combo of the two approaches
1. Copy the properties to the edge
2. Keep the discrete observation
vertex
Chunk
tstamp
value
Heap
Usage

31. © 2015. All Rights Reserved. 31
Chunk vertex with its observations
Vertex ID chunkStart chunkEnd obs. @ t2 obs. @ t1 obs. @ t0
Observations in time descending order

32. © 2015. All Rights Reserved. 32
Sample Gremlin queries
• observations > 1442162072000
• chunk.outE().has(“tstamp”, gt(1442162072000))
• observations between 1442162072000 and 1442162073000
• chunk.outE().has(“tstamp”, between(1442162072000, 1442162073000))
• Most recent observation before now
• chunk.outE().has(“tstamp”, lte(System.currentTimeMillis()).
order().by(“tstamp”, decr).limit(1)
• You can wrap this in your own time series specific API
• new SeriesQuery(series1).interval(startTstamp, endTstamp).decr().limit(1)

33. © 2015. All Rights Reserved. 33
Pros and cons vs. separate CQL or other tsdb
• Pros
• Allows for a single unified view of your IoT data, maintaining
direct connectivity between sensor data & the other entities
• Gremlin works well for processing streams of time series
data
• Cons
• Storage format is not as compact
• Extra overhead of managing ‘chunks’ versus CQL primary
key taking care of that for us (eg. chunk cache)

34. © 2015. All Rights Reserved. 34
Heap
Usage
Chunk
label: hasChunk
chunkStart: 1442880000000
chunkEnd: 1442966400000
Chunk
label: hasChunk
chunkStart: 1442966400000
chunkEnd: 1442966400000
A simple query - retrieve all the heap usage chunks
gremlin> g.V(4).out(‘hasChunk’).values(‘chunkStart’)
==> 1442880000000
==> 1442966400000

35. © 2015. All Rights Reserved. 35
Getting a vertex by id
gremlin> g.V(4)
==>v[4]
Yes
Does this vertex exist?
Vertex is now loaded in Titan
transaction cache

36. © 2015. All Rights Reserved. 36
Aside - a tool of the trade
Profiler with socket tracing

37. © 2015. All Rights Reserved. 37

38. © 2015. All Rights Reserved. 38

39. © 2015. All Rights Reserved. 39
Retrieving properties
gremlin> g.V(4).valueMap()
==>[sensorType:[heap usage], units:[bytes]]
Two properties
Retrieve properties
Vertex properties are now loaded
in the Titan transaction cache

40. © 2015. All Rights Reserved. 40
2 Round trips
Two properties
Retrieve properties
Yes
Does this vertex exist?
• Not a big deal for single
vertex lookup with
property retrieval but
can add up
• Exacerbated by
magnitude of latency
between Titan and
Cassandra

41. © 2015. All Rights Reserved. 41
Querying for adjacent vertices
gremlin>
g.V(4).out(‘hasChunk’).values(‘chunkStart’)
==> 1442880000000
==> 1442966400000
Does this vertex exist?
Get 1st chunk properties
Get edges
Get 2nd chunk properties

42. © 2015. All Rights Reserved. 42
Batch requests
Does this vertex exist?
Get 1st chunk properties
Get 2nd chunk properties
Get edges
Get 2nd chunk properties
• query.batch = true
• “Whether traversal queries should
be batched when executed against
the storage backend. This can lead
to significant performance
improvement if there is a non-trivial
latency to the backend.” - http://
s3.thinkaurelius.com/docs/titan/0.9.0-M2/titan-config-ref.html
gremlin>
g.V(4).out(‘hasChunk’).values(‘chunkStart’)
==> 1442880000000
==> 1442966400000

43. © 2015. All Rights Reserved. 43
Remove initial exists query
• storage.batch-loading = true
• WARNING - this disables
vertex ‘exists’ checks
gremlin>
g.V(4).out(‘hasChunk’).values(‘chunkStart’)
==> 1442880000000
==> 1442966400000
Does this vertex exist?
Get 1st chunk properties
Get 2nd chunk properties
Get edges
Get 2nd chunk properties

44. © 2015. All Rights Reserved. 44
Optimizing your write
gremlin> chunk.addEdge(“hasObservation”, chunk, “tstamp”, 1442162072000, “value”, 500.123)
Does this vertex exist?
Write new edge

45. © 2015. All Rights Reserved. 45
Optimizing your writes
gremlin> chunk.addEdge(“hasObservation”, chunk, “tstamp”, 1442162072000, “value”, 500.123)
Does this vertex exist?
Write new edge
• Remove the read from your write
path - storage.batch-loading =
true
• batch your commits, measure
latency and throughput on your
system to find a good commit size

46. © 2015. All Rights Reserved. 46
storage.batch-loading=false

47. © 2015. All Rights Reserved. 47
storage.batch-loading=true

48. © 2015. All Rights Reserved. 48
Quick and dirty write performance numbers
wps
0
22,500
45,000
67,500
90,000
• 9 m3.2xlarge nodes w/ C* 2.2, RF = 3, writing @ quorum, default C* settings
• 1 m3.2xlarge “client” w/ Titan 1.0-SNAPSHOT, 10 write threads writing 100
million points in total across 100,000 series

49. © 2015. All Rights Reserved. 49
In summary
• Understanding of underlying data storage format can help with performance
tuning
• Writes
• remove reads from the write path where possible
• test different batch commit sizes
• when writing vertices you may need to adjust ids.block-size and
ids.renew-percentage
• Reads
• batch communication between Titan and Cassandra with
query.batch=true
• make use of global and vertex centric indices when possible

50. What questions do you have and thanks!
Thanks to the Apache TinkerPop,TitanDB team, my awesome
coworkers, and the folks at DataStax for putting on an excellent
summit!
Ted Wilmes
Data Warehouse Engineer
@trwilmes
tedwilmes@wellaware.us
© 2015. All Rights Reserved. 50

51. Thank you