1. Acunu & OCaml:
Experience Report
Tom Wilkie
Founder & VP Engineering
tom@acunu.com
@tom_wilkie

2. What do we do?
1990
Small databases
BTree indexes
BTree File systems
RAID
Old hardware

3. What do we do?
2010
Distributed, shared-nothing databases
Write-optimised indexes Write-optimised indexes
BTree file systems BTree file systems
RAID ... RAID
New hardware New hardware

4. What do we do?
2011
Distributed, shared-nothing databases
Castle Castle
...
New hardware New hardware

6. What does this
have to do with
Functional
Programming?

7. Big
Appl
Java,
Amazon S3 compatible
Open API Erlang,
OCaml
Management
C
...
Deployment
Monitoring
Python, ...
...
... ...
Bash, Acunu Storage Core C
Perl
... ...
Cros
Manag

8. Management Stack

9. Autogeneranted HTML5/JavaScript External Monitoring
OCaml CLI User Interface Tools (Munin etc)
Routerd Another Routerd
enumeration, routing, clustering on a different machine
FSd Cassandrad S3d Miscd Clusterd Statsd AlertsD
Default_ Alert_
Disk Keyspace Bucket Base Host
Report Rule
Column NamedObje
Version Group Report Alert
Family cts
Collection Stat
Cassandra_
Filesystem S3_Node Service Source
Node
Castle Cassandra BigS3

10. FSd Cassandrad S3d Miscd
Disk Keyspace Bucket Base
Column NamedObje
Version
Family cts
Bridges to
Collection
other systems
Cassandra_
Filesystem S3_Node
Node
Castle Cassandra BigS3

11. Miscd Clusterd Statsd AlertsD
Default_ Alert_
Base Host
Report Rule
Clustering
NamedObje Group Report Alert
cts
Failure Detection
Stat
Monitoring
Service Source
Alerting

12. neranted HTML5/JavaScript External M
ml CLI User Interface Tools (Mu
Routerd
enumeration, routing, clustering
drad S3d Miscd Clusterd St
Routing & Aggregation De
ace Bucket Base Host
Re

13. Successes / Failures

14. Prototype
“Filesystem”

15. Aim: Investigate
algorithms for KV
storage
• CoW BTrees • Fractional Cascading
• Mod List BTrees • Stratified DAs
• LSM Trees • Multidimensional keys
• Doubling Arrays • Z curve packing

16. Doubling Array
2 2 9
9

17. Doubling Array
Inserts
11 2 9 2 8 9 11
8 8 11
etc...
Similar to log-structured merge trees (LSM), cache-
oblivious lookahead array (COLA), ...

18. Demo
https://acunu-videos.s3.amazonaws.com/dajs.html

19. 8KB @ 100MB/s, w/ 8ms seek 100 / 5
= 100 IOs/s = 20 updates/s
~ log (2^30)/log 100
= 5 IOs/update
Range Query
Update
(Size Z)
Log Structured O(logB N) O(Z/B)
B-Tree random IOs random IOs
O((log N)/B) O(Z/B)
Doubling Array sequential IOs sequential IOs
~ log (2^30)/100 8KB @ 100MB/s 13k / 0.2
= 0.2 IOs/update = 13k IOs/s = 65k updates/s
B = “block size”, say 8KB at 100 bytes/entry ~= 100 entries

20. Block Index BTree Disk Trace
Time (s)

21. Block Index Doubling Array Disk Trace
Time (secs)

22. Insertion Rate (kvps/s) OCaml Prototype Performance
# inserted kvps

23. The Dark Side...

24. Insert Rate (keys/s) Java Prototype Performance
Time (s)

25. What about
Castle?

26. Castle Performance

28. One more thing...

29. SH OT S*
SN AP
* And clones!

30. I’ll explain how....
“Castle: Re-inventing Storage
For Big Data”
London, 27th September
http://bit.ly/rduBia

31. Questions?
tom@acunu.com
@tom_wilkie
http://www.acunu.com
http://bitbucket.org/acunu
http://github.com/acunu

32. References
[LSM] The Log-Structured Merge-Tree (LSM-Tree)
Patrick O'Neil, Edward Cheng, Dieter Gawlick,
Elizabeth O'Neil Stratified B-trees and versioned dictionaries, - Andy
http://staff.ustc.edu.cn/~jpq/paper/flash/1996-The Twigg, Andrew Byde, Grzegorz Miłoś, Tim Moreton,
%20Log-Structured%20Merge-Tree%20%28LSM- John Wilkes, Tom Wilkie, HotStorage’11
Tree%29.pdf http://www.usenix.org/event/hotstorage11/tech/
final_files/Twigg.pdf
[COLA] Cache-Oblivious Streaming B-trees,
Michael A. Bender et al [RDA] Random duplicate storage strategies for
http://www.cs.sunysb.edu/~bender/newpub/ load balancing in multimedia servers, 2000, Joep
BenderFaFi07.pdf Aerts and Jan Korst and Sebastian Egner
http://www.win.tue.nl/~joep/IPL.ps
[DSST] Making Data Structures Persistent - J. R.
Driscoll, N. Sarnak, D. D. Sleator, R. E. Tarjan, Making Apache, Apache Cassandra, Cassandra, Hadoop, and
Data Structures Persistent, Journal of Computer the eye and elephant logos are trademarks of the
and System Sciences,Vol. 38, No. 1, 1989 Apache Software Foundation.
http://www.cs.cmu.edu/~sleator/papers/making-
data-structures-persistent.pdf