Apache Fluo (incubating) is an open source implementation of Percolator (which populates Google's search index) for Apache Accumulo. Fluo makes it possible to update the results of a large-scale computation, index, or analytic as new data is discovered. While working on developing Fluo, we made it a priority to attempt to develop applications for Fluo in tandem. Developing these applications resulted in many improvements to Fluo and we also learned a lot about writing Fluo applications. These lesson learned drove the implementation of the Fluo Recipes project. This talk will go over some of the lessons learned and Fluo Recipes. Hopefully this information will save time for anyone attempting to write a Fluo application. Some of the areas that will be covered are: • Organizing data for optimal performance • Achieving high cluster utilization • Exporting data from Fluo to query systems • Avoiding performance degradation over time — Speaker —  Keith Turner Software Engineer, Peterson Technologies Keith Turner has been working with big data since 2004. Keith started working on Accumulo in 2008 and Fluo in 2013. Keith has an MS in Computer Science from Purdue and a BS in Computer Science from the University of Louisiana at Lafayette. — More Information — For more information see http://www.accumulosummit.com/

1. Tips for writing Apache Fluo
Applications
Keith Turner
Peterson Technologies

2. Percolator : Google’s Use Case
● Terabytes of new data coming in each day
● To build index: join new data with petabytes of existing data.
● Joining new data with existing data via Map Reduce took
multiple days.
● Using Percolator, index update time dropped from days to
minutes.

3. Fluo Features
● Layer on top of Accumulo
● Snapshot Isolation : only see committed data.
● Cross Row Transactions
○ Read/write data from multiple nodes
○ Fail if two transactions modify same cell : collision
○ Correct in case of faults on multiple nodes
● Observers
○ User code, executes a transaction
○ Triggered by persistent notifications.
○ Observers can trigger other observers
○ Runs in parallel on many nodes

4. Fluo 101 - Architecture
Accumulo
HDFS
Zookeeper
YARN
Client Cluster
Fluo Client
for App 1
Fluo Client
for App 1
Fluo Client
for App 2
Fluo Application 2Fluo Application 1
Fluo Worker
Observer1 Observer2
Fluo Oracle
Fluo Worker
ObserverA
Fluo Oracle
Fluo Worker
Observer1 Observer2
Table1 Table2

5. Fluo Recipes Overview
● Separate project from Fluo
● Can have different release cadence and support older versions of Fluo
● Provides higher level abstractions that build on Fluo primitives
○ Collision Free Map : recipe for many-to-many updates
○ Export Queue : recipe for updating external systems in a fault tolerant manner
○ Recording TX : recipe for recording changes made by a TX
○ Each recipe is documented and tested
● Common code
○ Transient ranges
○ Table optimizations

6. Observers incrementally transform datasets
Input Dataset 1
Input Dataset 2
Derived Dataset
A
Derived Dataset
B
Observer Y
Observer X
Fluo
Client
Fluo
Client
External
System
Observer Z

7. Uneven data -> low utilization
Tablet
Server 1
Tablet
Server 2
Tablet
Server 3
Tablet
Server 4
Tablet
Server 5
Tablet
Server 6
Input dataset 1Input dataset 2
Derived dataset A
Derived dataset B

8. Balance is important

9. Evenly spread data -> maybe high utilization
Tablet
Server 1
Tablet
Server 2
Tablet
Server 3
Tablet
Server 4
Tablet
Server 5
Tablet
Server 6
Input dataset 1
Input dataset 2
Derived dataset A
Derived dataset B

10. Many instances of observers running
Tablet
Server 1
Tablet
Server 2
Tablet
Server 3
Tablet
Server 4
Tablet
Server 5
Tablet
Server 6
Input dataset 1
Input dataset 2
Derived dataset A
Derived dataset B
X X X X X X X X X X X X X X X X X
Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y Y

11. Balancing details
● Fluo Recipes provides an API to specify table ranges to
balance
● Other recipes use this balancing API
● Implementation uses Accumulo RegexGroupBalancer
introduced in 1.7.0

12. May still have hotspots
● Popular keys may cluster. Add short hash to prefix.
● <prefix>:<hash of key>:<key>
● p:a8j42:com.example/popular-link
● Fluo Recipes can help hash and evenly spread data.
○ Generates tablet splits for hashed prefix
○ Generates balancing config for hashed prefix
○ Util code to add/remove hash+prefix

13. Incrementing link counts
//When a document changes, determine changes in its links
foreach(url : newOutlinks)
C = tx.get(url, ‘inLinks’)
tx.set(url, ‘inLinks’, C + 1)
foreach(url : removedOutlinks)
C = tx.get(url, ‘inLinks’)
tx.set(url, ‘inLinks’, C - 1)

14. Collisions : One TX fails
Observer 1
Processing
Doc2
URL 1
URL 2
URL 3
Transactions attempting to
update URL in link count.
Do this by reading current
count, adding or subtracting
1, writing back.
Observer 1
Processing
Doc1

15. Queue updates for url
//When a document changes, determine changes in its links
foreach(url : newOutlinks)
tx.set(url, ‘queue:’+uuid, +1)
foreach(url : removedOutlinks)
tx.set(url, ‘queue:’+uuid, -1)

16. Use queues + multiple txs to avoid collisions
Observer 1
Processing
Doc2
URL 1
Update
Queue
URL 2
Update
Queue
URL 3
Update
Queue
Observer 1
Processing
Doc1
Observer 2
Processing
Queue
Observer 2
Processing
Queue
Observer 2
Processing
Queue
URL 1
URL 2
URL 3

17. Updates leave a trail
An Accumulo’s Tablet’s files
K1.1=45
K2.1=22
K3.1=99
.
.
A ton of
other key
values

18. Flush a file with some updates
An Accumulo’s Tablet’s files
K1.1=45
K2.1=22
K3.1=99
.
.
A ton of
other key
values
K1U.1=+1
K1U.2=-2
K2U.1=+5
K2U.2=-7

19. Flush a file with some deletes
An Accumulo’s Tablet’s files
K1.1=45
K2.1=22
K3.1=99
.
.
A ton of
other key
values
K1U.1=+1
K1U.2=-2
K2U.1=+5
K2U.2=-7
K1.2=44
K2.2=20
K1U.1
K1U.2
K2U.1
K2U.2

20. Compact smaller files into one
An Accumulo’s Tablet’s files
K1.1=45
K2.1=22
K3.1=99
.
.
A ton of
other key
values
K1.2=44
K2.2=20
K1U.1
K1U.2
K2U.1
K2U.2

21. Compacting all files drops delete markers
An Accumulo’s Tablet’s files
K1.2=44
K2.2=20
K3.1=99
.
.
A ton of
other key
values

22. Compactions will clean it up
● Too expensive to compact all data
● Accumulation of delete markers will cause slow down
● Accumulo supports compactions of ranges
● Organize data so that updates are in separate ranges.
● <update prefix>:<hash of key>:<key>
● <data prefix>:<hash of key>:<key>
● Use fluo scan raw command to see trail of data left by
transactions

23. Fluo Table
Derived
dataset A
Updates
for
A
Derived
dataset B
Updates
For
B
Input
dataset 2
Input
dataset 1
Can compact
just these
ranges.

24. Evenly spread update data
Tablet
Server 1
Tablet
Server 2
Tablet
Server 3
Tablet
Server 4
Tablet
Server 5
Tablet
Server 6
Input dataset 1
Input dataset 2
Derived dataset A Updates
Derived dataset A
Derived dataset B Updates
Derived dataset B

25. Transient data
● Fluo Recipes provides transient data registry
○ Used by other recipes (export Q and CFM)
● Fluo Recipes has a utility to compact transient ranges
○ Since data is evenly spread, compactions are also
○ Utility periodically compacts

26. Collision Free Map Recipe
● Recipe that implements queued update pattern
● Like map reduce, but continuous
● Two transactions
○ One queues updates to one or more values.
○ One processes queued updates.
● User provides two functions
○ Combiner
○ Update Observer

27. Exporting Data
● Export data from Fluo to external system
○ Useful for continuously updating an external user facing query system
● Assume external system needs to see every change

28. Exporting Data from Transaction
oldV = tx1.get(K1,”old”)
newV = tx1.get(K1,”new”)
exportDiffsExternalSystem(K1,oldV,newV)
//update old and new
tx1.commit()
Committ
ed
Key oldV newV
True K1 9 13
False K1 13 17
True K1 13 21
True K1 21 37
Assume external system deletes old and inserts new. Oh no
17 is inserted and never deleted.

29. Export Queue Recipe
● Only export committed data.
● Transactions add key/values to export queue
○ Gives each export entry a sequence number.
○ Only makes it in queue if tx succeeds
● User provides an idempotent export function.
○ Passed list of (key,seq,value)
● Receiving system must handle out of order and redundant
data.
○ Sequence can help
○ Use sequence number for timestamp when exporting to Accumulo
● Automatic balancing config, transient range config, and table
splits.

30. Export Queue
Transaction tx1
oldV = tx1.get(K1,”old”)
newV = tx1.get(K1,”new”)
queueExport(K1,oldV,newV)
tx1.commit()
Key Sequen
ce
oldV newV
K1 6 9 13
K1 13 13 21
K1 45 21 37

31. Export Queue Observer
● Transactions add to export queue and
notify observer
● An observer will export data on queue
○ May export same data multiple times
Ke
y
Sequen
ce
old
V
new
V
K1 6 9 13
K1 17 13 21
K1 45 21 37
Example of data being exported twice
● TX1 : Export K1,6,9,13
● TX1 : Export K1,17,13,21
● TX1 : Fault
● TX2 : Export K1,6,9,13
● TX2 : Export K1,17,13,21
● TX2 : Export K1,45,21,37
● TX2 : delete queued exports
● TX2 : commit

32. Evenly spread export data
Tablet
Server 1
Tablet
Server 2
Tablet
Server 3
Tablet
Server 4
Tablet
Server 5
Tablet
Server 6
Input dataset 1
Input dataset 2
Derived dataset A Updates
Derived dataset A
Derived dataset B Updates
Derived dataset B
Export Queue

33. Invert on export
● Compute information to index in Fluo
● Invert information when exporting
● Assume want to build a system that can answer following.
○ For a domain, which page has the most inbound links?
○ For a page, how many inbound links does it have?
○ For all pages, which page has the most inbound links?
● Index number of inbound links three ways.
○ Do not need to build 3 indexes in fluo.
○ Compute # of inbound links in fluo and export.
○ On export, update 3 indexes.
● Fluo orchestrates reliably and incrementally updating
external index.

34. Invert on export example
● URI1 incount changes from
90 to 115
● Queues URI1:90:115 for
export
● Makes 5 updates for one
export queue entry
Export actions
Action Row
Insert t:(99999-115) <uri>
Insert d:<domain>:(99999-115
) <uri>
Insert p:<uri> incount 115
Delete t:(99999-90) <uri>
Delete d:<domain>:(99999-90)
<uri>

35. Webindex video

36. Same row different columns
Tablet A
Row 1
Row 2
Tablet B
Row 3
Row 4
TX1
Fluo Client L
TX2
Fluo Client N
TX3
Fluo Client P
TX4
Fluo Client R
Col U
Col V
Col W
Col X

37. Row Locking
● Fluo uses Accumulo Conditional Mutations
● Tserver locks entire row to check conditions
● Concurrent transactions against a single row can slow each
other, even if they do not collide.
Example Schemas
Row Column Row Lock
Contention
<node1> edge:<node2> Likely
<node1>/<node2> edge Unlikely

38. How much work should an observer do?
● Reasons to do less work :
○ Has to fit into memory.
○ Should have a low probability of collisions.
● Reasons to do more work :
○ Minimizing total # of seeks
○ Avoiding recomputation

39. Fluo Releases
● Fluo 1.0.0 Release vote passed on October 4th
● Fluo Recipes 1.0.0 will be released soon
● Did well on 3 day cluster test
● Will follow semver

40. Fluo Tour
● Self guided set of exercises on website
● Considering stepping through it at hackathon tonight

41. Moved to Apache
https://fluo.apache.org