Aaron Cordova outlines how Accumulo helps provide the essential features of a "Data Lake": a system in which all types of data from all sources can be imported, secured, analyzed, and delivered to decision makers.

1. The Role of Accumulo in
the Data Lake
Koverse. Oct 16, 2014
Aaron Cordova
www.koverse.com

2. The Data Lake
• All your data
• Any source
• Ask questions
• Act quickly

3. Data Lake
Sources
Many
Formats
!
Security
Analytics
"
#
Fast
Results

4. Any Source
• File Systems
• Databases
• Streams
• APIs
• Relational Rows
• CSV, JSON, XML
• Word, PDF, Text
• Avro

6. Naive Approach
• Connect and read
• Copy to HDFS
Source
HDFS

7. Better Approach
• Connect and read
• Parse
• Index
• Profile
Source
Accumulo

8. Profiling

10. Common Data Abstraction

11. Record
• Flat
Field A Field B Field C Field D
abc 123 2014 44,33
• Nested Semi-Structured
• {people: [{name:’bob’},{name:’fred’}]}
• Unstructured
• Text: “the brown fox jumped over …”

12. Accumulo Records
• Each record -> blob (serialized)
• Each record -> set of key-value pairs

13. Analytics -> Portable

15. MapReduce Analytics
Apache HDFS
Apache Accumulo

16. Indexing
• dataset_field_value -> recordID
• very long rows
• need to stream key-value pairs

18. Access Control
• Each data set has a unique label
• Within a data set, records and fields can have
different labels
• Apply labels from an external system at query time

20. Query
• Indexing and Access control support query
• Point queries
• Lucene processing for text
• Ranges for numbers, dates
• Multi-dimensional

21. Query
Record Table Index Table
All Collections
Write
Queries
Read

23. Publish APIs
• Javascript SDK
• quickly develop apps
• no special Hadoop knowledge necessary

24. API
• query
• setup data flows
• control jobs
• administrate

25. Questions ?
@aaroncordova
@koverse
www.koverse.com