Presentation for the OpenStack Trove Day in Cambridge MA on 19 Aug 2014

1. Hadoop on OpenStack with
Sahara
August 19, 2014
Matthew Farrellee (@spinningmatt)
Emerging Technology and Strategy
CTO Office, Red Hat

2. Hadoop is
8/19/14 tesora.com
• Narrow definition - Apache Hadoop - a specific
Apache project originally from Yahoo!, based on
papers from Google
• Broad definition - the ecosystem of projects,
primarily within Apache, that integrate in some
form with Apache Hadoop
• I’m going to use the broad definition

3. Hadoop often looks like
8/19/14 tesora.com
• Multiple, loosely coupled
projects focused on data
storage and processing
• Includes: workload,
resource, system
management; data ingest
& storage; compute
frameworks and domain
languages

4. Hadoop is often used to
8/19/14 tesora.com
• Store data
• ETL data
• Analyze data
• Structured and unstructured

5. Data today
8/19/14 tesora.com
• Structured or unstructured
• >2.5x more unstructured
• Rate of growth for unstructured is 2x structured

6. Data problems
8/19/14 tesora.com
• It’s not just that processing data is expensive
• In hardware costs
• In computational time
• Most of all, in human time
• Data creation outpaces storage capacity

7. Value
Value
Data flows
8/19/14 tesora.com
DatabaseData
DB
Data
Many still look like this... ...but start to look like this...

8. The analysis itself is hard
8/19/14 tesora.com
• Data sources are hard to find, or create
• Data is always dirty and needs cleaning
• Clean data is always approximate
• Figuring out the right question to ask takes
iterations

9. Sahara’s goal
8/19/14 tesora.com
Make managing data processing (e.g. Hadoop)
infrastructure and tools so simple they just get out of
your way

10. Sahara’s history
8/19/14 tesora.com
• Started at the Portland summit (April 2013)
• Joint effort by Red Hat, Mirantis and
Hortonworks
• Originally called Savanna
• Incubated in Icehouse (released April 2014)
• Supported Apache and Hortonworks Hadoop
• Integrated for Juno (release October 2014)

11. Sahara’s use cases
8/19/14 tesora.com
• Cluster
• Start / stop / scale
• Different shapes and sizes
• Repeatable (template mechanism)
• Workload (Elastic Data Processing, a.k.a EDP)
• Job = Analysis code + Data urls
• Queued and run across clusters (ephemeral or
persistent)

12. Sahara’s architecture
8/19/14 tesora.com
Data
Sources
Sahara
Python
Client
RESTAPI
Cluster
Configuration
Manager
Horizon
Keystone
Auth
Data
Access
Layer
Swift
Sahara
Pages
Hadoop
VM
Vendors
Plugins
Hadoop
VM
Hadoop
VM
Hadoop
VM
Resources
Orchestration
Manager
Job
Sources Job
Manager
Heat
Nova
Glance
Cinder
Neutron
Trove DB
Sahara Service

13. Sahara’s vendor plugins
8/19/14 tesora.com
• It’s how users pick different software versions
• It’s how data processing frameworks are
integrated
• e.g. Vanilla (ref. impl. w/ Apache versions),
HDP (via Ambari), Spark (based on Vanilla),
CDH (spec approved), MapR (spec in review),
IDH (being removed)

14. Sahara’s API
8/19/14 tesora.com
• Both REST and Python (of course)
• Accessible from CLI and Horizon

15. Sahara’s basic structures
8/19/14 tesora.com
• Plugins - controller for specific software collections
• Images - in Glance, w/ special plugin specific tags
• Templates
• Two kinds, node group and cluster
• Combine node groups to form a cluster
template
• Clusters - the live clusters

16. Sahara’s EDP structures
8/19/14 tesora.com
• Data sources
• Input and output locations (Swift/HDFS/etc urls)
• Job binaries
• Often JARs or scripts, stored in a data source
• Jobs
• Templates for a job w/ parameters empty
• Job executions
• Instances of templates w/ parameters filled

17. Juno roadmap
8/19/14 tesora.com
https://review.openstack.org/#/q/sahara-specs+AND+status:merged,n,z
https://blueprints.launchpad.net/sahara
• Highlights -
• Dashboard merged into Horizon
• Spark w/ EDP
• CDH plugin
• Storm plugin
• Security group and Swift auth

18. 8/19/14 tesora.com
Demo video: http://youtu.be/vmry_kXqn4c
Questions?