More Related Content Similar to Introduction to the Hortonworks YARN Ready Program (20) More from Hortonworks (20) Introduction to the Hortonworks YARN Ready Program1. Page1 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Introduction to YARN Ready
John Kreisa – VP Strategic Marketing
Ajay Singh - Director, Partner Solutions
2. Page2 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
YARN Ready Program Overview
John Kreisa
3. Page3 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Why YARN Ready? -> Apache Hadoop YARN
• YARN is the architecture center of Apache Hadoop 2 enabling more
efficient cluster utilization
• More and more customers are asking about tools and applications that
are integrated into YARN to maximize the value of their Hadoop cluster
• YARN Ready is a program offered by Hortonworks the leading
developers of YARN
–YARN was conceived of by Arun Murthy Hortonworks founder
–He submitted JIRA MapReduce 279 in January 2008 and has been working on it since
–Hortonworks has written 80% of the YARN code
Page 3
4. Page4 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
The Two YARN Ready Program Goals
Expand the Hadoop ecosystem by enabling ISVs to build & validate
YARN-based applications, using Apache technologies
Allow enterprises to know which technologies maximize Hadoop
resources in an ecosystem-friendly way
5. Page5 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Two Achievements Available
HDP Certified
YARN Ready
Badge Description
HDP (Certification) Integrates with HDP 2.x
YARN Ready (Validation) Uses the YARN resource scheduler
6. Page6 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
YARN Technical Overview and Value
Ajay Singh
7. Page7 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Hadoop 1.0: Batch
HADOOP 1.0
Built for Web-Scale Batch Apps
Single App
BATCH
HDFS
Single App
INTERACTIVE
Single App
BATCH
HDFS
• All other usage patterns must
leverage that same
infrastructure
• Forces the creation of silos for
managing mixed workloads
Single App
BATCH
HDFS
Single App
ONLINE
8. Page8 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Hadoop 2.0: Multi Workload
HADOOP 1.0
HDFS
(redundant, reliable storage)
MapReduce
(cluster resource management
& data processing)
HDFS2
(redundant, reliable storage)
YARN
(cluster resource management)
MapReduce
(data processing)
Others
(data processing)
HADOOP 2.0
Single Use System
Batch Apps
Multi Purpose Platform
Batch, Interactive, Online, Streaming, …
9. Page9 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Key Driver Of Hadoop Adoption: Enterprise Data Lake
Flexible
Enables other purpose-built data processing
models beyond MapReduce (batch), such
as interactive and streaming
Efficient
Double processing IN Hadoop on the same
hardware while providing predictable
performance & quality of service
Shared
Provides a stable, reliable, secure
foundation and shared operational
services across multiple workloads
Data Processing Engines Run Natively IN Hadoop
BATCH
MapReduce
INTERACTIVE
Tez
STREAMING
Storm
IN-MEMORY
Spark
GRAPH
Giraph
ONLINE
HBase, Accumulo
OTHERS
HDFS: Redundant, Reliable Storage
YARN: Cluster Resource Management
10. Page10 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
YARN Development Framework
YARN : Data Operating System
°1 ° ° ° ° ° ° ° °
° ° ° ° ° ° ° ° °
°
°°
° ° ° ° ° ° °
° ° ° ° ° ° N
HDFS
(Hadoop Distributed File System)
System
Batch
MapReduce
Interactive
Tez
Engine Long Running
Slider
Direct
ISV
Apps
Scripting
Pig
SQL
Hive
Cascading
Java
Scala
NoSQL
HBase
Accumulo
Stream
Storm
API
ISV
Apps
ISV
Aps
Applications
Others
Spark
ISV Apps
ISV
Apps
11. Page11 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Tez – Next Generation Data Processing Framework
YARN ApplicationMaster to run DAG of Tez Tasks
Task with pluggable Input, Processor and Output
Tez Task - <Input, Processor, Output>
Task
ProcessorInput Output
12. Page12 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Slider – Onboarding Long Running Services To YARN
A YARN application to deploy existing distributed applications, monitor
them and make them larger or smaller as desired - even while the
cluster is running
Has a plug-in provider architecture to support different applications
Automatic Deploy, Manage, Monitor via Ambari
– Standard management APIs for Ambari to use
– Standard ways to collect metrics
Clusters can be stopped, "frozen" and restarted, "thawed" later
– Distribution of the deployed application across the YARN cluster is persisted.
13. Page13 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
# of Apps / Ease of Integration
Control
Integration Option Tradeoffs
YARN Native Apps
Tez Apps
Services on YARN (Slider)
MapReduce Apps
Hive
14. Page14 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
HADOOP
YARN
HDFS
SQL
Datanode
HDFS
Visual Data
& Analytics
Workflow
Actian Analytics Platform – YARN Native Integration Example
Transform Hadoop into a High Performance Analytics Platform
Datanode
HDFS
Datanode
HDFS
Datanode
HDFS
X100X100X100
Read
Load
Actian Vector
Blend &
Enrich
Data Science
& Analytics
Datanode
HDFS
X100
HDFS
Vector
• Original file format
• Standard block
replication
• Column-based
blocks
• Binary
• Compressed
• Partitioned
• Faster Loading
• Faster SQL
• Standard SQL
• Better Scaling
Namenode
High Performance,
Industrialized SQL
Database
High Performance,
Data Science &
Analytics
16. Page16 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
YARN Ready Program Details
John Kreisa
17. Page17 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Partner Benefits - Marketing
Recognition
Recognition as Hortonworks Certified Technology Partner on our website along with the YARN Ready
badge
Hortonworks quote for reference in partner marketing assets
Promotion to customers in Hortonworks led events and activities
Joint Marketing & Field Alignment
Invitation to Modern Data Architecture Webinar Series
Hortonworks Sales Briefing and alignment with Field BD
Automatic participation in digital marketing initiatives
LinkedIn and FB campaigns
Twitter and blog activity
18. Page18 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Partner Benefits - Technical
Tech Expertise
YARN implementation guide and sample code
Integration best practice guidance
Access to the architects and builders of YARN
Roadmap Input
YARN, Tez and Stinger roadmap visibility
Input into the direction of YARN and the YARN engines
Access to pre-release builds to speed integration with new features
19. Page19 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Process for Partners
Become a Hortonworks partner
Sign up at http://hortonworks.com/partners/become-a-partner/
Become Certified
Download the certification kit (a .zip archive) from the partner portal
Validate with HDP (Sandbox OK)
Complete the provided templates (samples are included) for
Solution Overview
Reference Architecture
Certification Report
Submit the three documents to technology-certification@hortonworks.com.
SLA: 10 working days for Hortonworks response.
Collaborate with Hortonworks on joint marketing – via ChannelMarketing@Hortonworks.com
20. Page20 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Certification Kit
A .zip file available to partners
Contents:
Program Guide
Solution Overview template (DOC)
Reference Architecture template (PPT)
Certification Report template & sample (DOC)
Sample YARN App code*
Sample Slider provider code*
YARN app test recommendations*
Logos (available after certification)
PR Template (available after certification)
README file (text)
*YARN Ready, Coming – target July
21. Page21 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Calls to Action
Engage us on your YARN initiative
Become HDP Certified and YARN Ready
Develop a joint Sandbox Tutorial
Collaborate on “Art of Possible” Demonstration
Co-Market & Co-Sell
22. Page22 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Questions?
Join us for the next series of webinars on integrating to YARN:
YARN native July 24 @ 9am PT
Slider integration August 7 @ 9am PT
Tez integration August 21 @ 9am PT
Page 22