Enterprise Data Science at Scale

1. 1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
From Data Science to Enterprise Data Science @ Scale

2. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Data Scientist Valentine Day Prediction
Spark with HDP
Improve Zeppelin
Data Science Platform

3. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
à #1 Pure Open Source Hadoop Distribution
à 1000+ customers and 2100+ ecosystem
partners
à Employs the original architects, developers
and operators of Hadoop from Yahoo!
à Best-in-class 24x7 customer support
à Leading professional services and training
à #1 Data Science Platform (Source: Gartner)
à OpenPOWER performance leadership
à Flexible, software defined storage
à #1 SQL Engine for complex, analytical workloads
à Leader in On-premise and Hybrid Cloud solutions
+
IBM + Hortonworks = Unlocking Actionable Insights

4. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Data Science For Modern Data Architecture
Ø Make Zeppelin great for Spark
Ø Enable Apps To Consume Predictions and become
smarter
Ø Make Zeppelin great for Spark
Ø Become easier, more accurate & faster to deploy
& manage
Ø Make Zeppelin great for Spark
Ø Bring predictive analytics to the IOT edge
Ø Make Zeppelin great for Spark
Ø Fully support data science life cycle

5. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Data Science Lifecycle

6. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Next Generation Data Science Problems
Multiple data sources & clusters
Data Scientists
Where is the data I need to answer the
business questions?
Data Engineers
How do I move that data into a central
repository?
How do I transform and cleanse that data?

7. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Next Generation Data Science Problems
Too many tools and technologies
Data Scientists
How do I learn the latest library/ technique?
I don’t (want to) know Hadoop/ Hive etc.
How do I bring my familiar R/ Python library
to the new data science platform?

8. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Next Generation Data Science Problems
Socializing insights is challenging
Data Scientists
How do I collaborate and share my work
with others in the organization?
Business Analyst
How do I move that data into a central
repository?
What is the best visualization to tell my
story?

9. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Next Generation Data Science Problems
Going from prototype to production is cumbersome
Data Scientists
I created this awesome Machine Learning
Model, how do I put it into production?
Data Scientists/ Data Engineers
How are my Machine Learning Models
performing & how to improve them?

10. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Enterprise Data Science At Scale
Enterprise
Secured,
governed and
managed
Tools
Leverage your
favorite tools,
technologies
and libraries
Deployment
From pilot to
production
Data
Build models
using all the
data

11. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Data Science Solution
Community Open Source Scale & Enterprise Security
• Find tutorials and datasets
• Connect with Data Scientists
• Ask questions
• Read articles and papers
• Fork and share projects
• Code in Scala/Python/R/SQL
• Zeppelin & Jupyter Notebooks
• RStudio IDE and Shiny
• Apache Spark
• Your favorite libraries
• Data Science at Scale
• Run Spark Jobs on HDP Cluster
• Secure Hadoop Support
• Ranger Atlas Support for Data
• Support for ABAC
Model Management
• Data Shaping Pipeline UI
• Auto-data preparation & modeling
• Advanced Visualizations
• Model management & deployment
• Documented Model APIs
Data Science Experience

12. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
DSX Demo

13. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Use Case
à All industries are effected by churn.
à Being able to predict churn helps
companies take action and keep
customers longer.
à The more historical data, the better
the model
à Data collected and labeled over time
based on churn.
à Using a Random Forest we will
predict future churners.
Customer Churn Architecture

14. © Hortonworks Inc. 2011 – 2017. All Rights Reserved

15. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Demo Scenario
Assessing Customer Churn Probability in Real Time
• Stored long term data on customer churn behavior
• New real time data coming in
• Predict a customers churn probability before they churn
• Alert the proper departments | manager
• Business monitors customer retention outlook & performance

16. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Demo Flow
Insights from Data Science to Production
Data Scientists
Where is the data I
need to answer the
business questions?
Business Users
Where is the insight
& predictions from
the data?
HDP Cluster
Knox
Admins
How do I meet SLA,
Performance, .., Feature
needs?

17. © Hortonworks Inc. 2011 – 2017. All Rights Reserved
Demo Scenario
Problems Solved
• Data Scientist collaborate, learn new tools & frameworks
• Choice of tools, notebooks and languages
• Run favorite notebook on all data in the HDP Cluster
• Deploy the model to production
• Leverage the production model to deliver insights to business
• Monitor models and retrain models as new data comes in

18. 18 © Hortonworks Inc. 2011–2018. All rights reserved.
Hortonworks confidential and proprietary information
Questions?