Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
Similaire à Data Science at Scale with Hortonworks
Similaire à Data Science at Scale with Hortonworks (20)
Dernier
Dernier (20)
Data Science at Scale with Hortonworks
- 1. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Future of Data Boston Data & Cognitive Developers Enterprise Data Science at Scale
- 2. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Agenda • Networking, Food and drink • Announcements • Main Presentation – Introducing Data Science at Scale – Building and Deploying Models Collaboratively – Training Models with all the Data – Putting Models to Work in a Streaming Application • Question and Answer • Networking and Wrap up
- 3. © Hortonworks Inc. 2011 – 2017. All Rights Reserved #1 Pure Open Source Hadoop Distribution 1000+ customers and 2100+ ecosystem partners Employs the original architects, developers and operators of Hadoop from Yahoo! Best-in-class 24x7 customer support Leading professional services and training #1 Data Science Platform (Source: Gartner) OpenPOWER performance leadership Flexible, software defined storage #1 SQL Engine for complex, analytical workloads Leader in On-premise and Hybrid Cloud solutions + Thanks to our Meetup Partners IBM + Hortonworks
- 4. © Hortonworks Inc. 2011 – 2017. All Rights Reserved About Carolyn Duby • Big Data Solutions Architect • High performance data intensive systems • Data science and Cyber Security SME • ScB ScM Computer Science, Brown University • LinkedIn: https://www.linkedin.com/in/carolynduby/ • Twitter: @carolynduby Github: carolynduby • Hortonworks – Innovation through data – Enterprise ready, 100% open source, modern data platforms – Engineering, Technical Support, Professional Services, Training
- 5. © Hortonworks Inc. 2011 – 2017. All Rights Reserved About Rich Tarro • Analytics Solutions Architect • IBM Corporation • Client insights through data • MS Electrical Engineering – Rensselaer Polytechnic Institute
- 6. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Data Science Lifecycle
- 7. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Next Generation Data Science Problems Multiple data sources & clusters Data Scientists Where is the data I need to answer the business questions? Data Engineers How do I move that data into a central repository? How do I transform and cleanse that data?
- 8. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Next Generation Data Science Problems Too many tools and technologies Data Scientists How do I learn the latest library/ technique? I don’t (want to) know Hadoop/ Hive etc. How do I bring my familiar R/ Python library to the new data science platform?
- 9. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Next Generation Data Science Problems Socializing insights is challenging Data Scientists How do I collaborate and share my work with others in the organization? Business Analyst How do I move that data into a central repository? What is the best visualization to tell my story?
- 10. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Next Generation Data Science Problems Going from prototype to production is cumbersome Data Scientists I created this awesome Machine Learning Model, how do I put it into production? Data Scientists/ Data Engineers How are my Machine Learning Models performing & how to improve them?
- 11. © Hortonworks Inc. 2011 – 2017. All Rights Reserved More data = better model but desktop is limited • Analyzing and training with portion of available data • Analysis or training too slow • Out of memory • Data accumulates over time
- 12. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Data Science Solution • Data Movement and Acquisition – Acquire and move data required for problem • Distributed Compute Platform – Store, clean, and organize historical data – Build and train models on historical data • Notebooks – Record data processes • Clean and prepare data • Build and train models – Collaborate with others • Model Deployment – Package model for use – Monitor performance
- 13. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Data Movement and Acquisition Constrained High-latency Localized context Hybrid – cloud/on-premises Low-latency Global context SOURCES REGIONAL INFRASTRUCTURE CORE INFRASTRUCTURE Data Lake
- 14. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Apache SPARK • Distributed processing efficiently crunches large data sets – Optimized – Horizontally scalable with multi tenancy – Fault tolerant • One platform for streaming, cleaning, analyzing • Elegant APIs – Scala, Python, Java, R • Many data source connectors – file system, HDFS, Hive, Phoenix, S3, etc
- 15. © Hortonworks Inc. 2011 – 2017. All Rights Reserved SPARK Libraries • Same API for all data sources • SQL - http://spark.apache.org/sql/ – Access structured data and combine with other sources • MLLIB - http://spark.apache.org/mllib/ – Machine learning for training models and predicting • GraphX - http://spark.apache.org/graphx/ – Connectivity algorithms • Streaming - http://spark.apache.org/streaming/ – Complex event processing and data ingest
- 16. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Spark Architecture Spark Driver Spark Application Master YARN container Spark Executor YARN container Task Task Spark Executor YARN container Task Task Spark Executor YARN container Task Task LivyNotebook
- 17. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Open Source Notebooks - Jupyter
- 18. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Open Source Notebooks - Zeppelin
- 19. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Sharing with DSX Projects
- 20. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Deploying a Model Virtual Model Deployment Physical Model Rest Service Physical Model Rest Service Physical Model Rest Service Model Algorithm and Parameters
- 21. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Big Picture Build Model Model 1 Model 2 Model 3 Model Deployment History MoveData Time Train Predict Streaming Application Evaluate Performance
- 22. © Hortonworks Inc. 2011 – 2017. All Rights Reserved DEMO
- 23. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Demo Scenario Customer Churn • Churn occurs when a customer stops subscribing or using a service, which affects all industries. • A company’s ability to predict customer churns allows them the opportunity to be proactive in efforts to retain them. • Historical customer churn data will be used to train the Machine Learning model, and it will be used to predict whether customer will stop using their services. • Random Forest Classifier will be used in this demo, which will well suited to handle variance in the training data set.
- 24. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Demo Flow Insights from Data Science to Production Data Scientists Where is the data I need to answer the business questions? Business Users Where is the insight & predictions from the data? HDP Cluster Knox Admins How do I meet SLA, Performance, .., Feature needs?
- 25. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Demo Scenario Problems Solved • Data Scientist collaborate, learn new tools & frameworks • Choice of tools, notebooks and languages • Run favorite notebook on all data in the HDP Cluster • Deploy the model to production • Leverage the production model to deliver insights to business • Monitor models and retrain models as new data comes in
- 26. © Hortonworks Inc. 2011 – 2017. All Rights Reserved Thank You
Notes de l'éditeur
- Hortonworks & IBM – Integration of HDP and DSX Who Are We IBM - #1 Data Science solution. Hortonworks – Largest Open Source Hadoop distribution. We believe this partnership optimizes the strengths our companies and uniquely positions our solution in the Data Science market. What Are We Talking About Today Integrating HDP and DSX creates a platform for organizations to unlock the potential of their data. Ultimately, it creates a pathway to innovative and valuable Data Science work flows. Presentation Overview 1. Walk through the Data Science Life Cycle. 2. Discuss challenges in the process. 3. Discuss how DSX & HDP solves these problems. 4. Demonstration of the technology.
- Problem Definition A successful Data Science practices begin with a well defined business problem. Ideally, the business has specific questions to ask their data. ETL – Feature Extraction Once the the problem has been defined, the process of data wrangling, transformation, and cleaning must be completed using various ETL processes. Once the data corpus has been curated, statistical analysis techniques are utilized to determine which features should be extracted. Learning After the features are selected, supervised or unsupervised Machine Learning models are be created for future prediction or classification. Model Deployment & Management In order for this process to be valuable, the organization must deploy these models into their production environment. Additionally, they most also monitor the performance and health of these models while they are operating.
- Data Science team consist of Data Scientist, Data Engineers, Business Analyst, and Application Developers. Challenge #1 – Multiple sources of data. Traditional – Structured (DB, EDW, CRM, Ect), Big Data – Unstructured (Social media, IOT), Hadoop based data stores Legacy – Spreadsheets Problem 1 – 20% of the time in the Data Science Lifecycle (DSLC) is spent on Data Scientists trying to find where the required data is located. Problem 2 – 60% of the the time in the DSLC is spent on Data Engineers centrally located the data, and preparing it to ensure data quality. *Key Take Away - Combined 80% of the time in the DSLC is spent on locating, moving, and preparing the data before machine learning models can be created or deployed.
- Challenge #2 – Data Science workflows lack standardization. Problem 1 - the open source community has created too many tools for expect a single person to know them all. Problem 2 – Data Science teams are often limited to using the tools that their data scientist and application developer are most familiar with including languages, and libraries. Problem 3 – There are no systems in place where an organization’s Data Science team can build reliable, standardized, and repeatable pipelines for managing models at a Big Data scale. *Key Take Away – Due to the number of open source tools, there are no standardizations in Data Science practices.
- Challenge #3 – Collaboration is difficult Problem 1 – Without a common framework, Data Scientists have difficultly collaborating with team members. They are unable to share code, results, or models with each other. Problem 2 - Due to lack of collaboration, Business Analyst struggle to find visualization tools that can integrate with the data repository. Key Take Away – Collaboration for a Data Science team is difficult leveraging existing tools.
- Challenge #4 – Deploying Models into Production Problem 1 – Data Science teams struggle migrating their prototype models to deployment in their production environment. Problem 2 – Using current tools, it is also difficult to monitor the health and performance on their models while they are operating in production. Key Take Away – Creating models in an isolated environment is relatively straight forward. The challenge begins when these models need to be deployed and monitored in production.
- Components of the end-to-end real-time insights dataflow platform MiNiFi : Edge Data Collection w/Provenance and centralized C&C NiFi: End to end dataflow management w/Provenance and Interactive C&C Kafka: High throughput durable replayable messaging Storm: High-scale Data Processing Right-sized solutions All optimized for delivery into HDP (HDFS, Hive, Spark, Hbase, etc…)