Double Your Hadoop Hardware Performance with SmartSense
•Télécharger en tant que PPTX, PDF•
2 j'aime•3,786 vues
Hortonworks SmartSense provides proactive recommendations that improve cluster performance, security and operations. And since 30% of issues are configuration related, Hortonworks SmartSense makes an immediate impact on Hadoop system performance and availability, in some cases boosting hardware performance by two times. Learn how SmartSense can help you increase the efficiency of your Hadoop hardware, through customized cluster recommendations. View the on-demand webinar: https://hortonworks.com/webinar/boosts-hadoop-hardware-performance-2x-smartsense/
Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (20)
En vedette
En vedette (20)
Similaire à Double Your Hadoop Hardware Performance with SmartSense
Similaire à Double Your Hadoop Hardware Performance with SmartSense (20)
Plus de Hortonworks
Plus de Hortonworks (20)
Dernier
Dernier (20)
Double Your Hadoop Hardware Performance with SmartSense
- 1. 1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Boost Apache Hadoop Hardware Performance 2X with SmartSense Paul Codding Product Management Director
- 2. 2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Hortonworks Connected Data Platforms and Solutions Data Services Hortonworks Solutions Enterprise Data Warehouse Optimization Cyber Security and Threat Management Internet of Things and Streaming Analytics Data Center Hortonworks Data Suite HDFHDP Hortonworks Connection Cloud Hortonworks Data Cloud AWS HDInsight Hortonworks Connection Enablement Subscription SmartSense™ Premier Operational Support Educational Services Professional Services Community Connection
- 3. 3 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Hortonworks Connection Ensures Success of Your Big Data Journey
- 4. 4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved 5 Reasons Why You Need More Than Just Open Source Software The open source community doesn’t ensure everything works together and is certified for the data center and cloud platforms you rely on. Hortonworks does.1 The unprecedented pace of open source innovation is both a benefit and a challenge. Hortonworks can help; it’s what we do.2 Your enterprise needs more than just support for the latest open source versions. Hortonworks supports and maintains the versions you rely on.3 The community doesn’t ensure that consistent security, governance, and operations are built in. Hortonworks takes enterprise needs seriously.4 The community is not responsible for your success with open source technologies and tools. Hortonworks success is built on your success.5
- 5. 5 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Increase Performance Prevent Issues Accelerate Case Resolution Understand Your Cluster
- 6. 6 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Issue: YARN @ capacity, struggling to add more use cases Before SmartSense Could only run 500 jobs concurrently 1100 jobs would be pending waiting for resources at peak hours
- 7. 7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved After Applying only 3 SmartSense Recommendations They can now run 1200 concurrent jobs ...with only 350 waiting jobs at peak hours Issue: YARN @ capacity, struggling to add more use cases Before SmartSense Could only run 500 jobs concurrently 1100 jobs would be pending waiting for resources at peak hours
- 8. 8 © Hortonworks Inc. 2011 – 2017. All Rights Reserved After Applying only 3 SmartSense Recommendations They can now run 1200 concurrent jobs ...with only 350 waiting jobs at peak hours Issue: YARN @ capacity, struggling to add more use cases Before SmartSense Could only run 500 jobs concurrently 1100 jobs would be pending waiting for resources at peak hours With SmartSense = 2X Throughput Improvement
- 9. 9 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Hardware ($$$) Hadoop Performance • Type of CPU & Core Count • Type & Amount of Memory • Type & Number of Disks
- 10. 10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Hardware ($$$) Operating System Hadoop Performance • Type of CPU & Core Count • Type & Amount of Memory • Type & Number of Disks • Kernel Configuration • Disk Mount/Tuning • Network Configuration
- 11. 11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Hardware ($$$) Operating System Hadoop Daemons Hadoop Performance • Type of CPU & Core Count • Type & Amount of Memory • Type & Number of Disks • Kernel Configuration • Disk Mount/Tuning • Network Configuration • YARN/MR/Tez Memory Configuration • HDFS Configuration • ZooKeeper Configuration
- 12. 12 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Hardware ($$$) Operating System Hadoop Daemons Hadoop Performance • Type of CPU & Core Count • Type & Amount of Memory • Type & Number of Disks • Kernel Configuration • Disk Mount/Tuning • Network Configuration • YARN/MR/Tez Memory Configuration • HDFS Configuration • ZooKeeper Configuration
- 13. 13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved What we do A M B A R I O P S S m a r t S e n s e S E R V E R B U N D L E G AT E WAY S m a r t S e n s e A n a l y t i c s S m a r t S e n s e S E RV I C E Collection Diagnostic Information Secure & Send Analyze & Recommend
- 14. 14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Hardware ($$$) Operating System Hadoop Daemons Hadoop Performance • Type of CPU & Core Count • Type & Amount of Memory • Type & Number of Disks • Kernel Configuration • Disk Mount/Tuning • Network Configuration • YARN/MR/Tez Memory Configuration • HDFS Configuration • ZooKeeper Configuration
- 15. 15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved YARN Memory Configuration Containers Unit of allocation for memory and compute Scheduler Configuration Minimum Container Size Maximum Container Size YARN NodeManager Configuration How much memory can be used by YARN on each cluster node YARN Cluster 1 2 3 4 5 6 7 64 GB
- 16. 16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved YARN Memory Configuration Containers Unit of allocation for memory and compute Scheduler Configuration Minimum Container Size: 5GB Maximum Container Size: 35GB YARN NodeManager Configuration How much memory can be used by YARN on each cluster node – 35GB YARN Cluster 1 2 3 4 5 6 7 64 GB
- 17. 17 © Hortonworks Inc. 2011 – 2017. All Rights Reserved YARN Memory Configuration 5 YARN ClusterApplication YARN Scheduler I need 5 2GB containers Min: 5GB Max: 35GB 201 2 3 4 5 6 7
- 18. 18 © Hortonworks Inc. 2011 – 2017. All Rights Reserved YARN Memory Configuration Application YARN Scheduler I need 5 2GB containers Min: 5GB Max: 35GB 5 5 5 5 5 YARN Cluster 5 201 2 3 4 5 6 7
- 19. 19 © Hortonworks Inc. 2011 – 2017. All Rights Reserved YARN Memory Configuration Application YARN Scheduler I need 5 2GB containers Min: 5GB Max: 35GB Application is taking 25GB of resources when it only needs 10GB 5 5 5 5 5 YARN Cluster 5 201 2 3 4 5 6 7
- 20. 20 © Hortonworks Inc. 2011 – 2017. All Rights Reserved YARN Memory Configuration Gatorade: $2.50 Machine only takes Cash EXACT CHANGE REQUIRED!
- 21. 21 © Hortonworks Inc. 2011 – 2017. All Rights Reserved YARN Memory Configuration Gatorade: $2.50 Machine only takes Cash EXACT CHANGE REQUIRED! Minimum Withdrawal: $20 Maximum Withdrawal: $500
- 22. 22 © Hortonworks Inc. 2011 – 2017. All Rights Reserved YARN Memory Configuration Containers Unit of allocation for memory and compute Scheduler Configuration Minimum Container Size: 2GB vs 5GB Maximum Container Size: 10GB vs 35GB YARN NodeManager Configuration How much memory can be used by YARN on each cluster node – 56GB vs 35GB YARN Cluster 1 2 3 4 5 6 7 64 GB
- 23. 23 © Hortonworks Inc. 2011 – 2017. All Rights Reserved After Applying only 3 SmartSense Recommendations They can now run 1200 concurrent jobs ...with only 350 waiting jobs at peak hours Issue: YARN @ capacity, struggling to add more use cases Before SmartSense Could only run 500 jobs concurrently 1100 jobs would be pending waiting for resources at peak hours With SmartSense = 2X Throughput Improvement
- 24. 24 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Increase Performance Prevent Issues Accelerate Case Resolution Understand Your Cluster
- 25. 25 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Support Cases by Type Configuration Environment Education No Response Product Defect Unreproducible Use Case Advice Works as Designed Other SmartSense Today – Prevent Issues
- 26. 26 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Support Cases by Type Configuration Environment Education No Response Product Defect Unreproducible Use Case Advice Works as Designed Other SmartSense Today – Prevent Issues
- 27. 27 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Support Cases by Type Configuration Environment Education No Response Product Defect Unreproducible Use Case Advice Works as Designed Other SmartSense Today – Prevent Issues 30% of support cases are configuration issues— this is where SmartSense adds incredible value
- 28. 28 © Hortonworks Inc. 2011 – 2017. All Rights Reserved SmartSense Today – Prevent Issues SmartSense analyzes Bundles for configuration issues – recommendations are produced and made available for each cluster in the Hortonworks Support Portal Recommendations prevent operational issues, and improve performance and overall cluster throughput.
- 29. 29 © Hortonworks Inc. 2011 – 2017. All Rights Reserved SmartSense Today – Prevent Issues SmartSense analyzes Bundles for configuration issues – recommendations are produced and made available for each cluster in the Hortonworks Support Portal Recommendations prevent operational issues, and improve performance and overall cluster throughput.
- 30. 30 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Increase Performance Prevent Issues Accelerate Case Resolution Understand Your Cluster
- 31. 31 © Hortonworks Inc. 2011 – 2017. All Rights Reserved SmartSense Today – Accelerate Case Resolution SmartSense provides Hadoop Operators with an Ambari Integrated tool to quickly capture diagnostic information for specific services and hosts into a single “Bundle” that’s automatically uploaded to Hortonworks Support. Significantly reduces the back-and-forth nature of troubleshooting issues. A M B A R I O P S H O R TO N WO R K S S U P P O R T S U P P O R T C A S E S m a r t S e n s e S E R V E R B U N D L E G AT E WAY
- 32. 32 © Hortonworks Inc. 2011 – 2017. All Rights Reserved SmartSense Today – Accelerate Case Resolution SmartSense provides Hadoop Operators with an Ambari Integrated tool to quickly capture diagnostic information for specific services and hosts into a single “Bundle” that’s automatically uploaded to Hortonworks Support. Significantly reduces the back-and-forth nature of troubleshooting issues. A M B A R I O P S H O R TO N WO R K S S U P P O R T S U P P O R T C A S E S m a r t S e n s e S E R V E R B U N D L E G AT E WAY
- 33. 33 © Hortonworks Inc. 2011 – 2017. All Rights Reserved SmartSense Today – Accelerate Case Resolution SmartSense provides Hadoop Operators with an Ambari Integrated tool to quickly capture diagnostic information for specific services and hosts into a single “Bundle” that’s automatically uploaded to Hortonworks Support. Significantly reduces the back-and-forth nature of troubleshooting issues. A M B A R I O P S H O R TO N WO R K S S U P P O R T S U P P O R T C A S E S m a r t S e n s e S E R V E R B U N D L E G AT E WAY
- 34. 34 © Hortonworks Inc. 2011 – 2017. All Rights Reserved SmartSense Today – Accelerate Case Resolution SmartSense provides Hadoop Operators with an Ambari Integrated tool to quickly capture diagnostic information for specific services and hosts into a single “Bundle” that’s automatically uploaded to Hortonworks Support. Significantly reduces the back-and-forth nature of troubleshooting issues. A M B A R I O P S H O R TO N WO R K S S U P P O R T S U P P O R T C A S E S m a r t S e n s e S E R V E R B U N D L E G AT E WAY
- 35. 35 © Hortonworks Inc. 2011 – 2017. All Rights Reserved SmartSense Data Capture Architecture L A N D I N G Z O N E S E RV E R G AT E WAY A M B A R I A G E N T A G E N T A G E N TA G E N TA G E N T A G E N T WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E S m a r t S e n s e A n a l y t i c s
- 36. 36 © Hortonworks Inc. 2011 – 2017. All Rights Reserved SmartSense Data Capture Architecture L A N D I N G Z O N E S E RV E R G AT E WAY A M B A R I A G E N T A G E N T A G E N TA G E N TA G E N T A G E N T B U N D L E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E Agent to Server: TLS Bundle: AES 256/RSA 1024 Landing Zone: SOC2 Certified S m a r t S e n s e A n a l y t i c s
- 37. 37 © Hortonworks Inc. 2011 – 2017. All Rights Reserved SmartSense Data Capture Architecture L A N D I N G Z O N E S E RV E R G AT E WAY A M B A R I A G E N T A G E N T A G E N TA G E N TA G E N T A G E N T B U N D L E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E Agent to Server: TLS Bundle: AES 256/RSA 1024 Server to Gateway: TLS Landing Zone: SOC2 Certified S m a r t S e n s e A n a l y t i c s
- 38. 38 © Hortonworks Inc. 2011 – 2017. All Rights Reserved SmartSense Data Capture Architecture L A N D I N G Z O N E S E RV E R G AT E WAY A M B A R I A G E N T A G E N T A G E N TA G E N TA G E N T A G E N T B U N D L E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E WO R K E R N O D E Agent to Server: TLS Bundle: AES 256/RSA 1024 Server to Gateway: TLS Landing Zone: SOC2 Certified Gateway to Landing Zone: HTTPS (TLS 1.2) or SFTP (AES) S m a r t S e n s e A n a l y t i c s
- 39. 39 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Increase Performance Prevent Issues Accelerate Case Resolution Understand Your Cluster
- 40. 40 © Hortonworks Inc. 2011 – 2017. All Rights Reserved “Who’s creating all of these small files in HDFS!?” “What are my top 10 most active users, and longest running jobs?” “How much should I charge users for their cluster resource use?” SmartSense Today – Understand Your Cluster
- 41. 41 © Hortonworks Inc. 2011 – 2017. All Rights Reserved SmartSense Today – Understand Your Cluster Chargeback Reporting
- 42. 42 © Hortonworks Inc. 2011 – 2017. All Rights Reserved SmartSense Today – Understand Your Cluster Chargeback Reporting HDFS Dashboards
- 43. 43 © Hortonworks Inc. 2011 – 2017. All Rights Reserved SmartSense Today – Understand Your Cluster Chargeback Reporting HDFS Dashboards YARN Dashboards
- 44. 44 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Impact of Hortonworks SmartSense 0 200 400 600 800 1000 1200 1400 Without SmartSense With SmartSense Concurrent Jobs B U N D L E 2X Throughput Improvement Address 30% of Issues Configuration Issues Avoid 10% of Sev1 Issues Production Down Single-Bundle Case Resolution 25% of the Time SmartSense Troubleshooting Bundle
- 45. 45 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Questions
- 46. 46 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Hortonworks Connected Data Platforms and Solutions Data Services Hortonworks Solutions Enterprise Data Warehouse Optimization Cyber Security and Threat Management Internet of Things and Streaming Analytics Data Center Hortonworks Data Suite HDFHDP Hortonworks Connection Cloud Hortonworks Data Cloud AWS HDInsight Hortonworks Connection Enablement Subscription SmartSense™ Premier Operational Support Educational Services Professional Services Community Connection
- 47. © DataWorks Summit and Hadoop Summit 2017. All Rights Reserved47 DataWorks Summit 2017 http://dataworkssummit.com
Notes de l'éditeur
- TALK TRACK Hortonworks Powers the Future of Data: data-in-motion, data-at-rest, and Modern Data Applications. [NEXT SLIDE]
- TALK TRACK At the highest level, this is what Hortonworks offers our customers. As I describe the various components, remember that all of these individual pieces interact to make your Data Plane work. The bottom of the diagram shows our unique flexibility with regards to implementation: in the data center or the cloud. Up to the left, we also offer pre-configured solutions that deliver the tech in optimal bundles to meet specific business needs that we see most often: Enterprise Data Warehouse Optimization Cyber Security and Threat Management Internet of Things and Streaming Analytics Of course, if none of these pre-configured solutions is what you need, we do customized solutions. At the center of all of this is Hortonworks Connection, which includes: Subscription Services Hortonworks SmartSense, which uses Hadoop machine learning for proactive recommendations to tune your Hadoop cluster. Premier Operational Support Educational Services Professional Servies, and Hortonworks Community Connection. [NEXT SLIDE]
- Hortonworks enables the people, processes, technology to all work together for your maximum benefit.
- We know the platform, we ensure it works together, and we’ll be there when something goes wrong to fix it for you. We are your voice to the community and we are able to make the broad cross-cutting changes Enterprises require. You don’t get that without Hortonworks Support.
- Calculations are a moving target – what was best practice last year is not this year Awareness of new features
- [CLOSE]
- TALK TRACK At the highest level, this is what Hortonworks offers our customers. As I describe the various components, remember that all of these individual pieces interact to make your Data Plane work. The bottom of the diagram shows our unique flexibility with regards to implementation: in the data center or the cloud. Up to the left, we also offer pre-configured solutions that deliver the tech in optimal bundles to meet specific business needs that we see most often: Enterprise Data Warehouse Optimization Cyber Security and Threat Management Internet of Things and Streaming Analytics Of course, if none of these pre-configured solutions is what you need, we do customized solutions. At the center of all of this is Hortonworks Connection, which includes: Subscription Services Hortonworks SmartSense, which uses Hadoop machine learning for proactive recommendations to tune your Hadoop cluster. Premier Operational Support Educational Services Professional Servies, and Hortonworks Community Connection. [NEXT SLIDE]