Effective Hadoop Cluster Management - Impetus Webinar Register at http://bit.ly/vXPfpd

1. Impetus Technologies Inc.
1 © 2014 Impetus Technologies
Effective Hadoop Cluster
Management
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

2. Outline
• HadoopTM - Quick overview
• HadoopTM clusters
• Addressing the HadoopTM cluster challenges
– Operational
– Cluster Setup
– Cluster Management
– Cluster Sharing
• Quick provisioning of Cloud-based HadoopTM clusters
• Available solutions for cluster setup and management
2 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

3. Hadoop Overview
 Hadoop
 Nodes, roles, services
 Configurations & modes
 Hadoop sub-components
Feature /
Cluster Mode
3 © 2014 Impetus Technologies
Production Development & Testing usage
Distributed Pseudo Distributed Standalone
How Services Run Truly Distributed Single Machine - Separate Java processes Single Java Process
HDFS Yes Yes No (Uses local File System)
Replication Yes No No
XML Conf Init required Yes Yes No
Services Start/Stop req. Yes Yes No
Node addition Yes No No
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

4. Hadoop Cluster- The Pain Areas
• Operations
• Cluster setup
• Cluster management
• HadoopTM compatibility
• Cluster sharing
4 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

5. Hadoop Cluster- What is Missing?
• Automation
• Setup support
• User friendly interface
• Cluster management capabilities
– Centralized management for multiple clusters
• Monitoring & notifications support
• Provision for cluster sharing
5 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

6. Hadoop Cluster- Key Challenges
Operations
Challenges
– Manual operations
– Console based interface
Addressing Challenges
– Automated activities
– User friendly GUI as an alternate interface
6 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

7. Hadoop Cluster- Key Challenges
Cluster Setup
Challenges
 Manual execution
 Interactive setup
 Setup procedure varies as per the situation
 Set-up environment - Cloud & local clusters
Addressing the challenges
 Context aware, non-interactive automation
 HadoopTM components initialization
 Click based cluster provisioning
7 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

8. Quick Provisioning of Cloud Clusters
Understanding the difference
Action Cloud Clusters Local Clusters
Instances Launch Required --- N.A. ---
Required additional Info
8 © 2014 Impetus Technologies
Machine image, Instance type,
Security groups, Key
Cluster machine types Homogeneous/ Heterogeneous Homogeneous /Heterogeneous
OS Normally same Could be different
Java Same on all Could be different
Password-less SSH Setup Already Exists Setup Required
Hadoop Components Setup of components nearly remains same in both case
Instances Termination Required, if requested --- N.A. ---
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

9. Quick Provisioning of Cloud Clusters
Challenges
– Manual operation
– Efforts increases with cluster
size
– Interface switching
Addressing the Challenges
• Automated provisioning
– Scripts
– Cloud APIs
9 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

10. Hadoop Cluster- Key Challenges
Cluster Management
Challenges
 Lack of appropriate management support
Addressing the challenges
 Support for managing
 Configuration
 Nodes & services
 Jobs
 Keys & users
 Monitoring & notifications
10 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

11. Hadoop Cluster- Key Challenges
HadoopTM Compatibility & Issues
Challenges
 Version compatibility
 Cluster modes
 Version, vendor, bundle format differences
 Vendor specific features
Addressing the challenges
 Workaround solution for compatibility issues
 Automation for version, vendor, bundle format
 Context aware management support
 Vendor specific tools
11 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

12. Hadoop Cluster- Key Challenges
Cluster Sharing
Challenges
 Ways of sharing
 No sharing / Sharing with limited security
Addressing these challenges
 Sharing through alternate interface
 Purpose specific role based sharing
12 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

13. How HCM Tools can Help?
• Quick turnaround
• Helps to minimize time, cost & efforts
• Integrated support (setup, management & monitoring)
• Easy execution
• Proven usefulness in production & testing
• Adhering to best practices
13 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

14. How HCM Tools can Help?
Setup
Automation for
setup of Cluster
Components local &
cloud environments
14 © 2014 Impetus Technologies
Management
Nodes, Services,
Configuration, Keys,
Users, Jobs, Job
Templates,
Performance
Tracking
Upgrade, Auditing,
Monitoring,
Import/Export,
Event Mail/Alerts/
Notifications, Active
working
Sharing
Alternate access
interface
Role based access
Security preserved
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

15. Some Available Options
• Amazon Elastic Map-Reduce
• Whirr
• Cloudera SCM
• Impetus Ankush
15 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

16. Ankush- Impetus HCM Tool
16 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

17. Ankush: Key Features
• Centralized multiple cluster management
• Common interface for cloud and local clusters
• Integrated support for setup, management & cloud
provisioning
• Anytime-anywhere access to cluster functionalities
• Cloud independent
• Reduces cluster setup time by 60 - 80%
• Bundle setup optimization
17 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

18. Summary
• Automation facilitates smooth HadoopTMworking
– Eases cloud cluster provisioning
• Alternate UI enables secure cluster sharing
• HadoopTM still in evolving stage
• Cluster management tools for pertinent problem areas
18 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

19. 19 © 2014 Impetus Technologies
About Impetus

20. Our Expertise
• Strategic partners for software product engineering and
R&D
• Thought leaders in cutting-edge technologies
• Mature processes and practices that are methodical, yet
flexible
• Diverse domain expertise
20 © 2014 Impetus Technologies
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51

21. 21 © 2014 Impetus Technologies
Q & A

22. 22 © 2014 Impetus Technologies
Thank You
Write to us at inquiry@impetus.com
Follow us on Twitter @impetustech
Recorded version available at
http://www.impetus.com/webinar_registration?event=archived&eid=51