Recommandé
Recommandé
Contenu connexe
Tendances
Tendances (19)
En vedette
En vedette (20)
Similaire à Cloudera User Group Chicago - Cloudera Manager: APIs & Extensibility
Similaire à Cloudera User Group Chicago - Cloudera Manager: APIs & Extensibility (20)
Dernier
Dernier (20)
Cloudera User Group Chicago - Cloudera Manager: APIs & Extensibility
- 1. Cloudera Manager – API’s & Extensibility Ma# Harris – Systems Engineer December 2013 1 CONFIDENTIAL -‐ RESTRICTED
- 2. Cloudera Manager End-‐to-‐End AdministraGon for CDH 1 Monitor 2 Diagnose 3 Integrate 4 Manage Easily deploy, configure & opGmize clusters Maintain a central view of all acGvity Easily idenGfy and resolve issues Use Cloudera Manager with exisGng tools 2 ©2013 Cloudera, Inc. All Rights Reserved.
- 3. IntegraGng with your IT Mgmt tools Datacenter Opera*ons Various op*ons of integra*ng Cloudera Manager into your exis*ng Installa;on, Datacenter Opera*ons/Tools Monitoring Deployment Aler;ng Tools tools Tools e.g. Orion, • Cloudera Manager API e.g. Chef, e.g Nagios, Tivoli, BMC Puppet etc. SNMP etc. • Introduced in CM4 (June 2etc. 012) • Installa*on & deployment • Monitoring • SNMP Alerts • Introduced in CM4.5 (Feb 2013) • Hadoop Opera*ons And more… Cloudera • Monitoring ‘tsquery’ (Feb 2013) Manager • User-‐defined triggers/alarms (new for C5!) • Service extensibility (new for C5!) 3 ©2013 Cloudera, Inc. All Rights Reserved.
- 4. Cloudera Manager (CM) API • • 4 API access was a new feature introduced in Cloudera Manager 4.0, providing programmaGc access to cluster operaGons (such as configuraGon and restart) and monitoring informaGon (such as health and metrics). The CM API is an HTTP REST API, using JSON serializaGon. The API is served on the same host and port as the CM web UI, and does not require an extra process or extra configuraGon. API users have the same privileges as they do in the web UI world. • Docs & Examples h#p://cloudera.github.io/cm_api/ h#ps://github.com/cloudera/cm_api • Java/Python clients h#p://blog.cloudera.com/blog/2013/05/how-‐to-‐ automate-‐your-‐hadoop-‐cluster-‐from-‐java/ ©2013Cloudera, Inc. All Rights Reserved.
- 5. Examples of integraGon with CM API • Installa;on & Deployment • • • Chef Puppet Dell Crowbar • • h#p://blog.cloudera.com/blog/2013/08/how-‐to-‐deploy-‐hadoop-‐clusters-‐automaGcally-‐with-‐dell-‐crowbar-‐and-‐cloudera-‐manager/ StackIQ • h#p://web.stackiq.com/blog/bid/312064/StackIQ-‐Cluster-‐Manager-‐now-‐integrated-‐with-‐Cloudera • • • WANdisco – non-‐stop NN setup Several other customers/partners leveraging the API’s as part of their install & deployment process Monitoring & Aler;ng • • Oracle Enterprise Manager (via Big Data Appliance) Nagios • • • h#ps://github.com/cloudera/cm_api/tree/master/nagios h#ps://github.com/harisekhon/nagios-‐plugins/blob/master/ check_hadoop_cloudera_manager_metrics.pl SNMP alerts integraGon with IBM Netcool Develop & Contribute your plug-‐in’s using Cloudera Manager API 5 ©2013 Cloudera, Inc. All Rights Reserved.
- 6. Cloudera Manager – Monitoring via ‘tsquery’ • Introduced as part of CM4.5 release (Feb 2013) • Great way to add interesGng charts (above & beyond what is provided by default) and monitor metrics that are relevant to your clusters • • The tsquery language s used to Manager Gme-‐series diata store specify statements for retrieving Gme-‐series data from the Cloudera Example: How do I compare all disk IO for all the DataNodes that belong to a specific HDFS service? select bytes_read, bytes_wriZen where roleType=DATANODE and serviceName=hdfs1 • Retrieved Gme-‐series data can be plo#ed via various opGons – line, bar, sca#er, heat maps, table list etc. • Extending this concept to create user-‐defined triggers/alarms (new for C5!). More details • h#p://www.cloudera.com/content/cloudera-‐content/cloudera-‐docs/CM5/latest/Cloudera-‐ Manager-‐DiagnosGcs-‐Guide/cm5dg_chart_Gme_series_data.html • 6 ©2013 Cloudera, Inc. All Rights Reserved.
- 7. Examples of Cloudera Manager ‘tsquery’ Example1: How do I track the aggregate Cluster Disk IO? select dt0(read_bytes_disk_sum), dt0(write_bytes_disk_sum) where category = CLUSTER and clusterId = $CLUSTERID Example2: How do I compare CPU usage across hosts? select dt0(total_cpu_user) / getHostFact(numCores, 1) * 100, dt0(total_cpu_system) / getHostFact(numCores, 1) * 100, dt0(total_cpu_nice) / getHostFact(numCores, 1) * 100, dt0(total_cpu_iowait) / getHostFact(numCores, 1) * 100, dt0(total_cpu_irq) / getHostFact(numCores, 1) * 100, dt0(total_cpu_so`_irq) / getHostFact(numCores, 1) * 100 Create & Contribute your ‘tsqueries’! h#ps://github.com/cloudera/cm_charGng_scrapbook 7 ©2013 Cloudera, Inc. All Rights Reserved.
- 8. Cloudera Manager – Service Extensibility • Introduced in C5 • SGll in Beta! • Some aspects (espcially Parcel mgmt) available in CM4.x • Example: CollaboraGon with Syncsort to deploy DMX-‐h libraries • Single management console for CDH, non-‐CDH services and ISV applicaGons • Similar look and feel as exisGng services • Easy to write (Java-‐free!) • Flexible • Independent release cycle ©2013Cloudera, Inc. All Rights Reserved.
- 9. Analogy from OperaGng Systems (OS) world ISV’s view of OS Systems Management Package Mgmt Process/ Resource Mgmt Security Mgmt Core OS kernel 9 ©2013Cloudera, Inc. All Rights Reserved. Data Access Mgmt
- 10. Bringing ISV Apps to CDH ISV’s view of Hadoop Cloudera Manager Parcels Resource Mgmt Security Mgmt CDK API’s Core Hadoop/CDH kernel 10 ©2013Cloudera, Inc. All Rights Reserved.
- 11. IntegraGng into the Cloudera Product Porpolio Features Examples -‐ Ability to easily package and distribute binaries/ jars via “Parcels” -‐ InformaGca -‐ Syncsort Resource Mgmt -‐ Ability to deploy applicaGons as stand-‐alone processes or via YARN* on the Hadoop grid -‐ Resource isolaGon of cluster resources -‐ SAS -‐ 0xData -‐ Accumulo Security Mgmt Cloudera Manager Descrip;on Package Mgmt ISV’s -‐ Support for Kerberos Mgmt -‐ Role bases access control for Tables/Views in Hive/Impala via Sentry Data Access Mgmt -‐ HDFS and HBase API abstracGon and simplificaGon Systems Mgmt Manage Monitor Non-‐CDH Apps… Accumulo, Spark, Giraph etc. -‐ Deploy and upgrade (rolling) services and pkgs -‐ Manage configuraGons -‐ ProacGve health checks -‐ Track resource uGlizaGon -‐ Custom metrics charts Diagnose -‐ Distributed log collecGon and searching -‐ Tag and track key events Integrate -‐ Access operaGonal tools via API -‐ Surface overall cluster metrics to ISV dashboard * Support for YARN planned as part of CM5.x in FY14 11 ©2013Cloudera, Inc. All Rights Reserved.
- 12. So.. How does it work? • A JSON file that describes your service • Set of control scripts • Packaged as a JAR file • As promised, Java-‐free ©2013Cloudera, Inc. All Rights Reserved.
- 13. Example: Cloudera Manager Extensions -‐ Spark ©2013Cloudera, Inc. All Rights Reserved.
- 14. Cloudera Manager Extensions ©2013Cloudera, Inc. All Rights Reserved.
- 15. Cloudera Manager Extensions: Spark ©2013Cloudera, Inc. All Rights Reserved.
- 16. Cloudera Manager Extensions: Spark ©2013Cloudera, Inc. All Rights Reserved.
- 17. Cloudera Manager Extensions: Spark ©2013Cloudera, Inc. All Rights Reserved.
- 18. The Code #!/bin/bash name : “spark”, roles : [{ name : "master", CMD=$1 MASTER_PORT=<read in from ./params.proper;es> startRunner : { program : "scripts/control.sh", args : [ "start_master", case $CMD in (start_master) exec $SPARK_HOME/scripts/spark-‐start.sh master" "./params.proper;es"] }, ;; parameters : [{ (*) echo "$;mestamp Don't understand [$CMD]" name : "master_port", type : "port", ;; esac default : 7077 }], configWriter : { generators : [{ filename : "params.proper;es" }] }] ©2013Cloudera, Inc. All Rights Reserved.
- 19. Next Steps • DocumentaGon & SDK as part of C5 Beta2 or later (definitely before GA!) • Working with select ISV’s (SAS, Syncsort, 0xData etc.) as part of Beta to further fine-‐ tune this feature Develop & Contribute your Cloudera Manager service extensibility plug-‐in’s ! ©2013Cloudera, Inc. All Rights Reserved.
- 20. Horizontal Extension Security ISV’s 0xData SAS Syncsort Informatica Revolution API Ops Apps Capacity Mgr Service Extensibility Vertical Extension Vision of CM Extensibility SLA Mgr Cost Optimizer CDH CM SNMP API Oracle OEM 20 Nagios Dell Chef/ Puppet ©2012Cloudera, Inc. All Rights Reserved. Accumulo Spark Giraph
- 21. Q&A ©2013Cloudera, Inc. All Rights Reserved.