Monitor your production workloads with Azure Log Analytics

3. • EdgenodeSSHWatchdog
• GatewayTCPPingWatchdog
• SSHTCPPingWatchdog
• RStudioWatchdog
• CertRolloverWatchdog
• JobSubmissionPingWatchdog
• OozieWatchdog
• DataNodesUpWatchdog
• NodeManagersUpWatchdog
• ResourceHealthWatchdog
• AzureNodeStatusWatchdog
• ClusterMALoggingHashWatchdog
• ClusterAvailabilityWatchdog
• ClusterHealthWatchdog
• ……..

8. Tool Purpose
Ambari Dashboard for monitoring health and status of the
Hadoop cluster
Yarn UI Monitor Yarn Application and logs
Tez View Track and debug the execution of jobs
Grafana Workload specific JMX metrics
Spark History Server The history server displays both completed and
incomplete Spark jobs
HMaster UI HBase provides a web-based user interface that you
can use to monitor your HBase cluster
Visual Studio /VS Code Monitor a Job status in VS with DataLake tools. Spark
Remote Job debugging

18. https://docs.microsoft.com/en-us/powershell/module/azurerm.hdinsight/enable-
azurermhdinsightoperationsmanagementsuite?view=azurermps-4.3.1

21. Region Servers Hive Resource Manager Livy
Zookeepers Hive Server 2 Node Manager Spark
HMaster Oozie Timeline Server Jupyter
HBase Rest Server WebHCat Executers
Apache Phoenix Metastore Drivers
RPC
Phoenix Query Server

22. Oozie Metastore
Hive Metastore
ADLS
Storage (Azure)
GW1
Head Node 1
Worker Node
Worker Node
Worker Node
Worker Node
Zookeeper1
Edge Node
HDI Management IP’s
168.61.49.99
23.99.5.239
168.61.48.131
138.91.141.162
VNET
Subnet
GW2Head Node 2
Zookeeper2 Zookeeper3

23. OMS Agent for
Linux
HDInsight nodes (Head, Worker ,
Zookeeper )
FluentD
HDInsight
plugin
1. Plugin for ‘in_tail’ for all Logs, allows
regexp to create JSON object
2. Filter for WARN and above for each
Log Type. `grep` filter plugin
3. Output to out_oms_api Type
4. Exec plugin for Metrics
HBaseConfigosmconfig
Spark
Hive
Storm
Kafka
Config
Config
Config
Config
Log Analytics(OMS) Service

26. HBase log_regionserver_CL
metrics_resourcemanager_clustermetrics_CL
log_hmaster_CL
metrics_regionserver_CL
metrics_regionserver_replication_CL
metrics_restserver_CL
metrics_restserver_jvm_CL
metrics_regionserver_ipc_CL
metrics_regionserver_wal_CL
metrics_regionserver_jvm_CL
metrics_regionserver_os_CL
metrics_hmaser_os_CL
metrics_hmaster_CL
metrics_hmaster_assignmentmanager_CL
metrics_hmaster_balancer_CL
metrics_hmaster_fs_CL
metrics_hmaster_ipc_CL
metrics_hmaster_jvm_CL
Hive hivemetastore_CL
log_oozie_CL
log_webhcat_CL
Kafka log_kafkaserver_CL
kafkametrics_CL
log_kafkacontroller_CL
Spark log_sparkappsexecutors_CL
log_sparkappsdrivers_CL
log_spark_CL
log_livy_CL
log_timelineserver_CL
log_jupyter_CL
VM Perf
Syslog
Yarn log_nodemanager_CL
metrics_resourcemanager_queue_root_CL
metrics_resourcemanager_queue_root_joblauncher_
CL
metrics_resourcemanager_jvm_CL
metrics_resourcemanager_queue_root_default_CL
log_resourcemanager_CL
log_mrjobsummary_CL