Hortonworks Data In Motion Series Part 3 - HDF Ambari

Harnessing Data-in-Motion with
Hortonworks DataFlow
HDF 2.0: Apache Ambari Integration
Ali Bajwa
Principal Partner
Solutions Engineer
Yolanda Davis
Senior Software
Engineer

Recap
HDF 2.0: Apache NiFi, Apache MiNiFi,
Apache Kafka, Apache Storm, Apache
Ambari

3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks DataFlow Manages Data-in-Motion
Core
InfrastructureSources
 Constrained
 High-latency
 Localized context
 Hybrid – cloud / on-premises
 Low-latency
 Global context
Regional
Infrastructure
Apache NiFi, Apache MiNiFi, Apache Kafka, Apache Storm are trademarks of the Apache Software Foundation

What is Apache Ambari?
Health Checks, Alerts
Stacks, Views
Lifecycle controls, Rolling
Restarts, Decommission/Re-
commission
Host Groups, Versioning,
Compare, Revert,
Recommendations, Security
Setup
Install Wizard (UI),
Blueprints (API)
Config
Management
ExtensibilityMonitoring
Service
Management
Cluster
Provisioning

Agenda
Apache Ambari Integration in HDF 2.0
• Deployment
• Configuration
• Debugging
• Monitoring

Ease of Deployment
Deploy Apache NiFi through:
– Apache Ambari install wizard or
– Operationalize via blueprints automation
Choose which nodes Apache NiFi can
be installed on
So either:
– Choose Apache NiFi hosts at time of
cluster install

Ease of Deployment
– Or add Apache NiFi to existing host (post-cluster install) and then start it

Ease of Deployment
–Can configure which user/group Apache NiFi runs as (when cluster installed or when
Apache NiFi is added to existing cluster)

Ease of Deployment
 Can remove Apache NiFi service from Apache Ambari
– Note: this does not remove bits
 Can stop/start/configure Apache NiFi across cluster via:
– Apache Ambari UI or
– Apache Ambari REST API

Ease of Deployment
 Can stop/start Apache NiFi on individual hosts via Apache Ambari UI and also via
REST API

Ease of Deployment
 Quicklink for easy access to Apache
NiFi UI
– URL dynamically determined based on:
• whether SSL enabled and
• which ports user configured

Demo #1: Deploy HDF 2.0
•Deploy HDF cluster using Apache Ambari Install Wizard:
http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.0/bk_ambari-installation/content/ch03s01.html
–Install Apache Ambari
–Install HDF mpack
–Open Apache Ambari WebUI
–Run through Install Wizard

Ease of Configuration
 Configure once via Apache Ambari (instead of repeating config on each node)
 Support for most Apache NiFi config files (e.g. nifi.properties, bootstrap.conf etc)
 Assistance for admin during Apache NiFi configuration:
– Help text displayed on hover with property descriptions
– Checkboxes instead of true/false values
– User friendly labels and default values
– ‘Computed’ values can be automatically handled (e.g. node address)
 Get benefit of Apache Ambari config features:
– Config history:
• diff between versions, revert to older version etc
– Update configs via Apache Ambari REST API

•Manage host specific configs using Config groups feature
–e.g. to bump memory for Apache NiFi on one of the nodes, click Override
button:

 ...create new config group and add desired host to it

 ‘Common’ configs grouped together and exposed in first config section
(‘Advanced nifi-ambari-config’) to allow configuration of:
– Ports (nonSSL, SSL, protocol)
– Initial and max memory (Xms, Xmx)
– Repo default dir locations (provenance, content, db, flow file)
– ‘Internal’ dir location - contains files NiFi will write to
• ‘conf’ subdir for flow/tar.gz, authorizations.xml
• ‘state’ subdir for internal state
• Can change subdir names by prefixing the desired subdir name with
‘{nifi_internal_dir}/’
– Sensitive props key (used to encrypt sensitive property values)
– Zookeeper znode for nifi

 nifi.properties exposed under ‘Advanced nifi-properties’ as key/value pairs with
helptext
– values replaced by Apache Ambari shown surrounded by double braces e.g.{{ }} but
can be overridden by end user
– Properties can be updated or added via ‘Custom nifi-properties’ and will get
written to all nodes
– ‘Computed’ values handled:
• ‘Node address’ fields replaced with each host's own FQDN

 Other property based config files exposed as jinja templates
– Values replaced by Apache Ambari shown surrounded by double braces e.g. {{ }} but can be overridden by
end user
– Properties can be added/updated and will get written to all nodes

Other xml based config files also exposed as jinja templates
– Values replaced by Apache Ambari shown surrounded by double braces e.g. {{ }} but can be overridden
– Elements can be updated/added and will get written to all nodes
– Config files written out with
either 0400 or 0600 perm
• Why?
• Some property files contain
plaintext passwords

Ease of Debugging
 Logsearch integration for ease of visualizing/debugging Apache NiFi logs w/o connecting via SSH to
hosts (e.g nifi_app.log, nifi_user.log, nifi_bootstrap.log)
 By default, monitors FATAL,ERROR,WARN messages (for all HDF services)

Ease of Debugging
 Can view/drill into errors at component level or host level
 Can filter errors based on severity (fatal, error, warn, info, debug, trace)
 Can exclude ‘noisy’ messages to find the needle in the haystack

Ease of Monitoring
Apache NiFi Service check: Used to ensure that Apache NiFi UI is up. Can also be
invoked via REST API for automation

Ease of Monitoring
Apache NiFi Alert: Host level Apache Ambari alert when Apache NiFi process
goes down
– Can temporarily be disabled by turning on maintenance mode

Ease of Monitoring
Alerts tab allows:
– Disabling and Configuring alerts (e.g. change interval)

Ease of Monitoring
 Can setup email or
SNMP notifications to
admins

Ease of Monitoring
 Apache Ambari Metrics (AMS) integration
– Service autocreates Apache Ambari reporting task, pointing to AMS collector host/port (autodetected)

Ease of Monitoring
This is achieved by setting up an initial flow.xml (configurable via Amari)...
Tip: can auto-deploy any Apache NiFi flow by pasting its flow.xml here

Ease of Monitoring
...and passing arguments via bootstrap.conf
 Ensures Apache NiFi autodetects updated location of AMS collector (in case it’s moved)

Ease of Monitoring
 Results in AMS dashboard
for Apache NiFi metrics,
such as:
– Flowfiles sent/received
– MBs read/written
– JVM usage/thread counts

Ease of Monitoring
 Dashboard widgets can:
– be drilled into to see
results from last 1,2,4
hours, day, week etc
– export metrics data as csv
or JSON

Ease of Monitoring
 Same metrics can be viewed in Grafana dashboard
– Access via quicklink under ‘Ambari metrics’ service
– Allows users to easily create custom dashboards for each component

Summary - Apache NiFi in Apache Ambari
• HDF available via management pack (mpack) that defines required services, stack
definitions etc
• Wizard Driven Setup
• Provides access to Apache NiFi configurations, start/stop/restart
• Has a Reporting Task for Apache Ambari Metrics
• In Cluster Mode by Default
• Leverages Zookeeper on HDF for State Management (not Embedded Zookeeper)
• Some Stack Advisor configurations (Ranger integration, Apache NiFi Certificate
Authority)
• Audit logging via Apache Ambari Infra (Internal Solr)
* You cannot install Apache Ambari to manage an HDF cluster on a system where
HDP is already installed (requires a dedicated instance)

Demo #2: Explore HDF 2.0
 Explore HDF 2.0
– Apache NiFi configurations
– Configuration groups
– Metrics
– Logsearch integration
– Service checks
– Alerts

Try it out!
 Deploy HDF cluster using either:
– a) Install Wizard: http://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.0/bk_ambari-installation/content/ch03s01.html
– b) Blueprints: https://community.hortonworks.com/articles/56849/automate-deployment-of-hdf-20-clusters-using-ambar.html
* Reminder: You cannot install Apache Ambari to manage an HDF cluster on a system where HDP is
already installed (requires a dedicated instance)

Questions? https://community.hortonworks.com/
Hortonworks Community Connection:
Data Ingestion and Streaming

Hortonworks Data In Motion Series Part 3 - HDF Ambari

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à Hortonworks Data In Motion Series Part 3 - HDF Ambari

Similaire à Hortonworks Data In Motion Series Part 3 - HDF Ambari (20)

Plus de Hortonworks

Plus de Hortonworks (20)

Dernier

Dernier (20)

Hortonworks Data In Motion Series Part 3 - HDF Ambari