Apache Ambari BOF Meet Up @ Hadoop Summit 2013 Blueprints http://www.meetup.com/Apache-Ambari-User-Group/events/119184782/

1. Apache Ambari
Hadoop Cluster Manifest/Blueprint
Sumit Mohanty
Member of Technical Staff
Hortonworks

2. Agenda
• Cluster Manifest
• Scenarios
• Cluster Blueprint
• Using Cluster Manifest
• What’s next?

3. Hadoop Cluster Manifest
• Declarative representation of a Hadoop Cluster
– Stack Definition
– Configuration
– Host Details
– Component Mapping
• A common spec. across tools/services
• Targets
– Package Author, Hadoop Admins, and System Admins

4. Cluster Manifest:
Package Definition
• Package metadata
• Repository details
• Constituent services and their components
• Service specific metadata
• Configurable parameters

5. Cluster Manifest:
Package Definition
{
"schemaVersion:" : "1”,
"version" : "1.3.0”,
"author" : "Hortonworks”,
"created" : "03-31-2013”,
"manifestId" : "GUID",
"stackVersion" : "1.3.0”, "stackName" : "HDP",
"context" : […],
"packages" : {
"type" : "rpm",
"osSpecificPackages" : […]
},
"services" : [
{
"name" : "HDFS",
"components" : [
{
"name" : "NAMENODE",
"category" : "MASTER",
…
},
{
"name" : "DATANODE", …
],
"configurations" : [
{
"type":"core-site.xml",
"properties" : [
{
"propertyName" : "fs.trash.interval",
"defaultValue" : "360",
"propertyDescription" : "..."
},
…
],
"isManageable": "true",
"isRequired": "true",
"packages": […],
"serviceContext" : […]
}
}

6. Cluster Manifest:
Package Configuration
• Configurable parameters and values
– Non-default
– Organization, environment, instance specific
• Service or component specific values
{
"schemaVersion:" : "1", …
"context" : [
{ "name" : "targetStackVersion", "value" : "1.3.0" },
],
"deployedServices" : ["HDFS”, … ],
"configuration" : [
{
"type":"core-site.xml",
"properties" : [
{ "name" : "fs.trash.interval", "value" : "300" },
...
]
},
…
"configOverrides" : [
/* delta changes on the top level changes */
{
"type" : "SERVICE”, "name" : "HDFS",
"configuration" : [
{
"type":"core-site.xml",
"properties" : [
{ "name" : "fs.trash.interval", "value" : "480" },
...
},
{
"type" : "COMPONENT"
"name" : "JOBTRACKER",
...
}

7. Cluster Manifest:
Host List
• List of hosts
– Can be fully specified
– Or, can be a set of requirements
– Or, can even be non-existent
{
"schemaVersion:" : "1", …
"context" : […],
"hostGroups" : [
{
"name" : "masterHosts",
"members" : {
"count" : "1",
"hosts" : [
{ "FQDN" : "host1.domain1.com", "ip" : "" }
]
},
"properties" : […]
},
{
"name" : ”slaveHosts",
"members" : {…},
"properties" : […]
},
{
"name" : "clientHosts",
"members" : {…},
"properties" : [
{ "name" : "host_type", "value" : "High-CPU Medium" }
]
},
...
]
}

8. Cluster Manifest:
Host Component Mapping
• A mapping of components to hosts
– Simple component mapping to named hosts
– A set of constraints that can be used to find best
match (e.g. evaluate against host properties)
• System resources
– users, groups, ports, etc.
• Host specific configuration
– Non-homogeneous cluster

9. Cluster Manifest:
Host Component Mapping
{
"schemaVersion:" : "1”, …
"context" : […],
"hostResourceMapping" : [
{
"hosts" : [
{
"predicate" : "name=*"
}
],
"systemResources" : {
"hadoopGroup" : "hadoop",
"groups" : [
{
"name" : "hadoop",
...
],
"users" : [
{
"groups" : [
"hadoop"
],
"name" : "hdfs",
"type" : "LOCAL”
],
"ports" : […]
...
],
"hostComponentMapping" : [
{
"hosts" : [
{
"predicate" : "name=masterhosts1"
"configOverrides" : [
{
"type":"core-site.xml",
"properties" : [
{ "name" : "fs.trash.interval", "value" : "480" },
...
],
"components" : [
"NAMENODE",
"JOBTRACKER",
...
]
},
...
}

10. Scenarios
• Define cluster templates
– and, host specific templates
• On demand cluster creation
– Cluster extension (e.g. add Datanodes)
• Export cluster manifest
• A uniform “language” across cluster managers
and environments

11. Cluster Blueprint
• Blueprint is manifest with “holes”
– Typically
• Hostnames
• Config parameters that use hostname
– But, any config params that a Hadoop admin
deems necessary to be parameterized
• Blueprint = Manifest + Parameter Values

12. Using Cluster Manifest

13. What’s Next?
• Apache Ambari JIRA 1783, is tracking this
project
– https://issues.apache.org/jira/browse/AMBARI-
1783
– Comments and suggestions, welcome
• In next releases, we will enhance Ambari to
add support for manifest and blueprints