leveraging ambari to build comprehensive management · pdf filefor your hadoop applications by...

Leveraging Ambari to Build Comprehensive Management UIs

For Your Hadoop Applicationsby Christian Tzolov

@christzolov

Whoami

Christian Tzolov Technical Architect at Pivotal, BigData, Hadoop, SpringXD, Apache Committer, Crunch PMC member

ctzolov@pivotal.io blog.tzolov.net @christzolov

How To Provision, Manage And Monitor Enterprise Applications At Scale?

Contents• Ambari Overview

• Stacks and Services (demo)

• Blueprints (demo)

• Views (demo)

• Metrics & Monitoring

• On the Cloud

Apache Ambari

A platform to provision, manage and monitor Hadoop and Enterprise Applications

Deployment Topology

Ambari Web

(REST API) Ambari Server

admin node

Ambari Agent

cluster node 1

Ambari Agent

cluster node 2

Ambari Agent

cluster node N

Special PowersServices & Stacks

Blueprints

Views

Monitoring

Services & Stacks

Service Concepts

• Stack - Set of (managed) Services and software packages for them

• Service - Set of (distributed) Components

• Component - Service building block of type Master, Slave, Client

Service Architecture

Service Layout|_ services |_ <service_name> metainfo.xml metrics.json |_ configuration {configuration files} |_ package |_ scripts {command scripts}

• metainfo.xml - defines service, components and the command management scripts

• configuration - service config files used in and deployed through the command scripts

• package/scripts - commandScripts and templates.

Metainfo.xml<metainfo> <schemaVersion/> <services> <service> <name/> <displayName/> <comment/> <version/> <components/> <osSpecifics/> <commandScript/> <requiredServices/> <configuration-dependencies/> </service> </services> </metainfo>

<components> <component> <name>CUST_MASTER</name> <displayName>Master</displayName> <category>MASTER</category> <cardinality>1</cardinality> <commandScript> <script>scripts/master.py</script> <scriptType>PYTHON</scriptType> <timeout>600</timeout> </commandScript> </component> ……… </components>

Service / Command Scriptimport sys from resource_management import * class Master(Script): def install(self, env): def stop(self, env): def start(self, env): def status(self, env): def configure(self, env): if __name__ == "__main__": Master().execute()

import sys from resource_management import * class Slave(Script): def install(self, env): def stop(self, env): def start(self, env): def status(self, env): def configure(self, env): if __name__ == "__main__": Slave().execute()

Service / Demo

Ambari Blueprint

Why?• Installation Automation

• Declarative definition of a cluster

• Specify a Stack, the Component layout and the Configurations to materialize a cluster

• REST API

Blueprint Flow

Ambari Server

node 1

Cluster 1

node 2 node N

Cluster 2

. . . .

Cluster Z

. . . .

Blueprint (json)

Host Mapping (json)

REST API

Blueprint Deploy{ "Blueprints" : { "stack_name" : "PHD", "stack_version" : "3.0" }, "configurations" : [ { "hawq-site" : { "properties" : { "hawq.master.port" : "5532" } } } ], "host_groups" : [ { "name" : "management_masters", "components" : [ { "name" : "NAMENODE" }, { "name" : "HAWQMASTER" } ], "cardinality" : "1" }, …… ]}

POST /api/v1/blueprints/my-blueprint

{ "blueprint" : "hdfs-hawq-blueprint", "default_password" : "secret-password", "host_groups" : [ { "name" : "management_masters", "hosts": [ { "fqdn":"ambari.localdomain" } ] }, { "name" : "masters_workers", "hosts": [ { "fqdn":"phd1.localdomain" }, { "fqdn":"phd2.localdomain" }, { "fqdn":"phd3.localdomain" } ] } ] }

POST /api/v1/clusters/my-cluster

Blueprint REST API/api/v1/blueprint

Cluster Logical Structure and configuration

/api/v1/clusterPhysical cluster mapping. Materializes a blueprint with real infrastructure

/api/v1/clusters/<cluster name>?format=blueprint

Exports blueprint for an existing cluster

Blueprint Demo

Ambari Views

View Components

Ambari Server

Ambari Web

Server Side (java)

Client Side (js,html..)

REST API

VIEW

PAC

KAG

E (J

AR)

• Client Side - any client-side technologies

• Server Side - Java application, Servlet, a REST service or ResourceProvider.

• View Package - JAR bundles of the view definition and resources

• View Context - Instance attributes and configuration

context

View Packageview.xml

<server packages + classes>

index.html <supporting UI assets>

WEB-INF (lib/*.jar)

View Context• On the Server Side

• View-instance Config

• View-instance Attributes

• Execution context, authenticated user principal

ViewContext

getUsername() : String getViewName() : String getInstanceName() : String getProperties() : Map<String,String> putInstanceData(…) getInstanceData(…) : String

View Dashboard

View REST APIGET /api/v1/views List deployed views

GET /api/v1/views/<View Name> List all View instances

GET /api/v1/views/<View Name>/versions/<View Version>

Show specific View instance

POST /api/v1/views/<View Name>/versions/<View Version>/instances/<Instance Name> [{“ViewInstanceInfo":{ "properties" : { "dataworker.defaultFs":“webhdfs://namenode.host:50070" }}}]

Create new View Instance

Google Polymerhttps://www.polymer-project.org/1.0/

View Demo

https://github.com/tzolov/ambari-webpage-embedder-view

https://youtu.be/qaR_6OBeKsQ

Ambari Monitoring

Ambari Metrics SystemCollect, Aggregate and Serve System Metrics

• Metric Collector - Collects and Aggregates

• Metric Monitor - System-level metrics

• Metric Hadoop Sinks - Service-level metrics

AMS Architecture

Ambari & Cloud

Virtualization & Cloud• Vagrant - PivotalHD

(https://github.com/tzolov/vagrant-pivotalhd)

• Cloudbreak (http://sequenceiq.com/cloudbreak/)

Thank You!