apache ambari: past, present, future
TRANSCRIPT
![Page 1: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/1.jpg)
1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache AmbariPast, Present and FutureOctober 2016
![Page 2: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/2.jpg)
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Who Am I
Jeff SposettiProduct Management, Cloud & Operations @ HortonworksApache Ambari Committer
https://www.linkedin.com/in/jsposetti
@jsposetti
![Page 3: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/3.jpg)
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda
Background, Release History and Demo Ambari Alerts and Metrics Ambari Blueprints Security Setup (Kerberos, RBAC) Log Search (Technical Preview) Automated Cluster Upgrade Extensibility: Stacks and Views What’s New and Next
![Page 4: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/4.jpg)
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What is Apache Ambari?
A completely open source management platform for provisioning, managing, monitoring and securing Apache Hadoop clusters. Apache Ambari takes the guesswork out of operating Hadoop.
![Page 5: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/5.jpg)
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Recent Releases
Ambari 2.2.0Dec 2015
Ambari 2.2.2Apr 2016
Ambari 2.4.0 GAAug 2016
Intro
duce
d
Expr
ess
Upgrade
Intro
duce
d
Grafan
a
Intro
duce
d
Log S
earch
TP
![Page 6: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/6.jpg)
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Alerts
![Page 7: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/7.jpg)
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Full Visibility into Cluster Health
• Centralized management of Health Alerts
• Pre-defined alerts, configured by default on cluster install
![Page 8: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/8.jpg)
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Customizing Alerts
• Control thresholds, check intervals and response text
![Page 9: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/9.jpg)
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alert Notifications
• What: Create and manage multiple notification targets
• Control who gets notified when
• Why: Filter by severity• Send only certain notifications to certain
targets based on severity
• How: Control dispatch method• Support for EMAIL + SNMP
![Page 10: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/10.jpg)
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alert Groups
• Create and manage groups of alerts• Group alerts further controls what alerts
are dispatched which notifications
• Assign group to notifications• Only dispatch to interested parties
![Page 11: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/11.jpg)
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alert Check Counts
Customize the number of times an alert is checked before dispatching a notification Avoid dispatching an alert notification (email, snmp) in case of transient issues
![Page 12: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/12.jpg)
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Configuring the Check Count
Set globally for all alerts, or override for a specific alert
Global Setting
Alert Override
![Page 13: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/13.jpg)
13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
State Change Types
SOFT state changes do not perform a dispatch HARD state changes (to non-OK) perform dispatch Regardless of change:
– The Ambari Web UI will show the current state (OK/WARN/CRIT)– The state change is written to ambari-alerts.log
2016-05-31 13:20:52,294 [CRITICAL] [SOFT] [AMBARI_METRICS] [grafana_webui] (Grafana Web UI) Connection failed to http://c6401.ambari.apache.org:3000 (<urlopen error [Errno 111] Connection refused>)2016-05-31 13:22:52,290 [CRITICAL] [HARD] [AMBARI_METRICS] [grafana_webui] (Grafana Web UI) Connection failed to http://c6401.ambari.apache.org:3000 (<urlopen error [Errno 111] Connection refused>)
Note: check counts are not configurable for AGGREGATE alert types. All state changes are considered HARD.
![Page 14: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/14.jpg)
14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Example: Check Count = 3
Check 1/3State: OK
Change: n/a
Check 1/3State: OK
Change: n/a
Check 1/3State: CRIT
Change: SOFT
Check 2/3State: CRIT
Change: n/a
Check 3/3State: CRIT
Change: HARD
Check 1/3State: OK
Change: HARD
DISPATCH
Check Interval Check Interval Check Interval Check Interval Check Interval
no state changestate changes to CRIT
performing multiple checks
back to OK
![Page 15: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/15.jpg)
15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Other Items
Alerts Log (AMBARI-10249)
• Alert state changes are written to /var/log/ambari-server/ambari-alerts.log
Script-based Alert Notifications (AMBARI-9919)
• Define a custom script-based notification dispatcher
• Executed on alert state changes
2015-07-13 14:58:03,744 [OK] [ZOOKEEPER] [zookeeper_server_process] (ZooKeeper Server Process) TCP OK - 0.000s response on port 21812015-07-13 14:58:03,768 [OK] [HDFS] [datanode_process_percent] (Percent DataNodes Available) affected: [0], total: [1]
![Page 16: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/16.jpg)
16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Alerting System1. User creates or modifies cluster2. Ambari reads alert definitions from
Stack3. Ambari sends alert definitions to
Agents and Agent schedules instance checks
4. Agents reports alert instance status in the heartbeat
5. Ambari responds to alert instance status changes and dispatches notifications (if applicable)
AmbariServer
1
2
4
Stack definitionalerts.json
5
AmbariAgent(s)
3
emailsnmp
![Page 17: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/17.jpg)
17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alert REST APIsREST Endpoint Description/api/v1/clusters/:clusterName/alert_definitions The list of alert definitions for the cluster.
/api/v1/clusters/:clusterName/alerts The list of alert instances for the cluster. Use partial response syntax to filter.
Example: find all alert instances that are CRITITAL/api/v1/clusters/c1/alerts?Alert/state.in(CRITICAL)
Example: find all alert instances for “ZooKeeper Process” alert def/api/v1/clusters/c1/alerts?Alert/definition_name=zookeeper_server_process
/api/v1/clusters/:clusterName/alert_groups The list of alert groups.
/api/v1/clusters/:clusterName/alert_history The list of alert instance status changes.
/api/v1/alert_targets/ The list of configured alert notification targets for Ambari.
![Page 18: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/18.jpg)
18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Metrics
![Page 19: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/19.jpg)
19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Metrics
Built using Hadoop technologies Three components: Metrics Collector, Metrics Monitors, Grafana
Local Filesystem or HDFS
HBaseATS
Phoenix
![Page 20: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/20.jpg)
20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Metrics Collection
1. Metric Monitors send system-level metrics to Collector
2. Sinks send Hadoop-level metrics to Collector
3. Metrics Collector service stores and aggregates metrics
4. Ambari exposes REST API for metrics retrieval
AmbariServer
Metrics Monitor
Metrics Collector
Host1
Sink(s)
3
Metrics Monitor
Host1
Sink(s)Metrics Monitor
Hosts
Sink(s)
1 2
4
![Page 21: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/21.jpg)
21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Metrics + Grafana: Introduced with Ambari 2.2
Including Grafana as a “Native UI” for Ambari Metrics
Including pre-build Dashboards Supports configuring for HTTPS
System Home, Servers HDFS Home, NameNodes, DataNodes YARN Home, Applications, Job History
Server HBase Home, Performance, Misc
Highlights List of Dashboards
![Page 22: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/22.jpg)
22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Grafana includes pre-built dashboards for visualizing the most important cluster metrics.
![Page 23: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/23.jpg)
23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Blueprints
![Page 24: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/24.jpg)
24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Blueprints
Two primary goals of Ambari Blueprints:
• Provide API-driven deployments based on self-contained cluster description
• Ability to export a complete cluster description of a running cluster Blueprints contain cluster topology and configuration information Enables interesting use cases between physical and virtual environments
![Page 25: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/25.jpg)
25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Stacks + Blueprints Together
Stack DefinitionComponent
Layout & Configuration
BLUEPRINT
BLUEPRINT INSTANTIATE CLUSTERHOSTS
![Page 26: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/26.jpg)
26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Blueprints API
BLUEPRINT POST /blueprints/my-blueprint
POST /clusters/MyCluster
1
2
CLUSTERCREATIONTEMPLATE
![Page 27: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/27.jpg)
27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Example: Single-Node Cluster
{ "configurations" : [ { ”hdfs-site" : {
"dfs.namenode.name.dir" : ”/hadoop/nn" } } ], "host_groups" : [ { "name" : ”uber-host", "components" : [ { "name" : "NAMENODE” }, { "name" : "SECONDARY_NAMENODE” }, { "name" : "DATANODE” }, { "name" : "HDFS_CLIENT” }, { "name" : "RESOURCEMANAGER” }, { "name" : "NODEMANAGER” }, { "name" : "YARN_CLIENT” }, { "name" : "HISTORYSERVER” }, { "name" : "MAPREDUCE2_CLIENT” } ], "cardinality" : "1" } ], "Blueprints" : { "blueprint_name" : "single-node-hdfs-yarn", "stack_name" : "HDP", "stack_version" : "2.2" }}
{ "blueprint" : "single-node-hdfs-yarn", "host_groups" :[ { "name" : ”uber-host", "hosts" : [ { "fqdn" : "c6401.ambari.apache.org”
} ] } ]}
BLUEPRINT
CLUSTER CREATION TEMPLATE
Description• Single-node cluster• Use HDP 2.2 Stack• HDFS + YARN + MR2• Everything on c6401
![Page 28: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/28.jpg)
28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Blueprints Add Host
• Add hosts to a cluster based on a host group from a Blueprint
• Add one or more hosts with a single call
POST /api/v1/clusters/MyCluster/hosts{ "blueprint" : "myblueprint", "host_group" : "workers", "host_name" : "c6403.ambari.apache.org"}
POST /api/v1/clusters/MyCluster/hosts[ { "blueprint" : "myblueprint", "host_group" : "workers", "host_name" : "c6403.ambari.apache.org" }, { "blueprint" : "myblueprint", "host_group" : ”edge", "host_name" : "c6404.ambari.apache.org" }]
https://issues.apache.org/jira/browse/AMBARI-8458
![Page 29: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/29.jpg)
29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Blueprints Host Discovery (AMBARI-10750)
Ambari
POST /api/v1/clusters/MyCluster/hosts
[ { "blueprint" : "single-node-hdfs-test2", "host_groups" :[ { "host_group" : "slave", "host_count" : 3, "host_predicate" : "Hosts/cpu_count>1” } ] }]
![Page 30: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/30.jpg)
30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Where Can I Learn More About Blueprints?
https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
![Page 31: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/31.jpg)
31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Kerberos Setup
![Page 32: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/32.jpg)
32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Hadoop + Kerberos
Strongly authenticating and establishing a user’s identity is the basis for secure access in Hadoop. Users need to be able to reliably “identify” themselves and then have that identity propagated throughout the Hadoop cluster.
Once this is done, those users can access resources (such as files or directories) or interact with the cluster (like running MapReduce jobs).
Besides users, Hadoop cluster resources themselves (such as Hosts and Services) need to authenticate with each other to avoid potential malicious systems or daemon’s “posing as” trusted components of the cluster to gain access to data.
![Page 33: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/33.jpg)
33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Hadoop + Kerberos
Service Component
A
Service Component
B
Hadoop Cluster
KDC
keytabkeytab
Service Component
C
keytab
Service Component
D
keytab
Service Component
X
Service Component
X
keytabkeytab
Service Component
X
keytab
Service Component
X
keytab
Kerberos is used to secure the
Components in the cluster. Kerberos
identities are managed via
“keytabs” on the Component hosts.
Principals for the
cluster are managed in
the KDC.
![Page 34: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/34.jpg)
34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automated Kerberos Security Setup
Wizard DrivenSimplify the experience of enabling and managing Kerberos security
Automated and Manual OptionsCritical for today’s enterprise
Works with Existing Kerberos InfrastructureSuch as Active Directory and MIT KDC
![Page 35: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/35.jpg)
35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automated vs. Manual Kerberos Options
Automated Manual
KDC Infrastructure MIT, Active Directory MIT, Active Directory, FreeIPA
Requires KDC administrative credentials Yes No
Installation of Kerberos clients Yes, optional No
Management of Kerberos client krb5.conf Yes, optional No
Creation of principals Yes No
Creation of keytabs Yes No
Distribution of keytabs Yes No
Cluster configuration Yes Yes
![Page 36: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/36.jpg)
36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automated: Principals and Keytabs1. User provides KDC Admin Account
credentials to Ambari2. Ambari connects to KDC, creates
principals (Service and Ambari) needed for cluster
3. Ambari generates keytabs for the principals
4. Ambari distributes keytabs to Ambari Server and cluster hosts
5. Ambari discards the KDC Admin Account credentials
AmbariServer KDC
1 2
4
3
5
Cluster
![Page 37: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/37.jpg)
37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automated: Customizable Principal Attributes
• Customize the directory principal attributes for your environment
• Particularly useful when using Active Directory for Kerberos
![Page 38: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/38.jpg)
38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automated: Kerberos Clients
• Option to manage krb5.conf
• Option to install clients
OS ClientRHEL/CentOS/OEL krb5-workstationSLES 11 krb5-clientUbuntu 12 krb5-user, krb5-
config
![Page 39: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/39.jpg)
39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automated: Kerberos Operations
Once enabled, Ambari works directly with the Kerberos infrastructure to automate these common tasks, removing the burden from the operator:
• Add/Delete Host
• Add Service
• Add/Delete Component
• Regenerate Keytabs
• Disable Kerberos
![Page 40: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/40.jpg)
40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Manual: Specify Realm, Client Utilities Path
![Page 41: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/41.jpg)
41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Manual: Principals and Keytabs
Configure Identities Download CSV
![Page 42: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/42.jpg)
42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Role-Based Access Control
![Page 43: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/43.jpg)
43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Role-Based Access Control
Use “roles” for more granular division of control for cluster operations
Old Permission New Role Notable Permissions
Operator Cluster Administrator Full operational control, including upgrades. Ambari Admins are implicitly granted this Role.
Cluster Operator Adding and removing hosts.
Service Administrator Manage configurations, move components.
Service Operator Service stop and start and service-specific operations such as HDFS Rebalance.
Read-Only Cluster User View cluster service and host information.
Note: Users flagged as “Ambari Admins” are implicitly granted Cluster Administrator permission.
![Page 44: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/44.jpg)
44 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Managing Cluster RolesAssign roles to users or groups
Manage roles in Block or List View
layouts
![Page 45: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/45.jpg)
45 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Managing Cluster Roles View users or groups
Change current role assignment
![Page 46: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/46.jpg)
46 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Log Search
TECH PREVIEW
![Page 47: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/47.jpg)
47 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Log Search
Solr
A M B A R ILog
Search
Search cluster Component Logs from within Ambari!
Goal: When issues arise, be able to quickly find issues across all cluster components
⬢ Capabilities– Rapid Search of all cluster component logs– Search across time ranges, log levels, and for keywords
⬢ Core Technologies: – Apache Ambari– Apache Solr– Apache Ambari Log Search
Tech Preview
![Page 48: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/48.jpg)
48 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Log Search Architecture
A M B A R I
L O G F E E D E R
L O G F E E D E R
L O GF E E D E R
L O GF E E D E R
L O G F E E D E R
L O G F E E D E R
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
Solr
L O G S E A R C H
U I
Tech Preview
![Page 49: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/49.jpg)
49 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Log Search Details
W O R K E RN O D E
L O G F E E D E R
Solr
L O G S E A R C H
U I
Solr
Solr
A M B A R I
Java ProcessMulti-output SupportGrok
Solr CloudLocal Disk StorageTTL
Tech Preview
![Page 50: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/50.jpg)
50 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automated Cluster Upgrade
![Page 51: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/51.jpg)
51 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automated Cluster Upgrade Terminology
Rolling UpgradeOrchestrates the cluster upgrade in an order that is meant to preserve cluster operation and minimize service impact. The tradeoff is more stringent prerequisites and a longer upgrade time.
Express UpgradeOrchestrates the cluster upgrade in an order that will incur cluster downtime. This method has less stringent prerequisites but completes in a faster upgrade time.
![Page 52: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/52.jpg)
52 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automated Upgrade: Rolling or Express Choice
Check Prerequisites Prepare Register +
InstallPerform Upgrade
Perform the upgrade. The steps depend on
upgrade method: Rolling or Express.
Finalize
Finalize the upgrade, making the target
version the current version.
Perform the preparation steps,
which include making backups of critical cluster metadata.
Review the prerequisites to
confirm your cluster configuration is ready
for upgrade
Register the software repository and install the target version on
the cluster.
![Page 53: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/53.jpg)
53 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automated Upgrade Choice: Rolling or Express
![Page 54: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/54.jpg)
54 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automated Upgrade: High-Level Process
Register Install PerformUpgrade Finalize
![Page 55: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/55.jpg)
55 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Register New Version
• Register new version with Ambari
• Provide the Repository Base URLs for new version
Register Install PerformUpgrade Finalize
![Page 56: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/56.jpg)
56 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Install Packages
Click “Install Packages” to start the install of the version across hosts
Register Install PerformUpgrade Finalize
![Page 57: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/57.jpg)
57 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Upgrade Choice
Upgrade Options dialog gives you a choice of method: Rolling or Express
Register Install PerformUpgrade Finalize
![Page 58: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/58.jpg)
58 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Upgrade Choice: Rolling or Express
Checks indicate if an option is available
Register Install PerformUpgrade Finalize
![Page 59: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/59.jpg)
59 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Upgrade Choice: Rolling or Express
New options to handle failures
Register Install PerformUpgrade Finalize
![Page 60: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/60.jpg)
60 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Upgrade Tolerance Options
Tolerance Option Description
Skip all Service Check failures Ambari will automatically skip any Service Check failures and complete the task without requiring user intervention to continue. After all the Service Checks have run in a task, you will be presented with summary of the failures and an option to continue the upgrade or pause.
Skip all Slave Component failures Ambari will automatically skip any Slave Component failures and complete the task of upgrading Slave components without requiring user intervention to continue. After all Slave Components have been upgraded, you will be presented with a summary of the failures and an option to continue the upgrade or pause.
Register Install PerformUpgrade Finalize
![Page 61: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/61.jpg)
61 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Wizard Driven Experience
With verification and validation
Register Install PerformUpgrade Finalize
![Page 62: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/62.jpg)
62 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Rolling Upgrade – Success!Register Install Perform
Upgrade Finalize
![Page 63: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/63.jpg)
63 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Extensibility
![Page 64: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/64.jpg)
64 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Extensibility Features
• To add new Services (ISV or otherwise) beyond Stack• To customize a Stack for customer specific environments
• To extend and customize the Ambari Web UI• Add new capabilities, customize existing capabilities
Stacks
Views
![Page 65: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/65.jpg)
65 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Anatomy of Ambari Extension Points
AmbariServer
AmbariAgentAmbari
AgentAmbariAgent
AmbariWeb
StacksStacks
Stacks
javajs python
Ambari Views Ambari Stacks
![Page 66: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/66.jpg)
66 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Stacks
![Page 67: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/67.jpg)
67 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Stacks
Defines a consistent Stack lifecycle interface that can be extended Encapsulates Stack Versions, Services, Components, Dependencies, Cardinality,
Configurations, Commands Dynamically add Stack + Service definitions
AMBARI{rest}
<ambari-web>Stacks
HDFS YARN MR2
Hive
Pig
OozieHBase
StormFalcon
![Page 68: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/68.jpg)
68 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Stack TerminologyTerm Definition Examples
STACK Defines a set of Services, where to obtain the software packages and how to manage the lifecycle. HDP-2.4, HDP-2.5
SERVICE Defines the Components that make-up the Service. HDFS, YARN, HIVE, SPARK
COMPONENT The building-blocks of a Service, that adhere to a certain lifecycle. NAMENODE, DATANODE, OOZIE_SERVER
CATEGORY The category of Component. MASTER, SLAVE, CLIENT
![Page 69: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/69.jpg)
69 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Stack Mechanics
Stacks define Services + Repos– What is in the Stack, and where to get the bits
Each Service has a definition– What Components are part of the Service
Each Service has defined lifecycle commands– start, stop, status, install, configure
Lifecycle is controlled via command scripts Ability to define “custom” commands
AMBARISERVER
Stack
Command Scripts
Service Definitions
AMBARIAGENT/SAMBARI
AGENT/SAMBARIAGENT/S
pythonxml
Repos
![Page 70: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/70.jpg)
70 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Stacks Support Inheritance
HDP 2.0 Stack
HDP 2.4 Stack
• Defines a set of Service definitions• Default service configurations and command scripts
• Overrides any Service definitions, commands and configurations
• Adds new Services specific to this Stack
![Page 71: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/71.jpg)
71 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Where Can I Learn More About Stacks?
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=38571133
https://github.com/apache/ambari/tree/trunk/ambari-server/src/main/resources/stacks/HDP/
![Page 72: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/72.jpg)
72 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Guided Configs
![Page 73: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/73.jpg)
73 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Guided Configurations
Take the guesswork out of cluster configuration Intuitive layout and grouping of configurations Smart UI controls to make it easier to set values Recommendations and cross-service
dependency checks Take the guesswork out of cluster configuration Driven by Stack definition
![Page 74: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/74.jpg)
74 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Layout and Grouping
Subtabs
Groups
Groups
New
Controls
New
Controls
![Page 75: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/75.jpg)
75 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
UI Controls
RecommendedBounds
RecommendedValue
SetRecommended
EscapeHatch
![Page 76: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/76.jpg)
76 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Control “Escape Hatch”
![Page 77: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/77.jpg)
77 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Dependency Checking
![Page 78: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/78.jpg)
78 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Driven By Stack Definition: Themes
![Page 79: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/79.jpg)
79 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Tabs
Tab1 Tab2
![Page 80: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/80.jpg)
80 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Sections
Section1 Section2
![Page 81: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/81.jpg)
81 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
SubSections
SubSection1 SubSection2 SubSection1
![Page 82: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/82.jpg)
82 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
UI Controls and Placement
Control + Placement
Control + Placement
Control + Placement Control + Placement
Control + Placement
![Page 83: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/83.jpg)
83 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Views
![Page 84: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/84.jpg)
84 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari Views Framework
Developers can extend the Ambari Web interface• Views expose custom UI features for Hadoop Services
Ambari Admins can entitle Views to Ambari Web users• Entitlements framework for controlling access to Views
![Page 85: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/85.jpg)
85 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Views Framework
Views Framework vs. Views
Views
Core to Ambari
Built by Hortonworks, Community,
Partners
![Page 86: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/86.jpg)
86 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Views Framework
Views Framework vs. Views
Views
Core to Ambari
Built by Hortonworks, Community,
Partners
![Page 87: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/87.jpg)
87 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Example Views
Capacity Scheduler“Queue Manager” View
Tez“Jobs” View
![Page 88: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/88.jpg)
88 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
View Components
Serve client-side assets (such as HTML + JavaScript) Expose server-side resources (such as REST endpoints)
VIEWClient-side
assets(.js, html)
AMBARI WEB
VIEWServer-side resources
(java)
AMBARI SERVER
{rest}
Hadoop and
other systems
![Page 89: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/89.jpg)
89 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
View Delivery
1. Develop the View (just like you would for a Web App)
2. Package as a View (basically a WAR)
3. Deploy the View into Ambari
4. Ambari Admins create + configuration view instance(s) and give access to users + groups
Develop DeployPackageCreate
Instance(s)
![Page 90: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/90.jpg)
90 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Versions and Instances
Deploy multiple versions and create multiple instances of a view Manage accessibility and usage
![Page 91: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/91.jpg)
91 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Choice of Deployment Model
For Hadoop Operators:– Deploy Views in an Ambari Server that is managing a Hadoop cluster
For Data Workers:– Run Views in a “standalone” Ambari Server
AmbariServer
HADOOPStore & Process
AmbariServer
Operators manage the cluster, may have Views deployed
Data Workers use the cluster and use a “standalone” Ambari Server for Views
![Page 92: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/92.jpg)
92 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Where Can I Learn More about Views?
https://cwiki.apache.org/confluence/display/AMBARI/Views https://github.com/apache/ambari/blob/trunk/ambari-views/docs/index.md https://github.com/apache/ambari/tree/trunk/ambari-views/examples https://github.com/apache/ambari/tree/trunk/contrib/views
![Page 93: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/93.jpg)
93 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What’s New and Next
![Page 94: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/94.jpg)
94 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What’s New in Ambari 2.4
Alerts: Retry tolerance (AMBARI-15686)
Alerts: Customizable SCRIPT Parameters (AMBARI-14898)
Alerts: New HDFS Alerts (AMBARI-14800)
New Host Page Filtering (AMBARI-15210)
Remove Service (AMBARI-14759)
Support for SLES 12 Technical Preview (AMBARI-16007)
Stability: Database Consistency Checking (AMBARI-16258)
Customizable Ambari Log + PID Dirs (AMBARI-15300)
New Version Registration Experience (AMBARI-15724)
Log Search Technical Preview (AMBARI-14927)
Operational Audit Logging (AMBARI-15241)
Role-Based Access Control (AMBARI-13977)
Automated Setup of Ambari Kerberos (AMBARI-15561)
Automated Setup of Ambari Proxy User (AMBARI-15561)
Customizable Host Reg. SSH Port (AMBARI-13450)
Core Features Security Features
AUGUST
View URLs (AMBARI-15821), View Refresh (AMBARI-15682)
Inherit Cluster Permissions (AMBARI-16177)
Remote Cluster Registration (AMBARI-16274)
Views Framework Features
![Page 95: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/95.jpg)
95 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What’s Next
Log Search GA (AMBARI-14927) Service and Component “auto-start” (AMBARI-2330) Ambari Server HA (Active-Passive + Load Balancer) (AMBARI-17126) Multi-service instances & versions (AMBARI-14714)
![Page 96: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/96.jpg)
96 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Learn More About Apache Ambari
Resource Location
Apache Ambari Project Page http://ambari.apache.org
Ambari Project Wiki https://cwiki.apache.org/confluence/display/AMBARI
Ambari Project JIRA https://issues.apache.org/jira/browse/AMBARI
Stacks https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=38571133
Blueprints https://cwiki.apache.org/confluence/display/AMBARI/Blueprints
Views https://cwiki.apache.org/confluence/display/AMBARI/Views
![Page 97: Apache Ambari: Past, Present, Future](https://reader036.vdocument.in/reader036/viewer/2022070516/587284231a28abc7068b6cc3/html5/thumbnails/97.jpg)
97 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank You