apache ambari - what's new in 2.4
TRANSCRIPT
![Page 1: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/1.jpg)
1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari 2.4.0What’s NewAugust 2016
![Page 2: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/2.jpg)
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What is Apache Ambari?
A completely open source management platform for provisioning, managing, monitoring and securing Apache Hadoop clusters. Apache Ambari takes the guesswork out of operating Hadoop.
![Page 3: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/3.jpg)
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What Ambari Does
Simplified Installation, Configuration and Management
Centralized Security Setup
Full Visibility into Cluster Health
Highly Extensible and Customizable
• Wizard-driven and automated cluster provisioning• Smart Configurations and Cluster Recommendations• Automated Rolling and Express cluster upgrades
• Reduce complexity to administer security across the platform• Automate setup Kerberos• Simplify the configuration of Apache Ranger
• Predefined alerts based on operational best practices• Advanced metrics visualization with Grafana
• Seamlessly fit into your enterprise environment• Bring custom Services under management via Ambari Stacks• Customize the UI with Ambari Views
![Page 4: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/4.jpg)
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What’s New in Ambari 2.4
Alerts: Customizable SCRIPT Parameters (AMBARI-14898)
Alerts: Retry Check Counts (AMBARI-15686)
Alerts: New HDFS Alerts (AMBARI-14800)
New Host Page Filtering (AMBARI-15210)
Remove Service (AMBARI-14759)
Support for SLES 12 Technical Preview (AMBARI-16007)
Stability: Database Consistency Checking (AMBARI-16258)
Customizable Ambari Log + PID Dirs (AMBARI-15300)
New Version Registration Experience (AMBARI-15724)
Log Search Technical Preview (AMBARI-14927)
Operational Audit Logging (AMBARI-15241)
Role-Based Access Control (AMBARI-13977)
Automated Setup of Ambari Kerberos (AMBARI-15561)
Automated Setup of Ambari Proxy User (AMBARI-15561)
Customizable Host Reg. SSH Port (AMBARI-13450)
Core Features Security Features
View URLs (AMBARI-15821), View Refresh (AMBARI-15682)
Inherit Cluster Permissions (AMBARI-16177)
Remote Cluster Registration (AMBARI-16274)
Views Framework Features
![Page 5: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/5.jpg)
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaWhat’s New in Ambari 2.4.0
Feature Highlights: Alert Retry Check Counts
![Page 6: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/6.jpg)
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alert Check Counts
Customize the number of times an alert is checked before dispatching a notification Avoid dispatching an alert notification (email, snmp) in case of transient issues
![Page 7: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/7.jpg)
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Configuring the Check Count
Set globally for all alerts, or override for a specific alert
Global Setting
Alert Override
![Page 8: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/8.jpg)
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
State Change Types
SOFT state changes do not perform a dispatch HARD state changes (to non-OK) perform dispatch Regardless of change:
– The Ambari Web UI will show the current state (OK/WARN/CRIT)– The state change is written to ambari-alerts.log
2016-05-31 13:20:52,294 [CRITICAL] [SOFT] [AMBARI_METRICS] [grafana_webui] (Grafana Web UI) Connection failed to http://c6401.ambari.apache.org:3000 (<urlopen error [Errno 111] Connection refused>)2016-05-31 13:22:52,290 [CRITICAL] [HARD] [AMBARI_METRICS] [grafana_webui] (Grafana Web UI) Connection failed to http://c6401.ambari.apache.org:3000 (<urlopen error [Errno 111] Connection refused>)
Note: check counts are not configurable for AGGREGATE alert types. All state changes are considered HARD.
![Page 9: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/9.jpg)
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Example: Check Count = 3
Check 1/3State: OK
Change: n/a
Check 1/3State: OK
Change: n/a
Check 1/3State: CRIT
Change: SOFT
Check 2/3State: CRIT
Change: n/a
Check 3/3State: CRIT
Change: HARD
Check 1/3State: OK
Change: HARD
DISPATCH
Check Interval Check Interval Check Interval Check Interval Check Interval
no state changestate changes to CRIT
performing multiple checks
back to OK
![Page 10: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/10.jpg)
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaWhat’s New in Ambari 2.4.0
Feature Highlights: Alert Customizable Params
![Page 11: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/11.jpg)
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alert Types and Thresholds
Ability to customize Thresholds for SCRIPT and SERVER alerts Ability to customize Connection Timeout for METRIC alerts
Alert Type Description Thresholds (units)
WEB Connects to a Web URL. Alert status is based on the HTTP response code. Response Code (n/a)Connection Timeout (seconds)
PORT Connects to a port. Alert status is based on response time. Response (seconds)
METRIC Checks the value of a service metric. Units vary, based on the metric being checked.
Metric Value (units vary)Connection Timeout (seconds)
AGGREGATE Aggregates the status for another alert. % Affected (percentage)
SCRIPT Executes a script to handle the alert check. Varies
SERVER Executes a server-side runnable class to handle the alert check. Varies
NEW!
![Page 12: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/12.jpg)
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alerts: Customizable METRIC Connection Timeout
Ability to set Connection Timeout threshold via Ambari Web UI
NEW!
![Page 13: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/13.jpg)
13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alerts: Customizable SCRIPT Thresholds
Ability to set various thresholds via Ambari Web UI
![Page 14: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/14.jpg)
14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Alerts: NEW!!! Ambari Server Performance Alert
Measures the Ambari Server REST API and Backend Database response
![Page 15: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/15.jpg)
15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaWhat’s New in Ambari 2.4.0
Feature Highlights: New HDFS Alerts
![Page 16: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/16.jpg)
16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
New HDFS Alerts Watch Trends
NameNode Client RPC Queue Latency (Hourly/Daily) NameNode Client RPC Processing Latency (Hourly/Daily) NameNode Service RPC Queue Latency (Hourly/Daily) NameNode Service RPC Processing Latency (Hourly/Daily) NameNode Heap Usage (Daily/Weekly) HDFS Storage Capacity Usage (Daily/Weekly)
![Page 17: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/17.jpg)
17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaWhat’s New in Ambari 2.4.0
Feature Highlights: New Host Filtering
![Page 18: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/18.jpg)
18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
New Host Filtering Control in Ambari Web
Ability to perform complex host filtering from Ambari Web Make it easier to find hosts
NEW!
![Page 19: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/19.jpg)
19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Search by Host Attribute, Service or Component
![Page 20: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/20.jpg)
20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Host Attribute Filtering
Host Name IP Host Status Cores RAM Stack Version + Version State Rack
![Page 21: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/21.jpg)
21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Service Filtering
![Page 22: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/22.jpg)
22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Component Filtering
![Page 23: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/23.jpg)
23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Host Filter: Examples
![Page 24: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/24.jpg)
24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaWhat’s New in Ambari 2.4.0
Feature Highlights: Remove Service
![Page 25: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/25.jpg)
25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Remove Service
Ability to perform Remove Service from Ambari Web Eliminates need to use Ambari REST API Checks for Service dependencies Service must be stopped All configuration information and history is also removed This operation is not reversible
NEW!
![Page 26: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/26.jpg)
26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaWhat’s New in Ambari 2.4.0
Feature Highlights: Other Items
![Page 27: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/27.jpg)
27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Customizable Ambari Log + PID Dirs (AMBARI-15300)
Ambari Server and Agents write log activity output to log files and use a PID-file that contains the process identification number (PID) for their running process.
Log Location PID Location
Ambari Server /var/log/ambari-server/ambari-server.log /var/run/ambari-server/ambari-server.pid
Ambari Agent /var/log/ambari-agent/ambari-agent.log /var/run/ambari-agent/ambari-agent.pid
![Page 28: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/28.jpg)
28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Customize Ambari Server Log + PID
vi /etc/ambari-server/conf/ambari.properties
pid.dir=/var/run/ambari-server
vi /etc/ambari-server/conf/log4j.properties
ambari.log.dir=${ambari.root.dir}/var/log/ambari-server
Ambari Server PID Ambari Server Log
1. Stop Ambari Server prior to modifying log or pid directories.2. You must manually create the new directories and be sure to set the directory
ownership + permissions to allow the Ambari Server process access.
![Page 29: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/29.jpg)
29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Customize Ambari Agent Log + PID
vi /etc/ambari-agent/conf/ambari-agent.ini
[agent]
logdir=/var/log/ambari-agent
piddir=/var/run/ambari-agent
1. Stop Ambari Agent prior to modifying log or pid directories.2. You must manually create the new directories and be sure to set the directory
ownership + permissions to allow the Ambari Agent process access.
![Page 30: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/30.jpg)
30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Customizable Host Registration SSH Port
Customize SSH Port when performing Host Registration automatically
NEW!
![Page 31: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/31.jpg)
31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Stability: Database Consistency Checking
On Ambari Server start, Ambari runs a database consistency check looking for issues. If any issues are found, Ambari Server start will abort and a message will be printed to
console “DB configs consistency check failed.” Check Ambari Server log file for more details:/var/log/ambari-server/ambari-server-check-database.log
Ability to “skip” check and force Ambari Server startambari-server start --skip-database-check
Important: if you “skip” the check to force Ambari Server start, do not make any changes to your cluster topology or perform a cluster upgrade until you correct
the database consistency issues.
![Page 32: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/32.jpg)
32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaWhat’s New in Ambari 2.4.0
Feature Highlights: View Framework Enhancements
![Page 33: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/33.jpg)
33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
View URLs (AMBARI-15821)
Ability to create a “short URL” or “vanity URL” for view instances Provide users with a non-version or instance specific URL to a view
/#/main/views/{viewName}/{viewVersion}/{viewInstanceName}/#/main/view/{viewName}/{shortURL}
![Page 34: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/34.jpg)
34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
View Refresh (AMBARI-15682)
Automatically deploy new views into Ambari Server w/o a restart
1. Copy view archive to: /var/lib/ambari-server/resources/views/
2. Ambari Server detects the new view, automatically extracts + deploys
3. View is available for creating instances
4. Click “Refresh” in Views UI
![Page 35: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/35.jpg)
35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Remote Cluster ConfigurationAMBARI-16274
![Page 36: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/36.jpg)
36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: View <-> Cluster Communication
Deployed Views “talk” with cluster using REST APIs (as applicable)
CLUSTER
ATS
RM
Ambari Server
Tez UIView
Tez UI View talks with cluster using
REST APIs toATS and ResourceManager
Ambari DB
LDAPAuthN
![Page 37: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/37.jpg)
37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Operational vs. Standalone Ambari Server
Ambari Agent
Host
Ambari Agent
Host
Ambari Agent
Host
Standalone Ambari ServerOne or More Ambari Server Instances
No Agents, no requirement to operate the cluster
Operational AmbariOne Ambari Server Instance
Talking with Agents, Managing the cluster
Ambari Server
Ambari DB
LDAPAuthN
Ambari Server
Ambari DB
LDAPAuthN
![Page 38: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/38.jpg)
38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Local Cluster vs. Non-Local
Ambari Server
Ambari DB
LDAPAuthN
Ambari Server
Ambari DB
LDAPAuthN
Standalone Ambari ServerOne or More Ambari Server Instances
No Agents, no requirement to operate the cluster
Operational AmbariOne Ambari Server Instance
Talking with Agents, Managing the cluster
LOCALCLUSTER
NON-LOCAL
CLUSTER
![Page 39: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/39.jpg)
39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Introducing Remote Cluster Configuration (AMBARI-16274)Option DescriptionLocal Cluster When you select this Local Cluster option, Ambari will automatically determine the
cluster configuration properties needed for the view instance.Criteria:• Ambari Server running the views is also managing the cluster
Remote Cluster When you select Remote Cluster option, Ambari will automatically determine the cluster configuration properties needed for the view instance.Criteria:• The cluster is not local to the Ambari Server running the views (i.e. Standalone)• Cluster is being managed by Ambari
Custom When you select Custom option, you must enter all configuration information, and are responsible for updating if the cluster configuration changes.Criteria:• The cluster running the view is not local to the Ambari Server• The cluster is not being managed by Ambari
NEW!
![Page 40: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/40.jpg)
40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Local vs Remote View Configuration
Ambari Server
Views
Cluster
Ambari Server
Views
ClusterAmbari Server
LOCAL CLUSTER
REMOTE CLUSTER
Operational Ambari
Manages cluster
Standalone Ambari
Manages cluster
Talks to cluster
Obtains view config
Obtains view config
Talks to cluster
Operational Ambari
![Page 41: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/41.jpg)
41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
View Configuration: Minimizing Need for Custom
Cluster Config Ambari Server Cluster Mgmt Ambari 2.2 or Earlier Ambari 2.4
No HA, No Kerberos Operational Ambari Local Local
HA or Kerberos Operational Ambari Custom Local
No HA, No Kerberos Standalone Ambari Custom Remote
HA or Kerberos Standalone Ambari Custom Remote
No HA, No Kerberos Standalone Non-Ambari Custom Custom
HA or Kerberos Standalone Non-Ambari Custom Custom
![Page 42: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/42.jpg)
42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Inherit Cluster PermissionsAMBARI-16177
![Page 43: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/43.jpg)
43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Inherit Cluster Permissions (AMBARI-16177)
Ability to automatically grant View “Use” permission based on Cluster role Note: Option is only available when using a Local Cluster Configuration
Explicitly grant users and groups Use
permission
Automatically grant users and groups Use permission based on
Cluster roles
![Page 44: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/44.jpg)
44 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaWhat’s New in Ambari 2.4.0
Feature Highlights: Log Search
TECH PREVIEW
![Page 45: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/45.jpg)
45 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Log Search
Solr
A M B A R ILog
Search
Search Cluster Component Logs from within Ambari
Goal: When issues arise, be able to quickly find issues across all components
⬢ Capabilities– Rapid Search of all cluster component logs– Search across time ranges, log levels, and for keywords
⬢ Core Technologies: – Apache Ambari– Apache Solr– Apache Ambari Log Search
Tech Preview
![Page 46: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/46.jpg)
46 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Log Search Architecture
A M B A R I
L O G F E E D E R
L O G F E E D E R
L O GF E E D E R
L O GF E E D E R
L O G F E E D E R
L O G F E E D E R
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
W O R K E RN O D E
Solr
L O G S E A R C H
U I
Tech Preview
![Page 47: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/47.jpg)
47 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Log Search Details
W O R K E RN O D E
L O G F E E D E R
Solr
L O G S E A R C H
U I
Solr
Solr
A M B A R I
Java ProcessMulti-output SupportGrok
Solr CloudLocal Disk StorageTTL
Tech Preview
![Page 48: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/48.jpg)
48 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Considerations
Log Feeders are CPU intensive, consider 1 dedicated core Solr instances should use dedicated hardware with at least 32GB of RAM dedicated to
the Solr instance By default, logs will age out after 7 days
Tech Preview
![Page 49: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/49.jpg)
49 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaWhat’s New in Ambari 2.4.0
Feature Highlights: RBAC
![Page 50: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/50.jpg)
50 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
New Role Based Access Control
Introducing new “roles” for more granular division of control for cluster operations
Old Permission New Role Notable Permissions
Operator Cluster Administrator Full operational control, including upgrades. Ambari Admins are implicitly granted this Role.
Cluster Operator Adding and removing hosts.
Service Administrator Manage configurations, move components.
Service Operator Service stop and start and service-specific operations such as HDFS Rebalance.
Read-Only Cluster User View cluster service and host information.
Note: Users flagged as “Ambari Administrators / Ambari Admins” are implicitly granted Cluster Administrator permission.
![Page 51: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/51.jpg)
51 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Managing Cluster RolesAssign roles to users or groups
Manage roles in Block or List View
layouts
![Page 52: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/52.jpg)
52 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Managing Cluster Roles View users or groups
Change current role assignment
![Page 53: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/53.jpg)
53 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaWhat’s New in Ambari 2.4.0
Feature Highlights: Security Enhancements
![Page 54: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/54.jpg)
54 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Summary of Security Enhancements
Automatic Setup of Ambari Server as a Proxyuser Automatic Setup of Ambari Server for Kerberos
![Page 55: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/55.jpg)
55 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automatic Setup of Ambari Server as a Proxyuser
![Page 56: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/56.jpg)
56 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Proxyusers
HDFS and WebHCat (as part of Hive) support the concept of a Proxyuser Proxyuser allows UserA to access the service on behalf of UserB (i.e. the proxyuser is
allowed to impersonate other users) Proxyuser is a commonly used capability of Hadoop
HDFS
“UserA” is setup as a proxyuser
UserA can access HDFS as “UserA” on behalf of “UserB”
HDFS ops performed are as “UserB”
![Page 57: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/57.jpg)
57 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: HDFS Proxyuser Setup
A proxyuser needs to be configured in core-site.xml configuration:hadoop.proxyuser.{proxyuser-name}.hosts
hadoop.proxyuser.{proxyuser-name}.groups
If these settings are not present, impersonation will not be allowed and connection to the service via proxyuser will fail
![Page 58: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/58.jpg)
58 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Ambari + Proxyuser
Ambari Views use proxyuser to access the cluster (such as Hive View and Pig View) Ambari Server needs to access a service on behalf of an authenticated user
Ambari Server HDFS
(running as user “ambari”)
“joe” authenticates to Ambari
(setup for proxyuser “ambari”)hadoop.proxyuser.ambari.hosts=*hadoop.proxyuser.ambari.groups=*
Ambari Server can talk to HDFS as “ambari” proxyuser on behalf of “joe”
Configuration of proxyuser is commonly “missed” when setting up Ambari Views
![Page 59: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/59.jpg)
59 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
New: Automatic Ambari Server Proxyuser Setup
Proxyuser configurations are automatically added for HDFS and WebHCat For example: Ambari Server as running as “ambari”, the following configurations are
added during HDFS service installhadoop.proxyuser.ambari.hosts
hadoop.proxyuser.ambari.groups
![Page 60: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/60.jpg)
60 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Automatic Setup of Ambari Server for Kerberos
![Page 61: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/61.jpg)
61 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Hadoop + Kerberos
Strongly authenticating and establishing a user’s identity is the basis for secure access in Hadoop. Users need to be able to reliably “identify” themselves and then have that identity propagated throughout the Hadoop cluster.
Once this is done, those users can access resources (such as files or directories) or interact with the cluster (like running MapReduce jobs).
Besides users, Hadoop cluster resources themselves (such as Hosts and Services) need to authenticate with each other to avoid potential malicious systems or daemon’s “posing as” trusted components of the cluster to gain access to data.
![Page 62: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/62.jpg)
62 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Hadoop + Kerberos
Service Component
A
Service Component
B
Hadoop Cluster
KDC
keytabkeytab
Service Component
C
keytab
Service Component
D
keytab
Service Component
X
Service Component
X
keytabkeytab
Service Component
X
keytab
Service Component
X
keytab
Kerberos is used to secure the
Components in the cluster. Kerberos
identities are managed via
“keytabs” on the Component hosts.
Principals for the
cluster are managed in
the KDC.
![Page 63: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/63.jpg)
63 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Automated Kerberos Setup with Ambari
![Page 64: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/64.jpg)
64 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Principal and Keytab Generation & Distribution
1. User provides KDC Admin Account credentials to Ambari
2. Ambari connects to KDC, creates principals (Service and Ambari) needed for cluster
3. Ambari generates keytabs for the principals
4. Ambari distributes keytabs to Ambari Server and cluster hosts
5. Ambari discards the KDC Admin Account credentials (optional)
AmbariServer KDC
1 2
4
3
5
Cluster
![Page 65: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/65.jpg)
65 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Ambari + Hadoop + Kerberos
Ambari Server communicates with the cluster to retrieve information (such as metrics) Especially important for Ambari Views (e.g. Files, Hive, Pig) Therefore: Ambari Server ALSO needs to be “setup for Kerberos”
Ambari Server Cluster
Kerberos enabled
![Page 66: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/66.jpg)
66 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Background: Manual Setup of Ambari Server for Kerberos
Manual setup of Ambari Server for Kerberos (outside of “Enable Kerberos” wizard):
1. Create principal for Ambari Server
2. Generate keytab for Ambari Server
3. Place keytab on Ambari Server host
4. Run “ambari-server setup-security” on Ambari Server
5. Restart Ambari Server
Configuration of Ambari Server for Kerberos is commonly “missed” when setting up Ambari Views
![Page 67: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/67.jpg)
67 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
New: Automatic Setup of Ambari Server for Kerberos
When enabling Kerberos and choosing an automated option (MIT or AD), Ambari Server will be setup for Kerberos automatically:
1. A principal will be created for Ambari Server
2. A keytab will be generated and placed on Ambari Server
3. Ambari Server is setup for Kerberos
Note: you will still need to perform the Ambari Server restart for the Kerberos identity to get picked-up by Ambari.
![Page 68: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/68.jpg)
68 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
What about Proxyuser + Kerberos?
![Page 69: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/69.jpg)
69 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
New: Automatic Proxyuser Setup with Kerberos
When a cluster has Kerberos enabled, the proxyuser needs to be configured based on the primary part of the Kerberos principal name
hadoop.proxyuser.{principal-name-primary}.hosts
hadoop.proxyuser.{principal-name-primary}.groups
Ambari will adjust proxyuser configurations during Kerberos setup
Ambari Server HDFS
(running as user “ambari”)(setup with principal “[email protected]”
“joe” authenticates to Ambari
(setup for proxyuser “ambari-server”)hadoop.proxyuser.ambari-server.hosts=*hadoop.proxyuser.ambari-server.groups=*
Ambari Server can talk to HDFS as “ambari-server” proxyuser on behalf
of “joe”
![Page 70: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/70.jpg)
70 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaWhat’s New in Ambari 2.4.0
Feature Highlights: Ops Audit Logging
![Page 71: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/71.jpg)
71 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Operational Audit Logging
Ambari will create entries in an audit log as Ambari + Cluster operations are performed Using the audit log, you can determine who performed the operation and when the
operation was performed as well as other operation-specific information The Ambari Audit log can be found at: /var/log/ambari-server/ambari-audit.log
![Page 72: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/72.jpg)
72 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
List of Operations
Stop/Start Service Stop all Services Add Service Move Component Turn On/Off Maintenance Mode Download Client Configurations Blueprint Export Update Configuration **
Login (success/failed) / Logout Create User, Group Delete User, Group Change Group Membership Change User Status, Admin Change User Password Grant/Revoke User, Group Cluster Roles
Service Operations User Operations
** Note: When a Service Configuration change is made, an entry is also written to a specific log file ambari-config-changes.log for configuration changes that provides even more detail on the change.
![Page 73: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/73.jpg)
73 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
List of Operations (continued)
Add/Remove Host Enable/Disable/Edit Alert Add/Update/Delete Alert Group Add/Upgrade/Delete Notification Enable/Disable Kerberos Regenerate Kerberos Keytabs Rename Cluster Add/Remove Remote Clusters
Register/Deregister Version Cluster Upgrade
Cluster Operations Upgrade Operations
Create/Delete View Instance Edit View Instance Grant/Revoke View Permissions Create/Delete View URLs
View Operations
![Page 74: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/74.jpg)
74 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Example: Change Group Membership
Add/Remove group members creates a “Membership change” audit entry
2016-06-02T23:12:09.930Z, User(admin), RemoteIp(192.168.64.1), Operation(Membership change), RequestType(PUT), url(http://c6401.ambari.apache.org:8080/api/v1/groups/customgroup/members), ResultStatus(200 OK), Group(customgroup), Members(joeuser)2016-06-02T23:12:34.700Z, User(admin), RemoteIp(192.168.64.1), Operation(Membership change), RequestType(PUT), url(http://c6401.ambari.apache.org:8080/api/v1/groups/customgroup/members), ResultStatus(200 OK), Group(customgroup), Members(joeuser, mike)
![Page 75: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/75.jpg)
75 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Example: Stop ZooKeeper
A single operation (like “Stop ZooKeeper”) might generate multiple audit entries Relate entries via RequestId()
2016-06-02T23:14:35.206Z, User(admin), RemoteIp(192.168.64.1), Operation(INSTALLED: ZOOKEEPER_SERVER/ZOOKEEPER on c6401.ambari.apache.org (MyCluster)), Host name(c6401.ambari.apache.org), RequestId(7), Status(Successfully queued)2016-06-02T00:31:56.016Z, User(admin), Operation(Stop ZooKeeper Server), Status(IN_PROGRESS), RequestId(7)2016-06-02T00:31:56.025Z, User(admin), Operation(STOP ZOOKEEPER_SERVER), Status(QUEUED), RequestId(7), TaskId(52), Hostname(c6401.ambari.apache.org)2016-06-02T00:31:57.370Z, User(admin), Operation(Stop ZooKeeper Server), Status(COMPLETED), RequestId(7)2016-06-02T00:31:57.370Z, User(admin), Operation(STOP ZOOKEEPER_SERVER), Status(COMPLETED), RequestId(7), TaskId(52), Hostname(c6401.ambari.apache.org)
![Page 76: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/76.jpg)
76 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
AgendaWhat’s New in Ambari 2.4.0
Feature Highlights: Version Registration Experience
![Page 77: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/77.jpg)
77 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Introducing the Version Definition File (VDF)
This is a meta file describing which Services are included and at which version
![Page 78: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/78.jpg)
78 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Ambari will “discover” Available Versions
Tabs for list of available Stacks
List of discovered Versions
List of Services w/version #
![Page 79: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/79.jpg)
79 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
“Default Version Definition” for Backwards Compat
Ambari provides a “default” Version
Definition.
![Page 80: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/80.jpg)
80 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Add New Version via File Upload or URL
![Page 81: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/81.jpg)
81 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Changes in Install / Version Registration Flow
Scenario Ambari 2.4 ChangeInternet Access / Public Repositories No change.
No Internet Access / Local repositories - Upload a VDF for the Local Repository you created- Set the Local Repository URLs
OR
- Choose the Default Version Definition- Set the Local Repository URLs
![Page 82: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/82.jpg)
82 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Other UX Changes: Local vs. Public Repository Radio
Explicit Choice
![Page 83: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/83.jpg)
83 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Other UX Changes: Local vs. Public Repository Radio
Choose Local
Must enter Base URLs
![Page 84: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/84.jpg)
84 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Other UX Changes: OS Add/Remove
![Page 85: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/85.jpg)
85 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Other UX Changes: RedHat Satellite/Spacewalk
Explicit Choice
- Ambari will not write the .repo files
- User must register the repositories channels via Satellite
![Page 86: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/86.jpg)
86 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Other UX Changes: Viewing, Install and Upgrade
![Page 87: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/87.jpg)
87 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Other UX Changes: Managing Versions
![Page 88: Apache Ambari - What's New in 2.4](https://reader034.vdocument.in/reader034/viewer/2022052514/587284441a28abc7068b6d15/html5/thumbnails/88.jpg)
88 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank You