apache hadoop security - ranger

25
Page 1 © Hortonworks Inc. 2011 – 2014. All Rights Reserved Apache Hadoop Security: Ranger Sep 16, 2015 Madhan Neethiraj

Upload: isheeta-sanghi

Post on 15-Apr-2017

913 views

Category:

Technology


5 download

TRANSCRIPT

Page 1: Apache Hadoop Security - Ranger

Page 1 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Apache Hadoop Security: Ranger

Sep 16, 2015

Madhan Neethiraj

Page 2: Apache Hadoop Security - Ranger

Page 2 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Agenda

Control access into

system

Flexibility in defining

policies

•Authorization & Auditing with Ranger•Centralized security administration for HDFS, Hive, HBase, Knox, Strom, YARN, Kafka, Solr, ..•Audit logs to Solr, HDFS, RDBMS, Log4j, ..•Extensible Architecture – custom conditions, context enrichers, easier addition of new components

Page 3: Apache Hadoop Security - Ranger

Page 3 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Security in Hadoop

AuthenticationAuthenticate users and systems Apache Knox, Native Kerberos

AuthorizationProvision access to data Apache Ranger

AuditMaintain a record of data access

Apache Ranger, Hadoop native audit

Data ProtectionProtect data at rest and in motion

HDFS encryption + Ranger KMS, Vendor solutions

AdministrationCentral management & consistent security Apache Ranger

Page 4: Apache Hadoop Security - Ranger

Page 4 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Authorization and Auditing with Ranger

HBase

Ranger Administration Portal

HDFS

Hive Server2

Ranger Policy Store

Ranger Audit Store

Ranger Plugin

Hadoop Components

Enterprise Users

Log4j

Knox

Storm

RDBMS

YARN

Kafka

Solr

HDFS

Solr

Ranger Plugin

Ranger Plugin

Ranger Plugin

Ranger Plugin

Ranger Plugin

Ranger Plugin

Ranger Plugin

Page 5: Apache Hadoop Security - Ranger

Page 5 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Central Security AdministrationApache Ranger• Delivers a ‘single pane of glass’

for the security administrator• Centralizes administration of

security policy• Ensures consistent coverage

across the entire Hadoop stack

Page 6: Apache Hadoop Security - Ranger

Page 6 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Ranger AuthorizationRanger Plugins authorize access to resources in following Hadoop components:Component Resources Access Types

HDFS Files/Directories Read, Write, ExecuteHive Databases, Tables, Columns Create, Alter, Drop, Select, Update, All

HBase Tables, Column-Families, Columns Read, Write, Create, Admin

Knox Topologies, Services AllowStorm Topologies Topology:

submit/activate/deactivate/reblance/kill/get/get-info/get-user/get-conf, File: upload/download, Get Nimbus Conf

YARN Queues Submit-application, Admin-queue

Kafka Topics Publish, Consume, Configure, Describe, Admin

Solr Collections Query, Update, Others, Admin

Page 7: Apache Hadoop Security - Ranger

Page 7 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Ranger Auditing• Ranger plugins generate detailed audit logs for accesses to protected resources.

Audit logs include details like: user, resource, type of access, time of access, client IP address, access-result, ID of the policy that allowed/denied the access

• Audit logs to one or more destinations – Solr, HDFS, RDBMS, Log4j, ...

• Interactive view of audit logs using Ranger Admin

Page 8: Apache Hadoop Security - Ranger

Page 8 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Ranger Policy - HiveAllow Marketing group users ‘select’ access on few columns in customer_details table

Page 9: Apache Hadoop Security - Ranger

Page 9 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Ranger Policy - HDFSAllow Marketing group users to access /demo/data/Customer* directories and files

Page 10: Apache Hadoop Security - Ranger

Page 10 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Ranger Policy - KafkaAllow Marketing group users to access Marketing topic

Page 11: Apache Hadoop Security - Ranger

Page 11 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Extensible Architecture

Page 12: Apache Hadoop Security - Ranger

Page 12 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Extensibility: Ranger Stacks

• Customers and partners can easily add Ranger authorization and auditing support for new components

• Describe component details (like resource structure, access-types) in JSON and register with Ranger

• Implement component authorizer to authorize resource accesses using Ranger policy engine

• Ranger Admin provides UI for policy administration, based on component details in registered JSON

Page 13: Apache Hadoop Security - Ranger

Page 13 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Extensibility: Ranger Stacks - example

Page 14: Apache Hadoop Security - Ranger

Page 14 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Extensibility: Dynamic Policy Conditions

• Provides ability to evaluate custom conditions to drive authorization decisions

• Custom conditions can evaluate various data available in the request – like user, groups, resource, IP-address, context, etc.

• Register custom conditions via component description JSON

• Ranger Admin provides UI to specify condition valuesto be satisfied

Allow accesses from 10.0.2.* IP addresses only!

Page 15: Apache Hadoop Security - Ranger

Page 15 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Extensibility: Dynamic Policy Conditions - sample• Register the custom condition in the component description JSON:

• Implement the custom condition and make it available to Ranger plugin:

• Ranger Policy Engine will call the custom condition while evaluating policies

Page 16: Apache Hadoop Security - Ranger

Page 16 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Extensibility: Context Enrichers

• Provides ability to add context data to access requests

• Context data added can be used by condition evaluators to drive authorization decisions

• An example: from the client IP address in the request, a context enricher adds location data (like COUNTRY, STATE, CITY, AREA-CODE) to the request context. A custom condition can then restrict access depending upon the location data in the context.

• Context enrichers should be specified in component description JSON

Page 17: Apache Hadoop Security - Ranger

Page 17 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Extensibility: Context Enrichers - sample• Register the context enrichers in the component description JSON:

• Implement the context enricher and make it available to Ranger plugin:

• Ranger Policy Engine will call all registered context enrichers before evaluating policies

Page 18: Apache Hadoop Security - Ranger

Page 18 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Extensibility: Context Enricher + Condition - sample• Implement a custom condition that verifies that the access is from specified countries only:

• Register the custom condition in the component description JSON

• On receiving authorization request, Ranger Policy Engine calls LocationDataProviderEnricher enricher, which adds location data to the request.

• When evaluating policies, Ranger Policy Engine calls LocationCountryCondition, which allows accesses only from the countries specified in the policy

• Ranger Policy Engine will call all registered context enrichers before evaluating policies

Page 19: Apache Hadoop Security - Ranger

Page 19 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

In Development: allow/deny/exceptions in policies

• Ability to explicitly deny access to resources• Ability to allow/deny access to a wider group, like employees/public,

but specify exceptions to a subset, like part-time employees/vendors/ip-addresses, etc.

• Policy evaluation order:• All deny-policies for the resource are evaluated first• If the request matches a deny-policy, and not its deny-exceptions, access will be denied• If the request is not denied by deny-policies, allow-policies will be evaluated• If the request matches an allow-policy, and not its allow-exceptions, access will be allowed

• Development in tag-policy branch of Apache Ranger

Page 20: Apache Hadoop Security - Ranger

Page 20 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Features under Development..

Page 21: Apache Hadoop Security - Ranger

Page 21 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

In Development: allow/deny/exceptions in policies

allow access to finance group

and falcon user

deny access from outside of Switzerland

for everyone,except falcon user

Policy to:- deny access from outside Switzerland to everyone, except falcon user- allow falcon user to access from anywhere- allow finance group users to access from Switzerland only

Page 22: Apache Hadoop Security - Ranger

Page 22 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

In Development: tag-based policies

• Ability authorize access based on tags associated with resources• A single tag-based policy, like for PII tag, to authorize access to

resources across components – like HDFS, Hive, HBase, ..• Available to all components that use Ranger authorization• Similar policy structure as existing resource-based policies• API to integrate with tag providers – like Apache Atlas• Development in tag-policy branch of Apache Ranger

Page 23: Apache Hadoop Security - Ranger

Page 23 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

In Development: tag-based policiesPolicy to authorize access to resources tagged as PII, in HDFS/Hive/HBase/Kafka/Solr, only to audit users

Page 24: Apache Hadoop Security - Ranger

Page 24 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Apache Ranger: how to contribute?

• Ranger Home Page - http://ranger.incubator.apache.org

• Ranger Wiki - https://cwiki.apache.org/confluence/display/RANGER

• Ranger JIRAs - https://issues.apache.org/jira/browse/RANGER

• Project Mailing Lists• Users: [email protected]• Developers: [email protected]• Commits: [email protected]

Page 25: Apache Hadoop Security - Ranger

Page 25 © Hortonworks Inc. 2011 – 2014. All Rights Reserved

Q&A…Discussion