apache hadoop security - ranger
TRANSCRIPT
Page 1 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Apache Hadoop Security: Ranger
Sep 16, 2015
Madhan Neethiraj
Page 2 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Agenda
Control access into
system
Flexibility in defining
policies
•Authorization & Auditing with Ranger•Centralized security administration for HDFS, Hive, HBase, Knox, Strom, YARN, Kafka, Solr, ..•Audit logs to Solr, HDFS, RDBMS, Log4j, ..•Extensible Architecture – custom conditions, context enrichers, easier addition of new components
Page 3 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Security in Hadoop
AuthenticationAuthenticate users and systems Apache Knox, Native Kerberos
AuthorizationProvision access to data Apache Ranger
AuditMaintain a record of data access
Apache Ranger, Hadoop native audit
Data ProtectionProtect data at rest and in motion
HDFS encryption + Ranger KMS, Vendor solutions
AdministrationCentral management & consistent security Apache Ranger
Page 4 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Authorization and Auditing with Ranger
HBase
Ranger Administration Portal
HDFS
Hive Server2
Ranger Policy Store
Ranger Audit Store
Ranger Plugin
Hadoop Components
Enterprise Users
Log4j
Knox
Storm
RDBMS
YARN
Kafka
Solr
HDFS
Solr
Ranger Plugin
Ranger Plugin
Ranger Plugin
Ranger Plugin
Ranger Plugin
Ranger Plugin
Ranger Plugin
Page 5 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Central Security AdministrationApache Ranger• Delivers a ‘single pane of glass’
for the security administrator• Centralizes administration of
security policy• Ensures consistent coverage
across the entire Hadoop stack
Page 6 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Ranger AuthorizationRanger Plugins authorize access to resources in following Hadoop components:Component Resources Access Types
HDFS Files/Directories Read, Write, ExecuteHive Databases, Tables, Columns Create, Alter, Drop, Select, Update, All
HBase Tables, Column-Families, Columns Read, Write, Create, Admin
Knox Topologies, Services AllowStorm Topologies Topology:
submit/activate/deactivate/reblance/kill/get/get-info/get-user/get-conf, File: upload/download, Get Nimbus Conf
YARN Queues Submit-application, Admin-queue
Kafka Topics Publish, Consume, Configure, Describe, Admin
Solr Collections Query, Update, Others, Admin
Page 7 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Ranger Auditing• Ranger plugins generate detailed audit logs for accesses to protected resources.
Audit logs include details like: user, resource, type of access, time of access, client IP address, access-result, ID of the policy that allowed/denied the access
• Audit logs to one or more destinations – Solr, HDFS, RDBMS, Log4j, ...
• Interactive view of audit logs using Ranger Admin
Page 8 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Ranger Policy - HiveAllow Marketing group users ‘select’ access on few columns in customer_details table
Page 9 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Ranger Policy - HDFSAllow Marketing group users to access /demo/data/Customer* directories and files
Page 10 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Ranger Policy - KafkaAllow Marketing group users to access Marketing topic
Page 11 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Extensible Architecture
Page 12 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Extensibility: Ranger Stacks
• Customers and partners can easily add Ranger authorization and auditing support for new components
• Describe component details (like resource structure, access-types) in JSON and register with Ranger
• Implement component authorizer to authorize resource accesses using Ranger policy engine
• Ranger Admin provides UI for policy administration, based on component details in registered JSON
Page 13 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Extensibility: Ranger Stacks - example
Page 14 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Extensibility: Dynamic Policy Conditions
• Provides ability to evaluate custom conditions to drive authorization decisions
• Custom conditions can evaluate various data available in the request – like user, groups, resource, IP-address, context, etc.
• Register custom conditions via component description JSON
• Ranger Admin provides UI to specify condition valuesto be satisfied
Allow accesses from 10.0.2.* IP addresses only!
Page 15 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Extensibility: Dynamic Policy Conditions - sample• Register the custom condition in the component description JSON:
• Implement the custom condition and make it available to Ranger plugin:
• Ranger Policy Engine will call the custom condition while evaluating policies
Page 16 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Extensibility: Context Enrichers
• Provides ability to add context data to access requests
• Context data added can be used by condition evaluators to drive authorization decisions
• An example: from the client IP address in the request, a context enricher adds location data (like COUNTRY, STATE, CITY, AREA-CODE) to the request context. A custom condition can then restrict access depending upon the location data in the context.
• Context enrichers should be specified in component description JSON
Page 17 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Extensibility: Context Enrichers - sample• Register the context enrichers in the component description JSON:
• Implement the context enricher and make it available to Ranger plugin:
• Ranger Policy Engine will call all registered context enrichers before evaluating policies
Page 18 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Extensibility: Context Enricher + Condition - sample• Implement a custom condition that verifies that the access is from specified countries only:
• Register the custom condition in the component description JSON
• On receiving authorization request, Ranger Policy Engine calls LocationDataProviderEnricher enricher, which adds location data to the request.
• When evaluating policies, Ranger Policy Engine calls LocationCountryCondition, which allows accesses only from the countries specified in the policy
• Ranger Policy Engine will call all registered context enrichers before evaluating policies
Page 19 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
In Development: allow/deny/exceptions in policies
• Ability to explicitly deny access to resources• Ability to allow/deny access to a wider group, like employees/public,
but specify exceptions to a subset, like part-time employees/vendors/ip-addresses, etc.
• Policy evaluation order:• All deny-policies for the resource are evaluated first• If the request matches a deny-policy, and not its deny-exceptions, access will be denied• If the request is not denied by deny-policies, allow-policies will be evaluated• If the request matches an allow-policy, and not its allow-exceptions, access will be allowed
• Development in tag-policy branch of Apache Ranger
Page 20 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Features under Development..
Page 21 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
In Development: allow/deny/exceptions in policies
allow access to finance group
and falcon user
deny access from outside of Switzerland
for everyone,except falcon user
Policy to:- deny access from outside Switzerland to everyone, except falcon user- allow falcon user to access from anywhere- allow finance group users to access from Switzerland only
Page 22 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
In Development: tag-based policies
• Ability authorize access based on tags associated with resources• A single tag-based policy, like for PII tag, to authorize access to
resources across components – like HDFS, Hive, HBase, ..• Available to all components that use Ranger authorization• Similar policy structure as existing resource-based policies• API to integrate with tag providers – like Apache Atlas• Development in tag-policy branch of Apache Ranger
Page 23 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
In Development: tag-based policiesPolicy to authorize access to resources tagged as PII, in HDFS/Hive/HBase/Kafka/Solr, only to audit users
Page 24 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Apache Ranger: how to contribute?
• Ranger Home Page - http://ranger.incubator.apache.org
• Ranger Wiki - https://cwiki.apache.org/confluence/display/RANGER
• Ranger JIRAs - https://issues.apache.org/jira/browse/RANGER
• Project Mailing Lists• Users: [email protected]• Developers: [email protected]• Commits: [email protected]
Page 25 © Hortonworks Inc. 2011 – 2014. All Rights Reserved
Q&A…Discussion