apache argus - how do i secure my entire hadoop cluster? olivier renault @ hortonworks

© Hortonworks Inc. 2011 – 2014. All Rights Reserved

Apache Argus Olivier RENAULT


Apache Argus: History

XASecure created in 2013

Hortonworks acquires XASecure in Mid-May 2014

Hortonworks fill Apache Argus proposal – mid July 2014

Can get the bits from:

-  hortonworks.com

-  http://argus.incubator.apache.org/


Security needs are changing

Administration Centrally management & consistent security

Authentication Authenticate users and systems

Authorization Provision access to data

Audit Maintain a record of data access

Data Protection Protect data at rest and in motion

Security needs are changing •  YARN unlocks the data lake •  Multi-tenant: Multiple applications for data access

•  Changing and complex compliance environment

•  ETL of non-sensitive data can yield sensitive data

Summer 2014 65% of clusters host multiple workloads

Fall 2013 Largely silo’d deployments with single workload clusters

5 areas of security focus


Security in Hadoop

Authorization Restrict access to explicit data

Audit Understand who did what

Data Protection Encrypt data at rest & in motion

•  Kerberos in native Apache Hadoop

•  HTTP/REST API Secured with Apache Knox Gateway

•  HDFS Permissions, HDFS ACL, •  Audit logs in with HDFS & MR •  Hive ATZ-NG

Authentication Who am I/prove it?

•  Wire encryption in Hadoop

•  Open Source Initiatives

•  Partner Solutions

•  HDFS, Hive and Hbase

•  Fine grain access control

•  RBAC

•  Centralized audit reporting

•  Policy and access history

•  Future Integration

HD

P 2.

1 A

rgus

Centralized Security Administration

•  As-Is, works with current authentication methods


Central Security Administration •  Delivers a ‘single pane of glass’ for

the security administrator •  Centralizes administration of

security policy •  Ensures consistent coverage across

the entire Hadoop stack


Setup Authorization Policies

6

file level access control, flexible definition

Control permissions


Monitor through Auditing

7


HDFS


What it means: HDFS API for Authorization

Today •  HDFS authorization is performed by JavaAgent based code

injection into namenode Tomorrow

•  Pluggable HDFS authorization is being added (HDFS-6826) •  Argus will replace the JavaAgent based code injection with a

custom authorization plugin •  Work being discussed currently


Hive


Hive Integration – Today

•  XA Secure/Argus uses multiple hooks in Hive hive.security.authorization.manager=com.xasecure.authorization.hive.authorizer.XaSecureAuthorizer hive.semantic.analyzer.hook=com.xasecure.authorization.hive.hooks.XaSecureSemanticAnalyzerHook hive.exec.post.hooks=com.xasecure.authorization.hive.hooks.XaSecureHivePostExecuteRunHook

– Not all information necessary to make authorization decision are available in Hive authorizer hooks

•  Local Grant/Revoke permission not integrated with Argus •  Storage based authorization only looks at POSIX

permissions


What it means: Tomorrow

12

•  New plug-in model in Hive to support external authorizers •  All information necessary to make authorization decision are provided to

authorizer plug-in •  XASecure/Argus Hive agent registers a single hook with Hive for

authorization hive.security.authorization.manager=com.xasecure.authorization.hive.authorizer.XaSecureHiveAuthorizerFactory


Integrate Grant/Revoke - Tomorrow

13

•  Integrate Grant/Revoke permissions •  New Hive Plugin enables Argus to handle Grant/Revoke permission •  Argus will store Grant/Revoke policy and enforce it, with auditing •  Option to disable Grant/Revoke •  Group/Roles mapped to Groups in Argus Admin


Storage Based Authorization - Tomorrow

•  In SBA, Hive used HDFS permissions for allowing operations

•  HDFS Permission Check •  Hive uses RPC to communicate with HDFS and validate permission on

HDFS folders •  If Argus is enabled, Hive will use permissions based on Argus policies in

HDFS •  Argus can be used for Storage based and regular Hive authorization


HBase


What it means: Hbase Integration

Today –  Hbase Agents supports table, CF, Column level permissions –  Local Permissions not integrated

Tomorrow –  Integrate local grant/revoke permissions –  New Argus/XA co-processor, no changes in HBase –  Hbase-site.xml

<property> <name>hbase.coprocessor.master.classes</name> <value>com.xasecure.authorization.hbase.XaSecureAuthorizationCoprocessor</value> <property> <name>hbase.coprocessor.region.classes</name> <value>com.xasecure.authorization.hbase.XaSecureAuthorizationCoprocessor</value>


About HBase Grant Revoke

17

•  Command Line Operations – Permission supported

–  Admin (A) – Create © – Write (W) – Read (R)

•  Can be performed at table, CF, column level grant <user> <permissions>[ <table>[ <column family>[ <column qualifier> ] ] ] #grants permissions revoke <user> <permissions> [ <table> [ <column family> [ <column qualifier> ] ] ] # revokes permissions


Storm


What it means?

•  Storm now support ACLs for authorization •  Argus provides administration for these ACLs, also enables access

auditing

•  Following permission support are enabled

•  Submit topology •  Kill topology •  Submit Topology •  File Upload •  Get Nimbus Conf •  Get Cluster Info •  File Download •  Kill Topology

•  Activate •  Deactivate •  Get Topology Conf •  Get Topology •  Get User Topology •  Get Topology Info •  Upload New Credential •  Rebalance


KNOX


What it means?

•  Knox currently performs service level authorization •  Allow group or user access to specific REST API (WebHDFS, WebHcat, JDBC over http etc) •  Can also restrict based on ip address •  Permissions maintained in a file

•  Manage these permissions through Argus Portal •  User experience similar to other components

•  Get access to auditing records in Argus portal


REST APIs


What does it mean?

•  Currently, Argus policies can only be managed through GUI •  Not a scalable model if there are large number of policies

•  Champlain work to expose REST APIs for the policy manager

•  Users can create/update/delete policies through these APIs


REST API’s Available

•  Repository management

REST API Request type Request URL* Get Repository GET service/public/api/repository/{id} Create Repository POST service/public/api/repository Update Repository PUT service/public/api/repository/{id} Delete Repository DELETE service/public/api/repository/{id} Search Repositories GET service/public/api/repository


REST API’s exposed in Champlain

•  Policy management

REST API Request type Request URL* Get Policy GET service/public/api/policy/{id} Create Policy POST service/public/api/policy Update Policy PUT service/public/api/policy/{id} Delete Policy DELETE service/public/api/policy/{id} Search Policies GET service/public/api/policy


Audit Log Storage in HDFS


What does it mean?

Today •  Argus audit data only in RDBMS (mysql) •  Issue with scalability

Tomorrow •  Option to write to RDBMS (mySQL or Oracle), HDFS •  Addition of Log4j file appender

•  HDFS destination can be specified in the appender •  Customer/Partners can add customer log4j appenders

•  Extensible HDFS LOG format •  Available as JSON format


Audit Logging to HDFS destination …

•  Argus Audit Logs To HDFS •  Log event is written to Local log file •  Local log file will be copied to HDFS destination (when

HDFS is available) •  Local log file and HDFS file rotated at a regular interval •  Design being enhanced


Questions ?

apache argus - how do i secure my entire hadoop cluster? olivier renault @ hortonworks

Technology

grantrevoke permission

argus storage

hive forauthorizationhive

hive integration

authorization tomorrow

rights reservedsecurity

hdfs acl

hdfs api