bluetalon auditing and authorization with hdfs on … · title: agenda – new hire hdp sessions...

1 © Copyright 2014 EMC Corporation. All rights reserved. © Copyright 2014 EMC Corporation. All rights reserved.

BLUETALON AUDITING AND AUTHORIZATION WITH HDFS ON ISILON ONEFS V8.0

Boni Bruno, Chief Solutions Architect, EMC


Data-Centric Security

Secure, Fast, Flexible Hadoop Data Security Solution for Enterprises

Data

users

Authorization

Masking

Hadoop Compute

HDFS Auditing

• Manage your enterprise data in a high-performance flexible grow-as-you-go storage system that scales-out

Info Sec

• Analyze data at any scale or speed with your favorite Hadoop framework

• Simplify data security with a central policy and audit


Hadoop

BlueTalon & Isilon provides Hadoop control & visibility at the Data Layer

• Transparent enforcement

– End users use existing apps without change

– Minimal performance overhead for security

• Contextual auditing

– Tagged with policy, role and actions

• Dynamic masking

– Selective for users without duplicating data

• Precise authorization

– Granular: file, sub-file, row, column, cell, sub-cell

– Decisions based on business data

Enforcement

Points Policy Engine

Audit Engine

Any Application

Data Stewards

Security admins

Auditors Data Scientists Analysts

Business users

Developers Machines


Example of policy, enforcement and audit in BlueTalon

Data users (e.g. analysts, data scientists)

Auditors

Data Stewards

Security admins


Performance benchmark with BlueTalon on Isilon

6.9

125.55

7.03

124.98

Teragen

Terasort

1 TB data Job Elapsed Time (mins)

Without BlueTalon With BlueTalon

• Teragen – Measures Write I/O from Hadoop cluster to Isilon cluster.

• Terasort – Measures entire MapReduce performance across HDFS I/O between Hadoop and Isilon, local disk I/O, CPUs usage, memory usage, etc.

Hadoop

Enforcement Points Policy Engine

Audit Engine

• 4 nodes, 100 TB, OneFS 8.0.0.1

• HDP 2.4, 7 compute nodes

• 40x7 cores, 252 GBx7 mem

• Minute performance difference with large map reduce jobs without and with BlueTalon


• BlueTalon Enforcement Points in diagram

– Filesystem EP (installed on each compute node)

• Policy and Audit in diagram

– Policy Engine and UI

– Audit Engine and UI

• Clients shown in diagram

– FsShell (hdfs)

– Hive cli (mapreduce)

This is how BlueTalon’s customers using a Hadoop cluster for compute and Isilon cluster for HDFS storage deploy BlueTalon. Security admins create rules and view audit through UI (or API) that drive run-time Policy and Audit Engines on a management node of the compute cluster. All file system requests from the compute cluster go through the local FSEP, which proxies the Isilon NameNode over HDFS (not webhdfs) protocol. There is one instance of FSEP per compute node. The FS EP proxy connects to OneFS using SmartConnect to maintain scalability and performance.

6

BlueTalon Validation in SA Lab

Isilon node 1


• Isilon storage cluster (Hopkington SA Lab)

• 4 node Isilon cluster with hdfs enabled and webhdfs not enabled.

• Webhdfs was disabled on Isilon to make sure BT only used HDFS on the backend with Isilon. It does. This means we can use BT to proxy both HDFS and WEBHDFS to Isilon HDFS on the backend!

• HDP compute cluster

• 8 node HDP cluster configured with HDFS, YARN, Ambari Metrics, etc.

• 7 compute nodes with 40 cores and 252 GB each

• Ambari UI: http://xx.solarach.lab.emc.com:8080

• BlueTalon EPs : Filesystem EP installed on each compute node

• BlueTalon Policy and Audit Engines and UIs installed on Ambari node

• Policy UI : http://xx.solarach.lab.emc.com:8111/BlueTalonConfig

• Audit UI : http://xx.solarach.lab.emc.com:8112/BlueTalonAudit

• Tests validated (see screenshots)

• FsShell –ls and –cat commands

• Teragen and Terasort mapreduce jobs with 1GB and 1TB data

• Screen on the bottom right shows write throughput on a teragen mapreduce job running through BlueTalon EP

7

Details of the BlueTalon validation in SA Lab


Details of Validation with OneFS Simulator

• Test1. Functional validation of storage, compute and storage+compute jobs

– 3 Node Isilon OneFS 8.0 Simulator with HDFS enabled on a ESXi host

o HDFS license enabled

o FreeBSD OS

– Single Node HDP 2.3 cluster on EC2 instance

o BlueTalon Policy, Audit and Filesystem EP

o HDFS clients (fs shell and yarn)

o CentOS 6.5 OS

– Ports opened between compute cluster on EC2 and Isilon storage cluster

o Port 8020 for NameNode and port 585 for DataNode process

– Configuration on HDP cluster

o core-site.xml changed to point to FSEP or Isilon for different tests

o Filesystem EP configured in proxy authentication

– Sizing for functional testing

o Isilon VMWare Host: 8 vCPU, 32 GB mem, 500GB disk

o HDP EC2 instance: m3.xlarge = 4 vCPU, 15 GB mem, 80GB disk

Note: BlueTalon Engineering runs an HDFS command test suite as part of its release exit criteria on native HDFS clusters. We ran this checklist Jenkins job against the Isilon cluster. All 118 tests passed successfully.


Comparison of storage queries with and without BlueTalon

core-site.xml on the compute cluster configures the filesystem

Compute cluster points to

Isilon storage cluster directly

Compute cluster points FSEP which

points to Isilon storage cluster

Without FSEP: Both alice & bob can list in alice’s home folder With FSEP: bob can’t list in alice’s home folder

Without FSEP: Both alice & bob can read data from a private file in alice’s home folder

With FSEP: bob can’t access data from a private file in alice’s home folder

Without FSEP: alice can’t move files in her home folder because filesystem is owned by hdfs & supergroup (required for Hadoop functionality)

With FSEP: alice can move data from private location to public location to share with bob

• Without BlueTalon • With BlueTalon


Enforcement and Policies applied on Isilon storage cluster

• global_default policy applies to all users

• If no rule is applicable, then deny is enforced

• Allowing recursive execute on / enables traversing the filesystem meta-data without exposing data.

• Allowing recursive read on /user/<username>/public enables users to share data with others through their home folder • <username>_default policy

applies to only that user

• Allowing recursive read and write on /user/<username> enables users to maintain their private files in their home folders

• Each user gets the effect of permissions from both global_default and their <username>_default policies

Compute cluster points to FSEP which points

to Isilon storage cluster

alice can list her folder

• Enforcement of BlueTalon policies in HDFS backed by Isilon • Policies created in BlueTalon Policy UI (or automated with rules API)

bob can’t list her folder

bob can view

alice’s public data

alice can make her private

data public by copying it to public folder

alice can view

her private data

bob can’t view her private data


Enforcement and Audit of requests on Isilon storage cluster

Compute cluster points to FSEP which points

to Isilon storage cluster

alice can list her folder

• Enforcement of BlueTalon policies in HDFS backed by Isilon • Audit of the requests captured by BlueTalon

bob can’t list her folder

bob can view alice’s public data

alice can make her

private data public by copying it to public folder

alice can view

her private data

bob can’t view her private data


MapReduce jobs through BlueTalon FSEP on Isilon storage cluster

• Subset of the audit captured during the map reduce test in BlueTalon UI

mapreduce compute cluster points to FSEP which points to Isilon storage cluster

alice doesn’t have any fs-test files

alice is running a mapreduce job that goes through BlueTalon FS EP

the file system read test run by alice completes successfully

the file system write test run by alice completes successfully


social security is selectively masked

data is restricted to west coast locations

Business Users, Data Scientists,

Developers

Security Admins

Policies in BlueTalon

• credit cards and social security are sensitive

• our contracts prohibits use of customer data outside west coast Data Stewards or

Business


GUI - BlueTalon HDFS Data Domain for Isilon OneFS


GUI - BlueTalon OpenLDAP User Domain for Users and Roles


Audit of HDFS with BlueTalon FS EP on Isilon OneFS

• Not only READ and WRITE, but also OPEN and GETFILESTATUS requests can be audited


Detailed audit of Hive

• user alice, beeline -e “select * from accounts” • user bluetalon, beeline -e “select * from accounts”

{ "Action": "LOGIN","AuditParams": "-", "Client": "-","ClientIp": "10.255.54.29", "ColumnList": "-","DataBase": "-", "Effect": "Authorized", "FinalQuery": "-", "GroupName": "bedrock", "LoggedUser": "alice", "OrignalQuery": "-", "PolicySet": "global_default,bedrock_default", "Schema": "-", "SessionID": "-", "Timestamp": "2016-02-04|03:09:34.354", "UniqueID": "-" }

{ "Action": "UNKNOWN","AuditParams": "", "Client": "","ClientIp": "10.255.54.29", "ColumnList": "-","DataBase": "default", "Effect": "Denied", "FinalQuery": "select * from ARCAccessDenied", "GroupName": "bedrock","LoggedUser": "alice", "OrignalQuery": "select * from accounts", "PolicySet": "global_default,bedrock_default", "Schema": "default","SessionID": "", "Timestamp": "2016-02-04|03:09:34.711", "UniqueID": "711655_2027923200_1246552766" }

{ "Action": "LOGIN","AuditParams": "-", "Client": "-","ClientIp": "10.255.54.29", "ColumnList": "-","DataBase": "-", "Effect": "Authorized","FinalQuery": "-", "GroupName": "bluetalon","LoggedUser": "bluetalon", "OrignalQuery": "-","PolicySet": "bluetalon_default,global_default", "Schema": "-","SessionID": "-","Timestamp": "2016-02-04|03:09:39.819", "UniqueID": "-" }{ "Action": "UNKNOWN","AuditParams": "", "Client": "","ClientIp": "10.255.54.29", "ColumnList": "ID,NAME,PHONE,BIRTHDATE,SOC_SEC_NO,ZIP,CREDIT_CARD,BALANCE", "DataBase": "default","Effect": "Policy", "FinalQuery": "select accounts.ID, accounts.NAME, accounts.PHONE, accounts.BIRTHDATE, hash(accounts.SOC_SEC_NO) SOC_SEC_NO, accounts.ZIP, 0 CREDIT_CARD, accounts.BALANCE from accounts WHERE (accounts.ZIP > /*<GCODE>WestCoastZips<GCODE>*/) ", "GroupName": "bluetalon","LoggedUser": "bluetalon", "OrignalQuery": "select * from accounts", "PolicySet": "bluetalon_default,global_default", "Schema": "default", "SessionID": "", "Timestamp": "2016-02-04|03:09:40.214", "UniqueID": "214805_2027923200_1282238494" }


Detailed audit of HDFS

{ "audit_type": "audit", "database": "", "group_list": ["bedrock"], "ipaddress": "10.255.54.29", "modified_request": ["Allow ","GETFILESTATUS ","/bedrock"], "policy_list": [], "policy_type": "", "request": ["GETFILESTATUS ","/bedrock"], "schema": "", "time_stamp": "2016-02-04|01:16:11", "unique_key": "0f885bb4-6635-4e56-9d6c-90c000f24f78", "user": "alice" }{ "audit_type": "audit", "database": "", "group_list": ["bedrock"], "ipaddress": "10.255.54.29", "modified_request": ["Allow ","READ ","/bedrock"], "policy_list": [], "policy_type": "", "request": ["READ ","/bedrock"], "schema": "", "time_stamp": "2016-02-04|01:16:11", "unique_key": "64d3e4b8-bbee-4e2a-a4f4-b6da6134e045", "user": "alice" }

{ "audit_type": "audit", "database": "", "group_list": ["users","bluetalon"], "ipaddress": "10.255.54.29", "modified_request": ["Allow ","GETFILESTATUS ","/bedrock"], "policy_list": [], "policy_type": "", "request": ["GETFILESTATUS ","/bedrock"], "schema": "", "time_stamp": "2016-02-04|01:16:15", "unique_key": "fa796aa9-1fb1-4ddc-8dfc-71dcc91981a5", "user": "bluetalon" }

• user alice, hdfs dfs -ls /bedrock • user bluetalon, hdfs dfs -ls /bedrock

Output from the bt-audit-kafka service

bluetalon auditing and authorization with hdfs on … · title: agenda – new hire hdp sessions...

Documents