big data security analytics (bdsa) with randy franklin

Sponsored byTop 5 Truths about Big Data Hype and Security

Intelligence

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.2

Thanks toMade possible by

www.hpenterprisesecurity.comSRIDHAR KARNAMHP ArcSight Product Marketing

http://www.hpenterprisesecurity.com/


Preview of Key Points

1. There’s More to Big Data than “Big” 2. The Real-Time Requirement for Big Data Security Analytics 3. There’s More to Big Data Security Analytics than Big Data

Technology 4. The Trap of Data Silos within Big Data Repositories 5. The 3 Vs of Big Data Aren’t New to Enterprise SIEM


1There’s More to Big Data than “Big”

Volume is only one dimension of “big”

Record quantity better metric than byte

About analysis or lots of information

BigDataIs.. Data

Science

Data Volume

Data Variety

Data Velocity

Put all data together; find relationships we didn’t know existed

Variety – total record types

Big data even with small volume

Velocity usually considered rate of new data to be stored

Not analyzed

But BDSA has a bigger velocity issue

The type of questions being asked and the analytical techniques being used to answer them is what distinguishes Big Data from traditional data

Cluster analysis Topological data analysis Machine learning Multi-linear subspace learning

Data visualization


2The Real-Time Requirement for Big Data Security Analytics

Big Data Security Analytics (BDSA) is a specialized application of the more general concept of Big Data. Most Big Data scenariosHigh velocity data aquisition

Human driven analysis

Long shelf life for conclusions drawn

3 types of velocityInsertion or append speed into Big Data repository

Processing speed for queries upon data rest

Analysis of events in real time

Human driven analysis has a place in BDSAImmediate tactical investigations in response to warning signs detected by automated correlation enginesForensic investigations Strategic research to tease out indicators of long-term, ongoing attacks



But what about tactical, second-to-second monitoring?Core of security operation center workAnalysis must be done automatically and in a streaming fashion

Current Big Data toolsRun a query, analyze results, tweak query, analyze results, repeatNot a streaming scenario in which a constantly updated tactical situation is plotted

But real-time analytics require a purpose-built correlation engine Enterprise SIEM correlation enginesDesigned to handle a constant stream in real timeMaintain in memory a massive amount of partial pattern match objectsChurning in and out of existence at a fantastic rate



SIEM Real-Time Correlation

Big DataBatch Analytics

Trigger for tactical investigations

Event feed

Context Criteria for better

correlation rules

Wide and deep trolling to identify

ongoing attacks too low and slow to

trigger SIEM alerts


3There’s More to Big Data Security Analytics than Big Data Technology

BDSA requires 3 kinds of advanced skills

Big data platform

technology

Still more of a concept and developer-level movement than a mature technology

platform with available off-the-shelf solutions

Data science

To make any sense of Big Data, analysts using Big Data farms need to

know how to use advanced analytics

Information

securityTo detect cyber-attacks and internal malicious

agents, analysts need to be more than data

scientists

Must also be technical information security professionals that

understand the organization’s IT infrastructure.

Network security, host security, data protection,

security event interpretation, and

attack vectors


3There’s More to Big Data Security Analytics than Big Data Technology


4The Trap of Data Silos within Big Data Repositories Point Solution for

Monitoring Application B

Point Solution for Monitoring

Application B

Point Solution for Monitoring

Application B

Big Data Repository

ApplicationA

ApplicationB

ApplicationC

ApplicationA

ApplicationB

ApplicationC

Even after migrating from point solutions to Big Data, the same silos can persist.


4The Trap of Data Silos within Big Data Repositories

Example: consider usernames and email addressesIf you are trying to track a user’s actions and communications through a variety of data, you must be cognizant of the fact that a given email address, such as [email protected], could be one of the following:

Email sender Email recipient Actor in a audit log event (e.g., jsmith opened a file)Object of an action in an audit log event (e.g., Bob changed jsmith’s reset password)Subject of a memo

Simply querying certain data can lead to extremely inaccurate results unless one of the following occurs:The analyst filters the results manually after the queryThe analyst builds knowledge into the query about the structure or format of the various data queried to do the filteringThe system understands the various formats and does the filtering automatically

mailto:[email protected]


4The Trap of Data Silos within Big Data Repositories

Silos in Big Data is failure to deal with variety Being able to store all types of data and query it for keyword occurrences does not satisfy BDSA requirements. Some enterprise SIEMs takes a more effective and pragmatic approach that embraces data variety Normalizing security events into a common event format Integrate non-event data sets into the correlation and analytics process. Directory informationIP reputation listsGeolocation datasocial network feeds


5The 3 Vs of Big Data Aren’t New to Enterprise SIEM

Big data architectureEnterprise SIEMs abandoned relational databases a long time agoProprietary correlation and storage engines• Allow rapid storage and query of massive amounts of event data

Real-time situational awarenessReal-time analysis is a manifest requirement of security analyticsEnterprise SIEMs analyze data as it arrivesCombines • real-time, in-memory, event-log data • asset awareness, asset vulnerability• identity correlation

Prioritize critical events and correlations to assist operating teams with immediate detection of threats

No data scientists required No silos


Bottom line

Hidden skill requirement of BDSA: data scientists Real-time requirement for security intelligence, often misunderstood in relation to Big DataRisk of data silos persisting in Big Data repositoriesInvesting in a Big Data cluster that runs search and a schemaless database is only the beginning of building a BDSA practiceAn enterprise SIEM like HP ArcSight provides BDSA that is specialized for event data

© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

How HP Solves

Big Data Security Analytics Problem? • With CORR• With Hadoop• With Autonomy• With HAVEn• Why HP?


Big data opportunities – won and lost

% of the Digital Universe that actually is being tagged and

analyzed

Competitive Advantage in the Digital UniverseMassive amounts of useful data are getting lost23%

3%% of data that would be

potentially useful IF tagged and

analyzed

% actually being tagged for Big Data Value (will grow to 33% by 2020)

0.5%¹Source: IDC The Digital Universe in 2020, December 2012


Collect, normalize, and categorize machine data such as logs, events, and flows from any device, any time, anywhere from any vendor

Collect & correlate up to 100,000 events per second from 350+ connectors

Collect, store, correlate, and analyze big data across IT

HP ArcSight Universal log management platform

High-performance universal log management to consolidate machine data across IT

HP ArcSight

The unified machine data through filtering and parsing is enriched with rich metadata, which allows you to search machine data through simple text-based keywords without the need of domain expertise

Search over 2,000,000 events per second

The unified data is stored through high compression ratio in any of your existing storage formats, eliminating the need for expensive databases and DBAs

Store years’ worth of data

Built-in content packs, algorithms, rules, and the unified machine data help you deploy IT security, IT operations, IT GRC, and log analytics

Analytics & intelligence


Volume

VarietyVelocity

Complexity

ArcSight CORRe for Big Data SecurityArcSight has been dealing with Big Data since 2007 with CORR engineVolume• Cross-device, real-time correlation of data across IT• Long term archival at 10:1 compression ratio with

ArcSight• Send it to Hadoop at over 100,000 EPS

Velocity• SmartConnectors collects logs, events, flows at over

100,000 EPS from almost any log generating source• Search data at over 2,000,000 EPS

Variety• Collects machine generated data from 350+ distinct

sources• Autonomy collects human generated data from 400+

distinct sources• Collect from Hybrid network such as physical, virtual,

and cloud


Success StoriesBeyond theory to practice:

U.S. Department of Health and Human Services “HP solutions have helped us transform from a reactive to a proactive IT Operations function, and to align our priorities to match the business and drive business value, delivering 300% ROI in one year.” - Dan Galik, CISO

Heartland Payment Systems

“ ArcSight solution will give us a more comprehensive threat and risk management platform that optimally enables enterprise-wide visibility to identify illegal activity in progress and take prompt, preemptive action.” - Kris Herrin, CTO


ArcSight and Hadoop

Security Intelligence

Storage Analytics Live/ Historical data

Hadoop

Live (Real-time, cross-device correlation of security events)

Historical (security intelligence)

ESM/Logger

ESM/Logger

Live (Real-time analytics on unlimited data)

Historical (Security analytics)Hadoop


Sentiment AnalysisArcSight with Autonomy

Meaning based security Predictive security – Moving from proactive securityAnswers critical questions:• Where is our sensitive information? Who has access to

it?• Which systems store sensitive information?• Do we have the right controls in place to

protect sensitive information?


HAVEn – big data platform

HAVEn

Social media IT/OT ImagesAudioVideoTransactional

dataMobile Search engineEmail Texts

Catalog massive volumes of distributed data

Hadoop/HDFS

Process and index all information

AutonomyIDOL

Analyze at extreme scale in real-time

Vertica

Collect & unify machine data

Enterprise Security

Powering HP Software+ your apps

nApps

Documents

hp.com/haven


How we help our customers?

5 minutes to generate IT GRC report Compliance packs generates IT GRC reports that otherwise would take 4 weeks3 days to run an IT auditSearch results yield audit-quality data that otherwise would take 6 weeks

10 minutes to fix an IT incidentFull-text based searching and integration with HP portfolio detects and corrects IT incident that otherwise would take 8 hours

4 hours to respond to a breachQuick forensic tools enable instant response to a data breach that otherwise would take 24 days

2 days to fix a threat vulnerabilityArcSight & TippingPoint solution builds threat immune that otherwise would take 3 weeks


HP Enterprise Security Momentum

HP Security SaaS

2.5Blines of code under SaaS subscription

HP ESP Customers

900+Customers

All Major BranchesUS Department of Defense

9 out of 10Major banks

9 out of 10Top software companies

10 of 10Top telecoms

35New ProductsReleased

in the last 12 months

HP Security Technology

#1

In all markets we play in#

2

10,000+Manage

d Securit

y Service

s

big data security analytics (bdsa) with randy franklin

Technology

big data thandata big

big data repositories

big data hype

sense of big data

big data farms

data protection

data scientiststo

big data technology9