ready for big data? a use case in threat prediction & prevention
DESCRIPTION
TRANSCRIPT
Turning Information Into InsightThreat Prediction & Prevention
Tim PaydosDirector, WW Government Big Data Team
International Business Machines (IBM)
Today’s DiscussionIf you only remember four things
1 Intensifying threat pressures and the explosion in Big Data have pushed our clients to a tipping point
Agency leaders embrace this, are defining the new requirements, and are demonstrating success
Achieving transformation requires a broad set of capabilities, and a combination of technology & expertise
The experience and capabilities exist to help you define a strategy and a roadmap to guide your transformation
2
3
4
“In our ever-changing world, America's first line of defense is timely, accurate intelligence that is shared, integrated, analyzed and acted upon quickly and effectively.”
- President Barack Obama, 1/7/10
This isn’t an Information Sharing Problem. Its an Information Management Problem. Information Sharing Is only one piece of
the larger problem.
General Michael Hayden, Former Director CIA, Former Director NSA
It is not just about sharing information. Its about making sense of it and exploiting it once we have it. There is too much information to identify that which is important. We
need tools that tell us what’s important, and what is relevant.
Juan Zarate, Deputy National Security Adviser for Combating Terrorism
© 2009 IBM Corporation
Today’s intensifying challenges mandate a fresh approach to managing threat information
Traditional approaches have become obsolete
Multiplication of threat types, and frequency with which they occur
Threats are increasingly asymmetrical
Explosion in complexity of threat identification
Multi cultural nature of citizenry
Frequency of transaction/interactions
Social Media and “Big Data” playing an increasingly important role
Transparency is clouding
Citizen, Policy Maker and Regulatory expectations and pressures are increasing
Information is compartmentalized – lack of full integration obscures visibility
Query State limits ability to address complexity of threats
Inability to manage and search across an expanding array of unstructured data sources
Inability to link unstructured content with structured data and manage together
Untimely – Sense & Respond vs. Predict & Act
Out of context – lack of visual analysis, collaboration, and support/guidance once threat is identified
Military & Intelligence Agency Challenges Leaders Often Stuck Between a Rock And a Hard Place
6
The Information Challenge Is Only Making It Harder… Multiple Levels Of Identity Ambiguation
Maiden Names, Deaths, Moves, New Accounts
Name / Address / DOB Deception
Intermediators, Introducers, Beneficiaries,
Pooled Accounts
NefariousUn-IdentifiedThird Parties
Data Islands/Silos
Online & Remote Interaction
Data Degradation / Data Drift
Multiple Name Variants
Phonetic Transposition Errors, Lester - Leicester
Name Order, “Maria del Carmen Bustamante de la
Fuente”
Multiple Titles, Prefixes Dr., Rev, Haj, Sri.,
ColAbdul, Fitz, O’, De La
Nicknames: Hammed, Mogs
?
The Information Challenge is Only Making It HarderBig Data Can Be Characterized By The Three “V’s” – Volume, Velocity, Variety
Variety of Information
Volume of Digital Data
Velocity of Decision Making
Every day, we create 2.5 Exabytes of data — so much that 90% of the data in the world today has been created in the last two years alone.
For every 1 minute in real-time, 60 hours of video is up-loaded to You Tube
12 terabytes of Tweets are created each day, providing insight into public sentiment
80% of all the data created daily is unstructured – videos, images, emails, and social media
Structured data now includes a massive range of sensors, click streams, log files, call records, transactions
5 million financial transactions occur every single day There are 30,000 commercial air flights every single day,
accounting for 1,500,000 air passengers – every single day
Leading the WayGovernment Leaders Moving to Address These Challenges
Proactive enterprise data activity monitoring & extrusion prevention
Insider ThreatCentralised
Screening Database
Automated content extraction, entity resolution and analysis from seized assets
Child Predator InvestigationWestern NationalLaw Enforcement
Registry of Identities, Objects and Events
Streamlined Information Sharing Across Fed and Local Agencies
Connect the dots, predict and prevent threatsProtecting the homeland
Perpetual credentialing and vetting across branches and bases
Insider Threat
State of The Art covert surveillance system based on Streams platformNational Border & Security
Real time network intrusion detection, sub-millisecond analysis and response
National Borders & Security:Cyber Security
Real time insider threat detection & prevention through Big Data
National Borders & Security: Threat Prediction & Prevention
Leading the WayGovernment Leaders Moving to Address These Challenges
Real time Information Sharing, Discovery & analysis
Crime Prediction & Prevention
Western Intelligence Agency
Greatly reduced frequency and severity of Traumatic Brain InjuryWarfighter Care
Defence Advanced Research Projects
Agency
Leading the WayNaval & Maritime Threat Intelligence
10
What We Have Learned Big Data Requires A Different Approach – It Breaks The Traditional Analytics Model
ITStructures the data to answer that question
ITDelivers a platform to
enable creative discovery
Business Explores what
questions could be asked
Business UsersDetermine what question to ask
Big Data ApproachTraditional Approach
Structured & Repeatable Analytics• Query Based -- Questions Drive Data• Citizen Surveys • Monthly, Weekly, Daily• Data At Rest
Iterative & Exploratory Analytics• Autonomic -- Insight Drives Answers
• Citizen Sentiment• Persistent & Ad Hoc• Data In Motion
VS.
What We Have LearnedA Complete Set of Capabilities Is Required To Address The Challenge
What’s Required To Bridge The Gap…Establish, Govern, Manage & Deliver Information That You Can Trust…
13
IBM InfoSphere Information Server
Parallel ProcessingRich Connectivity to Applications, Data, and
Content
Unified Deployment
Unified Metadata Management
Understand
Cleanse Transform Deliver
Discover, model, and govern information
structure and content
Standardise, merge,and correct information
Combine and restructure
information for new uses
Synchronise, virtualise and move information
What’s Required To Bridge The GapPersistent Relationship Awareness & Rules of Visibility
14
DATA
DATA
Relationship Awareness
Alerts sent to analyst proactively
!
Queries & Data Flow Through The Same “Smart” Channel
Trusted
Patented Entity
Resolution & Complex Event
Processing
Proactive
Discovery as soon as data is
available.
Efficient
Mitigate False Positives through
intelligent algorithms
Collaborative
Share within and among
departments securely
Persistent
Remember how people/
organizations relate
Governed
Configurable Rules of
Visibility and privacy
DATA
Ru
les of In
form
ation
V
isibility, In
form
ation
S
harin
g &
Privacy
Enterprise
Limited
Restricted
What’s Required To Bridge The GapPersistent Relationship Awareness – IBM Identity Insight
15
DATA
DATA
Relationship Awareness
Trusted
Patented Entity
Resolution & Complex Event
Processing
Proactive
Discovery as soon as data is
available.
Efficient
Mitigate False Positives through
intelligent algorithms
Collaborative
Share within and among
departments securely
Persistent
Remember how people/
organizations relate
Governed
Configurable Rules of
Visibility and privacy
DATA
Addresses The Thorny Issues Of:• Realtime Discovery• Enterprise Amnesia - Continuous• Enterprise Brain Hemorrhage -
Persistent• Complex and Expanding attribution• Degrees of Separation• Extensibility• Complex event processing Linked to
Identities• Information pedigree and audit trail of
merges/splits• Anonymization
A highly-specialized Identity Analytics repository providing real-time detection of obvious and non-obvious relationships between people, organizations, events, and other identity types
What’s Required To Bridge The GapRules of Visibility – IBM Initiate
16
Trusted
Patented Entity
Resolution & Complex Event
Processing
Proactive
Discovery as soon as data is
available.
Efficient
Mitigate False Positives through
intelligent algorithms
Collaborative
Share within and among
departments securely
Persistent
Remember how people/
organizations relate
Governed
Configurable Rules of
Visibility and privacy
Ru
les of In
form
ation
V
isibility, In
form
ation
S
harin
g &
Privacy
Enterprise
Limited
Restricted
Relationship & Hierarchy Management ViewsCollaborative, Visual Data Stewardship CapabilitiesData Security & PrivacySearch CapabilitiesHighly configurable/custom composite viewsOpen integration optionsDynamic implementation models Data security & privacyReporting and analytics
Provides accurate, real-time access to entity, object and event data across disparate sources, systems and networks
Taking it to the Next Level…with Advanced Business Analytics…
17
Structured Data & Unstructured Content
Descriptive Analytics
Prescriptive Analytics
Predictive Analytics
Made consumable and accessible to everyone
What if these trends
continue? Forecastin
g
How can we achieve the best
outcome and address
variability? Stochastic
Optimisation
What is happening
What exactly is
the problem?
How many, how often,
where?
What actions are
needed?
What could happen?
Simulation
How can we achieve the best
outcome? Optimisation
What will happen next if?
Predictive Modelling
Extracting concepts and relationships
Content Analytics
What Are People Talking About & Feeling
WebAnalytics
Language & Sentiment
Taking it to the Next Level…With Assisted Analysis & Visualization…
“I have seen a number of similar products and nothing comes close to Analyst’s Notebook.“
- Investigative Analyst G.M. Allen, Monroe High Intensity Drug Trafficking Areas
Taking it to the Next Level…With Social Network Analysis…
InsightfulComprehensive analytics to drive
new meaning from your data
Governed & Secure
Accurate & CompleteComplex and disparate data transformed, cleansed, reconcile
and delivered
Agency Leaders Telling Us What They Need Trusted Information -- On Multiple Levels
Can I Trust The Information?
Rules are in place and tools are deployed to limit visibility, secure sensitive information, and protect privacy
Can I Trust My Partner and My Own Agency?
Can I extract new Insight to make it all worthwhile?
1 Intensifying threat pressures and the explosion in Big Data have pushed our clients to a tipping point
Agency leaders embrace this, are defining the new requirements, and are demonstrating success
Achieving transformation requires a broad set of capabilities, and a combination of technology & expertise
The experience and capabilities exist to help you define a strategy and a roadmap to guide your transformation
2
3
4
Today’s DiscussionIf you only remember four things
Thank You !Tim PaydosDirector, WW Government Big Data [email protected]