security from the big data and analytics perspective
TRANSCRIPT
All Things Open
Agenda• Intro, who I am.• Cybersecurity • ONI now Apache Spot (incubating)• Apache Spot (incubating)• Demo• Call to Action.• Q&A
Cybersecurity• We have gaps… The analysis of billions of events , orchestrate our
data sources (logs in different forms), and sometimes the documentation of our security products is not the best.
The hacker community collaborates everyday, it’s time we do the same.
Services Products Training
Free$100Learn to Crack
WifiHack a Corporate
Email AccountAngler
Exploit Kits
$500
AssetInventory
ConfigurationGuidanceAnalysis
VulnerabilityAnalysis
ThreatAnalysis
IntrusionDetection
VulnerabilityAlert
ConfigurationGuidance
AssetDefinition
ThreatAlert
IncidentReport
IncidentManagement
Operational Enterprise Networks
Centralized ReportingEnterprise ITChange Management
Development & SustainmentSecurity ManagementProcesses
Assessment of System
Development,Integration, &Sustainment
Activitiesand
Certification &Accreditation
System &Software
AssuranceGuidance/
Requirements
Operations Security Management Processes
Enterprise IT Asset Management
CCE/CCSS/OVAL/ARF/XCCDF/CPE
CVE/CWE/CVSS/ARF/CCE/CCSS/ARF/CWSS/OVAL/CPE/XCCDF
CVE/CWE/CVSS/ARF/CCE/CCSS/OVAL/CWSS/XCCDF/CPE/CAPEC/MAEC
CVE/CWE/CVSS/ARF/.CCE/OVAL/CCSS/XCCDF/CPE/CAPEC/CWSS/MAEC/CEE
CPE/OVAL/ARF
CWE/CAPEC/CWSS/MAEC/OVAL/OCIL/XCCDF/CCE/CPE/ARF/SAFES/SACM
CWE/CAPEC/SBVR/CWSS/MAEC
OVAL/XCCDF/CCE/CCSS/CPE/ARF
CPE/OVAL XCCDF/OVAL/CCE/CCSS
CVE/CWE/OVAL/CVSS/CWSS
CAIF/IDMEF/IODEF/CVE/CWE/OVAL/CPE/MAEC/CCSS/CWSS/CEE/ARF
CVE/CWE/CVSS/CPE/CWSS/CAPEC/MAEC
SCAP
SwAAPCVE/CWE/CVSS/CCE/CCSS/OVAL/XCCDF/CPE/CAPEC/MAEC/CWSS/CEE/ARF
CVE/CWE/CVSS/CCE/CCSS/OVAL/XCCDF/CPE/CAPEC/MAEC/CWSS/CEE/ARF
TrustManagement
IdentityManagement
EMAP
ERAP ECAP
ESIP ITAPTAAP
ONI -> Apache Spot (incubating)• Apache spot (incubating) is an advanced analytic solution that will help us to
close the gaps that we are mention on the previous slides.
• Ingesting billions of records in HDFS and execute machine learning algorithms, to detect potential threats in our environment.
Apache Spot (incubating) Core
ON
I Dat
a So
urce
s
DNSInfrastructure Logs
ProxyInfrastructureLogs
Routers with Netflow Protocol Enabled onInterfaces
New Data Source
New DataSource
ONIVisualization
Server / iPython Server
ON
I GU
I
TLS Https 443
Security and Context Use
Cases Develop in Conjunction with
Intel Security
Assumes Cloudera Hadoop Environment
Data Integration
Data Store Machine Learning
CollectorsOnlineNoSQL(HBase)
Filesystem(HDFS)
Spot MLAlgorithms
Spark
Master Node (s) ClouderaManager/Navigator
Machine learningAlgoritms Output,ONI Recommendedthe Intel MPILibraries. Scala
Native AdministrationCloudera Manager
Cluster AuthenticationLDAP/KerberosAuthentication
Machine learningGenerates CSV Files
with the Results
Operational Analytics AddingContext Using Reputation Services
for Public IP Address (GTI)
Defining the Interface to Share
the Suspicious Connections with I-
Sec Products and Other Brands.
Product Architectural Overview
Apache Spot delivers…1. Scalable Data and Analytics Platform 2. Open Data Models3. Analytic Collaboration Across the Community4. Growing Application Ecosystem
… to address cybersecurity use cases.• Network Traffic Analytics• Threat Hunting• Incident Detection and Resolution• Cybersecurity Data Management• Custom Use Case
Platf
orm
Apache Spot, bringing all of the components together.
Data
Man
agem
ent
Apache Spot Sample Data Sets
Apache Spot Open Data Models (ODM)
Data Platform (CDH)
Ingestion (Kafka, Flume, Streamsets)
Anal
ytics Apache Spot OSS Analytics
Analytic Services (Jupyter, Apache Spark)
Apps Apache Spot ODM Marketplace
Infra Intel Hardware, On-Prem, AWS, Azure M
anag
emen
t, Se
curit
y, Go
vern
ance
(Dire
ctor
, Man
ager
, Sen
try,
Nav
igat
or)
Public or private clouds
Scalable storage and distributed processing
Provisioning, management, and security
Batch and stream data ingestion
Logical and physical models
Data Science workbench
Network traffic analytics, Add’l OSS analytics
ODM Compliant ecosystem, both open source and ISV
Community sourced, anonymized data sets for model development
Demo
Call to Action.Contribute for the Apache Spot (incubating) project.1. Develop connectors to ingest more data2. Develop new algorithms that help us to increase the detection rate of the tool3. Contribute to add Context to this results, adding threat intelligence feeds
connector to databases that will help us to present meaningful information to the end user.
4. Develop the User Interface, propose changes, technologies, operational summaries, reports, etc.
Call to Action.5. Integrate Apache Spot (incubating) with other security tools, that have the
capabilities to enforce / change security postures. (Firewall consoles, IPS consoles, Proxies, Endpoint Security Solutions, E-mail proxies)
6. Contac us • Web page: http://spot.apache.org/• slack: slack.apache-spot.io/ • twitter @ApacheSpot
7. Contribute to the Apache Spot (incubating) project.
With Apache Spot, you are joining a community.
Collaborate with industry leaders using a common framework.
Apache Spot(Incubating)
Join the community that is fighting cyber threats today.
spot.incubator.apache.org