issues with ingesting/staging/analyzing data in conmon implementation
TRANSCRIPT
Tieu Luu, Ben Stack Developer Days 2013 @ Mitre, McLean, VA
July 24, 2013
Background
Defining Continuous Monitoring
Supporting Data and Architecture
Ingest
Stage
Analyze
Future Architecture
SuprTEK has been at the forefront of Continuous Monitoring, working with and integrating technologies and standards from organizations such as the Defense Information Systems Agency (DISA), National Institute of Standards (NIST), National Security Agency (NSA), United States Cyber Command (USCYBERCOM), and Department of State (DoS)
Since 2010 SuprTEK has been working with DISA PEO-MA to develop and field the Department of Defense’s Continuous Monitoring and Risk Scoring (CMRS) system that enables USCYBERCOM and other DoD Enterprise level users to monitor and analyze the security posture of millions of devices deployed across the DoD’s networks.
Transforming and improving the DoD’s cyber security processes …
• Risk Management • Vulnerability Management • Certification & Accreditation
• Compliance and Reporting • Configuration Management
• Inventory Management
Improving security posture and reducing costs through continuous monitoring automation.
3
CMRS utilizes SCAP standards such as XCCDF, CPE, and CVE to continuously and automatically determine whether an asset is susceptible to vulnerabilities, its compliance level against required patches, and compliance against IAVAs, STIGs, and other enterprise security policies.
NIST SP 800-137:
Information security continuous monitoring is defined as maintaining ongoing awareness of information security, vulnerabilities, and threats to support organizational risk management decisions.
NIST IR 7756:
Continuous security monitoring is a risk management approach to Cybersecurity that maintains an accurate picture of an organization’s security risk posture, provides visibility into assets, and leverages use of automated data feeds to measure security, ensure effectiveness of security controls, and enable prioritization of remedies.
Asset
Configuration Compliance
Check Vulnerability
Software Inventory
Organization Location System
106
103
103 102
103 103
103 103
Source: NIST IR 7756 CAESARS Framework Extension
1. Ingest
2. Stage
3. Analyze
Web-based User Interface
Warehouse
Analysis Services OLAP Cubes
File
Processor
File
Processor
File
Processor
File
Processor
ARCAT ASCAT
Dimensional DB
Batch Jobs
Reporting ServicesBusiness Logic
File Processor Pool
File
Processor
…Risk
Dashboards
IAVM
Summary
Benchmark
Summary
Inventory
SummaryReports
ADS-Lite Web Service
HBSS
CMRSpreIOC
1. Ingest
2. Stage
3. Analyze
HBSS APS
HBSS APS
HBSS APS
ADS-Lite WS
ARF
ASR
SAN Filesystem
File Processor
File Processor File
Processor
Warehouse
continuously
20 hrs/day
A lot of publishers across DoD network ◦ Volume/configuration/versions
ARF & ASR XML Processing
CPU intensive
Complete “asset profile” distributed across multiple messages
Reconciliation with existing records in the warehouse
Asset identification
ADS-Lite Web Service and File Processor distributed across multiple nodes
Two-stage asynchronous architecture
Sequence-independent message processing
Custom shredding logic to reconcile new and existing records
Shred data into warehouse continuously (future)
Warehouse Dimensional DB OLAP Cubes nightly nightly
Rich data model to support new & evolving requirements
Data volume Efficiency & performance ◦ Finishing nightly jobs in allotted time window
Consolidate, Correlate, & Fuse Support for multiple interaction models ◦ A lot of writes ◦ Batch processing ◦ Interactive queries
Complex jobs to ETL data across 3 tiers
Three Tier Architecture ◦ Warehouse
◦ Dimensional
◦ OLAP Cubes
A lot of denormalizing ◦ Asset properties
◦ Findings
“Blue – Green” architecture for Dimensional DB and OLAP cubes (future)
Migration to HBase for warehouse (future)
IAVM Compliance
SOE Compliance
Scoring Ad Hoc Queries
Rollup & Drilldown
Canned Reports
Dimensional DB OLAP Cubes
Batch Jobs
Stored Procedures
Functions
SSDS SSRS SSAS
Data volume & performance Data quality Shrinking time windows to run nightly jobs Complex business logic ◦ Risk scoring ◦ IAVM compliance ◦ SOE compliance ◦ Benchmark compliance
Constantly evolving Ad hoc, interactive queries Data access control
Preprocess as much as possible
OLAP cubes for interactive queries
Tight algorithms and T-SQL coding
Agile approach ◦ “Expect it be wrong the moment we’re done”
◦ E.g. centralized tagging functionality
Enhance risk scoring algorithms (future) ◦ Weighting of assets
◦ Weighting of checks
Migration to Hadoop (future)
HBase
Analysis Services CMRS Reporting
HBSS
ADS-Lite Web Service
OLAP Cubes
Reporting ServicesBusiness Logic
Pig HiveMap/
ReduceHBase
API
ARF HBase
Shredder
ARF HBase
Shredder
ASR HBase
Shredder
ASR HBase
Shredder
HBase Shredder Pool
ACAS Other
Risk
Dashboard
Widgets
IAVM
Compliance
Widgets
Benchmark
Summary
Widgets
Inventory
Summary
Widgets
HBSS
Endpoint
Widgets…
Report
WidgetsOther Widget Other Widget Other Widget
OWF-Based User Interface
ARF HBase
Shredder
ASR HBase
Shredder
1. Ingest
2. Stage
3. Analyze
Tieu Luu Director of Research &
Product Development SuprTEK [email protected]
Ben Stack CMRSpreIOC
Development Lead SuprTEK [email protected]
www.panoptescyber.com