LOGGING AND MONITORING

PRINCIPLES OF WHY

Operational Needs

•Is the

environment

working well?

Security Needs

•Is the

environment

working securely?

PRINCIPLES OF HOW

Monitoring is

“Opt-Out”

•Bake monitoring

into the

provision and

configure

processes

Monitoring is

simple

•Leverage AWS

services where

cost efficient

•No on premise

dependencies

Monitoring is

useful

•Slack - ChatOps

for common

portal

•PagerDuty -

Alerting

NETWORK – POLICY COMPLIANCE

• REMOVED - SANITIZED

NETWORK – TRAFFIC ANALYSIS

• VPC Flow Logs

• Native integration available with AWS Elasticsearch service

• Currently only storing – no processing

• OpenVPN Logs

• CloudWatch storage

OPERATING SYSTEM

• System Monitoring

• OpsWorks provides CloudWatch detailed (1 minute vs 5 minute

sampling) monitoring for free

• Part of automatic (Vagrant) configuration process for non-OpsWorks

driven systems

• Only hypervisor visible stats – Disk usage and memory not tracked

• CloudWatch Logs

• Baseline configuration – Chef configuration management sets up

forwarding

• Core logging for AWS Linux (Redhat) and Ubuntu

AWS PLATFORM

AWS Config

•What is the state of our

environment?

AWS Cloudtrail

•How did we get in this

state?

AWS Lambda

•Are there nonstandard use

of resources?

AWS CloudWatch

•Are we using more

resources than normal?

•Are services working

correctly?

AWS Inspector (Beta)

•Are resources properly

configured?

PROOF OF CONCEPT – SSH LOGIN ALERTS

• Premise: Interactive shell access to systems is strongly discouraged.

• Monitoring

• Authentication logs are auto-forwarded to CloudWatch Logs

• Filter stream set to forward successful logins to Lambda

• Lambda function parses the log message and sends to SNS

• SNS sends to Slack (could also go to PagerDuty, Email, SMS, etc.)

• https://github.com/SCH-CISM/pylambda-login-alerter

CLOUDWATCH

• Pipeline monitoring

• Jobs running longer than expected

• Storage exceeding expected limits

• Service up/down monitoring

• StatusCake

• Custom CloudWatch metrics for processing status

• Limited use of CloudWatch dashboards

APPLICATION MONITORINGThere be dragons…

OTHER PLATFORMS

OPEN SOURCE

• Operations focused

• Zabbix

• Sensu

• Consul

• Security Focused

• Netflix Security Monkey (Internal deployment partially developed)

COMMERCIAL

• REMOVED - SANITIZZED