sk8ing on thin ice: a crash course in kubernetes & security · source: verizon dbir - 2016. the...
TRANSCRIPT
1The retirement benefit that benefits everyone
Matt Johansen Director of Security
Honest Dollar @mattjay
The retirement benefit that benefits everyone
A Simple, Portable & Affordable Savings Solution
October 2016
Sk8ing on Thin Ice: A Crash Course in Kubernetes & Security
2The retirement benefit that benefits everyone
Honest Dollar is a company dedicated to helping people save in
a society built on spending
3The retirement benefit that benefits everyone
Information Security Officer
Matt Johansen Long Term Security Guy, Sometimes Developer
SXSW, BlackHat, DEFCON, RSA, more++
4The retirement benefit that benefits everyone
Agenda
Security & Kubernetes
What is it? | What is Kubernetes anyway? (This portion will not include how to pronounce it)
Why use it? | “Because $Cool_Unicorn is using it” is not acceptable.
Security Pitfalls | Trust me. There are holes to fall into.
Security Benefits | You mean there are benefits to containerization outside of DevOps?
Examples | No Live Demos™ - But lets walk through some cool security monitoring, logging, alerting, and other tricks. Along with some basic security hygiene.
5The retirement benefit that benefits everyone
Kube
Kubernetes
6The retirement benefit that benefits everyone
Kube
Kubernetes
Source: Imesh Gunaratne - Intro to Kubernetes
7The retirement benefit that benefits everyone
Kubernetes
Things to keep in mind
YAML Land - Config Management is hard
Different Environments - We’re running a company here. Prod, Dev, QA, etc. need processes.
Persistence (and not) - Pods die. Its okay (and sometimes awesome). Manage your data intelligently.
Secret Sauce - Docker is not the magic. Orchestration is the magic. Learn and love your Services, RCs, and Pods.
Kube
Words I might say
Node - Worker VMs.
Pod - Group of Containers.
Replication Controller - Manages Pod lifecycle & config.
Service - Config for a set of Pods.
Cluster - Single network & group of Nodes.
8The retirement benefit that benefits everyone
Pitfalls
9The retirement benefit that benefits everyone
Cluster
10The retirement benefit that benefits everyone
Cluster
11The retirement benefit that benefits everyone
Cluster
Bad Idea
Nearly identical YAMLs. Duplication of work and error prone.
12The retirement benefit that benefits everyone
Cluster
Separation of Concerns
13The retirement benefit that benefits everyone
Environment Stability
Cluster
Options • Puppet (kinda)
• Shell scripts + templates
• rakefiles + templates (Us Currently)
Lessons Learned • Launch as much as possible, with the fewest
commands possible.
• Track versions and automate container building if possible.
• Automate testing / build pipeline to watch out for regressing over hot fixes.
• Track versions (and dependencies!) between environments
14The retirement benefit that benefits everyone
IP
Load Balancing
Watch your logs if you do this. You’ll lose origin IP.
Solution: Break nginx out of cluster (I know, I know)
15The retirement benefit that benefits everyone
Loophole
kubectl
Whats wrong with this picture?
16The retirement benefit that benefits everyone
Benefits
17The retirement benefit that benefits everyone
Security Benefits
Security
Containerization • Patching - Oh you mean `docker build`?
• One process per container - Less vulnerability surface for stuff you don’t use.
• Mobility - Host agnostic
• Segregation - Less pivot surface
• Typically short lifespan - Less old, stale, vulnerable systems.
Kubernetes • Upgrade process - Bring nodes down, bring new
ones up. Pods find their way to a new home.
• Log granularity - Know and prioritize your log events.
• Build pipeline and web hook friendly.*
• Rate of change it enables is incredibly fast.
• Transient by nature.
*Continuous Integration is no longer a ‘nice to have’. For security it is a must.
18The retirement benefit that benefits everyone
Fix Fast
Continuous Integration
Good luck with you two week release cycle.
Source: Verizon DBIR - 2016
19The retirement benefit that benefits everyone
Watch Everything
Monitoring
Log everything. Watch your logs.
Source: Verizon DBIR - 2016
20The retirement benefit that benefits everyone
Monitoring
Log all the things
Logging • ElasticSearch
• rsyslog + RELP
• auditd (go-audit)
• OSQuery
• nginx logs
• Docker logs
Alerting • ElastAlert
• Slack bots
• SSH Connections
• Suspicious commands (curl out of a prod server?)
• File watch
• Anomalies (Geo, IP, data movement, spikes, etc.)
21The retirement benefit that benefits everyone
Matt Johansen Director of Security
Honest Dollar @mattjay
The retirement benefit that benefits everyone
A Simple, Portable & Affordable Savings Solution
October 2016
Thank You