netflix open source meetup season 4 episode 3

SecuritySeptember 7, 2016

Agenda

6:00-7:00 Registration, Food/Drink, Networking

7:00-8:00 Talks:● Message Security Layer (MSL) - Wes Miaw, Netflix● Repoman - Patrick Kelley, Netflix● Lyft & BLESS - Ryan Lane and Vivian Ho, Lyft● OpenDNS/Cisco & Lemur - Chris Dorros,

OpenDNS/Cisco

8:00-9:00 Demos, Networking

Netflix Security OSS Timeline

2014 2015 2016June:Security

MonkeyOct: MSL

Aug: Scumblr, Sketchy, and Workflowable May: FIDO

Aug: Sleepy Puppy

Sep: Lemur

May: BLESS

Coming: RepoMan,

Dirty Laundry (Scumblr 2.0):

Security Release #10 Coming Soon!

Message Security Layer

a modern security protocol

● MSL in a nutshell● Motivations● Netflix and MSL● External Interest● Continuing Work

Agenda

MSL in a Nutshell

● Transport protocol.○ Security

■ encryption, integrity protection, non-replayability

○ Authentication

■ devices + servers + users

○ (Not Authorization)

Motivations (1)

● HTML5 Standards-Based Playback○ JavaScript, EME, MSE

○ Web browsers & HTML5 runtime environments

● Eliminate SSL/TLS○ initial handshake overhead

○ problematic PKI infrastructure

○ time is always wrong and never trustworthy

Motivations (2)

● Unified Authentication○ authenticate once

○ device + user auth anywhere (client + server)

● Platform & Services Integration○ device-based crypto (or no crypto)

○ third-party user authentication

Motivations (3)

● Updateable & Recoverable○ fixes and features pushed by Netflix

○ recovery from platform crypto or storage bugs

Netflix and MSL - Network Architecture

Once messages are processed by the MSL stack all applications trust entity + user identities.

Netflix and MSL - Trust

● Device Security○ securely identify device types

○ different devices satisfy different levels of content protection

● User Security○ user identity and data bound to the device

External Interest

● Financial firm trying to avoid HTTPS overhead.● Proxy-based service that want to inspect traffic w/o

compromising the communications security.● Company building microservices that require secure

communication and authentication.

Continuing Work

● New device authentication schemes.● Platform-based session keys.● Single-sign-on.● Integration into third-party applications.● Encoder abstraction.

Repoman

IAM Role Access ProfilingCross Account Permission Consistency

Repoman Agenda

● Review: Least Privilege● Dependency: RolliePollie● Workflow Overview● Introducing Role Groups● Access Profiling● Group Template Creation

Least PrivilegeAssigning the correct permissions is non-trivial.

* Too many permissions, nobody complains... until there is an incident.

* Too few permissions, the app is broken.

* There are currently around 2,500 unique AWS permissions. Almost impossible to guess which ones an app requires.

RolliePollieEnforcement Arm of Repoman.

Notifies Security Team, or reverts any changes, if role is ever modified and doesn’t match template.

Consistency is maintained across all AWS accounts.

Workflow Overview

Now: Chaos

Stage 1: Templated ChaosStage 2: Cross-Account ConsistencyStage 3: Access Profiling

Final State: Templated, consistent, least privilege

Role Groups

Especially useful for application roles deployed across many AWS accounts.● SecurityMonkey, Discovery, Lemur, Atlas

Treat a set of IAM roles as a single entity.

Keep their permissions consistent.

Search

Group Summary

Group Member Compare

Group CloudTrail Usage

Access Profiling

Only remove permissions that are supported by CloudTrail.

Handle wildcards & NotActionPreserve ConditionsPreserve Resource & NotResource

Access Advisor data is also incredibly useful.

Access Profiling

Only remove permissions that are supported by CloudTrail.

Handle wildcards & NotActionPreserve ConditionsPreserve Resource & NotResource

Access Advisor data is also incredibly useful.

Template Creation

Template Activation

History allows easy rollback.

All IAM changes require approval from the security team.

Lyft & BLESS

bless without the bastion

BLESS?

● BLESS = “Bastion's Lambda Ephemeral SSH Service”● Short lived (4-minute) certificates issued after strong

user authentication● Small codebase, running on Lambda in a separate AWS

account, as the Certificate Authority

BLESS on the endpoint?

Can we use the same principles as BLESS to allow ephemeral keys on our engineers’ laptops?● Enforce two-factor authentication when issuing a

certificate○ Less concern if their laptop is stolen or 0wned

● Improve employee onboarding/offboarding○ IT doesn’t have to generate the user’s private key○ No “base deploy” to add/remove user’s public key on infrastructure

BLESS + kmsauth

How do we ensure the user requesting the certificate matches the username logging into the server?

Lyft’s kmsauth to cryptographically bind the AWS user to certificate’s username

● Only the AWS user has the permissions to get a (kms encrypted) token for their username

● Lambda will only issue certificate with the kmsauth token’s username

Blessclient

● Small python script to get kmsauth token, assume “use-bless” role (requires MFA), and manage certificate on user’s laptop

● Use ssh_config’s “Match exec” to call python script whenever SSH is invoked○ However, script doesn’t have stdio/stdout bindings, so poor UX

● SSH wrapper script to call script before invoking SSH client for improved UX

Host Certificates

● Hosts gets a one-week certificate from Lambda● Use kmsauth to bind the instance identity to the

hostnames in the certificate● Blessclient manages CA keys on engineer laptops

OpenDNS/Cisco & LemurChris Dorros [cdorros@cisco.com]

Security Engineering @ OpenDNS

How to request a SSL Certificate

How to create an SSL cert

How to request a SSL Certificate

...and the deployment is manual too...

Problems

● Developers have to make the decisions about cryptography○ RSA vs ECDSA?○ 2048 vs 4096?○ Device compatibility vs security?

● Keys are littered everywhere○ Engineers often use laptop to create key/CSR

● Insanely manual, point-and-click, copy-paste process● ..etc

You can write and share plugins for all of these!

Lemur @ OpenDNS

● Wrote plugin for DigiCert○ Lemur plugin architecture FTW!

● Run in our Docker platform called Quadra● AWS RDS for Lemur DB● Keys transferred to Secrets storage service● Deployed from secrets storage to SSL endpoints

What’s Next?

● Increased usage of Lemur API for automation● Automatic certificate rotation● Short-lived certs● Integration with our HSMs

○ For internal CA● Let’s Encrypt● More self-service for devs

Demos in the Cafe!