london devops #9 - security at a startup
TRANSCRIPT
![Page 2: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/2.jpg)
1
![Page 3: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/3.jpg)
2
• At a huge competitive advantage compared to enterprises • No RFPs, Long winded procurement contracts • Outsource commodity services to others:
– Most obviously – Hosting – AWS – Monitoring – Dataloop.io – Alerting – Email – Source control and versioning – Messaging
Startups
![Page 4: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/4.jpg)
3
![Page 5: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/5.jpg)
![Page 6: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/6.jpg)
5
![Page 7: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/7.jpg)
6
![Page 8: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/8.jpg)
7
![Page 9: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/9.jpg)
8
![Page 10: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/10.jpg)
![Page 11: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/11.jpg)
10
?
![Page 12: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/12.jpg)
11
![Page 13: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/13.jpg)
12
![Page 14: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/14.jpg)
- A security oversight in one system can often be used to compromise another
- Are you aware when a new email address gets added? - What’s this IP address in our AWS security group? When was it
added? Who added it? If I remove it what will stop working? - Whose Github account is this? - What’s this hook on my github repo? - The list is endless, and is easily work for a dedicated team - That team will never exist in a startup!- I just want to focus on cutting our deploy time in half and
implementing Docker!
Security Jenga
![Page 15: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/15.jpg)
14
One potential solution… - A light weight process that actively audits security and alerts your
“Security team” when something doesn’t “smell” right - Needs to be extensible to keep up with all those new third party
services being added - Adding services needs to take up as little time as possible
![Page 16: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/16.jpg)
15
Single (secure) point of truth - Can be as simple as a text file on github - We chose LDAP - Deserving in its reputation as a PITA to setup: 6 months to learn, build, test, secure,
and populate (More on that later) - It integrates with everything (More on that later too) - Created a custom schema extension for a Beamly employees
- Github ID - Facebook User UID - Public SSH Key
- At first internal only - Now externally accessible
![Page 17: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/17.jpg)
16
Security “unit” tests • Now you’ve got a version controlled state • If you can programmatically extract a list of users from a
service, you can compare against that • Applies to more than just user accounts:
– AWS Security groups – Github hooks
• Store exceptions with explanatory comments • Track changes with git history
![Page 18: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/18.jpg)
17
Beamly Security Scripts • Implemented as Python “unit” tests (py.unit) • Short & quick to write • Executed every 10 minutes via a scheduled CD pipeline • Failures trigger an email to the platform team • Email can trigger whatever else you want (Pagerduty) • Platform team can login and take appropriate action
![Page 19: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/19.jpg)
18
![Page 20: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/20.jpg)
19
![Page 21: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/21.jpg)
20
![Page 22: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/22.jpg)
21
![Page 23: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/23.jpg)
22
collected 1280 items test_all_github_hooks_are_known.py ........................................................................................................................................................................................................................................................................................................................................................................................... test_aws_iam_users_with_passwords_have_2fa.py ..................................................................................... test_aws_security_groups.py ................................................................................................................................................................................................... test_github_users_and_in_ldap_and_2fa.py ............................................................. test_jira_users_not_in_ldap.py ....................................................................................................................................................................................... test_o365_users.py ........................................................................................................... test_onelogin_users_are_in_ldap.py ........................................................................................... test_pagerduty_users_not_in_ldap.py ........................ test_slack_users_not_in_ldap_and_2fa.py ............................................................F.............................................................................................. =================================== FAILURES =================================== _________________________ test_unknown_slack_users[60] _________________________ realname = ’New Joiner', two_factor_enabled = False def two_factor_enabled(realname, two_factor_enabled): > assert two_factor_enabled E assert False test_slack_users_not_in_ldap_and_2fa.py:42: AssertionError =================== 1 failed, 1279 passed in 145.26 seconds ====================
![Page 24: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/24.jpg)
23
![Page 25: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/25.jpg)
24
![Page 26: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/26.jpg)
25
![Page 27: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/27.jpg)
26
![Page 28: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/28.jpg)
27
Systems Integrated with LDAP - VPN access to VPC - Host access (Using SSH Public keys) - Sudo access (Via group) - Office appliances (NAS) - ….any application that supports it – GO, Grafana (Soon)
Couldn’t have done this if we’d have used a text file
![Page 29: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/29.jpg)
28
![Page 30: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/30.jpg)
29
• Herd all these 3rd party applications in to one place • Password manager on steroids • Centrally managed • Every application used by Beamly added by Platform team • Authenticate with LDAP account and 2FA • Single sign on just to services that users have rights to (LDAP groups) • Users never know the password for shared systems • Can bulk rotate shared passwords for all systems
OneLogin
![Page 31: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/31.jpg)
30
![Page 32: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/32.jpg)
31
![Page 33: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/33.jpg)
32
Now what?!
![Page 34: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/34.jpg)
33
“Preppers”!
![Page 35: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/35.jpg)
![Page 36: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/36.jpg)
![Page 37: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/37.jpg)
![Page 38: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/38.jpg)
37
BE A PREPPER!
![Page 39: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/39.jpg)
![Page 40: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/40.jpg)
Part I: Stock your stores - Backups • Taking backups isn’t good enough • 3 -2 -1 • 3 Copies of your data • 2 in different formats • 1 Offsite • S3 mutli-region ISN’T OFF SITE • S3 > (Another cloud provider || On site) • March 31st: World backup day
![Page 41: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/41.jpg)
Part II: Run through restoring from nothing • In the event of the Armageddon, you need to fight to get back up quickly • How long would if take to restore your AWS “Scaffolding” alone
– VPC Configuration – Security groups – Routing tables – Subnet configurations – DNS
• Ideally you’d have all this in Cloudformation…. • ….We set all this stuff up before Cloudformation was available • AWS Cloudformer
![Page 42: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/42.jpg)
41
To sum up: our approach • If a service supports 2FA, it’s mandatory.
– Currently Slack, Mailchimp, Github, AWS, and Onelogin • If a service has an API for exporting a security config: try and script an
audit for it • All third party services in Onelogin • Bulk rotate shared passwords regularly • Prepare for Armageddon:!
– Backup CD config & AWS setup – All data tiers backed up 3-2-1 (One offsite)
![Page 43: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/43.jpg)
42
Lessons Learned • Retro-fitting security is hard • Be prepared to break lots of things (temporarily) • Bring everyone along for the ride
– Explain why it’s important – Everyone gets the access they need – #security for interested parties
• Turn on AWS Cloudtrail and Config everywhere
![Page 45: London Devops #9 - Security at a startup](https://reader033.vdocument.in/reader033/viewer/2022051710/5a65a7ec7f8b9ab3488b4a43/html5/thumbnails/45.jpg)
LONDON
Drury House 34-43 Russell Street
London WC2B 5HA
NEW YORK CITY
84 Wooster Street Suite 703 New York NY 10012
SYDNEY
22-36 Mountain Street Suite 1.10
Ultimo NSW 2007
© Beamly 2014. All Rights Reserved.