attack-driven defense

A"ack-‐Driven Defense

[email protected] @zanelackey

Who am I?

•  Director of Security Engineering @ Etsy – Lead AppSec/NetSec/SecEng/RiskEng teams

•  Formerly @ iSEC Partners

This talk simply isn’t possible without a number of people:

–  Ben Hughes –  Brendan Adamson –  Corey Benninger –  Kai Zhong –  Ken Lee –  Kyle Barry – Marcus Barczak – Mike Arpaia – Omar Ahmed

Also a shout out to secteams we’ve enjoyed collaboraUng with:

Facebook/GitHub/Google/Square/Twi"er

What is Etsy?

Etsy? Security? … Why!?

Advanced Persistent Tapestries

This talk is about shiXing from historically un-‐contextualized defensive approaches

To building defenses around real world a"ack pa"erns

Un-‐contextualized?

Historically defense has: – Focused on the perimeter – Deployed security products that don’t address real a"ack scenarios

– Treated vulnerability enumeraUon (or worse, compliance) as “pentesUng”

These don’t address modern a"ack behavior

What should we be doing?

Fundamentally we have three goals:

1)  Raise cost to a"ackers

2)  Increase the odds of detecUng compromise

3)  Iterate defenses based on real a"ack pa"erns

Increasing detecUon

Build your defenses from an offensive mindset

Instrument detecUon mechanisms around key areas of the a"ack chain: •  IniUal compromise –  Defensive rootkidng

•  Persistence/C2 –  Host level –  OrganizaUonal level

•  Lateral Movement –  Network/systems discovery –  InformaUon discovery

IniUal compromise

Rootkit your endpoints before your a"ackers do

Focus on the combinaUon of system behaviors and commands executed

Specifically, log commands executed on endpoints and analyze this data

Analyze the data and build automaUc alerUng from anomalies

Anomalies?

From a macro level, bucket users into technical vs non-‐technical

Pa"erns then break down into: – Anomalous if by a non-‐technical user – Anomalous if by technical user – Always anomalous

Non-‐Technical bucket: –  Alert off any commands which show technical capabiliUes

•  It’s either an a"acker or your IT team

Technical bucket: –  Treat individual commands and behaviors as low quality signals •  Aggregate commands, look for unique combinaUons and bursts

Always anomalous (both buckets): –  Analyze a"ack pa"erns and idenUfy commands/behaviors strongly indicaUve of compromise •  We’re looking at you, `uname -‐a`

Sample output of auditd execuUon logging: header,163,11,execve(2),0,Mon Jul 29 20:31:11 2013, + 373 msec,exec arg,praudit,-l,current,path,/usr/sbin/praudit,path,/usr/sbin/praudit,attribute,100755,root,wheel,16777220,171538,0,subject,zlackey,root,wheel,root,wheel,78044,100005,50331650,0.0.0.0,return,success,0,trailer,163,

Persistence

Host level persistence: – Look for common pa"erns of persistence via programs executed on boot, kernel modules loaded, etc

Host level persistence: – Look for common pa"erns of persistence via programs executed on boot, kernel modules loaded, etc

– Understand that in pracUce sophisUcated persistence mechanisms won’t be detected •  Aim to detect the basics, and increase a"acker cost by forcing use of custom persistence mechanisms

Shout out to @mimeframe and the FB secteam for their work on BigMac:

h"p://www.slideshare.net/mimeframe/ruxcon-‐2012-‐15195589

PresenUng the Etsy version of a host IDS:

PresenUng the Etsy version of a host IDS: Tripyarn

Tripyarn’s goal is to alert off real world pa"erns of compromise and persistence

Lessons learned from detecUng hosUle persistence mechanisms

First, find the legiUmate OS-‐provided mechanisms you’re interested in instrumenUng

Treat addiUons to/modificaUon of these mechanisms as an event

For rare events, alert on every occurrence – Low false posiUve cost

Ex: – New SSH keys being added to a host – Crontabs being created – etc

For events that happen oXen, use data aggregated across the organizaUon to detect

anomalies

Example: Kernel modules

Goal: Detect a malicious kernel module loading on an endpoint – We thought kernel modules loading would be fairly rare aXer boot and we could alert off that alone. We were wildly wrong.

– WhitelisUng/blacklisUng kernel module names wouldn’t be effecUve

–  Instead, analyze a kernel module being loaded for organizaUonal uniqueness

“Did module X that just got loaded on endpoint Y get loaded on less than N systems across the

organizaUon in the last D days?”

Use a"ack post-‐exploitaUon techniques in a defensive context by separaUng your objecUves

from your tooling

Specifically, collect data on the endpoints and analyze/alert from that data on the server-‐side

When an a"acker discovers and analyzes your endpoint security mechanisms they shouldn’t be able to automaUcally learn all alerts in place

OrganizaUonal level persistence: –  LegiUmate remote access mechanisms or cloud systems with data desired by a"acker •  Ex: VPN and GMail

– Use a mixed approach of manual and automated anomaly detecUon for these systems •  GeneraUng daily rollups of new accounts/keys created •  AlerUng off account creaUon/modificaUon at unusual Umes, from unusual locaUons, etc

Example: GMail

Goal: Instrument GMail to detect compromise of domain admin accounts –  GMail provides logs of interesUng acUons via Admin Audit API and Email Audit API

–  Pull down logs via these APIs, store them locally so you have a record of acUons

–  Perform alerUng on strong signals of compromise and persistence: •  Signins from unusual locaUons/Umes •  CreaUon of new admin level accounts •  CreaUon of new mail-‐forwarding filters •  Any change to 2FA sedngs •  Etc

Lateral movement

Focusing on two areas of lateral movement: 1.  Network/systems discovery 2.  InformaUon discovery

Use endpoint firewalls as a detecUon mechanism (NOT a blocking one)

Build alerts around services unused on your network but likely interesUng to a"ackers

By alerUng on (but not blocking!) traffic you don’t immediately signal there’s a detecUon

mechanism in place

Also endpoint firewalls counter Uming-‐based evasions

Any traffic to targeted service, no ma"er how slow, causes alerts

InformaUon Discovery

What internal systems provide informaUon that

help an a"acker achieve their goals?

-‐ Wikis -‐ Source control -‐ Bug tracking -‐ Etc

Instrument these systems the way you would

other high-‐value pieces of infrastructure

Alert off behavioral anomalies such as:

– Usage outside of normal hours •  Your a"ackers are rarely in your Ume zone

–  Bursts of acUvity •  Viewing all security Uckets in the bug tracker isn’t even done by the security team

–  Etc

Increasing a"acker cost

Make compromise more expensive – We’ll discuss:

•  Reducing trusted CA roots •  Removing cheap exploitaUon vectors •  Forcing updates without the force •  LimiUng drive-‐by exposure •  PracUcal goals for security awareness training

How can you reduce the likelihood of a DigiNotar-‐like MITM happening against your

organizaUon?

If you remove unused CAs, when one is compromised it can’t silently MITM your

endpoints

We performed several months of anonymized traffic analysis to record what CAs were seen

during our employees Internet usage

We found less than 29% of SSL CerUficate AuthoriUes trusted by our endpoints were

actually used

21.29% EQUIFAX SECURE CERTIFICATE AUTHORITY 10.37% ENTRUST.NET SECURE SERVER CERTIFICATION AUTHORITY 10.07% DIGICERT HIGH ASSURANCE EV ROOT CA 8.97% GO DADDY CLASS 2 CERTIFICATION AUTHORITY 7.91% GEOTRUST GLOBAL CA 7.23% ADDTRUST EXTERNAL CA ROOT 6.48% HTTP://WWW.VALICERT.COM/ 6.04% GTE CYBERTRUST GLOBAL ROOT 4.45% VERISIGN CLASS 3 PUBLIC PRIMARY CERTIFICATION AUTHORITY -‐ G5 4.08% CLASS 3 PUBLIC PRIMARY CERTIFICATION AUTHORITY 3.82% BALTIMORE CYBERTRUST ROOT 3.22% CLASS 3 PUBLIC PRIMARY CERTIFICATION AUTHORITY -‐ G2 1.37% THAWTE PRIMARY ROOT CA 1.36% THAWTE PREMIUM SERVER CA 1.33% ENTRUST.NET CERTIFICATION AUTHORITY (2048) 0.65% GLOBALSIGN ROOT CA [The CAs which had < 0.5% traffic have been edited out for brevity. More info here: h"p://codeascraX.com/2013/07/16/reducing-‐the-‐roots-‐of-‐some-‐evil/]

Our raw results:

By removing only unused CAs you don’t teach

users to click through SSL errors

ConUnue traffic analysis, add/remove trusted CAs as appropriate

in the browser is: cheap, reliable, and

efficient (pick three!)

in the browser is: cheap, reliable, and

efficient (pick three!)

…for a"ackers

What did we learn when we removed Java web plugins from the enterprise?

•  Hardly any groups actually needed it – Network OperaUons, for legacy networking equipment

•  For them, we built dedicated Java jump boxes

Benefits:

1.  No Java on any laptops/desktops 2.  Boxes with Java can’t hit Internet 3.  Able to frequently re-‐image jump boxes 4.  Only a few boxes to patch

But…

Java shows back up when you apply Apple

patches.

But…

Java shows back up when you apply Apple

patches. Remove it on an ongoing basis

Browser updates

We wanted a less heavy handed approach to ensuring up to date browsers

Built browser detecUon logic into our internal SSO point

UX is key: Show in screenshots how quick it is to update,

provide a bypass mechanism

Simply asking users to update works shockingly well

Funny story, users will install malware because

an ad popup told them to.

Funny story, users will install malware because

an ad popup told them to. O8en.

You can almost enUrely kill this source of

compromise (for free!) by pushing adblocker plugins to the organizaUon

Security awareness training

Historically we’ve focused on reducing the number of people who fall for phishing

Historically we’ve focused on reducing the number of people who fall for phishing

This is the wrong goal

If you go from being 36% on fire to 27% on fire

you’re s;ll on fire

Instead, focus on incenUvizing users to:

The metric to track/increase is the likelihood of phishing emails being reported to security

Even if 36% sUll fall for phishing, as long as one

in the group reports it IR can begin

XXX

Running effecUve a"ack simulaUons

Problems with “pentesUng” are well understood in the offensive community but not as well in

the defensive community

Pentests typically result in a list of enumerated known vulnerabiliUes to be patched, not data on how a real a"acker would operate against a

given environment

A"ack simulaUons should be done to learn how a"ackers are likely to achieve goals against your

organizaUon

NOT to show compromise is possible (spoiler alert: it is.)

Use this a"ack data to focus where/how to build detecUon mechanisms

From an organizaUonal side, a"ack simulaUons compliment vulnerability enumeraUon/

compliance/etc

Vulnerability enumeraUon/compliance are checklists to make sure you’re covering the

basics

But checklists aren’t owning you, a"ackers are

Four keys to effecUve a"ack simulaUons:

–  Goal oriented •  “Obtain domain admin”, “read the CEOs email”, “view credit card data”, …

–  Full organizaUon in scope •  Have a"ack team call a contact if they’re about to do something risky

–  Simulate realisUc compromise pa"erns •  Start the a"ack team on a standard laptop/desktop endpoint inside the

organizaUon to simulate phishing/clientside compromise •  Start the a"ack team on a database or web server to simulate SQL injecUon/

RCE •  A"ack team should be encouraged to use 0days

–  Break simulaUon down into iteraUons: •  Don’t spend the full engagement Ume on only round of tesUng, once one team

achieve goal(s), then swap in new a"ack team to achieve the same goal(s) –  Ex: We try to run 3-‐4 iteraUons per several week simulaUon

Four keys to effecUve a"ack simulaUons:

–  Goal oriented •  “Obtain domain admin”, “read the CEOs email”, “view credit card data”, …

–  Full organizaUon in scope •  Have a"ack team call a contact if they’re about to do something risky

–  Simulate realisUc compromise pa"erns •  Start the a"ack team on a standard laptop/desktop endpoint inside the

organizaUon to simulate phishing/clientside compromise •  Start the a"ack team on a database or web server to simulate SQL injecUon/

RCE •  A"ack team should be encouraged to use 0days throughout engagement

–  Break simulaUon down into iteraUons: •  Don’t spend the full engagement Ume on only round of tesUng, once one team

achieve goal(s), then swap in new a"ack team to achieve the same goal(s) –  Ex: We try to run 3-‐4 iteraUons per several week simulaUon

The project output should be a>ack chains showing how a"ack team went from A-‐>B-‐>C to achieve goals, what steps they took and why

Just as importantly, what steps they didn’t take

Ex: “We didn’t try to find internal network diagrams on your wiki because zone transfers were enabled so we could got enough data about your network from that”

Remember, the goal is to simulate realisUc a"ack behaviors and pa"erns that can be used

to enhance detecUon

In addiUon, simulate varying a"ack profiles from quick & noisy to quietly maintaining persistence

Over mulUple iteraUons learn what behaviors overlap between a"ackers and what strong

signals of lateral movement in your environment look like

TL;DR (The secUon formerly known as “Conclusions”)

Defend like an a"acker

Where should defense focus? –  Increase a"acker cost by reducing cheap compromise vectors

–  Build detecUon mechanisms around real a"ack pa"erns •  Analyze past compromises, new offensive research, and conduct realisUc a"ack simulaUons to obtain data –  Depending on scale, have true offensive capabiliUes on staff or a call away

– Have product/tooling development capabiliUes within your security team •  Roughly one quarter of our team is soXware engineers who focus on building internal security products

Thanks!

[email protected] @zanelackey

attack-driven defense

Technology

ad popup told

apply apple

admin audit

automated

full engagement

unusual locajonsjmes

treat individual

acjons perform