the osquery file carver - osdfcon · #!/bin/bash nc -l -p 1337. what is osquery? ......

70

Upload: others

Post on 16-Jul-2020

9 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration
Page 2: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

The osquery File Carver

Nick AndersonSecurity Engineer

Page 3: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• Nick Anderson • Security Engineer at Facebook

[email protected] • Super legit, not an alias

• Github - github.com/muffins

• Twitter - twitter.com/poppyseedplehzr

• Slack - thor

C:\> Get-Host

Page 4: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

A quick story

Page 5: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

Analyst

Enterprise Endpoint

Page 6: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

Analyst

Enterprise Endpoint

/tmp/evil

Page 7: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

Analyst

Enterprise Endpoint

/tmp/evil

Page 8: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

Analyst

Enterprise Endpoint

/tmp/evil

#!/bin/bashnc -l -p 1337

Page 9: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

Analyst

Enterprise Endpoint

/tmp/evil

#!/bin/bashnc -l -p 1337

Page 10: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

What is osquery?

Page 11: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• FOSS host based IDS

What is osquery?

Page 12: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• FOSS host based IDS • Cross-platform

What is osquery?

Page 13: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• FOSS host based IDS • Cross-platform • Abstracts OS as SQLite tables

What is osquery?

Page 14: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• FOSS host based IDS • Cross-platform • Abstracts OS as SQLite tables • Performant

What is osquery?

Page 15: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• FOSS host based IDS • Cross-platform • Abstracts OS as SQLite tables • Performant • Extensible

What is osquery?

Page 16: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• FOSS host based IDS • Cross-platform • Abstracts OS as SQLite tables • Performant • Extensible • https://osquery.io

What is osquery?

Page 17: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

osquery architecture

Page 18: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

How does osquery work?dispatcher

SchedulerRunnerCarver

ExtensionRunnerCore

ConfigRefresh Runner

DistributedRunner

WatcherRunner

InternalRunnable

Page 19: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

How does osquery work?dispatcher

Carver

ExtensionRunnerCore

ConfigRefresh Runner

DistributedRunner

WatcherRunner

InternalRunnable

SchedulerRunner

Page 20: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

How does osquery work?dispatcher

Carver

ExtensionRunnerCore

ConfigRefresh Runner

DistributedRunner

SchedulerRunner

InternalRunnable

WatcherRunner

Page 21: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

How does osquery work?dispatcher

Carver

ExtensionRunnerCore

ConfigRefresh Runner

SchedulerRunner

InternalRunnable

WatcherRunner

DistributedRunner

Page 22: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

ad-hoc distributed live on-demand deployment

Page 23: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage

Analyst

cloud

off prem

enterprise

Page 24: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage

Analyst

cloud

off prem

enterprise

Page 25: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage

Analyst

cloud

off prem

enterprise

Page 26: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage

Analyst

cloud

off prem

enterprise

Page 27: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage

Analyst

cloud

off prem

enterprise

Page 28: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage

Analyst

cloud

off prem

enterprise

Page 29: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage

Analyst

cloud

off prem

enterprise

Page 30: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage

Analyst

cloud

off prem

enterprise

Page 31: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

distributed queries

Page 32: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storageSELECT p.pid, p.name, lp.port, lp.address, lp.path FROM processes AS p JOIN listening_ports AS lp USING (pid);

Analyst

cloud

off prem

enterprise

Page 33: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

Analyst

TLS

configuration

storage

SELECT p.pid, p.name, lp.port, lp.address, lp.path FROM processes AS p JOIN listening_ports AS lp USING (pid);

cloud

off prem

enterprise

Page 34: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage

AnalystSELECT p.pid, p.name, lp.port, lp.address, lp.path FROM processes AS p JOIN listening_ports AS lp USING (pid);

cloud

off prem

enterprise

Page 35: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage

SELECT p.pid, p.name, lp.port, lp.address, lp.path FROM processes AS p JOIN listening_ports AS lp USING (pid);

Analyst

cloud

off prem

enterprise

Page 36: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage

SELECT p.pid, p.name, lp.port, lp.address, lp.path FROM processes AS p JOIN listening_ports AS lp USING (pid);

Analyst

cloud

off prem

enterprise

Page 37: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage

SELECT p.pid, p.name, lp.port, lp.address, lp.path FROM processes AS p JOIN listening_ports AS lp USING (pid);

Analyst

cloud

off prem

enterprise

Page 38: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage[ {"address":"0.0.0.0","name":"rapportd","path":"","pid":"507","port":"49177"}, {"address":"::","name":"rapportd","path":"","pid":"507","port":"49177"}, {"address":"0.0.0.0","name":"SystemUIServer","path":"","pid":"601","port":"64562"}, {"address":"127.0.0.1","name":"scm_daemon","path":"","pid":"749","port":"15432"}, {"address":"127.0.0.1","name":"Dropbox","path":"","pid":"771","port":"17603"}]

SELECT p.pid, p.name, lp.port, lp.address, lp.path FROM processes AS p JOIN listening_ports AS lp USING (pid);

Analyst

cloud

off prem

enterprise

Page 39: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

Repurposing Runnables

Page 40: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

How does osquery work?dispatcher

ExtensionRunnerCore

ConfigRefresh Runner

SchedulerRunner

InternalRunnable

WatcherRunner

DistributedRunnerCarver

Page 41: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

How does osquery the carver work?dispatcher

ExtensionRunnerCore

ConfigRefresh Runner

SchedulerRunner

InternalRunnable

WatcherRunner

DistributedRunner

Carver

Page 42: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

How does osquery the carver work?dispatcher

ExtensionRunnerCore

ConfigRefresh Runner

SchedulerRunner

InternalRunnable

WatcherRunner

DistributedRunner

Carver

SELECT * FROM carves WHERE carve=1 AND ...

Page 43: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

How does osquery the carver work?dispatcher

ExtensionRunnerCore

ConfigRefresh Runner

SchedulerRunner

InternalRunnable

WatcherRunner

DistributedRunner

Carver

SELECT * FROM carves WHERE carve=1 AND ...

Carver::Carver

Page 44: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

How does osquery the carver work?dispatcher

ExtensionRunnerCore

ConfigRefresh Runner

SchedulerRunner

InternalRunnable

WatcherRunner

DistributedRunner

Carver

SELECT * FROM carves WHERE carve=1 AND ...

Carver::Carver

1. Create a temp file store

Page 45: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

How does osquery the carver work?dispatcher

ExtensionRunnerCore

ConfigRefresh Runner

SchedulerRunner

InternalRunnable

WatcherRunner

DistributedRunner

Carver

SELECT * FROM carves WHERE carve=1 AND ...

2. Grab the specified files

1. Create a temp file store

Carver::Carver

Page 46: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

How does osquery the carver work?dispatcher

ExtensionRunnerCore

ConfigRefresh Runner

SchedulerRunner

InternalRunnable

WatcherRunner

DistributedRunner

Carver

SELECT * FROM carves WHERE carve=1 AND ...

3. Create a .tar

Optionally compress

2. Grab the specified files

1. Create a temp file store

Carver::Carver

Page 47: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

How does osquery the carver work?dispatcher

ExtensionRunnerCore

ConfigRefresh Runner

SchedulerRunner

InternalRunnable

WatcherRunner

DistributedRunner

Carver

SELECT * FROM carves WHERE carve=1 AND ...

boost::filesystem::temp_directory_path()

Carver::carve(const boost::fs::path& p)

archive(std::set<boost::fs::path> paths)

compress(const boost::fs::path& p)

4. POST the data

3. Create a .tar

Optionally compress

2. Grab the specified files

1. Create a temp file store

Carver::Carver

Page 48: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

How does osquery the carver work?dispatcher

ExtensionRunnerCore

ConfigRefresh Runner

SchedulerRunner

InternalRunnable

WatcherRunner

DistributedRunner

Carver

SELECT * FROM carves WHERE carve=1 AND ...

Update DB

{ 'time': 1537852244.4733381, 'sha256': '<sha256>', 'size': 123, 'path': '<Path of Carves', 'status': 'SUCCESS' }

4. POST the data

3. Create a .tar

Optionally compress

2. Grab the specified files

1. Create a temp file store

Carver::Carver

Page 49: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

The file carver

Page 50: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

TLS

configuration

storage

cloud

off prem

enterprise

Page 51: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

configuration

storage

SELECT * FROM carves WHERE carve=1 AND path='/tmp/evil';

cloud

off prem

enterprise

TLS

Page 52: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

off prem

configuration

storage

SELECT * FROM carves WHERE carve=1 AND path='/tmp/evil';

cloud

enterprise

TLS

Page 53: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

configuration

storage

SELECT * FROM carves WHERE carve=1 AND path='/tmp/evil';

cloud

off prem

enterprise

TLS

Page 54: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

configuration

storage

SELECT * FROM carves WHERE carve=1 AND path='/tmp/evil';

cloud

off prem

enterprise

TLS

Page 55: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

Use Cases

Page 56: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• Auto Carving • New files • Case artifacts

Use Cases

Page 57: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• Auto Carving • New files • Case artifacts

• Asynchronous acquisitions • Endpoint is off corp network • Laptop is shut and then opened

Use Cases

Page 58: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

Surprises

Page 59: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

SELECT * FROM carves WHERE carve=1 AND path LIKE '/Users/%/Downloads/%';

Carver Wins

Page 60: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

Limitations

Page 61: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• Watchdog and size limits

Carver Limitations

Page 62: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• Watchdog and size limits • "carves"

Carver Limitations

Page 63: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• Watchdog and size limits • "carves" • Block POST retry

Carver Limitations

Page 64: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• Watchdog and size limits • "carves" • Block POST retry • Limited to TLS endpoints

Carver Limitations

Page 65: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• Watchdog and size limits • "carves" • Block POST retry • Limited to TLS endpoints • How does one carve?

Carver Limitations

Page 66: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

Carver Endpoints

Page 67: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• Zentral • github.com/zentralopensource/zentral

• SGT • github.com/OktaSecurityLabs/sgt

Carver Endpoints

Page 68: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• Doorman issue • github.com/mwielgoszewski/doorman/issues/120

• osquery integration test • osquery/tools/tests/test_http_server.py

Carver Endpoints Specs and Docs

Page 69: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration

• Nick Anderson • Security Engineer at Facebook

[email protected] • Super legit, not an alias

• Github - github.com/muffins

• Twitter - twitter.com/poppyseedplehzr

• Slack - thor

Thanks! Questions?

Page 70: The osquery File Carver - OSDFCon · #!/bin/bash nc -l -p 1337. What is osquery? ... Carver::Carver. The file carver. TLS configuration storage cloud off prem enterprise. configuration