hunting walkthrough with the sqrrl test drive vm · identifying potential persistent threats to...

Hunting Walkthrough with the Sqrrl Test Drive VM Sqrrl Enterprise Version: 2.8.x

Hunting Walkthrough with the Sqrrl Test Drive VM 2

©2013-2018 Sqrrl, Inc. All rights reserved.

Contents Setting up and logging in ............................................................................................. 4

Configuring and starting the VM ............................................................................. 4Logging in to the Sqrrl web application .................................................................... 4

About the CounterOps model ....................................................................................... 5What is the CounterOps model? ............................................................................. 5Overview of the CounterOps model structure ............................................................. 5

Network layout ....................................................................................... 5Network users and activity ......................................................................... 6Detection results ..................................................................................... 7

About the raw data .................................................................................................... 8Source connector data ........................................................................................ 8CarbonBlack endpoint data ................................................................................... 8Threat intelligence data ...................................................................................... 8

Getting your bearings in the web application .................................................................... 9Viewing the available sources ............................................................................... 9Exploring the CounterOps model structure .............................................................. 11Displaying and searching the Sqrrl documentation ..................................................... 12Returning to the risk dashboard ........................................................................... 13Exploring the risk dashboard ............................................................................... 13

Changing the risk dashboard time frame ....................................................... 13About the detection result scores ............................................................... 13Filtering and sorting detection results and entity instances ................................ 14

Overview of the threat hunting process ......................................................................... 15

Let's hunt for suspicious behavior! ............................................................................... 161. Create hypotheses ........................................................................................ 162. Investigate via tools and techniques ................................................................... 16

Question 1: Was this an attack? .................................................................. 17Reviewing the activity on the detection result profile ................... 17Finding the involved systems ................................................. 17Researching the destination domain ........................................ 17Analyzing the traffic ........................................................... 18Analyzing the timing of the connections .................................... 18Analyzing related suspicious activity ........................................ 19



Question 2: Was the attack successful? ......................................................... 19Question 3: What other resources were involved? ............................................ 20Question 4: What activities did the adversary conduct? ..................................... 21Question 5: What resources were compromised? ............................................. 21Question 6: What are the next steps? ........................................................... 21Bonus material: Advanced investigation queries .............................................. 22

Beaconing after connecting to webmail .................................... 223. Uncover new patterns and TTPs ........................................................................ 23

Saving your investigation ......................................................................... 24Exporting the results of your investigation .................................................... 24

4. Inform and enrich analytics ............................................................................. 24Creating a risk trigger ............................................................................. 24Updating the detector whitelist ................................................................. 26

Your turn! .............................................................................................................. 27



Setting up and logging in

Configuring and starting the VM We are using VirtualBox as the hypervisor.

1. Configure a host-only adaptor for 192.168.123.0/24 called vboxnet0 .

2. Set your host to be 192.168.123.1.

3. Import the OVA into VirtualBox.

4. Start it!

Logging in to the Sqrrl web application 1. Wait a minute or two for the VM to boot and to start all the services.

When it reaches this point, you typically see the CPU drop on the host system.

2. Connect to https://192.168.123.123:8443.

3. Accept the self-signed certificate.

4. Log in using the following credentials:

Username: sqrrldemo

Password: sqrrldemo

The Sqrrl web application opens with the risk dashboard.



About the CounterOps model

What is the CounterOps model? Sqrrl's CounterOps analytics model uses log data to generate an insight-rich hunting ground for identifying potential persistent threats to your environment.

For this version of the Test Drive VM, we've enhanced the network-based model data by adding two additional sources:

• CarbonBlack endpoint data

• Threat intelligence data

Overview of the CounterOps model structure

Network layout

One part of the model uses the source data to create a map of the network machines and domains:



Network users and activity

The model also tracks lists of users and accounts, and records general network activity:



Detection results

Finally, the model detectors automatically analyze the source data, using specialized algorithms to look for evidence of suspicious activity.

Based on this analysis, the detectors generate risk scores, and add instances of detection results:



About the raw data

Source connector data The raw data that we loaded into Sqrrl is from a dump of flow and Windows logs from the Los Alamos National Labs (LANL) cyber security research open test data set.

The data corresponds to a red-team test they did a few years ago and released publicly. It represents 58 days worth of data from their internal laboratory.

We do not have ground truth on this data. Their anonymization process altered the data, in particular by turning IP addresses into non-IP address values, so we had to map those values back.

The data set contains records of over 1.5 million individual events, and refers to:

• Over 12,000 users

• Over 17,000 computers

• Over 62,000 processes

CarbonBlack endpoint data The CarbonBlack data is from a local CarbonBlack instance created in our lab. It represents about a week's worth of data.

We then merged the CarbonBlack data with the LANL data set.

The CarbonBlack data contains over 500,000 individual events, including:

• Over 40,000 user login events

• Over 3,000 individual processes

Threat intelligence data The Threat Intelligence data comes from a minimal threat feed modified in our lab, and includes over 11,000 indicators.



Getting your bearings in the web application We’ll come back to the detections dashboard in a bit, but let's start with a short tour around the web application.

Viewing the available sources 1. In the Sqrrl menu on the left, click the sources icon.

The Sources page shows you what sources of raw data are available in this Sqrrl instance, and allows you to create additional ones.

2. Click one of the sources, and familiarize yourself with the data summary.

Keep in mind that the defined fields are a starting point. Other fields can be added automatically when data is loaded.



3. To get an idea what is actually populated for that source, click Browse Records.

4. The number of records is likely too large to display at once. When prompted to narrow the list, just click show a sample.



Exploring the CounterOps model structure 1. In the Sqrrl menu, click the models icon.

2. On the Models page, the only model you have is CounterOps, so go ahead and click that.

You see the model, and can move the entities around to make it more understandable.

3. To see the data summary, click Manage.

4. Go back, click an entity type such as IPAddress, then click Entity Mappings.

5. On the mappings page, play around with the UI to see how the sources map to the global fields.



Displaying and searching the Sqrrl documentation The menu in the top-right corner provides access to some general options:

1. Click the options icon, then click Sqrrl Enterprise Documentation.

The menu also contains options to set the time zone for displayed data, and to log out of the web application.

The Sqrrl documentation displays in a new tab.

2. At the top of the left panel, click Search. In the search text box, enter a search term such as "LDAP", then press Enter.

You can browse the results and see how to integrate with LDAP for authentication.

The documentation search is particularly useful when trying to write queries. Search for "Select" or "Match".

3. Now close the documentation tab, or just return to the Sqrrl UI tab.



Returning to the risk dashboard To return to the risk dashboard, in the Sqrrl menu, click the home icon.

Exploring the risk dashboard

Changing the risk dashboard time frame

By default, the risk dashboard displays data from the previous 7 days.

To change the timeframe for the dashboard, click the dropdown next to the heading text.

About the detection result and entity instance risk scores

Detection result risk scores represent a snapshot in time, and do not decrease over time.

Entity instance risk scores normally do decay, because they are designed to represent the current risk, but in this demo VM the risk score decay is disabled.



Filtering and sorting detection results and entity instances

For both detection results and entity instances, you can:

• Filter the list by detection result or entity type:

• Sort the list by risk score or when the item was last modified:



Overview of the threat hunting process Sqrrl's Threat Hunting Loop (https://sqrrl.com/solutions/cyber-threat-hunting) is a defined process for conducting a hunt. This document walks through a hunt using the loop as a guide, and explains how a security analyst can use Sqrrl Enterprise features to support the hunting loop.

Hunts start with hypotheses. Sqrrl has defined three categories of hypotheses as starting points for a hunting exercise:

Intelligence-Driven Hunts Use threat intelligence to proactively search for IOCs and TTPs

Behavior-Driven Hunts Use kill chain or data-driven frameworks to hunt for suspicious behaviors

Entity-Driven Hunts Hunt for threats that are targeting critical assets (e.g., crown jewel assets)



Let's hunt for suspicious behavior!

1. Create hypotheses The first step in hunting is to know what you’re hunting for. For example we are looking for "insiders" who are stealing our data. In this case, an "insider" could be either an authorized user (a real insider), or an APT actor who has stolen the credentials of an authorized user, which to the system is indistinguishable from a real insider.

There are numerous signs we can use to detect this type of activity, such as:

• Users accessing data they usually do not

• Users accessing data at unusual times

• Users consolidating a lot of data from around the network onto a single host

• Users printing an unusually large number of documents. Only a real insider, who can collect the printouts and walk out the door, would be able to do this.

• Users transferring an unusually high amount of data out of the network

We could go on, but let's start with this last one to build our hypothesis:

As part of an attack to steal our organization's data, we will see someone inside the network attempt to transfer an unusually high amount of data out of the network.

This hypothesis also aligns with Sqrrl's Behavior-Driven Hunt category, which includes our data exfiltration TTP detector.

2. Investigate via tools and techniques As part of a typical hunting exercise, Sqrrl recommends that hunters work through the following questions:

Question 1: Was this an attack?

Question 2: Was the attack successful?

Question 3: What is the scope of the involved resources?

Question 4: What additional activities did the adversary conduct?

Question 5: What is the scope of the compromised resources?

Question 6: What are the next steps?



During our example hunt, we go through each of these questions.

Question 1: Was this an attack?

Reviewing the activity on the detection result profile

On the detections dashboard, click EXFIL-13. We'll use its profile page as the starting point for the hunt.

What is it telling you about this activity? In particular, what is it telling you that supports or rejects the hypothesis that this may be a malicious exfil?

Perhaps the data rates are low, and the overage isn't significant, but is this a server that never communicates with the outside world? Shouldn't a small blip then be of interest?

In this case, we have a small blip above the background. This activity is not typically indicative of an attack. Generally, you do not see this activity on production systems with realistic amounts of traffic. However, if you do, contact your Sqrrl architect to ensure the system is properly tuned.

If the traffic volume was abnormally high, where do we go from here?

Finding the involved systems

What systems are involved?

1. From the profile page, click the explore icon.

2. Select all of the IP addresses.

To do this, right click a single IPAddress, then click Select all entities by selected types(s).

3. Right click anywhere, then click Expand Network Resolutions.

Researching the destination domain

Assuming that we did have a suspicious amount of traffic going out, would we expect that traffic to be going to neopets.com?

To answer that we need to consider - what is neopets.com? Here, you should use the same type of web tools you always have, such as whois, DNS records, domain tools, web of trust, etc. On Web of Trust, the summary for neopets is "Virtual Pet Community! Join up for free games, shops, auctions, chat and more!"

It's a game site, so we might expect it to be more interactive and have more data going to it than we would a static web site (whether the users should be going to a game site at all is outside the scope of this discussion).



Analyzing the traffic

We still expect more data coming back, though. Is that what's happening here?

1. Select the three IP Address entities again, then click the show all relationships icon on the toolbar.

You should now see the Connected To relationships in both directions.

2. We'll next compare the traffic in both directions.

To start, click one of the directions.

3. On the right hand details panel, if the Value sparkline shows "No data for current window", click the fit window to data icon next to the time range.

This selects the full time range you have data for. On a more active network, you may have to manually set the time range down to the times you are interested in.

4. You should now see data in the Value sparkline. To bring up the full-scale timeline, click the sparkline.

5. Make a note of the approximate times and values.

6. Now close that, select the relationship in the other direction, and then do the same thing.

How do they compare? What does that tell you? What if the values were 100 times larger in both directions?

7. Repeat for the pair of connections out to the other server.

In this case, with peaks around a MB (note that these are 15 minute windows) going out to a games site, with over twice that coming back does not seem suspicious, as we expect it to be interactive, but having 400kb going out to the images server (10.10.1.17 -> 198.172.121.10) does seem a bit excessive for requests, especially with less than twice that coming back.

Now if we were talking about 100MB going out (even with over twice that coming back), it would certainly be suspicious, even more so if it were 40MB going out to the images server.

Analyzing the timing of the connections

What does the timing of the connections tell you? In this case, we need the domain expertise to know where our 10.10.1.17 is located.

1. Click the Sqrrl options menu at the top right.



2. From the options menu, you can select the time zone you are viewing here, so let's select UTC, as if this system were in London.

We see that the activity occurred between 18:15 to 20:15 UTC. So what would that mean? Well, it could just be someone playing games after work.

3. But let's say that the system was in Sydney. Open the options menu again. Type "Australia/Sydney", and then select it when you see it.

Now the time looks like 4:15am to 6:15am. What could that mean? Let's set the time back to UTC and move on.

Analyzing related suspicious activity

What other suspicious activity have these machines been involved in?

1. Select all the IP addresses. Right click a single IPAddress, then click Select all entities by selected types(s).

Now we are going to look for any other detections or alerts that Sqrrl may have discovered about our 3 IPAddresses

2. Right click, then select Expand detections and alerts.

We find a new beacon, BEACON-6.

3. Repeat the same expansion - right click, then select Expand detections and alerts.

We did find a beacon going out to a new IP address.

4. Shift-click both 10.10.1.17 and 64.4.13.174, then click the show all relationships icon to see the relationships between just these two nodes.

5. When we click those links to see the time they were operating in, we get "No data for current window".

6. To change the current window, click the fit window to data icon.

What do you see about the time and what does it tell you?

This activity is happening a few hours before the exfil activity that we analyzed earlier.

Okay, so what do you make of this event?

At this point, I would say that this could be a false positive. Even if the original exfil is nothing more than someone playing games, the activity to 64.4.13.174 is indicative of a misconfiguration or bad information.

Question 2: Was the attack successful?

If this was indeed an attack (and we are now proceeding as if it is), then yes - we see indications that they have control of one of our internal machines and have started to exfiltrate our data.



Question 3: What other resources were involved? 1. To recenter our graph, click the fit and center icon, which is part of the navigation controls.

2. Now, select 10.10.1.17, then right-click and select Expand Connections.

Whoa! Okay, maybe not as bad as you’ll see on a system that has weeks or more of data, but still quite a bit.

One thing we can note here is that the only new neighbors are all IP addresses that this one connected to.

3. To undo the expansion, press Ctrl-Z.

4. Now let's use the conditional expansion option, which provides more control over which instances are added to the graph:

a. Select 10.10.1.17. On the toolbar, click the graph options menu, then click Expand > Expand.

b. Change the relationship type to IPAddress connectedTo> IPAddress.

c. Set the time window to 18:00 to 19:00, which is when the initial activity was occurring.

d. Click Expand. Much more manageable.

5. Looking at the DNS names, select Expand Network Resolutions

Looking at everything from the unresolved host (204.177.92.193), we see a bunch of small connections from a number of our internal hosts. This is all indicative of regular business use.

Now, if this is when the initial infection happened out to 64.4.13.174, it is quite possible that one of these legitimate sites was hacked or contained malvertising, or the user was phished via their Hotmail account.

If we look at the beacon out to 64.4.13.174 we see that activity started at 12:45… and the activity to zedo.com was at 12:45, and to fast.mediacharger.com (web tracker) started at 12:45 -- the other two sites were around the same time as 64.4.13.174.

We need to perform forensics on the machine and ask to see the message the user received to confirm.



At this point, it seems less certain that we have an attack, or if we do, that it might just be revolving around the BEACON.

So to answer Question 3, we expanded the scope to include 64.4.13.174, and no additional internal resources appear to be involved, based on the data available thus far.

Question 4: What activities did the adversary conduct?

As noted above, there may be command and control (C2) channels established to three servers. To determine anything further, we need more data from the endpoint.

Question 5: What resources were compromised?

It appears just 10.10.1.17. Select 10.10.1.17 again.

We should do an Expand user and account activity to see what accounts were logged in, as those credentials may be compromised… and, we don’t have that information.

Question 6: What are the next steps?

What do you think? Is this worthy of recording as an incident and dispatching IR?

Frankly, that depends on the environment. If I know that this is some run-of-the-mill desktop, I probably would not.

But if I do not know what this machine is - and right now I do not even have a Hostname entry for it - I might want to push this forward just to have more information for next time.

At this stage, the next step is further investigation to answer the questions:

• What is 10.10.1.17? A standard desktop / laptop, or something else?

• Who uses it, both in terms of specific accounts and their role in the enterprise?

• What did the user do when they opened an email? What message(s) did they get and what did they do with them? In particular, did they follow any links?

• What email did the user read? Did that email contain web beacons? Did the user click any links that would have taken them out to 64.4.13.174?

• Does a scan or analysis of 10.10.1.17 show any sign of malware or backdoors? Are there any executables that showed up on the system at or after 20:41 UTC?

• What user accounts were logged into 10.10.1.17 between 20:30 - 04:15 UTC?

If the answers to these questions are indicative of any malicious activity, then you want to do a deeper forensic dive into what resources the attacker might have accessed on 10.10.1.17 - or what services they might have accessed on the network where Sqrrl did not have east-west network visibility.



Bonus material: Advanced investigation queries

Now, there is still an open question if this was an incident, but there are a couple patterns here that could be useful in detecting suspicious activity:

• Beacon detection shortly following (within a minute) activity to a webmail server

• Domain authentication to an external host

We can do additional advanced searches at a few different levels:

1. Queries,

2. API scripts or programs, or

3. Spark jobs

Each of these is increasingly more powerful, but also complex, so let's start by seeing if we can detect these patterns using queries.

Beaconing after connecting to webmail

Let's start by doing a query for all IPs with beaconing activity:

1. Click the query text area at the top left.

2. In the query text area, enter:

MATCH Beacon <involved IPAddress

3. Okay, good - we can get the relationships between the Beacons and IP Addresses. Now let's add the other hosts they are going to:

MATCH Beacon <involved IPAddress WITH connectedTo> IPAddress

4. While this generates a pretty picture, it's hard to identify any activity.

Unfortunately, we can only have a single WITH clause, and we cannot do any filtering in the WITH clause, so let's look at this the other way:

MATCH DNSDomain *> IPAddress WHERE DNSDomain.instance_id() LIKE '%hotmail.com'

Note that the * there means "match any relationship type". In this case there's only one, so it's convenient shorthand.

The instance_id() function here is the quick way to get the entity identifier, but because we are looking for the end of a string, we can't use our indexes, so this requires a full table scan and is slow if you have lots of DNS domains, which you typically do in practice.

5. We can expand this out to see what internal hosts connected here:

MATCH DNSDomain *> IPAddress WHERE DNSDomain.instance_id() LIKE '%hotmail.com' WITH <connectedTo IPAddress



6. From here we could manually select the internal nodes to find which ones are beaconing, but because beacons can only originate from internal nodes, it is much faster to select all of the IP addresses.

To do this, right click a single IPAddress, then click Select all entities by selected types(s).

7. Right click, then select Expand detections and alerts.

Interesting that all the beaconing activity is around the one Hotmail node, and that it's named gateway.messenger.hotmail.com, suggesting that this activity might be associated with the Messenger IM service.

We should also note that it is in the same class-C as the host we had the odd external authentication to above. If we have cross-domain SSO set up with Microsoft to host our Messenger instance, this makes a lot more sense.

8. If we take the selected beacons and select Expand detections and alerts, we see that most of the beacons are going to this same subnet, which answers some of the key questions we had before, and means that in this case, the pattern does not seem to be suspicious.

If we saw beaconing start after connecting to any of those other Hotmail nodes though, it might be. In that event, we might want to eliminate the messenger nodes using something like:

MATCH DNSDomain *> IPAddress WHERE DNSDomain.instance_id() LIKE '%hotmail.com' AND DNSDomain.instance_id() NOT LIKE '%messenger.hotmail.com' WITH <connectedTo IPAddress

If you run that, you'll still find one beacon, as the 10.10.1.60 IP was connected to both Messenger and another Hotmail server.

9. To take this further, you could script it to perform all of the steps in one place - which also allows you to check the temporal locality of the events - but that's a topic for another session.

3. Uncover new patterns and TTPs The key activities in this step of the hunting loop are:

• Saving and documenting your findings

• Sharing this information with other tools



Saving your investigation

Let’s save what we have done up to this point so we can come back to it later.

1. In the upper right corner, click Save, then Save Investigation as....

2. Give the investigation a name to identify the activity. If you opened a ticket for it, it is typically the ticket ID.

3. Save it!

In an actual investigation, you can annotate the individual steps in the investigation as you saw fit, but we'll skip that for now.

Exporting the results of your investigation

From the graph, you can export the current graph display to a .png image. You can also export a CSV list of the displayed entity instances and the entities they belong to.

4. Inform and enrich analytics

Creating a risk trigger

At the end of our investigation, we ran the following query looking for activity to %hotmail.com but not %messenger.hotmail.com.

MATCH DNSDomain *> IPAddress WHERE DNSDomain.instance_id() LIKE '%hotmail.com' AND DNSDomain.instance_id() NOT LIKE '%messenger.hotmail.com' WITH <connectedTo IPAddress

Lets use that query as the base to create a risk trigger that is looking for spikes of activity to hotmail.com. Click Create risk trigger.



Configure the resulting dialog box as shown, with the WHERE statement we were using above.

We are creating a risk trigger to look for spikes in DNS traffic to hotmail.com.

After you click Create, click the Explore icon. In the exploration list, verify that the risk trigger was created.

This risk trigger does not immediately contribute any additional risk to entity instances, because of the nature of the data set we are working with in this walkthrough. You can, however, explore some pre-built risk triggers on the 10.10.1.4 entity.

For additional information about risk triggers, check out the documentation included in the Demo VM, or request the SqrrlRiskTriggerQuickReference.pdf.



Updating the detector whitelist

Okay, so above we determined that the beacons to Messenger were not a problem. Typically, activity like that clears up after the detector learns what's normal with a week or two of data, but if you don't want to wait, or if it's not going away, you can add it to the whitelist:

1. Click the models icon, then click the model. In this case it is the Sqrrl default model, CounterOps.

2. On the details panel, click the detector in question. In this case, it's the Malicious Beacon Detector.

3. To edit the whitelist, click the WhiteList Items selection.

4. Enter the relevant info. I like to be as focused as possible, so in this case:

a. Set Destination to 64.4.13.0/24.

b. Set Port to 80

c. Add a quick note describing what this is.

5. Click Save, and you're done!



Your turn! Now you try it:

1. Pick one of the detection results on the risk dashboard and use it to guide a hunt and figure out what's going on.

2. At the end, you should have:

• A summary of what you found

• Next steps

• What patterns you either saw or it made you think of

• Where you want to go on your next hunting trip

3. Assume you are hunting for something and do not have detection results for the data you have loaded, or want to look for something that there is not a built-in detector for.

4. Go through the hunting loop, see what you find, and prepare a summary of:

• What you were looking for

• How you looked for it

• What you found

• Patterns for future use

• Where it made you want to go next

hunting walkthrough with the sqrrl test drive vm · identifying potential persistent threats to...

Documents