tyler owen - threat intelligence & siem - 2014-11-09 ·...

Threat Intelligence & SIEM

Tyler Owen

Lewis University

Abstract

While many SIEM implementations are completed due to auditory or compliance

requirements, the next step to provide valid alerts for an organization is often neglected. The

SIEM solutions might have an inadequate asset model and do not utilize up to date threat

intelligence.

This paper details the creation of an additional standalone virtual system that can

address both of these issues. The system will gather knowledge automatically by collecting

Known Bad Actors from the Internet and user-‐supplied information. This knowledge can be

provided to a SIEM for correlation and alerting. In addition to direct integration into the SIEM,

the system provides a mechanism for additional context to be gathered about a host. This

system also provides a powerful user interface that allows analysts the ability to research

information about a particular host. The knowledge contained in the system can also be shared

with other organizations via a built-‐in mechanism.

With the addition of this system to an environment and integration into a SIEM or other

point, security solutions can immediately provide a better security stance to an environment

with very low effort.

Table of Contents

Abstract .............................................................................................................................. 1

Chapter 1: Introduction ...................................................................................................... 4

Chapter 2: Moat Backend ................................................................................................... 8

Moat Host Model ........................................................................................................... 8

Populating Data in Moat ............................................................................................... 11

Post Processing ............................................................................................................. 12

Risk Score .................................................................................................................. 14

Chapter Summary ......................................................................................................... 16

Chapter 3: Moat Web User Interface ............................................................................... 18

Dashboard/Main Page .................................................................................................. 18

Host Information .......................................................................................................... 24

Contextual Help ............................................................................................................ 26

API ................................................................................................................................. 27

Authentication .......................................................................................................... 28

Retrieving Information ............................................................................................. 28

Adding Information .................................................................................................. 29

Chapter Summary ......................................................................................................... 31

Chapter 4: SIEM Integration ............................................................................................. 32

Feeding Data Back To the SIEM .................................................................................... 32

Correlation .................................................................................................................... 33

Alerting in the SIEM ...................................................................................................... 34

Direct Query of Moat from the SIEM ............................................................................ 36

Chapter Summary ......................................................................................................... 38

Chapter 5: Sharing Data .................................................................................................... 40

Sharing from Moat ....................................................................................................... 41

Chapter Summary ......................................................................................................... 42

Chapter 6: Summary ......................................................................................................... 44

Appendix: Database Schema ............................................................................................ 46

Audit ............................................................................................................................. 46

Auth .............................................................................................................................. 46

Generators .................................................................................................................... 46

Host .............................................................................................................................. 47

Host_generator ............................................................................................................ 47

Host_id ......................................................................................................................... 48

Host_ip_mac ................................................................................................................. 48

Host_kba ....................................................................................................................... 48

Host_software .............................................................................................................. 49

Vuln ............................................................................................................................... 49

Appendix: Known Bad Actors Sources .............................................................................. 51

References ........................................................................................................................ 52

Chapter 1: Introduction

Companies implement Security Information/Event Management (SIEM) in order to meet

audit or regulatory requirements. These requirements often do not address how the SIEM

solution should be integrated into the organization’s environment. Therefore, many of these

implementations face a lack of integration and tuning after they are initially implemented.

These installations are performed and devices are connected, but no real actionable value is

garnered out of the SIEM. The few organizations that do implement real time alerting often fail

due to lack of proper integration into the environment, or they “have trouble building

correlation rules so essential to that real-‐time threat detection” [1]. Often this is due to the

fact that many organizations do not know what to actually alert on, or the system is too

complex. The fact is that the staffs at many organizations are not adequately trained or they

are over tasked and spread too thin. Other times, implementations fail due to lack of proper

planning, and organizations just do not know what they want, so they turn everything on and

get lost in the volumes of meaningless alerts [2].

As a way to combat the lack of actionable alerts and steer organizations in the

appropriate direction to protect their environment, threat intelligence can be utilized. Threat

Intelligence is a relatively new concept for SIEM technologies. It can range from IP Addresses to

malicious URLs to known malware hashes. IP Addresses, hostnames and URLs are often lumped

together and referred to as known bad actors. When known bad actor information is applied to

an environment, it can enrich the value of the SIEM alerting output and allow security analysts

and engineers to perform much faster with a higher degree of confidence.

Many SIEM solutions have the ability to perform information look ups for IP Addresses

or hostnames with data that is already stored or “known” by the SIEM. This data is traditionally

loaded in to the SIEM by an administrator. This paper will explore the concept of

supplementing this administrator-‐loaded data with known bad actor information that is

harvested from the Internet. Security research teams publish repositories, for the consumption

of the community, to the Internet that can be utilized. In addition to loading data into the

SIEM, a new system will be built that will allow anyone to query the server for information

about a host.

The building of a supplemental system and adding in Threat Intelligence into the SIEM is

designed to address some of the challenges that face organizations. If the SIEM solution can

leverage information that is already on the internet about hosts that have been known to

perform malicious activity, it can make smarter decisions and provide an organization with

legitimate actionable alerts. By implementing a system that will not only find known bad actors

but also determine key information about them, an overtasked security analyst can make

educated decisions faster.

The supplemental system concept was roughly based on an open source solution called

Collective Intelligence Framework [3]. The concept of the Collective Intelligence Framework

(CIF) solution is sound but is far too complex than is required by many organizations to be

successful. In addition to the complexity to operate CIF, the installation and upkeep is far too

cumbersome. After running CIF for a while, the scripts that feed the data into the database will

begin to hang. When this happens, no new data will be collected from the Internet and the

data will have gaps in it. So as a result, it is hard to ultimately trust the system. In addition to

the missing data, the database schema does not lend itself to fast retrievals of information.

Once the database contains a few million records it significantly slows down, causing queries to

take 1-‐5 minutes. The CIF team does offer some PostgreSQL tuning advice on their site, but this

had no impact on the test implementation created as part of this research.

The system that was born from this effort is called Moat. Every castle needs a

secondary means of protection such as a moat that can separate the castle from potential

attackers. The aim of Moat that will be laid out in this paper is meant to be a turnkey solution..

It is to be a virtual-‐machine-‐based Linux system that will contain everything that is required to

begin operating as soon as it is powered on in an environment. Moat will drop in to any

environment and begin providing value to a Computer Incident Response Team (CIRT) or

Security Operations Center in a matter of hours.

Moat is a flexible system that attempts to utilize as many sources of information about

hosts and assets as possible in an environment and the Internet. The system not only relies on

known bad actors gathered from the Internet, but it will have an interface that will allow an

organization to continually build knowledge of its environment and the internet as it is

encountered. After the system knows about a host, it will perform a post processing task that

will complete a host record by looking up the country of origin, ISP information, WHOIS

information, geographic information, any known vulnerability and software information.

Moat will not be worth the electricity that powers it unless the data is easily retrieved.

Therefore, a graphical user interface was built to allow for easy retrieval of the knowledge it has

built along with automated integration into other portals and products. To achieve integration,

an API has been developed and documented.

Chapter 2: Moat Backend

Moat is a standalone system that runs via CentOS Linux 6 based virtual-‐machine and

comes with all the required packages and dependencies loaded on it. The core of Moat is a

MySQL database that is loaded on the system. Information is inserted into the database though

a number of standalone applications written in Perl, Python, Bash and PHP.

Moat was designed to be flexible and allow for extensions by the end user to bolster the

information gathered by the system. To achieve this goal of flexibility, a handful of helper

scripts were built for use with the system in addition to an API. An example of this flexibility is

discussed in the host model section which allows the end user to modify how Moat calculates

how a host is determined.

The database backend stores information about a host, which can include data from

vulnerability scans, port scans, information gathered from the internet, or provided by a Moat

user. This data is all stored in a set of tables. Once the data is inserted into the database, it is run

through a post processing system that will add additional information gathered passively through

the Internet, and a risk score will be created. This risk score is designed to give analysts an idea

of how dangerous a particular host is.

Moat Host Model

One of the key design elements of Moat is the host or asset model. The driving design

principal for the host model was a mechanism to attempt to determine what a host was and

attempt to eliminate duplicates. For Moat the decision was made to create a host ID to tie all

information stored in the database together.

There are a number of logical choices to create the Host ID, including an IP

address, MAC address or Hostname. Each of these items has their challenges.

Many current SIEM asset models are based on an IP Addresses. While this works, it

presents problems when an IP Address changes. Unfortunately, IP Address changes are the

norm in most environments as DHCP is utilized to assign IP Addresses to hosts. As leases change

and hosts come on and off the network, their IP changes. While a user’s laptop might have

fifteen IPs in a day, as they come and go or login remotely though a VPN, it is always the same

host. In many environments, the hostname are statically assigned to a host, so no matter what

the IP address, the hostname will remain.

The best mechanism to determine a host is to utilize a marker on a host that does not

change, such as a MAC Address. However, this is typically not possible as the host’s MAC

address gets obfuscated as packets traverse routers and the Internet. MAC Addresses could be

utilized on local networks but would potentially require an agent or other piece of software on

each system that Moat sees. This is not feasible when the Internet is added to the equation.

Hostnames can change, but given the Internet and a local network with statically

assigned hostnames, it seems to be the most stable solution. By utilizing hostnames and

tracking when an IP and hostname change, there is a potential to isolate attacks such as

FastFlux [4] as well.

The mechanism that was chosen for Moat was to base the ID on an MD5 hash. At the

time of writing, this hash is currently based on the hostname of the system. The reason for

utilizing a hash is so the end user could build a set of criteria that is utilized to make up the host

ID hash that best fits their needs and environment. Utilizing a hash also allows for future

expansion of Moat without any database schema changes. Using a hash also provides for

leveraging the use of multiple pieces of identifying information. The more specific the markers,

such as MAC address, BIOS serial number or asset tag number that help uniquely determine a

host, the better the system will become at decisively distinguishing hosts.

To allow for easier development, a Perl module was created which will create the host

ID. This Perl module can be modified to manipulate how Moat creates host IDs while and all of

the other pieces of Moat will begin to utilize this newly created structure on the fly. The

framework of Moat allows for any utility or language to be utilized to create the host ID, but the

current implementation utilizes the Perl programming language.

Once a host ID is generated, it is stored in a table in the database along with the

hostname that was initially used to generate it in the host_id table. As supporting data is added

to complete the information known about a host it is all referenced to this host ID hash. So as

the information about a host evolves over time, the entity that is this particular host is all tied

to a single ID in the database.

Populating Data in Moat

Moat comes out of the box ready to gather data on its own from the Internet. On a

daily interval, Moat will query a number of Open Source Intelligence (OSI) feeds and load the

data into the database. It will then begin the post processing phase to build as much

knowledge about the system; the post processing subsystem will be covered in a later chapter.

The OSI feeds consist of a list of known malware hosting hosts, command and control

hosts, malicious scanning servers and a list of TOR exit nodes (a complete list of OSI feeds can

be found in Appendix: Known Bad Actors). These OSI feeds can have different levels of impact

to different organizations, therefore a weighting mechanism was provided. This weight is

initially set by Moat, but an end user has the ability to modify this setting through the User

Interface.

Another source of information is vulnerability scan data. Moat is able to ingest files that

are generated by many popular vulnerability scanners such as Tenable Network Security’s

Nessus [5] or Trustwave’s SpiderLabs Vulnerability Scanner [6]. Vulnerability scan data can

provide key information about a host, as it can lead an analyst to either trust an asset or be

suspicious. Vulnerability scanning can also bring to light misconfigurations that hackers can

take advantage of [7]. A security analyst should have as much information at their fingertips

when performing an investigation. Once this data is loaded into Moat, the standard post

processing will occur.

In addition to traditional vulnerability scan data, port scan data can also be loaded in to

Moat from NMAP [8]. For NMAP scans, it is beneficial to add the options to identify the

Operating System and grab the banners from listening ports on the remote host. This data is

valuable and will be stored in the host and software database tables. Moat can also execute

NMAP scans on its own. These scans are initiated by the end user though the UI. The UI will

then execute a Python script that leverages the NMAP libraries [9] and then loads the data into

the backend database.

Over the years, many different frameworks have been created to share or structure IT

Security information [10]. The Department of Homeland Security has put its support and

funding behind a standardized format that is being developed by Mitre called Structured Threat

Information eXpression, commonly known as STIX [11]. Due to such large support by DHS, the

adoption of STIX will likely be rapid. In preparation, Moat can accept data in STIX XML format

for consumption. This will allow for data sharing from other organizations and products in an

environment. The STIX XML files are processed by a Python script that utilizes the STIX Python

module developed by Mitre and posted as open source [12] and loads the IPs and URLs into the

database. After the data is loaded, it is then passed to the post processing functions.

Post Processing

Once hosts have been added to the Moat database, more information needs to be

added to the host. Currently, Moat will gather GeoIP information, ASN and WHOIS data for

each new host. To determine if a host is new to the table, the post process script will look for

any host that does not have a country code and has not been updated in the past five (5) days.

When the script finds a host to process, it will take the IP Address and look up the GeoIP

information utilizing Maxmind’s Open Source GeoLiteCity binary database [13]. This database is

not as precise as the paid version that Maxmind offers, but it still 83% accurate to the city level

[14] within 40km. The paid version would be more accurate at the city level, but the country

accuracy is 99.8%. When researching a security threat, the country is as specific as is typically

required. An analyst will usually know if their organization does business with a particular

country and can make a quick decision based on this data. The GeoIP lookup will provide two-‐

character country code, country name, region, city, state, latitude and longitude.

In addition to the GeoIP information, the IP Address is also used to look up the

Autonomous System Number (ASN) which will tell an analyst who the network operator is for a

given IP Address or range of IPs [15]. There are many Internet Service Providers around the

world that are known to be very lenient toward hacking, spam and general malicious activity

[16].

WHOIS data will show an analyst who a domain is registered to. This will also, at times,

provide an abuse or technical contact that an analyst might be able to reach out to in effort to

stop an attack that is originating from their domain. Many times the listed contacts do not care

or the address is an unmonitored mailbox. On occasion however, they the ISP will care and will

investigate and remove the offending device from their network. There are also times where a

company will get the courts involved to take a domain or set of domains to stop the spread of

malicious software; Microsoft is one organization that has done this. A documented case of

this is when Microsoft took No-‐IPs domains off line and interrupted their service. This was a

widely publicized event that had great success in putting a serious dent in the command and

control families of malware named Bladabindi and Jenxcus [17].

Risk Score

The post processing script will also create a threat score for each host. The threat score

is meant to be a quick indicator of how malicious the particular host is. The threat score is

calculated by an algorithm that takes into account when a host was seen, the number of times

a host has been seen by a known bad actor source, location of the IP/host, the port and the

history of each threat giving a cumulative score to base the severity of risk this IP/host is. It is

based on a scale of 10 with 0 being the lowest score a threat can have and 10 being the highest.

The algorithm will first calculate the top 50 countries. This is based on data that Moat

has collected over time. This list of countries can be altered by the end user to provide an

exclusion list. If the system primarily has local data, then the likelihood of the country that will

be seen as the biggest contributor will be the local country where the assets are in. If the host

is in one of the top 50 countries, the score is increased by two.

Next, Moat will calculate the top 10 ports that have been observed as being malicious.

If the host being processed has ports associated with it, the algorithm will determine if any are

in the top 10 list. If a port is in the top 10 then the risk score is increased by 2.

After the ports calculation, the last time the host was seen by Moat is taken into

account. If the host has been seen in one (1) week or less, the score is increased by 2. If the

host has not been seen for a week, but less than a month ago, the score is reduced by 2. If the

host has not been detected in over a month the score is reduced by 3. By the time the score

has been reduced significantly, point security solutions will have had time to adapt. This

adaptation could be updated intrusion detection system (IDS), intrusion prevention system (IPS)

or anti-‐virus signatures

Next, the number of times a host has been seen by Moat is leveraged to influence the

score. If a host has been witnessed less than 50 times, the score is increased by 2. If the count

is greater than 50, but less than 75, the score is increased by 3. If the host has been spotted

more than 75 times the score is increased by 4.

The second to last step in the process is to utilize the user-‐configured source weighting.

This allows the end user the flexibility to tailor Moat’s known bad actor sources to meet their

needs a bit better. The sources come out of the box with a default setting and the end user can

modify the value and also set them back to the default. This will allow a user who, for instance,

may not care about SSH scanners to set the score to a 0 so that they have less impact on the

risk score. Conversely, they can set TOR exit nodes to a 6 as they pose a real threat to their

environment (See Figure 1. Generator Configuration UI).

Figure 1. Generator Configuration UI

The final step in the risk score calculations is a level setting evaluation. If the score is

higher than 10 it will be adjusted down to 10 and if the score is less than 1 it will be set to 1.

Chapter Summary

This chapter covered the standalone, supplemental Linux virtual machine called Moat.

Moat allows organizations to collect information about hosts and present that information to

analysts to make informed decisions as they are performing information security analysis. This

chapter also covered the host model which allows for flexibility in how Moat determines what a

host is so that it can adequately track a single host across IP Address changes in a dynamic

environment. Moat would not be nearly as valuable without the addition of hosts and the

automatic feeding of Known Bad Actors, which was also covered in this chapter. After hosts

have been added, an additional step of post processing is completed; this allows Moat to

complete the picture with GeoIP, ISP, WHOIS and risk score information. The risk score is an

algorithm that gives a user the ability to determine how risky a particular host is to their

environment and allows them the ability to influence the score.

Chapter 3: Moat Web User Interface

Getting information into the hands of an analyst is one of the most important tasks of

any information security tool. Analysts must respond to threats in an informed and timely

fashion. The Moat web user interface (UI) allows an analyst to be more proactive in their

response [18]. A good user interface can be the difference between a hugely successful

application and its failure to make an impact [19]. This UI was developed in PHP and leverages

a number of open source libraries and tools to make development faster and easier. The UI

contains a dashboard to provide an analyst the ability to spot trends along with the ability to

lookup information about a specific host. In addition to finding host information, the user

interface provides a contextual help system. The UI is not only a standalone mechanism to

locate information about hosts, but also has an application programming interface (API) that

allows for data entry and retrieval. This API also leverages an authentication system that will

limit the capabilities of users for role based access control.

Dashboard/Main Page

The main page of the user interface is setup to be a dashboard that provides quick data

points and a great jump off point for delving into the data that Moat has gathered. The

dashboard is a designed to provide the end user with a rudimentary situational awareness and

overall threat awareness of the environment that Moat has learned about [20]. The dashboard

contains 5 charts, Top 10 Source Countries (Figure 2. Top 10 Source Countries Chart), Regional

Source Location of Attacks (Figure 3. Regional Source Location of Attacks Chart), Current

Number of Attacks (Figure 4. Current Number of Attacks Chart), Ports Currently Under Attack

(Figure 5. Ports Currently Under Attack Chart) and Current Number of Verified Malware (Figure

7. Current Number of Verified Malware).

Figure 2. Top 10 Source Countries Chart

Figure 3. Regional Source Location of Attacks Chart

Figure 4. Current Number of Attacks Chart

Figure 5. Ports Currently Under Attack Chart

The Top 10 Source Countries chart is drillable. It will allow a user to click on a country

and see the top 10 IP Addresses, sorted by number of times seen, that originate from that

country. The initial view can be seen in figure 2, and the view after a user has selected a country

can be seen below in Figure 6. Top 10 Source Countries -‐ Drilled In

Figure 6. Top 10 Source Countries -‐ Drilled In

Figure 7. Current Number of Verified Malware

The Current number of Verified Malware chart, Figure 7. Current Number of Verified

Malware provides additional details for a particular malware when the user clicks on a malware

column. When the user clicks on the malware, a pop up window will open that provides details

similar to the host information, which will be covered in the next section. An example of the

additional context and help information can be seen in Figure 8. Context based malware

information popup. This information can be viewed in a normal page, as well for malware that

Moat knows about. This information can be seen by visiting the appropriate URL.

Figure 8. Context based malware information popup

The main page also includes a navigation menu that provides a world map displaying the

Top 25 Known Bad Actors (Figure 9. Top 25 Known Bad Actors Map), Top 25 IP/ Hosts (Figure

10). It also provides access to a Frequently Asked Questions section. The Top 25 KBAs map

view has data points that represent IP addresses. Each of these points is clickable to provide

detailed information about the host. More information about this view will be detailed in the

Host Information section below. These views of the data provide another means to observe the

data allowing analysts to identify dangerous hosts.

Figure 9. Top 25 Known Bad Actors Map

Figure 10. Top 25 IP Host Attacks View

Host Information

The host information page is designed to be a single page to view all information that

Moat has collected about a host. This page is divided into six sections. Each section groups

similar information together. The first section is “host info” (see Figure 11). It focuses on

information that has been gathered about the host such as the threat score, first, last and

number of times the host has been seen, country information, and ISP information.

Figure 11. Host Information section

The next section displays which data sources have reported this host to Moat. The Geographic

information is displayed next (see Figure 12) which shows the latitude and longitude of the

host, as well as a small map.

Figure 12. Sources that have seen the host and geographic information section

The WHOIS section (see Figure 13) is next, which simply shows the WHOIS information as

gathered from the Internet. This data is not stored in the database, but gathered each time the

page is displayed.

Figure 13. WHOIS section

The final two sections (see Figure 14) are vulnerability and software information. These

sections show any banner or software information that has been collected by Moat through

either vulnerability scans, NMAP scans or manually loaded.

Figure 14. Vulnerability and Software Information section

Contextual Help

The contextual help system in Moat is an attempt to provide analysts with

supplementary data in a single location (see Figure 15) and to allow analysts with less

information security knowledge make quicker, more informed decisions. Malware and

intrusion vectors change and morph at such a rapid pace that it can be challenging to stay on

top of it. The contextual help system is a way to help combat this. Currently, the system does

not have an automated pull of information from online sources. The data must be entered

manually. This system allows an organization to validate the information prior to its

dissemination. The help section can also be used to craft a corporate message that can be

delivered to customers or internal resources.

Figure 15. Context Sensitive Help Example

API

To make Moat as flexible as possible, an API was built. This API was modeled after the

RESTful API [21]. A full RESTful implementation was scoped out for inclusion, but after

investigation it appeared to be more than was required for this application. A RESTful API calls

for the handling of multiple types of HTTP calls such as GET, POST or DELETE [22] which are not

necessary for Moat at the time of writing. The API utilizes the authentication system hashes to

ensure that the requesting machine or URL has the appropriate privileges to perform the

requested action.

Authentication

In an effort to ensure that Moat only has relevant data being inserted, and is not subject

to data poisoning efforts, an authentication system was put in place. This system utilizes a 32

character MD5 hash as the key. This hash is currently based on the user’s full name but can be

generated however the end user sees fit. The authentication system utilizes a simplistic

permission set which consists of read, write and enabled permissions. The read privilege is

required to be able to retrieve data out of Moat, while the write privilege is required to add

data to Moat. The enabled option allows for accounts to be disabled so that they may not

access data within Moat.

Retrieving Information

Retrieving information out of Moat is done via a HTTP GET request to the appropriate

URL. This request requires the appropriate area and item that is to be queried. For example, to

query information about an IP address, visit http://moat/ip/x.x.x.x where x.x.x.x is the IP to be

looked up. See Table 1. Moat API URLs for a list of URLs and their query type. In order to

facilitate easier integration with 3rd party tools, the host, ip, malware and term URLs do not

utilize the authentication system. This ensures the data is easily accessible from other tools as

need and URLs are easily crafted. These URLs also do not support the ability to write back into

the system.

Table 1. Moat API URLs

URL Functionality

/host/<hostname> Display the Host Info page for the hostname provided (see Host Info section)

/host_id/<entity to

lookup>?auth_id=<auth_id>

Lookup and display the Host ID for an entity given what is provided, if it does not exist the system will create and return the host ID

/ip/<ip address> Display the Host Info page for the IP Address provided (see Host Info section)

/kba/<format> Display Known Bad Actors gathered by Moat in the given format. Format options currently include pipe (pipe separated), csv (comma separated), newline (one kba per line)

/malware/<malware> Display information about the given malware (see context sensitive help section)

/share/<source or target IP

address>

Generate a STIX formatted XML for sharing.

/term/<term> Display information about the given term. This can be utilized to display information about known topics or further explain malware such as “what is malware” (see context sensitive help section)

When retrieving a host ID, an authentication hash is required. This will need to be

supplied via the auth_id variable in the URL. This helps protect the inner workings of Moat

from data leakage and, if desired, is the beginning of restricting the entire Moat ecosystem.

Adding Information

Adding information to Moat is done via a specially crafted URL. The hostadd function is

used to process information via a URL for insertion. This process utilizes a different structure in

order to support multiple pieces of information about a host via a single GET command. A

single GET command can support all of the variables listed in Table 2 for insertion. One

mandatory variable that must be included in every call is the auth_id and the user’s associated

ID. The ID that is provided must have the write option and must be enabled. When inserting

Common Platform Enumeration (CPE), Banner and port information, it is best to combine these

variables in to a single command, as they will all be tied together. CPE information is a standard

initially developed by Mitre and transitioned to NIST. CPE is a “structured naming scheme for

information technology systems, software, and packages” [23].

Table 2. Hostadd Variables

Variable Description

auth_id Authentication token for a given user, this must be supplied and have the write option

banner The banner from a port or service. This could be the service and version information of a service on a given port

cpe Common Platform Enumeration value.

generator The host or program that generated the information that is being inserted into Moat. A couple of examples might be NMAP or OpenVAS [24]

hName Hostname of the system being inserted

hOSsp Operating System Service Pack level

hOSV Operating System Version

host_id Host ID hash

ip IP Address of the system being inserted

mac Media Access Control (MAC) address. The systems will also look up the Organizational

Unique Identifier (OUI) and store this information as well.

os Operating System of the system being inserted into Moat

port Port number

Chapter Summary

The User Interface (UI) is a valuable piece of the system which allows users to extract

the information housed within Moat. The UI was designed to allow users to look up a single

host and to allow dashboards to spot trends and determine an overall threat awareness of their

environment. The user interface also provides a contextual help system that will allow

companies to enter information that will assist junior analysts with information about terms

and malware. The help system allows for a common message to be distributed about threats

and malware ensuring consistency among the Information Security team.

To facilitate integration into SIEM and other point solutions, Moat contains a

documented API. This API allows the automatic retrieval and addition of information into

Moat. The API provides an authentication system to prevent data poisoning and provide role

based access.

Chapter 4: SIEM Integration

Moat is a powerful tool that can be additionally magnified by integrating it with other

point security solutions such as a SIEM. This power is achieved by extracting the knowledge out

of Moat and putting it into a SIEM for correlation and alerting. Moat can also learn from the

SIEM. This is achieved by creating access from the SIEM into Moat via context menus.

Feeding Data Back To the SIEM

Like a standard SIEM, Moat can consume data from multiple sources such as

vulnerability systems and port scanners. However, one difference between the two is that

Moat covers Known Bad Actors while many SIEMs. Feeding known bad actors into a SIEM can

make the system much more valuable to an environment. Knowing that an internal host has

been in contact with a known bad actor can allow an analyst and the SIEM to take a more

critical view of that host and its traffic. So pairing Moat with a SIEM significantly increases its

value. Getting the known bad actor data into the SIEM can be done in a couple different ways;

either through a specially crafted database query or via the /kba/<format> URL. The multiple

formats will allow for a greater adoption by SIEM solutions, a future update to Moat will likely

include the Comment Event Format (CEF). CEF is a standard that was developed by ArcSight

[25] and had been adopted by many SIEMs and product.

For Trustwave SIEM, the pipe format can be used in conjunction with a Perl script

crafted to convert the data into the appropriate asset and zoning information for direct import.

By importing the KBAs as an asset/zone into Trustwave SIEM, the SIEM will tag events as they

enter the SIEM as a KBA. By tagging hosts as soon as they enter the SIEM the KBA information

will be available to base correlation and alerting on. For domain and URL based Known Bad

Actors they are placed into lists that the SIEM can leverage.

Correlation

The power of a SIEM is the ability to correlate disparate pieces of information together.

The SIEM can make much smarter decisions by leveraging the information gathered by Moat.

By importing the Known Bad Actor data, correlations like those listed in Table 3 can also be

built. This is only a start to correlation. As threats evolve, correlations can be altered or added

to the SIEM as an enterprise requires.

The correlations in table 3 were created for the Trustwave SIEM. These correlations can

be created or altered to work in many other SIEM products.

Table 3. Correlations

Correlation Name Description

Critical Asset Targeting Known Bad Actor A host identified as critical has been detected contacting a known bad actor

DMZ Asset Targeting Known Bad Actor A DMZ host has been detected contacting a known bad actor

Internal Asset Accessing Known Botnet

Domain

An Internal Host has been detected talking to a Host in the Domain-‐Botnet Dynamic List

Internal Asset Accessing Known Botnet URL An Internal Host has been detected talking to a Host in the URL-‐Botnet Dynamic List

Internal Asset Accessing Known Malware

Domain

An Internal Host has been detected talking to a Host in Domain-‐Malware_List List

Internal Asset Accessing Known Malware URL An Internal Host has been detected talking to a Host in the URL-‐Malware Dynamic List

Internal Asset Accessing Known Phishing

Domain

An Internal Host has been detected talking to a Host in the Domain-‐Phishing Dynamic List

Internal Asset Accessing Known Phishing URL An Internal Host has been detected talking to a Host in the URL-‐Phishing Dynamic List

Internal Asset Targeting KBA An Internal host has been detected talking to a known bad actor

Known Bad Actor Targeting Critical Asset A Known Bad Actor is trying to talk to a host that has been identified as Critical

Known Bad Actor Targeting DMZ Asset A Known Bad Actor is trying to talk to a host located in the DMZ

Known Bad Actor Targeting Internal Asset A Known Bad Actor is trying to talk to a host located in the internal network

Multiple Enterprise Hosts Targeting Known

Bad Actors

Multiple inside hosts have been detected targeting multiple known bad actors

Multiple Known Bad Actors Targeting

Enterprise Hosts

Multiple known bad actors detected talking to internal hosts

Post Known Bad Actor Compromise Activity A host that has already been contacted by a Known Bad Actor is now talking to a Known Bad Actor

Alerting in the SIEM

Many organizations install a SIEM to be able to provide actionable alerting for their

environment. By leveraging the correlations defined in the previous section very pointed and

specific alerting can be developed. By tuning the correlations and alerting analysts, staff can

focus their efforts on high value incidents.

In an effort to make these high value alerts be more noticeable in the Trustwave SIEM, a

local extension was created to alter the coloring. This extension sets the background color of

an alert black and the text white (see Figure 16). This is color scheme is different than any

other alert colors that ship out of the box. This will allow an alert that is associated with any of

the above listed correlations to pop out on the screen and alerts the analysts to their severity.

These colors can be altered by modifying the correlations themselves and setting the

appropriate keys to any desired color combination.

Figure 16. Trustwave SIEM Alerting View

These alerts are only the beginning to what can be possible when more intelligence is

added into the SIEM in an environment. The more familiar analysts become with the SIEM and

the environment, the more alerts that can be created and tuned. These alerts and correlations

provide a great starting point and will provide value instantly to any SIEM in which they are

implemented.

Direct Query of Moat from the SIEM

By leveraging specially crafted URLs from the SIEM’s user interface, analysts can query

Moat. These queries will allow analysts to make smarter and faster decisions. Enabling

analysts to perform their job in a quicker fashion not only makes the enterprise more secure

but makes analysts happier. No analyst wants to have to perform the same tasks over and

over, especially if these tasks can be automated.

To automate the tasks in the Trustwave SIEM, context menu rules were created (see

Figure 17). These menus were created under a sub-‐menu for Moat and include the ability to:

lookup information on the source, target and generator IP Addresses, malware and also add the

source or target to Moat for post processing. These context menus will pop open a browser

window that contains the host information page, detailed in the Host Information section, and

as seen in Figure 18. This is the same view that is seen if a user navigates directly to the web

page in a standalone fashion.

Figure 17. Context Menus

The ability to add hosts to Moat is limited to only certain users. This functionality is not

only controlled through the authentication system, as detailed earlier, but also by the SIEM’s

built in permissions functionality. This will allow only seasoned analysts or power users to be

able to add information to Moat avoiding the entry of erroneous data.

Figure 18. Host Information Popup

Chapter Summary

Being able to integrate with a SIEM is what makes Moat valuable. This allows an

organization to utilize the intelligence that Moat has built over time. This information is based

on Known Bad Actors (KBA) as well as enterprise information that has been added by analysts

through the UI and API. The best use of this knowledge is through the usage of a SIEM’s

correlation engine. This chapter discussed numerous correlations that were built and

implemented in the Trustwave SIEM. These can easily be modified and implemented in any

SIEM solution that has a correlation engine. These correlations create alerts for analysts to act

upon. A custom rule was built for the Trustwave SIEM allowing it to distinguish known bad

actor alerts from other alerts by making the background black and the text white. Additional

context menus were also created to allow analysts the ability to quickly add or view data within

Moat directly from the SIEM interface without leaving the SIEM UI.

Chapter 5: Sharing Data

Data sharing in the information security field is becoming more prevalent with the

recent development and maturity of frameworks such as STIX and TAXII. Data and threat

sharing is not a new concept [26]; it has been around for a while and has even been put into

production in places like Argonne National Laboratory [27]. While many previous

implementations have focused on a single type of information, STIX provides a mechanism to

share Indicators of Compromise (IOC), campaigns, techniques, tactics, and procedures (TTP),

complete incidents and courses of actions.

Given the rise of Information Security breaches and the amount of information exposed,

it is becoming evident to corporations and the Information Security community that the

primary way to protect themselves is through threat intelligence sharing [28]. Currently,

Information Sharing and Analysis Centers (ISAC) are being created for industry specific needs

such as Retail (R-‐CISC), Financial Sector (FS-‐ISAC), Research and Education (REN-‐ISAC) and

electricity sector (ES-‐ISAC) to name a few. These groups are an excellent start, as they are all

sharing threat information among their validated members for the betterment of an industry

vertical.

Validation in a community is extremely important. If a known bad actor were to

infiltrate a sharing community, the information could be poisoned or create denial of service

attacks across members to hide their own activity. Only trusted entities should be added to the

sharing infrastructure. This validation could be handled through onsite meetings or phone calls.

The addition of trusted entities should also be a manual process so that a known bad actor

cannot automatically add themselves to the sharing without the knowledge of administrators.

A model for this sharing and validation could be based off of the web of trust model that is

utilized in public key infrastructure (PKI) [29].

Sharing from Moat

Moat is a valuable system from which to share as it will be fed with systems that are

seen on the network. These systems could potentially be part of an incident. A prime use case

for Moat would be, a user interface that allows a user to select suspicious hosts and create STIX

formatted XML. This XML can then be sent to other members of a particular organization’s

ISAC. This would not only benefit the company hosting the Moat server, but all of their peer

organizations as well.

Sharing from Moat is extremely simple. The IP Address(es) that the user would like to

share is isolated and the appropriate link is selected. This can also be completed inside the

Trustwave SIEM via the “Moat-‐Share” context menu (see Figure 19). These menu options will

appropriately craft a URL and send the GET request to the /share/ section of the API. This will,

in turn, create a STIX formatted XML file. This file can then be sent to the appropriate party.

Figure 19. Sharing Context Menus

The next logical step for Moat to take is to add TAXII functionality. “TAXII defines a set

of services and message exchanges that, when implemented, enable sharing of

actionable cyber threat information across organization and product/service boundaries

to enable automatic sharing.” [30] This implementation is beyond the scope of this paper as

TAXII bring its own challenges that need to be addressed, such as authenticating users and

setting up federation that would be required.

Chapter Summary

Sharing of threat information is gaining popularity in the information security space.

This is going to be a necessary practice for organizations to remain secure. Many of the barriers

to this sharing are coming down, especially with fully functional frameworks maturing such as

STIX. To assist in this sharing, Moat has the ability to create STIX formatted XML files for

organizations to send to other partner organizations. These XML files can be created through

the API or directly through a context menu that was created for the Trustwave SIEM.

Chapter 6: Summary

With the stress of compliance requirements, small staff sizes and undereducated

employees that plague Information Security teams, a SIEM implementation can be a burden.

Integrating a SIEM into an environment and achieving valuable results can be a daunting task.

By utilizing threat intelligence in the SIEM, the time to value can be reduced drastically.

The supplemental system, Moat, outlined in this paper can improve the overall security

stance of an organization. While Moat is not the latest, shiny technology to come out of a

marketing department somewhere, it can improve the lives of security analysts. Putting more

information at the fingertips of security operators and analysts allows these employees to make

more informed decisions faster. The security of an enterprise becomes robust when analysts

are able to act in a quicker and more accurate fashion on a consistent basis.

Moat aims to be not only a knowledge repository of systems that users feed it, but it

also actively learns from a multitude of open source intelligence feeds on the Internet. For

every host that Moat learns about, it gathers more information. This supplemental data

includes GeoIP, WHOIS and ISP information. All of this data is accessible through an API and

web user interface. The web interface includes dashboards that can help analysts isolate trends

and locate the biggest offenders.

Moat also provides enterprises with threat awareness by helping organizations identify

suspicious behavior, incorporating knowledge of external threats and assisting organizations in

threat sharing for possible indicators of compromise and warnings [31].

The sharing that Moat allows is via the STIX framework. This has been supported by the

US Department of Homeland Security (DHS) and adopted by many Information Sharing and

Analysis Centers (ISAC). Sharing in the Information Security sector has seen an increase in

recent years. Many business sectors are realizing that there is safety in numbers and no

company should be its own island.

All of the information that is gained from Moat and Known Bad Actors (KBA) is notably

more useful when utilized in a real-‐time alerting fashion. Moat provides facilities to integrate

directly into SIEM solutions so that they can harness the power to alert organizations. These

alerts are high value alerts of known malicious hosts that are communicating with their assets.

These alerts can be created with the correlations outlined in this paper or others that are more

relevant to a particular organization.

The goal of this work is to provide more value to an organization that has an existing

SIEM solution. This will allow the organization to have a stronger security stance with regard to

the utilization of known bad actor data.

Appendix: Database Schema

Audit

ID Int(11) – Primary Key

Severity Varchar(75)

Message Varchar(255)

date Timestamp

Auth

Auth_id Char(32)

Name Varchar(255)

Added Timestamp

Updated Timestamp

Write Tinyint

Read Tinyint

Enabled Tinyint

Generators

Generator_id Int(11)

Generator Varchar(255)

Weight Int(10)

Weight_default Int(10)

Host

Host_id Char(32)

Latitude Decimal(18,8)

Longitude Decimal(18,8)

Asn Varchar(255)

Country_code Varchar(5)

City Varchar(255)

State Varchar(255)

Postal_code Varchar(20)

Address Varchar(255)

Notes Varchar(500)

Os_vendor Varchar(255)

Os_version Varchar(255)

Updated Timestamp

Fingerprint Text

First_seen Timestamp

Last_seen Timestamp

Risk_score Smallint(5)

Host_generator

Host_id Char(32)


Host_id

Host_id Char(32)

Hostname Varchar(255)

Updated Timestamp

Host_ip_mac

Host_id Char(32)

Ip Varchar(100)

Mac Varchar(255)

Mac_vendor Varchar(255)

Updated Timestamp

Host_kba

Host_id Char(32)

First_seen Timestamp

Last_seen Timestamp


Times_seen Int(10)

Host_software

Host_id Char(32)

Software Varchar(255)

Updated Timestamp


Port Int(10)

Cpe Varchar(255)

Id Int(10)

Vuln

Host_id Char(32)

Vuln_code Varchar(255)

Vuln_name Varchar(255)

Description Text

Remediation Mediumtext

Protocol Varchar(25)

Severity Varchar(20)

Banner Text

Cve Varchar(50)

Bugtraq_id Varchar(50)

Cvss_basescore Varchar(20)

Cvss_vector Varchar(100)

Reference Mediumtext

Service Int(11)

Port Smallint(20)

Evidence Text

Updated Timestamp

Id Int(10)

Current Tinyint(1)


Appendix: Known Bad Actors Sources

http://isc.sans.edu/ipsascii.html?limit=100

http://rules.emergingthreats.net/blockrules/rbn-malvertisers-ips.txt

https://www.openbl.org/lists/base.txt

https://zeustracker.abuse.ch/blocklist.php?download=ipblocklist

https://spyeyetracker.abuse.ch/blocklist.php?download=ipblocklist

https://feodotracker.abuse.ch/blocklist/?download=ipblocklist

http://www.projecthoneypot.org/list_of_ips.php

http://www.blocklist.de/lists/ssh.txt

https://palevotracker.abuse.ch/blocklists.php?download=ipblocklist

http://reputation.alienvault.com/reputation.generic

https://www.dan.me.uk/tornodes

References

[

1]

Network Computing, "Report: SIEM Tools Still Pose Deployment Challenges," 05 July

2012. [Online]. Available: http://www.networkcomputing.com/careers-‐and-‐

certifications/report-‐siem-‐tools-‐still-‐pose-‐deployment-‐challenges/d/d-‐id/1233766?.

[

2]

D. Sher, "Survey: The Trouble With SIEM," 14 March 2013. [Online]. Available:

http://www.securitybistro.com/?p=5600.

[

3]

CSIRT Gadgets, "CSIRT Gadgets Foundation," 03 September 2014. [Online].

Available: http://csirtgadgets.org/.

[

4]

J. Riden, "HOW FAST-‐FLUX SERVICE NETWORKS WORK," 16 08 2008. [Online].

Available: http://www.honeynet.org/node/132. [Accessed 24 09 2014].

[

5]

"Nessus," 03 September 2014. [Online]. Available:

http://www.tenable.com/products/nessus/nessus.

[

6]

Trustwave, "Trustwave Spiderlabs Vulnerability Management," 03 September 2014.

[Online]. Available: https://trustwave.com/Services/SpiderLabs-‐Services/Vulnerability-‐

Management/.

[

7]

E. Carabott, "Why You Need to Run a Vulnerability Assessment," 31 May 2011.

[Online]. Available: http://www.gfi.com/blog/vulnerability-‐assessment/.

[ G. L. a. Fyodor, "NMAP," 24 09 2014. [Online]. Available: http://www.nmap.org.

8] [Accessed 24 09 2014].

[

9]

A. Norman, "python-‐nmap : nmap from python," 23 06 2014. [Online]. Available:

http://xael.org/norman/python/python-‐nmap/. [Accessed 24 09 2014].

[

10]

J. Ginn, "The Key to Success for the Cybersecurity Framework," Sedona Cyber Link,

December 2013. [Online]. Available: http://www.sedonacyberlink.com/2013/12/the-‐key-‐

to-‐success-‐for-‐the-‐cybersecurity-‐framework/. [Accessed 19 October 2014].

[

11]

US-‐CERT, "Information Sharing Specifications for Cybersecurity," 24 09 2014.

[Online]. Available: https://www.us-‐cert.gov/Information-‐Sharing-‐Specifications-‐

Cybersecurity. [Accessed 24 09 2014].

[

12]

Mitre, "STIXproject/stix-‐python," 24 09 2014. [Online]. Available:

https://github.com/STIXProject/python-‐stix. [Accessed 24 09 2014].

[

13]

Maxmind, "GeoIP City Database Installation Instructions," Maxmind, 2014. [Online].

Available: http://dev.maxmind.com/geoip/legacy/install/city/. [Accessed 02 10 2014].

[

14]

Maxmind, "How accurate are the GeoIP databases?," Maxmind, 2014. [Online].

Available: http://dev.maxmind.com/faq/how-‐accurate-‐are-‐the-‐geoip-‐databases/.

[Accessed 02 10 2014].

[

15]

APNIC, "Autonomous System numbers -‐ FAQs," APNIC, 2014. [Online]. Available:

http://www.apnic.net/services/services-‐apnic-‐provides/helpdesk/faqs/asn-‐faqs. [Accessed

02 10 2014].

[

16]

J. Conway, "Ecatel’s harboring of SpamBots and Malware causes BGP Peers to stop

peering with them.," SudoSecure, 30 November 2008. [Online]. Available:

http://www.sudosecure.com/ecatels-‐harboring-‐of-‐spambots-‐and-‐malware-‐causes-‐bgp-‐

peers-‐to-‐stop-‐peering-‐with-‐them/. [Accessed 02 10 2014].

[

17]

D. Fisher, "Latest Microsoft Malware Takedown Causes Waves in Security

Community -‐ See more at: http://threatpost.com/latest-‐microsoft-‐malware-‐takedown-‐

causes-‐waves-‐in-‐security-‐community#sthash.FM5oeuju.dpuf," Threatpost, 1 July 2014.

[Online]. Available: http://threatpost.com/latest-‐microsoft-‐malware-‐takedown-‐causes-‐

waves-‐in-‐security-‐community. [Accessed 02 October 2014].

[

18]

Thales, "Thales Group," June 2014. [Online]. Available:

http://www2.thalesgroup.com/press/eurosatory2014/english/Security/Thales,%20data%2

0sheet%20-‐%20Cybersecurity_Juin2014.pdf. [Accessed 09 October 2014].

[

19]

DigitalZoo, "The importance of great User Interface Design," 25 March 2014.

[Online]. Available: http://www.digitalzoo.co.uk/blog/item/the-‐importance-‐of-‐great-‐user-‐

interface-‐design. [Accessed 09 October 2014].

[

20]

Mitre, "Situation Awareness," Mitre.org, 2014. [Online]. Available:

http://www.mitre.org/capabilities/cybersecurity/situation-‐awareness. [Accessed 09

October 2014].

[

21]

R. T. Fielding, "Architectural Styles and the Design of Network-‐based Software

Architectures," 2000. [Online]. Available:

http://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arch_style.htm. [Accessed 19

October 2014].

[

22]

Parse.com, "REST API," Parse.com, 2014. [Online]. Available:

https://parse.com/docs/rest. [Accessed 19 October 2014].

[

23]

National Institute of Stadards and Technology, "Official Common Platform

Enumeration (CPE) Dictionary," NIST NVD, 17 October 2014. [Online]. Available:

http://nvd.nist.gov/cpe.cfm. [Accessed 21 October 2014].

[

24]

OpenVAS, "The world's most advanced Open Source vulnerability scanner and

manager," Open Vulnerability Assessment System, October 2014. [Online]. Available:

http://www.openvas.org/. [Accessed 21 October 2014].

[

25]

ArcSight, "Common Event Format," 17 July 2009. [Online]. Available: http://mita-‐

tac.wikispaces.com/file/view/CEF+White+Paper+071709.pdf. [Accessed 21 October 2014].

[

26]

B. Jordan, "STIX and TAXII: On the road to becoming the de facto standard," Blue

Coat, 26 August 2014. [Online]. Available: https://www.bluecoat.com/security-‐blog/2014-‐

08-‐26/stix-‐and-‐taxii-‐road-‐becoming-‐de-‐facto-‐standard. [Accessed 21 October 2014].

[

27]

R. Klump and M. Kwiatkowski, "Distributed IP Watch List Generation for Intrustion

Detection on the Electrical Smart Grid," 16 March 2010. [Online]. Available:

http://www.thei3p.org/docs/events/ifip2010/distributedipwatchlistgenerationforintrusion-‐

klumpkwiatkowskicip.pdf. [Accessed 21 October 2014].

[

28]

R. Westervelt, "The Rise Of Threat Intelligence Sharing," CRN, 21 July 2014. [Online].

Available: http://www.crn.com/news/security/300073317/the-‐rise-‐of-‐threat-‐intelligence-‐

sharing.htm. [Accessed 21 October 2014].

[

29]

E. Al-‐Hajery, "Trust Model in PGP and X.509 Standard PKI," 2002. [Online]. Available:

http://www.giac.org/paper/gsec/625/trust-‐model-‐pgp-‐x509-‐standard-‐pki/101441.

[Accessed 02 November 2014].

[

30]

Mitre, "Trusted Automated eXchange of Indicator Information," Mitre, 16 October

2014. [Online]. Available: https://taxii.mitre.org/. [Accessed 22 October 2014].

[

31]

Mitre, "Cybersecurity: Situational Awareness," Mitre, 2014. [Online]. Available:

http://www.mitre.org/capabilities/cybersecurity/situation-‐awareness. [Accessed 23

October 2014].

tyler owen - threat intelligence & siem - 2014-11-09 ·...

Documents