0012 - total visibility 2017-09-05 - senki...2017/09/05 · • by eliminating spoofed traffic, we...

Total Visibility

The Importance of Detection and Classification

• In order to operate and ensure availability of the network, we must have the ability to detect undesirable network traffic and to classify it appropriately

• We cannot contain/mitigate what we cannot detect• All the mitigation technology in the world isn’t helpful if we’ve no visibility into threats to network availability

• Detection and classification must be part of the network architecture and operational security practice• Otherwise, we’re left scrambling to figure out what’s happening—or even if anything is happening at all—instead of

how what we’re going to do about it

• In order to detect the abnormal, and possibly malicious, we have to know what’s normal—we must establish a baseline of network activity, traffic patterns, etc.

• Classification is key—it provides the context for further action

Topics for Discussion

• In this session, we’ll be talking about ways to ‘listen to the network’ by making use of specific features and tools

• Our aim is to explore some of the many ways to gain visibility into network behavior, integrating it into our operational security framework

• This is not a session on Intrusion Detection Systems (IDS)—there are plenty of IDS-specific tutorials and documentation available elsewhere. Instead, we’ll be delving into other technologies and techniques

• We are at the intersection of ‘traditional’ information security and network operational security (opsec)—it is important to distinguish between policy/compliance, penetration attempts, and opsec

• This session is focused on opsec

Six-Phased Methodology forIncident Handling• Preparation • Detection/Identification • Classification• Traceback • Reaction • Post Mortem

This Session Covers Multiple Steps

• Preparation • Detection/Identification • Classification• Traceback • Reaction • Post Mortem

Principles of Detection

Principles of Detection

• When discussing detection, there are general principles which apply across the entire network topology

• There are also specific concepts and requirements which apply to various portions of the network

• We can characterize these groupings of concepts and requirements into principles which apply to each portion of the topology

Principles of Detection—Peering Edge

• At the peering edge, we’re concerned with traffic ingressing and egressing our network

• SPs typically have more than one peer, as well as more than one peering edge router—this means that traffic may (and almost certainly is) routed asymmetrically

• Because of this asymmetry, it’s important that the various methods and systems employed to detect and classify network traffic allow visibility across the peering edge

• This means we don’t have to simulate the topology in our heads

IXP-B

IXP-A

Customer-B Customer-

A

IDC

SP-B

SP-A

Peer-B

Peer-A

Principles of Detection—Peering Edge

PeeringEdge

PeeringEdge

PeeringEdge

PeeringEdge

Principles of Detection—Customer Edge

• While we care about traffic exiting our network towards the customer, the focus at the customer edge ison ingress

• Why? Because if interesting or undesirable traffic is heading downstream, we will detect it ingressing somewhere else

• From the SP standpoint, DoS, worms, from customer networks are problematic—they must be detected and mitigated as close to the ingress point as possible

• It is also important to protect the SP peering links and other SPs (and their customers) from attack

IXP-B

IXP-A


A

IDC

SP-B

SP-A

Peer-B

Peer-A

Principles of Detection—Customer Edge

CustomerEdgeCustomer

Edge

CustomerEdge

Principles of Detection—IDC• The Internet Data Center (IDC) is a vital part of the SP

infrastructure—shared services such as DNS, status Web pages, etc., are often housed within the IDC

• Hosted and co-located customers also depend upon the IDC for connectivity to the world

• Because it is an edge of the network, we generally see topological symmetry at the interconnection between the IDC and the rest of the network—this gives us good bidirectional visibility

• We must have the capability to examine trafficbetween hosts within the IDC as well as traffic to/from the Internet

IXP-B

IXP-A


A

IDC

SP-B

SP-A

Peer-B

Peer-A

Principles of Detection—IDC

IDCEdge

Principles of Detection—Core

• In the core, router health—CPU load, memory, etc.—is paramount. Problems in the core = problems elsewhere

• While we do not generally perform first-order detection and classification within the core itself, it is often necessary to trace traffic through the core to its origin(s) and/or destination(s)

• In many instances, we must examine the same traffic at various points in the topology in order to gain a clear understanding of a given event

• It is important that detection and classification technologies utilized within the core have minimal impact on performance

IXP-B

IXP-A


A

IDC

SP-B

SP-A

Peer-B

Peer-A

Principles of Detection—Core

Telemetry — A Conceptual Overview

Te·lem·e·try— a technology that allows the remote measurement and reporting of information of interest to the system designer or operator. The word is derived from Greekroots tele = remote, and metron = measure

What Is Meant by ‘Telemetry’?

Check List

• Check SNMP. Is there more you can do with it to pull down security information?

• Check RMON. Can you use it?• Check Netflow. Are you using it, can you pull down more?• Check Passive DNS• See addendum for lots of links.

Holistic Approach to System-Wide Telemetry

Cardiologist

OphthalmologistNeurologist

NephrologistHematologist

Podiatrist

Holistic Approach to Patient CareUses a system-wide approach, coordinating with various specialists, resulting in the patient’s better overall health and wellbeing.

Holistic Approach to System-Wide Telemetry

Data Center:• Inter as well as Intra Data

Center traffic

Customer Edge:• Shared resources and

services should be available

Core:• Performance must

not be affected SP Peering:• Ability to trace

through asymmetric

traffic

P

P

P

P

PE

P

P

PE(s) L2 Agg.

Broadband, Wireless (3G,

802.11), Ethernet, FTTH, Leased

Line, ATM, Frame-Relay

CPE(s)

P P

Data/Service Center

CPE/ACCESS/AGGREGATION CORE PEERINGDATA/SVC Center

ISP / Alt. Carrier

Listen ListenListen

Listen

Network Telemetry

• Network telemetry offers extensive and useful detection capabilities

• This telemetry is often coupled with dedicated analysis systems to collect, trend, and correlate observed activity

• There are several forms of telemetry available from routers, switches, and other network devices

• There are a number of open source and commercial tools available which greatly enhance the utility of network telemetry

• Getting started with network telemetry is both inexpensive and relatively easy

Network Telemetry—Time Synchronization

• When dealing with network telemetry, it is important that dates and times are both accurate and synchronized

• Enabling Network Time Protocol (NTP) is the common method of time synchronization — it is supported by routers, switches, firewalls, hosts, and other network-attached devices

• Without time synchronization, it’s very difficult to correlate different sources of telemetry

• More information on NTP can be found at• http://www.ntp.org

Network Telemetry—OOB Management

• In-Band access to network infrastructure, hosts, etc., works very well — until there’s a problem on the network

• In order to maximize reachability of and control over the network even during disruptive events, it is necessary to build an isolated Out-of-Band (OOB) management network

• Many devices such as routers and switches have serial console ports; others have Ethernet management interfaces

• Transmitting network telemetry over the OOB network minimizes the chance for disruption of the very information which gives us network visibility

Network Telemetry—Antispoofing• There are many mechanisms available in modern network infrastructure devices to disallow

spoofed traffic from transiting the network - Unicast Reverse Path Forward (uRPF), DHCP Snooping with IP Source Guard, Cable IP Source Verify, etc.

• Spoofed traffic is by definition invalid traffic - there is no reason to allow spoofed traffic to ingress and transit your network. Disallowing spoofed traffic is a basic step in improving network resiliency

• By eliminating spoofed traffic, we remove clutter from the ‘data horizon’ generated by analyzing network telemetry

• This greatly reduces the traceback problem - with antispooing measures in place, we know that purported source IPs originating from network edges under our control are valid, and we eliminate bogon-sourced traffic from the peering edge

Source: University of Wisconsin

Open Source Tools for NetFlow Analysis Visualization — FlowScan

Investigate the spike

An identified cause of the outage

Netflow & IPFIX

NetFlow Records and Key Fields

• NetFlow maintains per-’conversation’ flow data in Flow Records in a cache on a NetFlow-enabled device, and optionally exports that flow data to a collection/analysis system

• It is a form of network telemetry which describes traffic conversations headed to/passing through a router• Key Fields

• Key field values define a Flow Record• An attribute in the packet used to create a Flow Record• If the set of key field values is unique, a new flow is created

NetFlow Non-Key Fields and Statistics• Non-key fields are not used to define a flow and are exported along

with the flow and provide additional information• NetFlow non-key fields

• Source and destination AS's• Source and destination IP prefix masks• IP address of next-hop router• TCP flags• Output interface

• NetFlow features provide per flow statistics• Number of packets and bytes in flow• Time-stamps for first and last packets in flow

What Constitutes a Flow?

1. Inspect a packet’s 7 key fields and identify the values 2. If the set of key field values is unique, create a new

flow record or cache entry3. When the flow terminates, export the flow to the

collection/analysis system

NetFlowExport

PacketsReporting

NetFlow Key Fields

1

2

3

NetFlow Key Fields CreatingFlow Records

Inspect Packet

Key Fields Packet 1

Source IP 1.1.1.1

Destination IP 2.2.2.2

Source port 23

Destination port 22078

Layer 3 Protocol TCP - 6

TOS Byte 0

Input Interface Ethernet 0

Source IP Dest. IP Dest. I/F Protocol TOS … Pkts

1.1.1.1 2.2.2.2 E1 6 0 … 11000

1. Inspect packet for key field values

2. Compare set of values to NetFlow cache

3. If the set of values are unique create a flow in cache

4. Inspect the next packet

Inspect Packet

Key Fields Packet 2

Source IP 3.3.3.3

Destination IP 2.2.2.2

Source port 23

Destination port 22078

Layer 3 Protocol TCP - 6

TOS Byte 0

Input Interface Ethernet 0

Source IP Dest. IP Dest. I/F Protocol TOS … Pkts

3.3.3.3 2.2.2.2 E1 6 0 … 11000

1.1.1.1 2.2.2.2 E1 6 0 … 11000

Add new flow to the NetFlow CacheCreate Flow Record in the Cache

Example 1 Example 2

NetFlow Cache on a Router1. Create and update flows in NetFlow cache

Srclf SrclPadd Dstlf DstlPadd Protocol TOS Flgs Pkts SrcPort

SrcMsk

SrcAS

DstPort

DstMsk

DstAS NextHop Bytes/

Pkt Active Idle

Fa1/0 173.100.21.2 Fa0/0 10.0.227.12 11 80 10 11000 00A2 /24 5 00A2 /24 15 10.0.23.2 1528 1745 4

Fa1/0 173.100.3.2 Fa0/0 10.0.227.12 6 40 0 2491 15 /26 196 15 /24 15 10.0.23.2 740 41.5 1

Fa1/0 173.100.20.2 Fa0/0 10.0.227.12 11 80 10 10000 00A1 /24 180 00A1 /24 15 10.0.23.2 1428 1145.5 3

Fa1/0 173.100.6.2 Fa0/0 10.0.227.12 6 40 0 2210 19 /30 180 19 /24 15 10.0.23.2 1040 24.5 14

• Inactive timer expired (15 sec is default)• Active timer expired (30 min (1800 sec) is default)2. Expiration

Srclf SrclPadd Dstlf DstlPadd Protocol TOS Flgs Pkts SrcPort

SrcMsk

SrcAS

DstPort

DstMsk

DstAS NextHop Bytes/

Pkt Active Idle

Fa1/0 173.100.21.2 Fa0/0 10.0.227.12 11 80 10 11000 00A2 /24 5 00A2 /24 15 10.0.23.2 1528 1800 4

4. Export version

5. Transport protocol ExportPacket

Payload(Flows)

Non-Aggregated Flows—Export Version 5 or 9

Hea

de r

30 Flows per 1500 byte export packet

Key fields in yellowNon-key fields white

NetFlow Version 5—Flow Format

• Source IP Address• Destination IP Address

• Packet count• Byte count

Usage

QoS

Timeof Day Application

PortUtilization

From/To

Routing and

Peering

• Input ifIndex• Output ifIndex

• Type of service• TCP flags• Protocol

• Start sysUpTime• End sysUpTime

• Source TCP/UDP port• Destination TCP/UDP port

• Next hop address• Source AS number• Dest. AS number• Source prefix mask• Dest. prefix mask

• Source IP address• Destination IP address

NetFlow v9 Export Packet Format

• Matching ID #s is the way to associate template to the data records• The Header follows the same format as prior NetFlow versions so collectors will be

backward compatible• Each data record represents one flow• If exported flows have the same fields then they can be contained in the same

template Record E.G. Unicast traffic can be combined with multicast records• If exported flows have different fields then they can’t be contained in the same

template record e.g. BGP next-hop can’t be combined with MPLS Aware NetFlow records

Data FlowSetTemplate FlowSet Option TemplateFlowSet

FlowSet ID #1Data FlowSetFlowSet ID #2

Template ID

(SpecificField Types

and Lengths)

(Version, # Packets,

Sequence #, Source ID)

Flows from Interface A

Flows from Interface B

To Support Technologies Such asMPLS or Multicast, this Export Format CanBe Leveraged to Easily Insert New Fields

Option DataFlowSetFlowSet IDTemplate

RecordTemplate ID #1(Specific Field

Types and Lengths)

DataRecord

(FieldValues)

Template Record

Template ID #2

(Specific Field Types and Lengths)

DataRecord

(FieldValues)

DataRecord

(FieldValues)

Option Data

Record

(Field Values)

Option Data

Record

(Field Values)

Key Concept — NetFlow Scalability

• Packet capture is like a wiretap• NetFlow is like a phone bill• This level of granularity allows NetFlow to

scale for very large amounts of traffic• We can learn a lot from studying the phone bill!• Who’s talking to whom, over what protocols and

ports, for how long, at what speed, for whatduration, etc.

• NetFlow is a form of telemetry pushed from the routers/switches — each one can be a sensor

35

Netflow Setup•Don’t have a copy of netflow data b/c IT won’t share?

•Many products have the ability to copy flow data off to other destinations

Collector

Peakflow NetQoS

Storage

Export netflow data to OSU Flowtools Collector

Regionalized collection to minimize WAN impact

Netflow data copied to other destinations with flow-fanout

NetFlow Collection at Cisco

• DMZ Netflow Collection (4 servers)• Data Center Netflow Collection (20+ servers)• Query/Reporting tools (OSU Flowtools, DFlow, Netflow Report Generator) 200K pps

3 ISP gateways600GB ~ 3 months

Key Concepts—Anomaly Detection

• No signatures—instead, uses observed behavior as the baseline

• No false positives—everything reported is actually happening (setting thresholds for tuning severity)

• Can detect ‘minute-0’ attacks• Can highlight behaviors which are not indicative of attack traffic,

but are still of interest• Can be used as an indicator to focus more closely using packet-

capture, IDS, etc.• Not limited to large-scale events—user-defined thresholds

determine severity/alert scaling

38

OSU Flowtools - Netflow Collector Setup

• Tool: OSU FlowTools• Free!• Developed by Ohio State University

• Examples of capabilitiesDid 192.168.15.40 talk to 216.213.22.14?What hosts and ports did 192.168.15.40 talk

to?Who’s connecting to port TCP/6667?Did anyone transfer data > 500MB to an

external host?

39

OSU Flowtools ExampleWho’s Talking?

• Scenario: New botnet, variant undetected• You need to identify all systems that ‘talked’ to the botnet C&C• Luckily you’ve deployed netflow collection at all your PoPs

put in specific query syntax for the example[mynfchost]$ head flow.acl

ip access-list standard bot permit host 69.50.180.3ip access-list standard bot permit host 66.182.153.176

[mynfchost]$ flow-cat /var/local/flows/data/2007-02-12/ft* | flow-filter -Sbot -o -...

Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP0213.08:39:49.911 0213.08:40:34.519 58 10.10.71.100 8343 98 69.50.180.3 313370213.08:40:33.590 0213.08:40:42.294 98 69.50.180.3 31337 58 10.10.71.100 83

flow.acl file uses familiar ACL syntax. create a list named ‘bot’ concatenate all

files from Feb 12, 2007 then filter for src or dest of ‘bot’ acl

we’ve got a host in the botnet!

Custom NetFlow Report GeneratorQuery by IP

41

Know Thy Subnets• Critical to providing context to an incident

• Is the address in your DMZ? lab? remote access? desktop? data center?

• Make the data queryable• Commercial & open source products available

• Build the data into your security devices• SIMS - netForensics asset groups• SIMS - CS-MARS network groups• IDS - Cisco network locale variables

variables DC_NETWORKS address 10.2.121.0-10.2.121.255,10.3.120.0-10.3.127.255,10.4.8.0-10.4.15.255variables DMZ_PROD_NETWORKS address 198.133.219.0-198.133.219.255variables DMZ_LAB_NETWORKS 172.16.10.0-172.16.11.255

eventId=1168468372254753459 eventType=evIdsAlert hostId=xxx-dc-nms-4appName=sensorAppappInstanceId=6718 tmTime=1178426525155 severity=1 vLan=700 Interface=ge2_1 Protocol=tcpriskRatingValue=26 sigId=11245 sigDetails=NICK...USER" src=10.2.121.10 srcDir=DC_NETWORKSsrcport=40266 dst=208.71.169.36 dstDir=OUTdstport=6665

data center host!

Network Telemetry - MRTG/RRDTool• Not just netflow, can also use SNMP to grab telemetry• Shows data volumes between endpoints

You must understand

your network traffic volume!

43

•Network traffic data

•Subnet information - IP address management data

• 10.10.0.0/19 A (Active) Data Centers• |-- 10.10.0.0/20 A (Active) Building 3 Data Center• | |-- 10.10.0.0/25 S (Active) Windows Server Subnet• | |-- 10.10.0.128/25 S (Active) Oracle 10g Subnet• | |-- 10.10.1.0/26 S (Active) ESX VMWare Farm• | |-- 10.10.1.64./26 S (Active) Web Application Servers

• 10.10.0.0/16 A (Active) Indiana Campus• |-- 10.10.0.0/19 A (Active) Data Centers• |-- 10.10.32.0/19 A (Active) Site 1 Desktop Networks• | |-- 10.10.32.0/24 S (Active) Building 1 1st floor• | |-- 10.10.33.0/25 S (Active) Building 1 2nd floor• | |-- 10.10.33.128/25 S (Active) Building 2

Based on our design,

environment, and these aggregate traffic levels with

spikes above 400Mbps, We need

an IPS 4260

Blanco Wireless: Network

Source: UNINETT

NetFlow - Stager

Source: http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/

Other Visualization Techniques Using SNMP Data with RRDTool

Anomaly for DNS Queries

Thru’put Spike

RTTSpike

Source: http://www.ntop.org

Displaying RMON—ntop Examples

Detailed Analysis i.e. TTL

BGP Example—SQL Slammer

Correlating NetFlow and Routing Data

Matching data collected from different tools

DNS for Attack Detection

Utilizing DNS Telemetry for Detection

• The Domain Name System (DNS) is a ‘background’ service we often don’t think about, but in actuality use many, many timeseach day

• Many types of application use name-based lookups—Web browsers, email servers, Web servers—and malware such as trojans and bots running on compromised hosts

• By examining DNS logs and statistics, we can detect activity which should be further investigated

• Correlating DNS-related info with other forms of telemetry (NetFlow, packet capture, application logs, etc.), we can often infer the causes and effects of unusual network activity

Example—dnstop query types

0 new queries, 38 total queries Wed Jun 1 17:35:51 2005

Query Type count %

---------- --------- ------

A? 9 23.7

NS? 1 2.6

SOA? 1 2.6

PTR? 15 39.5

MX? 10 26.3

TXT? 2 5.3

Example—dnstop sources output

0 new queries, 38 total queries Wed Jun 1 17:35:51 2005

Sources count %

---------------- --------- ------

172.19.61.44 19 50.0

172.19.60.28 9 25.0

172.19.61.33 9 25.0

Example—dnstop source, record output0 new queries, 38 total queries Wed Jun 1 17:35:51 2005

Source 3LD count %

---------------- -------------------- --------- ------

172.19.61.44 cnn.com 10 26.3

172.19.61.33 107.128.in-addr.arpa 5 13.2

172.19.61.44 19.172.in-addr.arpa 5 13.2

172.19.60.28 cisco.com 5 13.2

172.19.61.33 71.171.in-addr.arpa 2 5.3

172.19.61.44 24.172.in-addr.arpa 2 5.3

172.19.61.44 www.slacker.com 2 5.3

172.19.61.44 google.com 2 5.3

172.19.60.28 speakeasy.net 1 2.6

172.19.61.44 168.192.in-addr.arpa 1 2.6

172.19.61.33 . 1 2.6

172.19.60.28 www.cnn.com 1 2.6

172.19.61.44 telecomplete.co.uk 1 2.6

Example—dnstop destination output0 new queries, 38 total queries Wed Jun 1 17:35:51 2005

Destinations count %

---------------- --------- ------

172.19.226.120 29 77.7

10.158.254.13 5 15.0

172.19.220.131 4 9.3

DNS—Correlation

• By correlating DNS queries with other forms of telemetry and graphing, we can spot trends and infer root causes

• Kumamoto University in Tokyo have published several insightful papers on this subject—the tools they used were server logs (BIND, sendmail, QPopper), IDS, and grep

• In other words, you can try this at home!

Example—DNS Queries,Compromised PC

Source: Kumamoto University, Tokyo

Example—DNS Queries, Compromised PC-Based Firewall


Example—Correlating DNS and SMTP


RRDTool Graph of DNS Queries/Sec


DNS—More Information

• dnstop home• http://dns.measurement-factory.com/tools/dnstop/

• dnslogger home• http://www.enyo.de/fw/software/dnslogger/

• Kunamoto University Papers on DNS-based Detection• http://www.cc.kumamoto-u.ac.jp/~musashi/musashicsec27.pdf• http://www.cc.kumamoto-u.ac.jp/~musashi/dsm32-12.pdf• http://www.cc.kumamoto-u.ac.jp/~musashi/info2006-mcfsit.pdf

• Dan Kaminsky on DNS as a Covert Channel• http://www.doxpara.com/dns_bh

• DNS as an IDS• http://staff.science.uva.nl/~delaat/snb-2005-2006/p12/report.pdf

• Detecting Mass-Mailing Worms via DNS• http://www.sigcomm.org/sigcomm2005/paper-IshToy.pdf

BGP Used as Attack Detection

BGP• Large-scale network security events such as worms, DDoS attacks, etc., often produce side-

effects visible in the globalrouting table

• Correlating BGP information with other forms of telemetry (NetFlow, SNMP, RMON, etc.) can be effective in determining the true impact of incidents

• Zebra (http://www.zebra.org) and Quagga (http://www.quagga.net) are two open source BGP daemons which can log BGP updates for further analysis

• Arbor Peakflow SP Traffic provides BGP visualization, trending, NetFlow traffic correlation, additional functionality (http://www.arbornetworks.com/products_sp.php)

• RIBs/updates available from http://archive.routeviews.org/, http://www.ripe.net/ris/index.html, http://www.renesys.com (commercial, useful monitoring tools/services for your ASN)

BGP Example—SQL Slammer

Arbor Example—Correlating BGPand NetFlow

Narus - BGP Activity, Topology

Packet Design RouteExplorer—Detecting BGP Backdoor Routes

BGP Example—Two Backdoor Routes

• Hard to notice in sh ip bgp output• This incident was due to a department having a ‘special

arrangement’ to reach a certain AT&T customer• Backdoor peering can have severe impact

• What if AT&T started sending full routes?

Packet Design—BGP Peer Reset

Gain

BGP—More Information

• Slammer/BGP analysis—• http://www.nge.isi.edu/~masseyd/pubs/masse

y_iwdc03.pdf

• Team CYMRU BGP Tools—• http://www.cymru.com/BGP/index.html

• Packet Design Route Explorer—• http://www.packetdesign.com/products/rex.htm

Syslog for Attack Detection

Syslog

• De facto logging standard for hosts, network infrastructure devices, supported in all most routers and switches

• Many levels of logging detail available—choose the level(s) which are appropriate for each device/situation

• Logging of ACLs is generally contraindicated due to CPU overhead—NetFlow provides more info, doesn’t max the box

• Can be used in conjunction with Anycast and databases such as MySQL (http://www.mysql.com) to provide a scalable, robust logging infrastructure

• Different facility numbers allows for segregation of log info based upon device type, function, other criteria

• Syslog-ng from http://www.balabit.com/products/syslog_ng/ adds a lot of useful functionality—HOW-TO located at http://www.campin.net/newlogcheck.html

Syslog

• Syslog output can become overwhelming very quickly—ensure adequate storage and processing capacity are available. Open source databases (MySQL, PostGres) are often used to store syslog output for postprocessing and searching

• Routers have Facility numbers are the key to organizing syslog output on a syslog server—by default, Cisco routers export syslog as Facility 7

• The level of logging detail is also an important consideration. Too little information is useless—likewise, too much information is overwhelming

Router(config)# memory free low-watermark processor 20000

Router(config)# memory free low-watermark io 20000

Router(config)# memory reserved critical 1000

000029: *Aug 12 22:31:19.559: %SYS-4-FREEMEMLOW: Free Memory has dropped below 20000kPool: Processor Free: 66814056 freemem_lwm: 20480000

000032: *Aug 12 22:33:29.411: %SYS-5-FREEMEMRECOVER: Free Memory has recovered 20000kPool: Processor Free: 66813960 freemem_lwm: 0

Memory Thresholding Syslog Example

Syslog—Example SEC Rules# Watch for 10 denies within 10 seconds. Especially useful to monitor

# for certain trojans and mass mailers

type=SingleWithThreshold

ptype=RegExp

pattern=\s*.*Deny\s+(\w+)\s+src.*:(.*)/.*:(.*)/(\b2\d\b).*$

desc=Unusual Failures:$1 $4/$2 -> $3

action=create ffo_$1; add ffo_$1 %t; add ffo_$1 $0; report ffo_$1 /bin/mail -s "%s" [email protected]; delete ffo_$1

window=10

thresh=10

# Failure of secondary (standby) firewall while primary is active

# Works for PIX 7.x

# $1 is the IP address of the primary firewall

#type=Single

continue=takenext

ptype=RegExp

pattern=\s*\S+\s(\d+\.\d+\.\d+\.\d+).*PIX-1-102001.*$Primary$.*$

desc=Secondary firewall for $1 - failure/reload

action=create ffo_$1; add ffo_$1 %t; add ffo_$1 $0; report ffo_$1 /bin/mail -s "%s" [email protected]; delete ffo_$1

Syslog—CS-MARS

Syslog—Sawmill

Syslog—More Information

• Syslog.org• http://www.syslog.org/

• Syslog Logging w/PostGres HOWTO—• http://kdough.net/projects/howto/syslog_postgresql/

• Agent Smith Explains Syslog—• http://routergod.com/agentsmith/

• Simple Event Correlator (SEC)• http://kodu.neti.ee/~risto/sec/

• Cisco CS-MARS• http://www.cisco.com/en/US/products/ps6241/index.html

SNMP for Attack Detection

SNMP• Routers can be configured to generate Simple Network Management Protocol

(SNMP) traps when various events take place. SNMP statistics can be polled from routers, switches, and other network devices

• Cisco IOS CPU Thresholding enables routers to generate SNMP traps when a specified CPU performance envelope, expressed as a high-water mark and a low-water mark, is transgressed

• NET-SNMP, MRTG, Cricket, RRDTool, Nagios, are all open source tools which make use of SNMP telemetry. Arbor PeakFlow DoS and Cisco CS-MARS combine SNMP polling of interface speeds with NetFlow telemetry in order to perform traffic profiling and anomaly-detection

SNMP—High CPU

• Spikes in CPU load on routers, switches, servers, and other devices is often an indication that an event is taking place

• High CPU is not always an indicator of malicious activity. Trending is important

• Correlating CPU utilization with other information such as network traffic statistics, routing-table changes, etc., is very useful

• A baseline of CPU utilization over time is a good idea from a network management standpoint, and also allows us to determine if further investigation is warranted

SNMP—Network Traffic Statistics

• Excessive bandwidth (bps) and/or throughput (pps) can be an indicator of undesirable traffic

• DoS attacks and DoS-like worms can cause high amounts of traffic• Most Network Management Systems (NMS) can alert on high bps and/or high

pps• High amounts of traffic are not always indicative of problems—it’s situationally-

dependent, another element to consider. This is another reason it’s important to have a baseline

• NMS such as HP OpenView, Nagios, etc. often support alerting based upon low-water-marks as well as high-water-marks—it can be just as important to be notified of a drop in link speed (or number of routes, etc.) as it is to learn of an increase

SNMP—Link-Flap Notification

• Link-flaps can also indicate that something is amiss• They’re often a sign of misconfiguration, backhoe incidents

and the like—but they can also result from malicious activity, such as a DoS attack

• Routers and switches can be configured to notify monitoring systems when link-flaps occur

• Link-flaps are not always indicative of problems—it’s situationally-dependent, another element to consider

SNMP—CPU Thresholding Example

Router(config)# snmp-server enable traps cpu threshold

Router(config)# snmp-server host 192.168.0.1 traps public cpu

Router(config)# process cpu threshold type total rising 80 interval 5 falling 20 interval 5

Router(config)# process cpu statistics limit entry-percentage 40 size 300

Example—Cisco MIB Browser

Example—iReasoning MIB Browser

Example—NET-SNMP snmpwalk

luser@linuxbox:$~snmpwalk -m ALL 172.19.24.1 public cisco

enterprises.cisco.local.lcpu.1.0 = "..System Bootstrap, Version 12.3(4)T, [htseng 180] EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)..Copyright (c) 2003 by Cisco Systems, Inc..."enterprises.cisco.local.lcpu.2.0 = "reload"enterprises.cisco.local.lcpu.3.0 = "cisco"enterprises.cisco.local.lcpu.4.0 = "ora.com"enterprises.cisco.local.lcpu.5.0 = IpAddress: 127.45.23.1enterprises.cisco.local.lcpu.6.0 = IpAddress: 0.0.0.0enterprises.cisco.local.lcpu.8.0 = 131888844enterprises.cisco.local.lcpu.9.0 = 456enterprises.cisco.local.lcpu.10.0 = 500enterprises.cisco.local.lcpu.11.0 = 17767568enterprises.cisco.local.lcpu.12.0 = 0enterprises.cisco.local.lcpu.13.0 = 0enterprises.cisco.local.lcpu.14.0 = 104enterprises.cisco.local.lcpu.15.0 = 600

Graphing SNMP Statistics with MRTG

Source: mrtg.org

Graphing SNMP Statistics with RRDTool


SNMP—More Information

• Cisco SNMP object tracker—• http://www.cisco.com/pcgi-

bin/Support/Mibbrowser/mibinfo.pl?tab=4• Cisco MIBs and trap definitions

• http://www.cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml• MRTG

• http://people.ee.ethz.ch/~oetiker/webtools/mrtg/

Network Time Protocol

Network Time Protocol

• Synchronize time across all devices• When security event occurs, data must have consistent

timestamps• From external time source

• Upstream ISP, Internet, GPS, atomic clock• From internal time source• Router can act as stratum 1 time source• ntp source loopback0• ntp server 10.1.1.1 source loopback0

Benefits of Deploying NTP

• Very valuable on a global network with network elements in different time zones

• Easy to correlate data from a global or a sizable network with a consistent time stamp

• NTP based timestamp allows to trace security events for chronological forensic work

• Any compromise or alteration is easy to detect as network elements would go out of sync with the main ‘clock’

Open Source Netflow Tools


• NTop (or Ntopng) Probably the most well-known open source traffic analyzers, Ntop, is a web-based tool that runs on Ubuntu x64 versions, CentOS/Redhat x64 Linux flavors, Windows x64 Operating systems, BeagleBoard ARM, Ubiquity networks EdgeRouter and even Mac OSX per their github site. nTopng also includes suuport for sFlowand IPFIX (through nProbe add-on), as its becoming a new standard that many manufacturers are using for flow analysis. RRD is used for databases and storing of data on a per-host level.


• Flow-tools - Flow-tools is a toolset that can be used to Collect, Send, Process and generate Reports for Netflowdata flows and provides an API for developing custom features and applications. Flow Tools is hosted at http://flow-tools.googlecode.com.


• FlowScan Flowscan is more of a visualization tool that analyzes and reports Netflow data and can produce visual graphs that are in “near” real-time to see whats going on in your network. Flowscan can be deployed on a GNU/Linux or BSD system and uses some of the following packages in order to correctly collect and process flows: “cflowd” to as the flow collector, “flowscan” which is a perl script that makes up the software package itself (“FlowScan”) and is responsible for loading and executing reports and the last major component is “RRDtool” which is used to store all flow information in its database.


• EHNT EHNT (which is pronounced “ent”) is an acronym for Extreme Happy NetFlow Tool. This is a commandline tool that supports Netflow Version 5 only and provides reports for intervals between 1 min to 24 hrs and provides information about Ip Protocols, TCP/UDP ports and more.

• BPFT (which stands for Berlekey Packet Filter Traffic collector) is a built on top of the BPF “pseudo-device” and libpcap for capturing IP traffic, including Source/Destination IP’s & Ports, number of transmitted/received bytes which are all stored in one compact form binary file.


• Maji Maji is an implementation of an IPFIX meter which is based on libtrace, a packet capturing and processing library. Maji seems to have an array of information per their website and the latest release was from 07/2011. One of the major benefits to maji is the custom templates you can develop with as many elements included into them as you want, and can be exported via Network over SCTP/TCP/UDP, SQLite database or the terminal.


• Cflowd - cflowd is a tool that is made for analyzing Netflowenabled devices and includes modules for collecting, storing and analyzing netflow data. Apparently cflowd is no longer being supported per their website, and is directing users to use flow-tools with FlowScan in order to take advantage of cflowd and its modules.

• AnonTool - AnonTool is more of an anonymization tool for netflow v5 & v9 traces.


• Panoptis - According to the sourceforge page, this project is no longer being developed or supported and was an open-source project that used NetFlow data to help detect and stop (Distributed) Denial of Service attacks. It is no longer support or being updated, so use at your own risk. Check out their Sourceforge page for more information and a download link.


• Pmgraph - pmGraph is a great open source tool for graphing and monitoring bandwidth using pmacct, which is a network monitoring and auditing tool. pmacct collects and monitors traffic using Netflow or Sflow on network devices (including firewalls, routers and switches) into a database and allows for analysis of that data using pmGraph. The software was developed by Aptivate staff and volunteers and looks to still be active.


• InMon sFlow Toolkit - sFlow toolkit is an open source software package the is used for analyzing sFlow data and can be used with other utilities including tcpdump, ntop and Snort for further analysis. “sflowtool” is the main component of the sFlow toolkit software and is a command-line utility that gives you the ability to view network traffic devices in real-time and interface with other software packages for mapping out graphical images of IP flow. sflowtool is also available for windows as well per their website.


• NDSAD Traffic Collector - NDSAD, which stands for NetUP’s Data Stream Accounting Daemon, was developed by NetUP as a tool to capture packets and generate Netflow v5 data streams and was specifically used for ISP billing purposes. The software still seems to be supported as well.


• NFsen/NFDump NFsen, which is short for Netflow Sensor, is a web-based front-end tool for nfdump to present the user a nice graphical image of all the data nfdump pumps out. You have the ability to generate reports of your netflow data with information including Flows, Packets and bytes using RRD database tool, as well as setup alerts and view historical data. nfsen project is still very active and can be downloaded from its Sourceforgepage here and runs on any Unix/Linux systems. You’ll need PHP, PERL (along with Perl Mail::Header and Mail::Internet modules), RRD Tools module and Nfdump tools installed on your system in order to use it correctly.

Total Visibility - Addendum

NetFlow—More Information

• Cisco NetFlow Home—http://www.cisco.com/warp/public/732/Tech/nmp/netflow

• Linux NetFlow Reports HOWTO—http://www.linuxgeek.org/netflow-howto.php

• Arbor Networks Peakflow SP—http://www.arbornetworks.com/products_sp.php

More Information about SNMP

• Cisco SNMP Object Tracker— http://www.cisco.com/pcgi-bin/Support/Mibbrowser/mibinfo.pl?tab=4

• Cisco MIBs and Trap Definitions— http://www.cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml

• SNMPLink—http://www.snmplink.org/• SEC-1101/2102 give which SNMP parameters should be looked at.

RMON—More Information

• IETF RMON WG—http://www.ietf.org/html.charters/rmonmib-charter.html

• Cisco RMON Home—http://www.cisco.com/en/US/tech/tk648/tk362/tk560/tech_protocol_home.html

• Cisco NAM Product Page—http://www.cisco.com/en/US/products/hw/modules/ps2706/ps5025/index.html

BGP—More Information

• Cisco BGP Home—http://www.cisco.com/en/US/tech/tk365/tk80/tech_protocol_family_home.html

• Slammer/BGP analysis—http://www.nge.isi.edu/~masseyd/pubs/massey_iwdc03.pdf

• Team CYMRU BGP Tools—http://www.cymru.com/BGP/index.html

Syslog—More Information

• Syslog.org - http://www.syslog.org/• Syslog Logging w/PostGres HOWTO—

http://kdough.net/projects/howto/syslog_postgresql/• Agent Smith Explains Syslog—

http://routergod.com/agentsmith/

Packet Capture—More Information

• tcpdump/libpcap Home—http://www.tcpdump.org/

• Vinayak Hegde’s Linux Gazette article—http://www.linuxgazette.com/issue86/vinayak.html

Remote Triggered Black Hole

• Remote Triggered Black Hole filtering is the foundation for a whole series of techniques to traceback and react to DOS/DDOS attacks on an ISP’s network.

• Preparation does not effect ISP operations or performance.• It does adds the option to an ISP’s security toolkit.

More Netflow Tools

• NfSen - Netflow Sensor• http://nfsen.sourceforge.net/

• NFDUMP• http://nfdump.sourceforge.net/

• FlowCon• http://www.cert.org/flocon/

Okay—Tell Me Where to Start From?

1. NetFlow enablement on the network elements2. NetFlow data correlation and analysis 3. SNMP / RMON [SNMP more prevalent]

1. CPU / Memory util2. Link usage and display with MRTG

4. SysLog collection and analysis5. Monitoring to Routing, DNS queries, etc. [BGP, DNS]6. Local and remote packet capture facility [Most have it

today with sniffer, ethereal]

Summary

• Define telemetry strategy—ASAP• Local and remote

• Need to start deployment today where the most bang for the buck is offered. However, the end goal is to achieve the holistic view

• Telemetry: Deploy, Understand and Practice• For any security event – Proactive Telemetryor

telemetry during the incident, if ‘SECOPS’ trained then they can use it with familiarity of ‘back of their hand’

• Telemetry builds foundation to be successful with all the other 5 of 6 steps methodology

Additional References

• Product Security: • Cisco’s product vulnerabilities; a page that every SE MUST know!!!

[http://www.cisco.com/warp/public/707/advisory.html]• Security reference information: Various white papers on DoS attacks and

how to defeat them [http://www.cisco.com/warp/public/707/ref.html]

• ISP Essentials:• Technical tips for ISPs every ISP should know

[ftp://ftp-eng.cisco.com/cons/isp/]

• Technical tips:• Troubleshooting High CPU Utilization on Cisco Routers

[http://www.cisco.com/warp/public/63/highcpu.html]• The “show processes” command

[http://www.cisco.com/warp/public/63/showproc_cpu.html]• NetFlow performance white paper

[http://www.cisco.com/warp/public/cc/pd/iosw/prodlit/ntfo_wp.htm]

• Mailing lists: • Cust-security-announce: All customers should be on this list • Cust-security-discuss: For informal discussions

Recommended Reading

Silence on the Wireby Michael ZalewskiISBN: 1593270461

Recommended Reading

The Tao of Network Security Monitoringby Richard BejtlichISBN: 0321246772

Recommended Reading

TCP/IP Illustrated, Vol. I (The Protocols)by W. Richard StevensISBN: 0201633469

Recommended Reading• Continue your Networkers at

Cisco Live learning experience with further reading from Cisco Press

• Check the Recommended Reading flyer for suggested books

Available Onsite at the Cisco Company Store

Silence on the Wire

by Michael Zalewski

ISBN: 1593270461





The Tao of Network Security Monitoring

by Richard Beitlich

ISBN: 0321246772





The TCP/IP Guideby Charles M. KozierokISBN: 159327047X

0012 - total visibility 2017-09-05 - senki...2017/09/05 · • by eliminating spoofed traffic, we...

Documents