advanced asa and fwsm firewallsd2zmdbbm9feqrf.cloudfront.net/2011/eur/pdf/brksec-3020.pdf · cisco...

BRKSEC-3020

Advanced ASA and FWSM Firewalls

© 2010 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKSEC-3020 2

What we will NOT talk about

IOS Firewall

ASA versions prior to 8.3

VSG:

Take a look at BRKVIR-2011 instead

VPNs

The session for that is BRKSEC-3013


What We WILL Talk About

FWSM deployment considerations

ASA architecture and packet flow

Selected inspection engines

DNS inspection

HTTP inspection

Failover

Monitoring


FWSM deployment considerations


A granular policy is a lot of rules

Object-groups:

sources (10 addresses)

destinations (21 addresses)

ports (33 ports)

Result: 10x21x33 = 6930 rules

Nested object-groups:

Group App-servers used in 20 places

+1 server = many new rules

Keep an eye on the effect of the policy changes on FWSM:

Access Rules Download Complete: Memory Utilization: 90%


FWSM: ACL rule limits

A balance:

Larger ACLs, more space spent on backup tree

Less space for backup tree, smaller ACLs

Tree 0 : Active

100,567 ACEs

Backup Tree:

100,567

(mirror of active tree)

177612

combined

total ACEs

Tree 0 : active = 14,801 ACEs












Tree 12 : backup

Multi-ContextSingle Context


Access-list rules allocation knobs

FWSM 2.3 introduced

resource acl-partition - set the number of ACL partitions

allocate-acl-partition - assigns a context to a specific partition

FWSM 3.2 introduced

resource-rule - allows further customization of a partition

FWSM 4.0 introduced

resource partition - customize the size of individual partitions

access-list optimization enable - merges and/or deletes redundant and conflicting ACEs without affecting the policy

'hitcnt=*' in show access-list means ACE has been optimized out


FWSM – Hardware Limits

Full list in FWSM documentation, appendix A (specifications)

2.3 (Multimode) 3.1 (Multimode) 4.0 (Multimode)3.2 / 4.0

Configurable

ACEs 56,627 (9,704) 72,806 (11,200) 100,567 (14,801) X

AAA Rules 3,942 (606) 6,451 (992) 8,744 (1,345) X

Global Statements 1K (1K) 4K (4K) 4K (4K)

Static NAT Statements 2K (2K) 2K (2K) 2K (2K)

Policy NAT ACEs 3,942 (606) 1,843 (283) 2,498 (384) X

NAT Translations 256K (256K) 256K (256K) 256K (256K)

Connections 999,990 (999,990) 999,990 (999,990) 999,990 (999,990)

Route Table Entries 32K (32K) 32K (32K) 32K (32K)

Fixup/Inspect Rules 32 (32 per) 4147 (1,417) 5621 (1,537) X

Filter Statements 3942 (606) 2764 (425) 3747 (576) X

Increase over 2.3 Increase over 3.1


Classifier in Multimode

Inbound traffic is classified to context CTX3, based on the global IP in the static

VL

AN

3—

10

.14.3

.x

Inside

10.1.2.2

Inside

10.1.1.2

Inside

10.1.3.2

Inbound Packet

Outside

VLAN 4

VLAN 5

VLAN 6

CTX1

CTX2

CTX3

MSFC

.1

.2

.3

DST IP SRC IP

10.14.3.89 192.168.5.4

static (inside, outside) 10.14.3.89 10.1.3.2

Shared

Interface

Example


Classifier in Multiple Context Mode

FWSM has a single MAC address for all interfaces

Cisco ASA has single MAC for shared interfaces (physical interfaces have unique MACs)

Cisco ASA 7.2 introduced mac-address autooption to change this

Classify to determine the receiving context

Packets are classified based on:

Unique ingress interface/VLAN

Packet’s destination IP matches a global IP


Packet Capture: Limitations on FWSM

Capture functionality is available on the FWSM starting in 2.3

However, only packets processed by the control point could be captured

FWSM 3.1(1) added support to capture packets in hardware

Only ingress packets were captured

FWSM 3.1(5) both ingress and egress transient packets can be captured which flow through hardware

Capture requires an ACL to be applied

Capture copies the matched packets in hardware to the control point where they are captured; be careful not to flood the control point with too much traffic

Control Point (CP) Central CPU

C6K Backplane Interface

Session Manager

NP 3

Fast Path

NP 1

Fast Path

NP 1

FW

SM


Fast Path

Flow Identification,

Security Checks and

NAT in Hardware

FWSM Architectural Overview

C6K Backplane Interface

Session Manager

NP 3

Control Point (CP)

Central CPU

Fast Path

NP 1

Software

Hardware

FW

SM

Control Point

ACL Compilation,

Fixups, Syslog, AAA

in Software, IPv6

Session Manager

Session Establishment

and Teardown, AAA

Cache, ACLs

Fast Path

NP 2


NP oversubscription

Usually caused by bursts in a single TCP flow

Disable TCP offloading (TSO) may help

Sometimes caused by other factors

Large number of fragments

Large volume of traffic to a non-existent host

FWSM# show np blocks

MAX FREE THRESH_0 THRESH_1 THRESH_2

NP1 (ingress) 32768 32736 524 312368 2148096

(egress) 521206 521176 0 0 0

NP2 (ingress) 32768 32672 28099 932899 4564525

(egress) 521206 521144 0 0 0

NP3 (ingress) 32768 32768 91830 185963 1546629

(egress) 521206 521206 0 0 0


Architecture and Packet Flow


ASA Architectural Overview

Interface cards

Control Point (CP)

Software

AS

A

Control Point

ACL Compilation,

Fixups, Syslog, AAA

Accelerated Security

Path (ASP)

Session Establishment

and Teardown, AAA

Cache, ACLs, Flow

Identification,

Security Checks and

NAT

Data plane


NIC Driver

Crypto

Load Balancer

… FTP…………

Feature/Inspect

Modules

……

HTTP

ASA packet flow


ASA# show interface g0/1

Interface GigabitEthernet0/1 "DMZ2", is up, line protocol is up

Hardware is i82546GB rev03, BW 1000 Mbps, DLY 10 usec

Auto-Duplex(Full-duplex), Auto-Speed(1000 Mbps)

Input flow control is unsupported, output flow control is unsupported

MAC address 0024.97f0.4edb, MTU 1500

IP address 10.10.10.1, subnet mask 255.255.255.0

39645 packets input, 4980966 bytes, 0 no buffer

Received 192 broadcasts, 0 runts, 0 giants

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

37599 L2 decode drops

6011 packets output, 756890 bytes, 0 underruns

0 pause output, 0 resume output

0 output errors, 0 collisions, 1 interface resets

0 late collisions, 0 deferred

0 input reset drops, 0 output reset drops, 0 tx hangs

input queue (blocks free curr/low): hardware (255/252)

output queue (blocks free curr/low): hardware (255/251)

...

ASA NIC driver


ASA# show cpu detailed

Break down of per-core data path versus control point cpu usage:

Core 5 sec 1 min 5 min

Core 0 16.8 (16.7 + 0.2) 15.2 (14.9 + 0.3) 7.5 (7.4 + 0.1)

Core 1 15.7 (15.6 + 0.1) 14.4 (14.1 + 0.3) 7.1 (7.0 + 0.1)

Core 2 15.0 (14.9 + 0.1) 13.9 (13.6 + 0.2) 6.9 (6.8 + 0.1)

Core 3 14.2 (14.1 + 0.1) 12.7 (12.5 + 0.2) 6.3 (6.2 + 0.1)

Core 4 16.1 (15.9 + 0.1) 14.3 (14.0 + 0.3) 7.0 (6.9 + 0.1)

Core 5 14.7 (14.6 + 0.2) 13.5 (13.2 + 0.3) 6.5 (6.4 + 0.1)

Core 6 16.4 (16.3 + 0.1) 14.6 (14.4 + 0.2) 7.3 (7.2 + 0.1)

Core 7 15.1 (15.0 + 0.1) 13.3 (13.1 + 0.2) 6.6 (6.5 + 0.1)

Current control point elapsed versus the maximum control point elapsed for:

5 seconds = 77.8%; 1 minute: 77.8%; 5 minutes: 77.7%

ASA#

Multi-core models only (ASA-558x)

Balancing multiple RX queues on 10Gbps

Arbitration of data plane processing by cores

ASA packet load-balancer


CPU Scheduling (ASA5580-40)

Core 0Data PathThread








ControlPointThread

Network Interface 1

Network Interface 2

Network Interface N

…

Core 2CPThread

Core 6CPThread



Parse the headers of the packets

Virtual Reassembly

Determine target context if in multimode

ASA packet decode

ASA# ASA# show fragment

Interface: outside

Size: 200, Chain: 24, Timeout: 5, Reassembly: virtual

Queue: 0, Assembled: 207392, Fail: 2035, Overflow: 1937

Interface: dmz



Interface: inside



Interface: DMZ2



ASA#


Drops in ASP

Accelerated security path counts all drops

(NB: FWSM: applies only to control point traffic!)

Frame drops: per packet, flow drops: per flow

Drop counters are documented in the command reference, under show asp drop

ASA# show asp drop

Frame drop:

Invalid encapsulation (invalid-encap) 10897

Invalid tcp length (invalid-tcp-hdr-length) 9382

Invalid udp length (invalid-udp-length) 10

No valid adjacency (no-adjacency) 5594

No route to host (no-route) 1009

Reverse-path verify failed (rpf-violated) 15

Flow is denied by access rule (acl-drop) 25247101

First TCP packet not SYN (tcp-not-syn) 36888

Bad TCP flags (bad-tcp-flags) 67148

TCP option list invalid (tcp-bad-option-list) 731


ARP punts to control plane

DHCP punts to control plane

MAC access-list (transparent mode)

Configured captures

L2 action lookup

ASA# show capture cap

4 packets captured

1: 17:40:48.795613 802.1Q vlan#1527 P0 192.168.2.10.12345 >

192.0.4.126.80: S 0:492(492) win 8192

2: 17:40:48.795613 802.1Q vlan#1527 P0

3: 17:40:48.796818 802.1Q vlan#1527 P0 192.0.4.126.80 >

192.168.2.10.12345: S 3900802120:3900802120(0) ack 1000 win 3129 <mss 536>

4: 17:40:48.796955 802.1Q vlan#1527 P0 192.168.2.10.12345 >

192.0.4.126.80: R 1000:1000(0) win 0

4 packets shown

ASA#


Flows are user-visible as connections

Flow lookup

ASA# show conn long

5 in use, 19 most used

Flags: A - awaiting inside ACK to SYN, a - awaiting outside ACK to SYN,

B - initial SYN from outside, b - TCP state-bypass or nailed, C –

. . .

TCP outside:192.0.4.126/23 (1.1.1.1/23) inside:192.168.2.10/56642

(192.0.4.2/43688), flags UIO, idle 5s, uptime 5s, timeout 1h0m, bytes 1030

ASA#

Translated

address

Non-translated

address


Outbound Connection Inbound Connection

TCP Connection FlagsFor YourReference


Every passed packet is part of a flow

Main point of policy enforcement

Decide on further flow processing (inspect)

NAT in 8.3+ effectively happens before ACL

Flow creation


Single translation rule table, first match wins

Manual NAT

Automatic NAT

Manual “after” NAT

Flow creation: NAT in 8.3 and later

ASA(config)# nat ?

configure mode commands/options:

( Open parenthesis for

(<internal_if_name>,<external_if_name>)

pair where <internal_if_name> is the Internal or prenat

interface and <external_if_name> is the External or postnat

interface

<1-2147483647> Position of NAT rule within before auto section

after-auto Insert NAT rule after auto section

source Source NAT parameters

ASA(config)#


Automatic NAT: single rule per object

Useful for less complex scenarios

Lexicographic order of statements within “auto” section


ASA# show running-config object network

object network SERVER

host 192.168.2.10

nat (inside,outside) static 192.0.192.1 service tcp www 8080

ASA# show xlate


Flags: D - DNS, i - dynamic, r - portmap, s - static, I - identity, T - twice

TCP PAT from inside:192.168.2.10 80-80 to outside:192.0.192.1 8080-8080

flags sr idle 0:00:09 timeout 0:00:00

TCP PAT from inside:192.168.2.2/43392 to outside:192.0.4.2/10966 flags ri

idle 0:55:10 timeout 0:00:30

ASA#


Manual NAT (a case of policy NAT)


ASA# show run object


host 192.168.2.10

object network WEB

host 10.1.1.1

object service HTTP

service tcp eq www

ASA# show run nat

nat (inside,dmz) static SERVER WEB service HTTP HTTP

ASA#

ASA# show run nat

nat (dmz,inside) source static any any destination static WEB SERVER service

HTTP HTTP

ASA#


Multiple “virtual hosts” into a single real host

Useful for renumbering

Advanced NAT translation: N to 1

ASA# show run object


host 192.168.2.10

object network WEB-old

host 10.1.1.1

object network WEB-new

host 10.1.1.2

object service HTTP

service tcp source gt 1024 destination eq www

ASA# show run object-group

object-group network WEB

network-object object WEB-old

network-object object WEB-new

ASA# show run nat

nat (dmz,inside) source static any any destination static WEB SERVER service

HTTP HTTP

ASA#


Starting 8.3 several features expect “Real IP” in ACLs

Real address: with which the host can be reached from ASA

Flow creation: Real IP

Real IP Translated IP

ASA# show running-config access-list DMZ

access-list DMZ extended permit ip any host 192.168.2.10

ASA# show running-config access-group

access-group DMZ in interface dmz

ASA#

ASA# show running-config access-list DMZ

access-list DMZ extended permit object HTTP any object SERVER

ASA# show access-list DMZ

access-list DMZ; 1 elements; name hash: 0x55d29ba9

access-list DMZ line 1 extended permit object HTTP any object SERVER 0x77bfec73

access-list DMZ line 1 extended permit tcp any gt 1024 host 192.168.2.10 eq www

(hitcnt=6) 0x77bfec73

ASA#


Access-group

MPF

WCCP redirect ACL

Botnet traffic filter

AAA match access-list

Features that use Real IPFor YourReference


RPF checks

Check for symmetricity of the flows (NAT)

No ARP entry – packet dropped, counter incremented

Flow creation: Route/ARP lookup

ASA# show arp statistics

Number of ARP entries in ASA: 4

Dropped blocks in ARP: 15

Maximum Queued blocks: 2

Queued blocks: 0

Interface collision ARPs Received: 0

ARP-defense Gratuitous ARPS sent: 0

Total ARP retries: 28

Unresolved hosts: 0

Maximum Unresolved hosts: 2

ASA#


How do I process this flow after setup ?

Compiled information from ACL, MPF, etc.

Flow creation: feature classifier

ASA# show asp table classify domain ?

exec mode commands/options:

aaa-acct

aaa-auth

aaa-user

accounting

app-redirect

arp

autorp

. . .


Verify that data path punted packets to inspect

Example: classifier for DNS inspect

ASA# show asp table classify interface inside domain inspect-dns hits

Input Table

in id=0xb3da0d08, priority=70, domain=inspect-dns-np, deny=false

hits=1, user_data=0xb3d9ffe0, cs_id=0x0, use_real_addr, flags=0x0,

protocol=17

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=0.0.0.0, mask=0.0.0.0, port=53, dscp=0x0

input_ifc=inside, output_ifc=any

Output Table:

L2 - Output Table:

Last clearing of hits counters: Never

ASA#


Selected Inspection Engines


What do the inspects do ?

ingress egress

control plane

data plane

Flow inspected ?

yes

Unwrap L7 payload

Nat embedded IPs

Open needed pinholes

Enforce L7 policies

Wrap it back & send

The flows are classified for inspection based on the configured service-policy.

The inspected TCP flows are usually subject to TCP normalizer before the

inspection.


Inspects at a glance

show service-policy: inspection policies applied and the packets matching them

ASA# show service-policy

Global policy:

Service-policy: global_policy

Class-map: inspection_default

Inspect: dns maximum-length 512, packet 92, drop 0, reset-drop 0

Inspect: ftp, packet 43, drop 0, reset-drop 0

Inspect: h323 h225, packet 0, drop 0, reset-drop 0

Inspect: h323 ras, packet 0, drop 0, reset-drop 0

Inspect: http, packet 562, drop 0, reset-drop 0

Inspect: netbios, packet 0, drop 0, reset-drop 0

Inspect: rsh, packet 0, drop 0, reset-drop 0

Inspect: rtsp, packet 0, drop 0, reset-drop 0

Inspect: skinny, packet 349, drop 0, reset-drop 0

Inspect: esmtp, packet 0, drop 0, reset-drop 0

...

Interface outside:

Service-policy: VoIP

Class-map: voice_marked

Priority:

Interface outside: aggregate drop 0, aggregate transmit 349


Zoom-in: show service-policy flow

What policies will a given flow match within MPF(modular policy framework) ?

ASA# show service-policy flow tcp host 10.1.9.6 host 10.8.9.3 eq 1521

Global policy:


Interface outside:

Service-policy: outside

Class-map: oracle-dcd

Match: access-list oracle-traffic

Access rule: permit tcp host 10.1.9.6 host 10.8.9.3 eq sqlnet

Action:

Input flow: set connection timeout dcd


DNS inspection


DNS: The Protocol

www.cisco.com ?

Recursive

resolver

1

2

Root Server

Who is

authoritative

for “com” ?

3

“x.x.x.x”

4

“cisco.com”?

x.x.x.x : “.com”

5“y.y.y.y” y.y.y.y : “cisco.com”

8: “z.z.z.z”www.cisco.com


DNS: Cache Poisoning

www.cisco.com ?

Recursive

resolver

1

Root Server

2

Who is

authoritative

for “com” ?

a.a.a.a

malware

4: “a.a.a.a”

Easier

With

Trigger



CERT advisory: http://www.kb.cert.org/vuls/id/800113



Some implementations

keep source port

keep transaction ID or reuse small number of them

make multiple outstanding requests for the same query


DNS: but the advisory is 2 years old!

But there are more…

http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-0290


DNS: but we randomized the IDs and ports!

They do not help a whole lot…

Source: http://nbo.icann.org/meetings/nairobi2010/presentation-dnssec-dns-attacks-in-cz-08mar10-en.pdf


How does DNS inspect help ?

Allow only one response per request

dns-guard, enabled by default

Transaction ID randomization

7.2.1+, via MPF

DNS Header Flag Filtering

Clear the RD (recursion desired) flag on open resolvers

7.2.1+, need configuration via MPF

DNS message size limitations

By default, limited to 512 bytes.

This needs tuning – more on this later.

Logging

http://www.cisco.com/web/about/security/intelligence/dns-bcp.html


Showtime!


DNS inspect tuning

%ASA(config-subif)# show run all policy-map | beg ayourtch

policy-map type inspect dns ayourtch

parameters

no message-length maximum client

message-length maximum 512

no message-length maximum server

dns-guard

protocol-enforcement

nat-rewrite

no id-randomization

no id-mismatch

no tsig enforced


DNS inspect: solution of poisoning ?

No – but it helps to significantly lower the risk

The real long-term solution is DNSSEC

DNSSEC Intro RFC (RFC 4033)

DNSSEC Records RFC (RFC 4034)

DNSSEC Protocol RFC (RFC 4035)

Extremely short summary:

cryptographically signed DNS

Corollary of DNSSEC usage:

The DNS packets become bigger than 512 bytes!


DNS inspect default + DNSSEC

DNSSEC uses EDNS0 extension

Packets longer than the default 512 bytes limit

%ASA-4-410001: Dropped UDP DNS reply from outside:x.x.x.x/53 to

inside:y.y.y.y/12345; packet length 2876 bytes exceeds configured limit of 512

policy-map type inspect dns preset_dns_map

parameters

message-length maximum client auto


Adjust the configuration of the DNS inspect policy map


Advanced DNS Inspect Configuration

Log all DNS packets that contain “example.com” in the Question

regex EXAMPLE-DOT-COM "example.com―

class-map type inspect dns match-all EXAMPLE

match domain-name regex EXAMPLE-DOT-COM

match question

policy-map type inspect dns ayourtch

parameters


class EXAMPLE

log

Warning: adds the CPU processing

%ASA-6-410004: DNS Classification: Received DNS request (id 19380)

from inside:192.168.2.10/49671 to outside:144.254.10.123/53;

matched Class 23: EXAMPLE

%ASA-6-410004: DNS Classification: Received DNS reply (id 19380)

from outside:144.254.10.123/53 to inside:192.168.2.10/49671;

matched Class 23: EXAMPLE


HTTP inspection


HTTP Inspect Without the HTTP Map

Logs URLs into syslogs

Helps AAA authen/author (if configured)

Helps URL filtering (if configured)

Basic protocol sanity checks

%ASA-6-302013: Built outbound TCP connection 764 for dmz:192.168.1.2/8080

(192.168.1.2/8080) to inside:192.168.2.10/60886 (192.168.2.10/60886)

%ASA-5-304001: 192.168.2.10 Accessed URL 192.168.1.2:http://192.168.1.2:8080/

%ASA-6-302014: Teardown TCP connection 764 for dmz:192.168.1.2/8080 to

inside:192.168.2.10/60886 duration 0:00:00 bytes 3778 TCP FINs


HTTP Inspect With the HTTP Map

Parse the HTTP headers fully

Monitor violations

Enforce L7-based actions

More resource-intensive


HTTP: The Protocol

GET /test.html HTTP/1.0

User-Agent: Wget/1.11.4

Accept: */*

Host: 192.168.1.2:8080

Connection: Keep-Alive

HTTP/1.0 200 OK

Connection: close

Vary: Accept-Encoding

Content-Type: text/html

Accept-Ranges: bytes

ETag: "-1443736970"

Last-Modified: Sun, 15 Nov 2009 03:14:06 GMT

Content-Length: 3410

Date: Mon, 20 Dec 2010 01:27:02 GMT

Server: lighttpd/1.4.19

. . . Content goes here . . .

Running “wget http://192.168.1.2:8080/test.html”


HTTP Advanced Inspect

regex WGET "Wget"

class-map type inspect http match-all WGET-CLASS

match request header user-agent regex WGET

!

policy-map type inspect http DROP-WGET

parameters

class WGET-CLASS

reset log

!

access-list HTTP-8080-ACL extended permit tcp any any eq 8080

class-map HTTP-8080

match access-list HTTP-8080-ACL

!

policy-map global_policy

class HTTP-8080

inspect http DROP-WGET

Let’s “prevent” wget on port 8080

Same logic as with DNS inspect

Inspect policy map with parameters

Applied in global policy as necessary


HTTP Advanced Inspect

%ASA-5-415008: HTTP - matched Class 25: WGET-CLASS in policy-map DROP-

WGET, header matched - Resetting connection from

inside:192.168.2.10/37953 to dmz: 192.168.1.2/8080

%ASA-5-304001: 192.168.2.10 Accessed URL

192.168.1.2:http://192.168.1.2:8080/testing123

%ASA-4-507003: tcp flow from inside:192.168.2.10/37953 to

dmz:192.168.1.2/8080 terminated by inspection engine, reason - reset

unconditionally.

%ASA-6-302014: Teardown TCP connection 778 for dmz:192.168.1.2/8080 to

inside:192.168.2.10/37953 duration 0:00:00 bytes 0 Flow closed by

inspection

Testing the configuration

# wget 192.168.1.2:8080/testing123

--2010-12-20 01:45:46-- http://192.168.1.2:8080/testing123

Connecting to 192.168.1.2:8080... connected.

HTTP request sent, awaiting response... Read error (Connection reset by

peer) in headers.

Retrying.


Inspect tuning dependency graph

global/intf

service

policy

class-map

class

policy map

type

inspectinspect …

inspect

knobsparameters

class-map

type

inspect

class

regexL7-specific

criteria

match …


TCP Normalizer

Reassemble the first packet in the flow

Verify the correctness of the TCP fields

Put packets in order if needed (queueing)

ASA# sh asp drop frame tcp?

exec mode commands/options:

tcp-3whs-failed tcp-ack-syn-diff tcp-acked

tcp-bad-option-list tcp-buffer-full tcp-buffer-timeout

tcp-conn-limit tcp-data-past-fin tcp-discarded-ooo

tcp-dual-open tcp-dup-in-queue tcp-fo-drop

tcp-global-buffer-full tcp-invalid-ack tcp-mss-exceeded

tcp-not-syn tcp-paws-fail tcp-reserved-set

tcp-rst-syn-in-win tcp-rstfin-ooo tcp-seq-past-win

tcp-seq-syn-diff tcp-syn-data tcp-syn-ooo

tcp-synack-data tcp-synack-ooo tcp_xmit_partial

tcpnorm-rexmit-bad tcpnorm-win-variation

ASA(config-pmap-p)#


Case Study: Out-of-Order Packets

Lots of out-of-order packets seen

Default out-of-order buffer too small to hold

Poor TCP throughput due to lot of retransmits

Problem

OutsideInside

Client Server

Packet 10

10.16.9.2192.168.1.30

Packet 12

Packet 13

Packet 14

Packet 15

Buffer

Dropped by Firewall

Packet 11Dropped on Network



Inspections require ordered packets

Packets sent to the SSM require ordered packets

Cisco ASA will buffer up to three packets by default

Buffering can be increased on ASA by using the queue-limit option under the tcp-map



How to detectASA# show asp drop

Frame drop:

...

TCP packet buffer full 90943

...

How to fix

access-list OOB-nets permit tcp any 10.16.9.0 255.255.255.0

!

tcp-map OOO-Buffer

queue-limit 6

!

class-map tcp-options

match access-list OOB-nets

!

policy-map global_policy

class tcp-options

set connection advanced-options OOO-Buffer

!

service-policy global_policy global



How to verify ?

ASA# show service-policy

Global policy:


Class-map: inspection_default

...

Class-map: tcp-options

Set connection policy:

Set connection advanced-options: OOB-Buffer

Retransmission drops: 0 TCP checksum drops : 0

Exceeded MSS drops : 0 SYN with data drops: 0

Out-of-order packets: 2340 No buffer drops : 0


Failover


Failover

Active/Standby vs. Primary/Secondary

Stateful failover

Both firewalls swap MAC and IP addresses when a failover occurs

Failover stateful link and failover interface never swap MAC addresses

Secondary

Standby

Primary

Active

Failover LAN

Stateful link

Internet

Corp


Verifying Failover ConfigurationASA# show failover

Failover On

Failover unit Primary

Failover LAN Interface: failover Redundant5 (up)

Unit Poll frequency 200 milliseconds, holdtime 1 seconds

Interface Poll frequency 500 milliseconds, holdtime 5 seconds

Interface Policy 1

Monitored Interfaces 2 of 250 maximum

Version: Ours 8.1(2), Mate 8.1(2)24

Last Failover at: 13:05:44 UTC May 29 2009

This host: Primary - Active

Active time: 1366024 (sec)

slot 0: ASA5580 hw/sw rev (1.0/8.1(2)) status (Up Sys)

Interface outside (10.8.20.241): Normal

Interface inside (10.89.8.29): Normal

Other host: Secondary - Standby Ready


slot 0: ASA5580 hw/sw rev (1.0/8.1(2)24) status (Up Sys)

Interface outside (10.8.20.242): Normal

Interface inside (10.89.8.30): Normal

Stateful Failover Logical Update Statistics

Link : stateful Redundant6 (up)

Stateful Obj xmit xerr rcv rerr

General 424525 0 424688 0

sys cmd 423182 0 423182 0

Interface

monitoring


What triggers a failover ?

Power loss/reload (this includes crashes) on the active firewall

SSM interface/module failure

The standby becoming healthier than the active firewall


What triggers a failover ?

Two consecutive hello messages missed on any monitored interface forces the interface into testing mode

Both units first verify the link status on the interface

Next, both units execute the following tests

Network activity test

ARP test

Broadcast ping test

The first test passed causes the interface on that unit to be marked healthy

only if all tests fail will the interface be marked failed


How Well do You Understand Failover ?

The LAN interface communication is severed?

You disable failover? (By issuing no failover)

You RMA/Replace the Primary unit?

You don’t define Standby IPs

A member-interface in a Redundant interface fails?

What happens when…


What to Do After a Failover

Always check the syslogs to determine root cause

ASA-4-411002: Line protocol on Interface inside, changed state to down

ASA-1-105007: (Primary) Link status ‘Down’ on interface 1

ASA-1-104002: (Primary) Switching to STNDBY—interface check, mate is healthier

Example: port failed on inside interface of active ASA

Syslogs from Secondary (Standby) unit:

ASA-1-104001: (Secondary) Switching to ACTIVE—mate want me Active

Syslogs from Primary (Active) unit:


What to Do After Failover

show failover state – will provide specific details about the failure reason.

This information is not saved across reboots

ASA# show failover state

State Last Failure Reason Date/Time

This host - Primary

Failed Ifc Failure 12:56:00 UTC May 6 2009

Inside: Failed

Other host - Secondary

Active None

====Configuration State===

Sync Done

====Communication State===

Mac set

For YourReference


What to Do After Failover

Starting with FWSM 2.3 and Cisco ASA 7.0, the reason for failover is recorded in the failover history



ASA# show failover history

==========================================================================

From State To State Reason

==========================================================================

Disabled Negotiation Set by the CI config cmd

Negotiation Just Active No Active unit found

Just Active Active Drain No Active unit found

Active Drain Active Applying Config No Active unit found

Active Applying Config Active Config Applied No Active unit found

Active Config Applied Active No Active unit found

Active Failed Interface check

==========================================================================

For YourReference


Other Useful Failover Commands

failover exec mate – allows you to execute commands on the peer and receive the response back.

failover reload-standby – only valid on Active unit

prompt – changes the prompt to display failover priority and state.

ASA(config)# prompt hostname priority state

ASA/sec/act(config)#

For YourReference


Failover Prompt Configuration

The firewall’s prompt may be changed to display certain keyword

Usage

prompt <keyword> [<keyword> ...]

Keywords:

hostname Configures the prompt to display the hostname

domain Configures the prompt to display the domain

context Configures the prompt to display the current context (multi-mode only)

priority Configures the prompt to display the failover LAN unit setting

state Configures the prompt to display the current traffic handling state (Active/Standby)

slot Configures the prompt to display the slot location (when applicable)

For YourReference


Active-Active Failover

Secondary

Failover LAN

Stateful link

Internet

Corp

Primary

Group 2

Standby

Group 1 - ActiveGroup 1 - StandbyGroup 2 - Active


A real-world story about failover

Upon the reboot of Primary, it always becomes active (without preempt!). This failover takes a long time to happen and is disruptive

Secondary

Standby

Primary

Active

Failover LAN

Stateful link

Corp

CoreThe primary and seconary sites are geographically distributed

Connected by a 802.1q trunk, which gets the second 802.1q tag in the core due to Q-in-Q

Internet


SecondaryPrimaryMAC_1

data

MAC_1

failover

MAC_2

data

MAC_2

failover ActiveStandby

Zoom-in: let's see what happens with MAC addresses

MAC_1 : left

MAC_2 : right

802.1q #1

802.1q #2


SecondaryPrimaryMAC_2

data

MAC_1

failover

MAC_1

data

MAC_2

failover

Zoom-in: let's see what happens with MAC addresses

Problem: The core switch does not know about the ―inner‖ VLANs, so it sees the same MAC addresses from both sides => losses of connectivity, MAC moves on the core switch

Solution: send the failover VLANs over different long-haul link.

@#$%!?!


Failover: IPv6

Starting with ASA 8.2.2 and 8.3.1

ASA# show failover

Failover On

Failover unit Primary

Failover LAN Interface: fover GigabitEthernet0/2.100 (up)

Unit Poll frequency 200 milliseconds, holdtime 800 milliseconds

Interface Poll frequency 5 seconds, holdtime 25 seconds

Interface Policy 1

Monitored Interfaces 0 of 160 maximum

Version: Ours 8.3(1), Mate 8.3(1)

Last Failover at: 21:35:30 UTC Dec 3 2011

This host: Primary - Active



Interface outside (192.0.3.2/fe80::21f:caff:febb:f8f3):

Interface dmz (192.168.1.1/fe80::21f:caff:febb:f8f3):

Interface inside (192.168.2.1/fe80::21f:caff:febb:f8f3)

slot 1: empty

Other host: Secondary - Standby Ready



Interface outside (0.0.0.0/fe80::21d:70ff:fe8e:d633):

Interface dmz (0.0.0.0/fe80::21d:70ff:fe8e:d633):

Interface inside (0.0.0.0/fe80::21d:70ff:fe8e:d633):

slot 1: empty


Failover: IPv6

Best practice: always define standby IPv6 address

ASA# sh run int gig0/1.10

!

interface GigabitEthernet0/1.10

vlan 1505

nameif outside

security-level 0

ip address 192.0.3.2 255.255.255.128

ipv6 address 2001:db8:cafe:3000::2/64 standby 2001:db8:cafe:3000::3

ASA#


IPv6 support in ASA: summary

ACLs, static routing, neighbor discovery

Transparent mode (starting with 8.2)

Management: ASDM

Inspections: ICMP, FTP, HTTP, SIP, SMTP

VPN remote access is v6-over-v4 only

VPN site-to-site IPv6 support starting from 8.3


Monitoring


“The environment is everything that isn’t me.”

Albert Einstein


Show CPU usage

Under normal conditions the CPU should stay below 50% (baseline as per network); if the CPU reaches 100% the firewall will start dropping packets

FWSM CPU is used for limited traffic processing; during ACL compilation CPU is expected to be near 100% until ACL is compiled

The show cpu usage command displays the CPU over time as a running average

ASA# show cpu usage

CPU utilization for 5 seconds = 5%; 1 minute: 4%; 5 minutes: 4%


Show conn count

Baseline and monitor your connection counts

Beware the platform limits*

FWSM: 1mln (500K + 500K)

ASA558x: 2mln (ASA5585/SSP-20: 1mln)

(ASA5505: Inside hosts # according to license)

* For the complete list, please refer to

http://www.cisco.com/en/US/products/ps6120/prod_models_comparison.html

ASA# show conn count



Show traffic

The show traffic command displays the traffic received and transmitted out each interface of the firewall

* For the complete list, please refer to

http://www.cisco.com/en/US/products/ps6120/prod_models_comparison.html

ASA# show traffic

outside:

received (in 124.650 secs):

295468 packets 167218253 bytes

2370 pkts/sec 1341502 bytes/sec

transmitted (in 124.650 secs):

260901 packets 120467981 bytes

2093 pkts/sec 966449 bytes/sec

1 minute input rate 1830 pkts/sec, 973849 bytes/sec

1 minute output rate 1580 pkts/sec, 839743 bytes/sec

1 minute drop rate, 0 pkts/sec

5 minute input rate 1283 pkts/sec, 693528 bytes/sec

5 minute output rate 1102 pkts/sec, 581942 bytes/sec

5 minute drop rate, 0 pkts/sec

. . .


SNMP OIDs

CPU usage

1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 (5 sec)

1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 (1 min)

1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 (5 min)

Connections

1.3.6.1.4.1.9.9.147.1.2.2.2.1.5.40.6 (Current total)

1.3.6.1.4.1.9.9.147.1.2.2.2.1.5.40.7 (Max total)

Traffic

1.3.6.1.2.1.2.2.1.{10|16}.n (in/out octets)

Use SNMPwalk to verify the interfaces!

For YourReference


ASA Syslog Level vs. Number of Messages

Level Description

Number of

messages

Ver. 6.3 Ver. 7.0 Ver. 7.2 Ver. 8.0 Ver. 8.1 Ver. 8.2

0 Emergencies 0 0 0 0 0 0

1 Alerts 41 (41) 62 (62) 77 (77) 78 (78) 87 (87) 87 (87)

2 Critical 21 (62) 29 (91) 35 (112) 49 (127) 50 (137) 56 (143)

3 Errors 74 (136) 274 (365) 334 (446) 361 (488) 363 (500) 384 (527)

4 Warnings 56 (192) 179 (544) 267 (713) 280 (768) 281 (781) 315 (842)

5 Notifications 21 (213) 161 (705) 206 (919) 216 (984) 218 (999) 237

(1079)

6 Informational 95 (308) 234 (939)

302 (1221)

335 (1319)

337 (1336)

368 (1447)

7 Debugging 15 (323) 217 (1156)

258 (1479)

266 (1585)

267 (1603)

269 (1716)


What Are Modifiable Syslog Levels?

Modifiable syslog levels

Allows one to move any syslog message to any level

Problem

You want to record what exec commands are being executed on the firewall; syslog ID 111009 records this information, but by default it is at level seven (debug)

%ASA-7-111009: User ‘johndoe’

executed cmd: show run

The problem is we don’t want to log all 1715 other syslogs that are generated at debug level

Levels

0—Emergency

1—Alert

2—Critical

3—Errors

4—Warnings

5—Notifications

6—Informational

7—Debugging

[no] logging message <syslog_id> level <level>


How To Modify Syslog Levels

Lower syslog message 111009 to level 3 (error)

Solution

ASA(config)# logging message 111009 level 3

%ASA-3-111009: User ‘johndoe’ executed cmd: show run

ASA(config)# no logging message 111009 level 3

Now our syslog looks as follows

To restore the default syslog level


Show local-host A local-host entry is created for any IP

Groups the xlates, connections, and AAA information

Very useful for seeing the connections terminating on servers

ASA# show local-host 10.1.1.9

Interface inside: 1131 active, 2042 maximum active, 0 denied

local host: <10.1.1.9>,

TCP connection count/limit = 1/unlimited

TCP embryonic count = 0

TCP intercept watermark = 50

UDP connection count/limit = 0/unlimited

AAA:

user 'cisco' at 10.1.1.9, authenticated (idle for 00:00:10)

absolute timeout: 0:05:00

inactivity timeout: 0:00:00

Xlate(s):

Global 172.18.124.69 Local 10.1.1.9

Conn(s):

TCP out 198.133.219.25:80 in 10.1.1.9:11055 idle 0:00:10 Bytes 127 flags UIO


Show local-host - also IPv6

ASA# show local-host 2002:c01d:cafe:3002:218:51ff:fec9:5352

Interface folink: 1 active, 1 maximum active, 0 denied

Interface inside: 2 active, 5 maximum active, 0 denied

local host: <2002:c01d:cafe:3002:218:51ff:fec9:5352>,

TCP flow count/limit = 1/unlimited

TCP embryonic count to host = 0

TCP intercept watermark = unlimited

UDP flow count/limit = 0/unlimited

Conn:

TCP outside 2002:c01d:cafe:3000::1:23 inside

2002:c01d:cafe:3002:218:51ff:fec9:5352:53975, idle 0:01:18, bytes 1090,

flags UIO

Interface dmz: 0 active, 5 maximum active, 0 denied

Interface outside: 2 active, 9 maximum active, 0 denied

Interface fover: 1 active, 1 maximum active, 0 denied

ASA#


Capturing Packets Dropped by the ASP

Capture all packets dropped by the ASPASA# capture drops type asp-drop all

ASA# capture drops type asp-drop invalid-tcp-hdr-length

ASA# capture drop type asp-drop ?

acl-drop Flow is denied by configured rule

all All packet drop reasons

bad-crypto Bad crypto return in packet

bad-ipsec-natt Bad IPSEC NATT packet

bad-ipsec-prot IPSEC not AH or ESP

bad-ipsec-udp Bad IPSEC UDP packet

bad-tcp-cksum Bad TCP checksum

bad-tcp-flags Bad TCP flags

Capture on a specific drop reason


Show Processes CPU-usage

The show processes cpu-usage non-zero command displays the amount of CPU per-process, for those processes with nonzero usage

ASA# show processes cpu-usage non-zero

PC Thread 5Sec 1Min 5Min Process

081aa124 d51ab230 0.2% 2.0% 2.0% Dispatch Unit

08b99aec d5195d98 3.9% 0.5% 0.1% ssh

...

*Command first Introduced in Cisco ASA Version 7.2(4.11), 8.0(4.5), 8.1(1.100), 8.2(1).


Show processes cpu-hog

List of processes, and the function stack (Traceback) which executed and has lead to running on the CPU longer than the minimum platform threshold

ASA# show processes cpu-hog

Process: ssh_init, NUMHOG: 18, MAXHOG: 15, LASTHOG: 10

LASTHOG At: 14:18:47 EDT May 29 2009

PC: 8b9ac8c (suspend)

Traceback: 8b9ac8c 8ba77ed 8ba573e 8ba58e8 8ba6971

8ba02b4 8062413

CPU hog threshold (msec): 10.240

Last cleared: None

Corresponding syslog when the entry is created:

May 29 2009 14:18:47: %ASA-7-711002: Task ran for 10 msec,

Process = ssh_init, PC = 8b9ac8c, Traceback = 0x08B9AC8C

0x08BA77ED

0x08BA573E 0x08BA58E8 0x08BA6971 0x08BA02B4 0x08062413


Thoughts to take home

Are you using the power of inspects on the ASA?

Do you collect enough data to know the baseline ?

IPv6 over WLAN for 4

years thanks to

Google whitelisted

www.cisco-live6.com or

Ciscolive-ipv6.com for statistics


BRKSEC-3020 Recommended Reading


We value your feedback - don't forget to complete your online session evaluations after each session. Complete 4 session evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Networkers 20th Anniversary t-shirt.

All surveys can be found on our onsite portal and mobile website: www.ciscoliveeurope.com/connect/mobi/login.ww

You can also access our mobile site and complete your evaluation from your mobile phone:

1. Scan the Access Code(See http://tinyurl.com/qrmelist for software,

alternatively type in the access URL)

2. Login

3. Complete and Submit the evaluation

Please complete your Session Survey

http://www.ciscoliveeurope.com/connect/mobi/login.ww

http://tinyurl.com/qrmelist

advanced asa and fwsm firewallsd2zmdbbm9feqrf.cloudfront.net/2011/eur/pdf/brksec-3020.pdf · cisco...

Documents