brkapp-3003

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco PublicBRKAPP-3003 1

Advanced Troubleshooting the Cisco Application Control EngineBRKAPP-3003


Core Message

Understanding the architecture and flow management will help troubleshoot the Application Control Engine


Session Objective

ACE ArchitectureUnderstand the ACE architecture and connectivity through ACEVerify software images, licenses and image recoveryUse the real-time “TCP-DUMP” commandUnderstand access list and ACL merge on ACE

Flow Management Understand the difference between “L4” and “L7” processingCheck for possible asymmetric flowsProvide layer 7 troubleshootingAbility to monitor performance and troubleshoot resourcesUnderstand high availability

At the End of the Session, You Will Be Able To:


Session Agenda ACE Architecture

Discuss the ArchitectureFunctions of control plane and data planeCommon debugging commandsPacket Capturing and loggingTraffic Forwarding on ACEAdmin Context and ACL Merge

Flow ManagementConnection Handling on ACELayer 4/7 Troubleshooting and PerformanceHealth Monitoring on ACEHigh Availability on ACE


ACE Architecture


ACE20 Module Hardware Architecture

SwitchFabric

Interface

16G

DaughterCard 1

DaughterCard 2

8G

8G

SSLCrypto

10G

2G

Consoleport

SupConnect

100M

ControlPlane

NetworkProcessor 1

NetworkProcessor 2

10G10G

ClassificationDistribution

Engine(CDE)


ACE30 Module Hardware Architecture

SwitchFabric

Interface

16G

2G

Consoleport

SupConnect

100M

ControlPlane

8G

Daughter Card 1NP1 NP2

8G

Daughter Card 2NP1 NP2

ClassificationDistribution

Engine(CDE)


2x 700MHz MIPS1 GB Memory

Control Plane Software

SupervisorConnection

DBUS

16 GbpsBus

RBUS

EOBC

CiscoASIC

100 Mbps 8 Gbps

8 Gbps

1 Gbps

ACSW OS

60Gbps switching CapacityIPv4, IPv6 Classifications

TCP Checksum Generation/Verification

Variable Load Distribution

Daughter Card 1

16 Gbps

CEF720 Linecard

20 Gbps

20 GbpsSwitch Fabric

ACE30 Detailed Hardware Architecture

CPU

Classification DistributionEngine (CDE)

NetworkProcessor

1

Verni FPGA

DRAM 4 GB

DRAM 4 GB

NetworkProcessor

2shared memory

Daughter Card 2

NetworkProcessor

3

Verni FPGA

DRAM 4 GB

DRAM 4 GB Network

Processor4shared memory

Cavium Octeon CN5860 (OcteonPlus)16 core, 600 MHz CPUs with 4G DRAM 32k iCache, 16k dCache, 2MB L2 cache

On chip support for Encryption/Decryption Coprocessors for Compression/Decompression


Data Traffic vs. Management Traffic

ACE30 Control plane architecture is very similar to ACE20

Device controlConfiguration manager (CLI, XML API, SSH, …)Server health monitoring (native probes, TCL scripts)Syslog's, SNMP, …ARP, DHCP relayHigh-AvailabilityACL Compilation

ACE30 data plane architecture is very similar to ACE 4710

Connection managementTCP terminationAccess listsNATSSL OffloadRegular expression matchingLoad Balancing & forwarding


Common Debugging


Common Debugging

Show commands on the Catalyst 6500 Supervisor show version

show clock

show module

show power

show asic slot <n>

show interface TenGigabitEthernet <n>/1

show interface TenGigabitEthernet <n>/1 trunk

show svclc vlan-group

[no] power enable <module>

show svclc module <n> traffic

Make sure the module status is OK

VLAN‟s used by ACE must be configured in the MSFC


Common Debugging Show commands available on ACE

show version

show cde health

show ft group status

show ip int br

show int vlan <n>

show arp

show service-policy

show serverfarm

show rserver

show probe

show conn

show stat

show ip traffic

show resource usage

show np 1 me-stats “-s norm”

show np 1 me-stats “-s norm –M1”

System Information

L2, L3

Performance,ResourcesDebuggingFlows

L4, L7

This provides the DELTA

If incorrect version, check „boot‟ parameter


Show Module from the Catalyst 6500 Supervisor

cat6k#show mod

Mod Ports Card Type Model Serial No.

--- ----- -------------------------------------- ------------------ -----------

1 1 Application Control Engine 10G Module ACE20-MOD-K9 SAD12345678

2 48 48 port 10/100 mb RJ45 WS-X6348-RJ-45 SAD04450L44

5 2 Supervisor Engine 720 (Active) WS-SUP720-3BXL SAD08300D5L

Mod MAC addresses Hw Fw Sw Status

--- ---------------------------------- ------ ------------ ------------ -------

1 0001.0002.0003 to 0001.0002.000a 2.4 8.7(0.22)ACE A2(2.3a) Ok

2 00d0.d32e.1b42 to 00d0.d32e.1b71 1.5 5.4(2) 8.5(0.46)RFW Ok

5 000f.f7be.b17c to 000f.f7be.b17f 4.0 8.1(3) 12.2(PP_R31_ Ok

Mod Sub-Module Model Serial Hw Status

---- --------------------------- ------------------ ----------- ------- -------

5 Policy Feature Card 3 WS-F6K-PFC3BXL SAD083006N2 1.3 Ok

5 MSFC3 Daughterboard WS-SUP720 SAD082905VE 2.1 Ok

Mod Online Diag Status

---- -------------------

1 Pass

2 Pass

5 Pass

Module status shows OK


Verifying Version and Licenses

ACE/Admin# show version

Cisco Application Control Software (ACSW)

<snip>

Software

loader: Version 12.2[121]

system: Version A2(2.3a) [build 3.0(0)A2(2.3a)

system image file: [LCP] disk0:c6ace-t1k9-mz.A2_2_3a.bin

installed license: ACE-08G-LIC ACE-VIRT-250 ACE-SSL-15K-K9

Hardware

Cisco ACE (slot: 1)

cpu info:

number of cpu(s): 2

cpu type: SiByte

cpu: 0, model: SiByte SB1 V0.2, speed: 700 MHz

cpu: 1, model: SiByte SB1 V0.2, speed: 700 MHz

Installed Licenses


Available System Memory and Uptime

ACE/Admin# show version – Continuation of output

[...]

memory info:

total: 827128 kB, free: 335372 kB

shared: 0 kB, buffers: 3540 kB, cached 0 kB

cf info:

filesystem: /dev/cf

total: 1014624 kB, used: 529472 kB, available: 485152 kB

last boot reason: NP 2 Failed: NP ME Hung

configuration register: 0x1

ACE kernel uptime is 7 days 23 hours 42 minute(s) 25

second(s)

Displays ACE module uptimeUseful information in case of system reload


ACE File System

Use the dir command to view directory listing for filesACE/Admin# dir ?

core: Directory or filename

disk0: Directory or filename

image: Directory or filename

probe: Directory or filename

volatile: Directory or filename

The internal File system is mapped as below/mnt/cf - Image:

Also the following compressed file systems are used

/TN-HOME = disk0:

/TN-CONFIG = Startup config

/TN-LOGFILE = Internal Storage for audit logs

/TN-CERTKEY-STORAGE : internal storage for Cert and Keys

/TN-COREFILE = core:


ACE File System Load debug plug-in to access ACE file system

Startup configuration located at /mnt/cf/TN-CONFIG

ACE will generate / fix any missing or corrupted file systems during boot

When to use the format command?If you receive the following error

Warning!! This will erase everything in the compact flash including startup configs for all the contexts and reboot the system!!

ACE/Admin# write memory

ERROR!config filesystem is not mounted on compact flash


Working with Core Files

If ACE creates a core file you can locate the files in the core directory

All cores files are stored in dir core: (core names are self explanatory)

ACE/Admin# dir core:

99756 Apr 5 17:57:05 2007 ixp2_crash.txt

13047 Apr 5 17:56:59 2007 loadBalance_core_log.tar.g

Ixpx_crash.txt will have some details on the core dump

If it is a kernel crash , then a file named crash info will be available in core

Show version will show last reload reason


System Logging


Logging Features Each virtual context generates logs independently and sends to

specified destinationsSyslog server, console, buffer, SNMP station, etc..

Rate limiting of syslog messages is recommended. Never log to the console using level 7

ACE can log connection setup/teardown at the connection speed

Access-List deny entries are logged

Use the terminal monitor command to display log message when not using console

Useful commands to troubleshoot syslog issues:show logging statistics show logging historyshow logging queue

Make sure logging queue size is set properly


Basic Configuration to Enable Logging Enable logging on the ACE

logging enable

logging timestamp

logging monitor 4

logging trap 4

logging buffer 4

logging history 4

logging queue 1024

no logging message 111008

It is recommended to disable or change the severity level of some syslog messages. Use logging message syslog_id [level severity_level] command

To enable the logging of connection setup and teardown messages, use the logging fastpath command. Use the logging rate-limit to limit the rate at which the ACE generates messages to the syslog server


Real-Time “TCP Dump”


Real-Time “TCP Dump” Supportability and analysis of load balanced traffic is a

major requirement in today's load balanced environment ACE can capture real-time packet information for the

network traffic that passes through it The attributes of the packet capture are defined by

an ACL The ACE buffers the captured packets, and you can copy

the buffered contents to a file in flash memory on the ACE or to a remote server

User can also display the captured packet information on your console or terminal; capture can also be exported and viewed using Ethereal or Wireshark


Real-Time “TCP Dump” To enable the packet capture on ACE use the capture

commandcapture c1 interface vlan 211 access-list FILTER bufsize 64

Buffer in Kbytes(can be circular)

Pre-defined ACL toidentify relevant traffic

Interface to applycapture

One capture session per context Capture triggered at flow setup Capture configured on client interface where flow is

received


Real-Time “TCP Dump” ACE can capture traffic based on a configured access-list

and interface

Follow the following procedure to capture traffic on ACE:1. Specify an ACL2. Capture on an interface or globally

access-list FILTER line 10 extended permit tcp any any eq www

capture c1 interface vlan 211 access-list FILTER

Show capture status show status and buffer sizeACE/Admin# show capture c1 status

Capture session : c1

Buffer size : 64 K

Circular : no

Buffer usage : 1.00%

Status : stopped


Real-Time “TCP Dump” Start the capture on the ACEACE/Admin# capture c1 start

23:40:37.236868 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 58:

172.16.11.190.443 > 209.165.201.11.1180: S

1389739009:1389739009(0) ack 617249474 win 17408 <mss 1460>

(ttl 255, id 2401, len 44, bad cksum 0!)

23:40:37.239102 0:12:43:dc:93:bb 0:0:c:7:ac:a 0800 54:

172.16.11.190.443 > 209.165.201.11.1180: . ack 71 win 17408

(ttl 255, id 2402, len 40, bad cksum 0!)

ACE/Admin# capture c1 stop

To copy the packet capture to disk0: use the copy captureACE/Admin# copy capture c1 disk0: c1

Maximum buffer size is 5MB of data


Traffic Forwarding on ACE


ACE Load Balancer Policy Lookup Order

There can be many features applied on a given interface, so feature lookup ordering is important

The feature lookup order followed by data path in ACE is as follows:1. Access-control (permit or deny a

packet)2. Management traffic3. TCP normalization/connection

parameters4. Server load balancing5. Fix-ups/application inspection6. Source NAT 7. Destination NAT

The policy lookup order is implicit, irrespective of the order in which the user configures policies on the interface


ACE in Routed Mode

IP subnets cannot overlap within a context but can across two contexts

Non-Load balanced traffic is routed. ACE needs to ARP for destination before forwarding packet

Client MAC ACE MAC

Client IP VIP

Random Port VIP Port

ACE MAC SelectedServer MAC

Client IP Server IP

Random Port Server Port


ACE in Bridge Mode

Non-Load balanced connections are bridged from client to server vlan

Client MAC ACE MAC

Client IP VIP

Random Port VIP Port

Client MAC SelectedServer MAC

Client IP Server IP

Random Port Server Port


Checking VLAN Configuration Show interface provides you with valuable information ACE/Admin# show interface vlan 211

vlan210 is up

Hardware type is VLAN

MAC address is 00:16:36:fc:b3:36

Virtual MAC address is 00:0b:fc:fe:1b:02

Mode : routed

IP address is 172.16.10.21 netmask is 255.255.255.0

FT status is active

Description:WAN Side

MTU: 1500 bytes

Last cleared: never

Alias IP address is 172.16.10.23 netmask is 255.255.255.0

Peer IP address is 172.16.10.22 Peer IP netmask is 255.255.255.0

Assigned on the physical port, up on the physical port

499707 unicast packets input, 155702918 bytes

1485258 multicast, 5407 broadcast

0 input errors, 0 unknown, 0 ignored, 0 unicast RPF drops

497610 unicast packets output, 46804782 bytes

6 multicast, 8201 broadcast

0 output errors, 0 ignored


MAC Addresses

Virtual MAC (VMAC) is used for the alias IP, VIP address

Alias IP and Virtual IP (VIP) are associated with a VMAC only if high availability is configured

Active context responds to ARPs for alias IP with VMAC

One unique VMAC per FT Group 00:0b:fc:fe:1b:XX(XX=FT group number in hex)

Packets destined to the VMAC are blocked on standby context


MAC Addresses

The VMAC is a function of ft-group-id. Therefore different cards must have different ft-group-ids

Use the show interface internal iftable to locate the VMAC

Each ACE supports 1,024 shared VLAN‟s, and uses only one bank of MAC addresses randomly selected at boot time

ACE‟s may select the same address bank so avoid this conflict use the shared-vlan-hostid command


Key Things to Know About ARP on ACE

For unicast packets, if the destination MAC is unknown ACE will drop the packet, instead of flooding it

So IP-address-to-MAC mapping and outgoing interface needs to happen first

ARP entries are populated as follows:With ARP requestsLearning through incoming ARP requestsGratuitous ARP packets

Layer 2 mode:ARP is the only way to learn IP to MAC and interface mapping


Admin Context Resource Reservation



If Admin context is not configured correctly, Admin could be starved of all resources

When configuring resource allocations in ACE, it is possible to allocate 100% of resources to non-Admin contexts, so that the Admin context is no longer reachable via ICMP, telnet, SNMP, etc

In some cases, this could cause FT between a pair of HA ACE modules to fail, and create an active/active situation

Highly recommended to put some safeguard in place to ensure that the Admin context always receives at least a small percentage of resources


Admin Context Resource Reservation Shows starved resources and drops for throughputACE/Admin# show resource usage context Admin

Allocation

Resource Current Peak Min Max Denied

-------------------------------------------------------------------------------

Context: Admin

conc-connections 9 9 0 0 0

mgmt-connections 2 12 0 0 0

proxy-connections 0 0 0 0 0

xlates 0 0 0 0 0

bandwidth 0 4715 0 0 3704068

throughput 0 4247 0 0 3704068

mgmt-traffic rate 0 468 0 125000000 0

connection rate 0 7 0 0 8

ssl-connections rate 0 0 0 0 0

mac-miss rate 0 1 0 0 0

inspect-conn rate 0 0 0 0 0

acl-memory 26816 26880 0 0 0

sticky 0 0 0 0 0

regexp 0 0 0 0 0

syslog buffer 1024 4096 0 1024 0

syslog rate 0 7 0 0 118

No resources reserved



Shows heartbeats missed increasing. Heartbeats are not reaching the peer. Possibility for both ACE‟s to go Active/Active

ACE/Admin# sh ft stats

HA Heartbeat Statistics

------------------------

Number of Heartbeats Sent : 1095573

Number of Heartbeats Received : 1092586

Number of Heartbeats Missed : 2987

Number of Unidirectional HB's Received : 2640

Number of HB Timeout Mismatches : 0

Num of Peer Up Events Sent : 1

Num of Peer Down Events Sent : 1

Successive HB's miss Intervals counter : 0

Successive Uni HB's recv counter : 0



Below shows the problem why ACE is starved of all resources

resource-class admin

limit-resource all minimum 0.10 maximum equal-to-min

Suggest the following reserved resources for Adminresource-class Admin

limit-resource conc-connections min 5.00 max equal-to-min

limit-resource mgmt-connections min 5.00 max equal-to-min

limit-resource rate bandwidth min 5.00 max equal-to-min

limit-resource rate ssl-connections min 5.00 max equal-to-min

limit-resource rate mgmt-traffic min 5.00 max equal-to-min

limit-resource rate conc-connections min 5.00 max equal-to-min


Access-Control Lists and ACL Merge


ACL Merge Process and Enhancements New ACL merge enhancements added to ACE

ACL merge is responsible for merging all the features and generating a single merged list for an given interface. ACL compiler is responsible for programming the merged list into MTrie data structure – “Fast retrieval of data”

ACL memory usage has been optimized to better support incremental changes

The new implementation provides a consistent ACL memory usage during system bootup time and during incremental changes after the system comes up

This feature also provides an early detection of failure if the configuration needs more ACL resources than available

Also, note ACL masks are in 255.255.x.x format (not 0.0.y.y)


View Total Action Nodes

Use the show np 1 access-list resource to view action nodes

ACE/Admin# show np 1 access-list resource

ACL Tree Statistics for Context ID: 3

=======================================

ACL memory max-limit: None

ACL memory guarantee: 0.00 %

MTrie nodes(used/guaranteed/max-limit):

6 / 0 / 262143 (compressed)

2 / 0 / 21999 (uncompressed)

Leaf Head nodes (used/guaranteed/max-limit):

3 / 0 / 262143

Leaf Parameter nodes (used/guaranteed/max-limit):

7 / 0 / 524288

Policy action nodes used: 4

memory consumed: 4696 bytes resource-limited 128 bytes other

4824 bytes total.

min-guarantee: 0 bytes total.

max-limit: 78610432 bytes total, 0 % consumed


Connection Handling in ACE


Flow ManagementLevel of Flow Processing Type of Processing Feature of Function

Layer 3 and Layer 4 Balance on first packet Basic Load Balancing

Applies to TCP/UDP for layer 4 rules Source IP Sticky

Applies to all other IP protocols TCP/IP Normalization

Layer 7 TCP Splicing Terminate TCP Connection HTTP Layer 7 rules based on first request (URL LB)

Buffer request, inspect, LB Cookie Sticky (Persistence)

Create Hardware Shortcut Generic TCP Payload Parsing

Layer 7 Re-proxy TCP Splicing + ability to parse subsequent HTTP requests within the same TCP

HTTP Layer 7 rules with HTTP 1.1 connections keepalive(“persistence rebalance”)

Layer 7 Full-Proxy Fully terminate clients connection SSL Offload

TCP re-use

HTTP 1.1 Pipelining

Protocol Inspection (FTP,SIP)


Internal Mapping of TCP/UDP Flows

TCP and UDP Flows = 2 X Internal Half Flows

ACE/Admin# show conn

conn-id np dir proto vlan source destination stat-------------+--+----+--------+-----+--------------------------+-------------------------------+---------+

9 1 In TCP 211 209.165.201.11:1867 172.16.11.190:80 ESTAB

6 1 Out TCP 411 192.168.1.11:80 209.165.201.11:1867 ESTAB

Client IP:port VIP Address

Server IP Returning half flow automatically created for both TCP and UDP flows

INIT, SYNACK,ESTAB, CLOSED

SYN_SEEN, SYN_SEEN,ESTAB, CLOSED

Non TCP shows as “--”

Use conn-id to track flow through ACE

Check the Network Processor


Troubleshooting Connections Use the show stats connection command to show

connections statistics

Use the clear stats connection command to clear these counters

ACE/Admin# show stats connection

+------------------------------------------+

+------- Connection statistics ------------+

+------------------------------------------+

Total Connections Created : 288232

Total Connections Current : 2

Total Connections Destroyed: 283404

Total Connections Timed-out: 892

Total Connections Failed : 3934

Note: ACE does not destroy connection. These are connections closed correctly!!!


Troubleshooting Connections Use the show stats loadbalance command to view the load

balance statistics

To clear the load balance statistical information stored in the ACE buffer, use the clear stats loadbalance command ACE/Admin# show stats loadbalance

+------------------------------------------------------------+

+------- Loadbalance statistics ----------------------+

+------------------------------------------------------------+

Total version mismatch : 0

Total Layer4 decisions : 0

Total Layer4 rejections : 0

Total Layer7 decisions : 24

Total Layer7 rejections : 0

Total Layer4 LB policy misses : 0

Total Layer7 LB policy misses : 0

Total times rserver was unavailable : 0

Total ACL denied : 0


Troubleshooting Individual Connections Use the NP and connection ID from ‟show conn‟ command

to view the front-end and back-end connection statistics using show np <#> me-stats “-c <connection ID> -v”ACE/Admin# show np 1 me-stats “-c 4096 –v”

+------------------------------------------------------------+

+------- Individual connection statistics -------------------+

+------------------------------------------------------------+

Connection ID:seq: 4096[0x1000].2

Other ConnID : 8194[0x2002].14

Proxy ConnID : 0[0x0].0

Next Q : 0[0x0]

10.1.1.22:23 -> 12.2.2.14:8739 [RX-NextHop: TX] [TX-NextHop: CP]

Flags: PAT: No DynNAT: No Implicit PAT: No On_Reuse: No

L3 Protocol : IPv4 L4 Protocol : 6

Inbound Flag : 0

Interface Match : Yes

Interface MatchID:24

……… <snip>


Troubleshooting Individual Connections To further debug and check if the traffic pattern matches the

correct rule, the following command can be used: show np 1 access-list trace vlan <inbound vlan> in protocol <IP protocol #> source <source IP> <source port or „0‟> destination <destination IP> <destination port>

ACE/Admin# show np 1 access-list trace vlan 10 in protocol 6

source 10.10.10.1 0 destination 10.20.30.40 80

<snip> <look for NAT pool ID, vserver ID, etc.>

src nat 0x0 dst nat 0x0 vserver 0x66 fixup 0x0

<snip> <vserver ID here is 0x66 or 102 decimal>

Now, the internal vserver ID 102 can be looked up in the config: ACE/Admin# show cfgmgr internal table l3-rule | inc 102

102 224 249 0 0 DATA_VALID

Internal Policy Map # is 224 and Class Map # is 249: ACE/Admin# show cfgmgr internal table policy-map | inc 224

224 MyPolicy9 0 DATA_VALID

ACE/Admin# show cfgmgr internal table class-map | inc 249

249 MyClass4 0 DATA_VALID


Troubleshooting VIPACE/Admin# show service-policy client-vips detail

Status : ACTIVE

Description: -

-----------------------------------------

Interface: vlan 211

service-policy: client-vips

class: VIP-HTTPS

VIP Address: Protocol: Port:

172.16.11.190 tcp eq 443

loadbalance:

L7 loadbalance policy: HTTPS-POLICY

VIP Route Metric : 77

VIP Route Advertise : DISABLED

VIP ICMP Reply : ENABLED-WHEN-ACTIVE

VIP State: INSERVICE

curr conns : 22 , hit count : 22

dropped conns : 0

client pkt count : 0 , client byte count: 0

server pkt count : 0 , server byte count: 0

max-conn-limit : 0 , drop-count : 0

conn-rate-limit : 0 , drop-count : 0

bandwidth-rate-limit : 0 , drop-count : 0

L7 Loadbalance policy : HTTPS-POLICY

class/match : class-default

LB action :

primary serverfarm: backend-ssl

backup serverfarm : -

hit count : 22

dropped conns : 0


Troubleshooting Serverfarm

Best command for checking server status and load

ACE/Admin# show serverfarm HTTPS-FARM detail

serverfarm : HTTPS-FARM, type: HOST

total rservers : 4

active rservers: 4

description : -

state : ACTIVE

predictor : ROUNDROBIN

failaction : -

back-inservice : 0

partial-threshold : 0

num times failover : 0

num times back inservice : 0

total conn-dropcount : 0

----------connections-----------

real weight state current total failures

---+---------------------+--------+---------------------+-----------+------

rserver: linux-1

192.168.1.11:0 8 OPERATIONAL 0 0 0

max-conns : - , out-of-rotation count : -

min-conns : -

conn-rate-limit : - , out-of-rotation count : -

bandwidth-rate-limit : - , out-of-rotation count : -

retcode out-of-rotation count : -


Layer 7 Troubleshooting


Layer 7 Policy Hits

Expanding the show service-policy using the detail option to provide hit count for layer 7 matches

ACE/Admin# show service-policy client-vips detail

Status : ACTIVE

Description: -

-----------------------------------------

Interface: vlan 211

service-policy: client-vips

<snip>

L7 Loadbalance policy : pslb

class-map : curl1

LB action :

serverfarm: s1

hit count : 3

dropped conns : 0

class-map : curl2

LB action :

serverfarm: s2

hit count : 0

dropped conns : 0

Shows hit count for layer 7 load balanced policy


Match URL Hit Count

Expanding the show service-policy using the url-summaryoption to provide visibility on which match http url are getting hit

ACE/Admin# show service-policy url-summary

Service-Policy: VIRTUAL-HOSTING-01 L3-Class: WEB-SSL L7-Class: VH-01

match http url /ACCOUNTING/.* hit: 42

Service-Policy: VIRTUAL-HOSTING-02 L3-Class: WEB-SSL L7-Class: VH-02

match http url /BUSINESS/.* hit: 93

match http url /SALES/.* hit: 102

match http url /SPECIAL/.* hit: 67

match http url /BUSINESSOBJECTS/.* hit: 78

match http url /CUSTOMERS/.* hit: 84

Use the show service-policy <service-policy-name> class-map <L3-class map-name> url-summary to provide better granularity


Troubleshooting HTTP To effectively troubleshoot HTTP use the show stat http

commandACE/Admin# show stats http

+------------------------------------------+

+-------------- HTTP statistics -----------+

+------------------------------------------+

LB parse result msgs sent : 6288 , TCP data msgs sent : 9143

Inspect parse result msgs : 0 , SSL data msgs sent : 6041

TCP fin/rst msgs sent : 135 , Bounced fin/rst msgs sent: 19

SSL fin/rst msgs sent : 13 , Unproxy msgs sent : 0

Drain msgs sent : 3107 , Particles read : 37917

Reuse msgs sent : 1539 , HTTP requests : 3145

Reproxied requests : 0 , Headers removed : 1549

Headers inserted : 1598 , HTTP redirects : 2

HTTP chunks : 0 , Pipelined requests : 0

HTTP unproxy conns : 0 , Pipeline flushes : 0

Whitespace appends : 0 , Second pass parsing : 0

Response entries recycled : 3032 , Analysis errors : 0

Header insert errors : 1509 , Max parselen errors : 0

Static parse errors : 9 , Resource errors : 0

Invalid path errors : 0 , Bad HTTP version errors : 0


Troubleshooting HTTP Cookies ACE parses HTTP requests for cookies with the name

given in the configuration and can skip a certain number of bytes and look for another specific number of bytes.

If the cookie is not found, then the ACE looks for a string in the URL, starting with one of the characters /?&#+ and followed by a "=", then parses that value.

If no cookie or HTTP URL cookie exists ACE defaults to the predictor for that farm

ACE can parse HTTP headers (includes cookies) up to 64kB (default header max parse length is 2048k)

Make sure that sticky timeout (note this is more like an idle timeout) matches the session timeout on the application


Troubleshooting TCP Re-Use When using TCP connection re-use,"Connection: keep-

alive" is inserted and "Connection: close" is removed from the clients HTTP request, to avoid closing the server connection early

User needs to configure Source NAT in the policy map when using TCP connection re-use

Use the show stats http | include Reuse counters to check if see if TCP Re-use is in effectACE/Admin# show stats http | include Reuse

Reuse msgs sent : 1 , HTTP requests : 4

„sh conn detail‟ will also show information about server side connection reuse


Troubleshooting HTTP Compression


HTTP Compression Overview

ACE uses Cavium Octeon zip engineImplement deflate block as defined in RFC 1951Hardware determines fixed or dynamic Huffman encodingHistory buffer is supported to achieve better compression ratio

Support two output file formats. GZIP (RFC1952) or X-GZIP (RFC2616) and ZLIB (aka DEFLATE) RFC1950

Compression is used with HTTP connection only

Compression only supports HTTP 1.1 protocol

No decompression support

Feature Available on ACE 4710 and ACE30


HTTP Compression

Searching for “cisco”in www.google.com

Compressed data

http://www.google.com/


ACE Compression Traffic Flow Example

2. ACE rewritesClient‟s request

GET / HTTP/1.1

Accept-Encoding: gzip,

deflate

1. Request before ACE

GET / HTTP/1.1

Accept-Encoding: identity

Request after ACE

4. ACE Inspects response

HTTP/1.1 200 OK

Content-type: text/html

Content-Encoding: deflate

Transfer-Encoding:

chunked

6. Response after ACEServer sends uncompressed HTTP payload of 5963 bytes

7. Client receives compressed HTTP payload 2577 bytes

Cisco ACEClient LAN

HTTP/1.1 200 OK

Content-type: text/html

Content-Length: 5963

3. Response before ACE

5. ACECompresses

Response

Server

WAN


Default Compression Controls

Parameter-map type http compression

Minimum content size (512 bytes) to compresscompress minimum-size 100 - Compress if content length is 100 bytes or more

User-Agent Exclusion (Null)compress user-agent UnknownBrowser - Disallow compression for Unknown Browser

Compress only http text/* typecompress mimetype image/jpeg - Compress jpeg content


Debugging HTTP Compression Check the following if there no configuration errorFrom client side:

1.Accept-Encoding is not present or has invalid type

2.User-Agent is being excluded from the configuration

3.HTTP version is not 1.1 or higher

From server side

1.Invalid HTTP response header

2.HTTP response code not 200

3.Content type is not allowed

4.Content length is too small

5.Chunk encoding has invalid format

Get request from client:

GET HTTP/1.1

Host: www.yahoo.com

User-Agent: Mozilla/5.0 Windows; U; Windows NT 5.1;

Accept: text/html,application/xhtml+xml,

Accept-Language: en-us,en;q=0.5

Accept-Encoding: gzip,deflate

Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7

Keep-Alive: 300

Connection: keep-alive


Debugging HTTP Compression (Cont.) look at the stats from show np x me-stat “-s http”

Analysis errors: 0 0 General HTTP internal error

Static parse errors: 0 0 General HTTP parsing error

Compression reqs sent: 0 0

Compression rsps rcvd: 0 0

Compression bytes in : 0 0

Compression bytes out: 0 0

Compression rx data in rsp wait: 0 0

Compression no paticles: 0 0 Not enough internal buffer for

compressed output Compression no buffers fpa:0 0 Not enough internal buffer for

hardware

Compression no buffers sglist: 0 0 Not enough internal buffer for

hardware

Compression no buffers result zip: 0 0 Not enough internal buffer for

hardware

Compression session gone: 0 0 HTTP session is deleted

Compression session cleaned: 0 0

Compresssion rslt non-success: 0 0 Hardware compression error

Compression out alloc 0 0

Compression out dealloc 0 0

Compression chunk error 0 0 HTTP input chunk error

Compression error reset 0 0 HTTP compression session reset

Compression session alloc 0 0

Compression session free 0 0

Compression history set 0 0


Troubleshooting Secure Socket Layer (SSL)


Troubleshooting SSL Configuration of SSL on ACE is relatively simple. However

if you experience an issue, how to troubleshoot?

Make sure the certificate and key used in ssl-proxy are valid. Use the crypto verify commandACE/Admin# crypto verify RSA2048.key RSA2048.cert

Keypair in RSA2048.key matches certificate in RSA2048.cert

Check the size and location of the key. Use the show crypto key commandACE/Admin# show crypt key all

Filename Bit Size Type

-------- -------- ----

RSA2048.key 2048 RSA


Troubleshooting SSL Review the certificate details. Use the show crypto

certificate command ACE/Admin# show crypto certificate cisco-sample-cert

Certificate:

Data:

Version: 3 (0x2)

Serial Number:

ad:e4:e2:f1:50:b7:ce:bd

Signature Algorithm: sha1WithRSAEncryption

Issuer: C=IN, ST=KA, L=BLR, O=CISCO, OU=ADBU, CN=SSL-TEST

Validity

Not Before: Apr 3 09:50:55 2009 GMT

Not After : Apr 1 09:50:55 2019 GMT

Subject: C=IN, ST=KA, L=BLR, O=CISCO, OU=ADBU, CN=SSL-TEST

Subject Public Key Info:

Public Key Algorithm: rsaEncryption

RSA Public Key: (1024 bit)

Modulus (1024 bit):

00:cf:a2:60:66:5b:ce:b6:38:6f:94:df:0d:1c:61:

26:af:7a:05:49:ed:8d:93:3b

Exponent: 65537 (0x10001)


Troubleshooting SSL: CRL Download Check to make sure you can download the CRLACE/Admin(config-ssl-proxy)# do show crypto crl test2 detail

test2:

URL: http://119.60.60.23/test.crl

Last Downloaded (Cached): Sat Aug 8 16:14:24 2009 UTC

Total Number Of Download Attempts: 1

Failed Download Attempts: 0

Successful Loads: 1 Failed Loads: 0

Hours since Last Load: 0 No IP Addr Resolutions: 0

Host Timeouts: 0 Next Update Invalid: 0

Next Update Expired: 0 Bad Signature: 0

CRL Found-Failed to load: 0 File Not Found: 0

Memory Outage failures: 0 Cache Limit failures: 0

Conn failures: 0 Internal failures: 0

Not Eligible for download: 3 HTTP Read failures: 0

HTTP Write failures: 0

Looking for all best-effort CRLs in the system and their download status. Use the show crypto crl best-effort command

http://119.60.60.23/test.crl


Advanced SSL Debugging This command provides the current crypto statistics

ACE/Admin# sh np 1 me-stats "-s crypto”

Crypto Statistics: (Current)

------------------

ARC4 operations: 376572 0

TCP msgs received: 285260 0

APP msgs received: 235151 0

Nitrox messages forwarded to XScale: 381041 0

SSL ctx allocated: 47758 0

SSL ctx freed: 47758 0

SSL received bytes: 61070430 0

SSL transmitted bytes: 283256220 0

SSL received application bytes: 7679113 0

SSL transmitted application bytes: 275120867 0

SSL received non-application bytes: 53391317 0

SSL transmitted non-application bytes: 3292887 0

Bulk flush operations: 95037 0

ME records sent to XScale: 285808 0

ME records received from XScale: 47723 0

ME hw responses: 471516 0

First segments received: 47400 0

Handshake failure alert: 94 0

CM close: 446 0


Advanced SSL Debugging The show stats crypto server command provides

statistics of the SSL handshakeACE/Admin# show stats crypto server

+---- Crypto server termination statistics -----+

+------- Crypto server alert statistics --------+

+--- Crypto server authentication statistics ---+

+------- Crypto server cipher statistics -------+

+------ Crypto server redirect statistics ------+

+---- Crypto server header insert statistics ---+

These statistics provide details of the SSL packets for example; which version client interacted with ACE, which cipher is used, whether re-handshake happened, whether session id reuse happened and which SSL alerts are received or sent by ACE


Health Monitoring on ACE


Fundamentals for ACE Probing ACE probes are fundamental to the system. It is key to not

oversubscribe the ACE health monitoring system

Use the show resource internal socket to determine how many sockets ACE has open. This is an Admin command

ACE/Admin# show resource internal socket

Application MaxLimit Current Creates Frees

--------------------------------------------------------------

SYSTEM 4000 0 0 0

CRITICAL 50 0 0 0

AAA 256 0 0 0

MGMT 256 0 0 0

XINETD 512 1 12 11

HEALTH_MON 2500 532 193494 192962

USER_TCL 200 0 0 0

SYSLOG 256 10 14 4

VSH 256 0 0 0

OverAll - 650 194812 194162

Non Reg App Usage: 107


Health Monitoring Process If you see probing issues, check the health monitoring

process. The show proc cpu command provides very useful information

ACE/Admin# show proc cpu

CPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35%

PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process

972 1072965 613352 1749 35.9 18.5% 21.67% 20.90% arp_mgr

HM process is only consuming 1.40%. Why is the control plane CPU running at 30%? Check what process is consuming CPU

ACE/Admin# show proc cpu | inc hm

CPU utilization for five seconds: 30%; one minute: 37%; five minutes: 35%

PID Runtime(ms) Invoked uSecs 1Sec 5 Sec 1 Min 5 Min Process

987 90257 57805 1561 0.0 1.40% 1.46% 1.43% hm

988 90198 58952 1530 0.0 1.49% 1.49% 1.44% hm

989 851 2947 288 0.0 0.0 % 0.1 % 0.0 % hm

990 0 2 56 0.0 0.0 % 0.0 % 0.0 % hm


Health Monitoring on ACE Use the show probe detail command to determine the

status of the probe or possible last failureACE/Admin# show probe detail – Cut output

--------------------- probe results --------------------

probe association probed-address probes failed passed health

------------------- ---------------+----------+----------+----------+-------

rserver : CAS1

10.7.53.55 24 24 0 FAILED

Socket state : CLOSED

No. Passed states : 0 No. Failed states : 1

No. Probes skipped : 0 Last status code : 403

No. Out of Sockets : 0 No. Internal error: 0

Last disconnect err : Received invalid status code

Last probe time : Wed Nov 25 18:48:16 2009

Last fail time : Wed Nov 25 18:25:16 2009

Last active time : Never


High Availability on ACE


High Availability Basic Building Blocks

FT PEEROnly one FT peer per ACE device1:1 peer relationship

FT GROUPOne FT group per ACE virtual context

FT VLANDesignated VLAN between the redundant peersAll HA related traffic sent over this VLANFT VLAN can be trunked between two Catalyst 6500 ChassisShould not be used for normal traffic

Admin Context

Context A

Context B

Context A

Context B

ACE2 (FT PEER)

FT VLAN

FT Group 2

FT Group 3

ACE1 (FT PEER)

FT Group 1


High Availability Control Traffic

TCP Connection between FT PeersState Machine (Election, Preempt, Relinquish)Configuration syncState Sync for ARP

Heartbeats between FT peersHeartbeats are sent over UDPMonitors the health of the peerHeartbeat interval and count are configurable


ACE High Availability State Machine

Active/Standby Election (assuming both peers are initialized at same time)

Based on a priority scheme Member with highest priority becomes ACTIVEOther member enters the STANDBY_CONFIG state If priorities are equal, member with the higher IP address wins

STANDBY_CONFIG StateStartup Configuration Sync from Active to StandbyRunning Configuration Sync from Active to StandbyKnob to turn on/off


ACE High Availability State Machine STANDBY_BULK State

ARP Sync (knob to turn on/off)Connection Table SyncSticky Database Sync (knob to turn on/off)

STANDBY_HOT StateStandby FT group member is ready to take overIncremental Configuration Sync from Active to StandyIncremental State Sync from Active to Standby

STANDBY_COLD StateDue to error during Config Sync or Incremental Config Sync No Config or State Sync happens from Active to Standby‟

STANDBY_WARM StateMajor version mismatch between peers (example 2.x and 4.x)


ACE High Availability State Machine

Mismatch in software versionFT Peer may become INCOMPATIBLE ACTIVE ACTIVE state on both FT group members

Mismatch in Virtual Context LicensesConfiguration Sync (all types) for Admin context is disabledState Sync for Admin context will continue to happenFor matching user contexts – Configuration State Sync will work

Mismatch in Other LicensesConfiguration and State Sync will work After switchover, new Active will handle traffic as per its licenses


ACE Redundancy Query VLAN

When no heartbeat is received, ACE can use the Query Vlan to check the HA status

ACE tries to do a ping to the destination via the Query VLAN

If ping fails, the Standby will transition to the ACTIVE state

If ping succeeds, the Standby will transition to a STANDBY_COLD state

To configure a query interface, enter the following:ACE/Admin(config-ft-peer)# query-interface vlan 110


More Debugging Commands


Additional Debugging Some more ACE debugging commands

show np <#> me-stats -cpu

show np <#> me-stats –Q

show np <#> me-stats “-s fp”

show np <#> me-stats “-s tcp”

show np <#> me-stats “-s icm”

show np <#> me-stats “-s ocm”

show proc cpu

show netio stats

Show service-policy summary

brkapp-3003

Documents

cisco andor

cisco publicbrkapp

ace flow management

data traffic

data planecommon

advanced troubleshooting

performancehealth monitoring

core message