troubleshooting common network related issues with netscaler

Troubleshooting Common Network Related Issues with NetScaler

Michael Dean & Raghu Varma Tirumalaraju

Citrix Support Secrets Webinar Series

NetScaler Support Escalation

May 2014

© 2014 Citrix. Confidential.2

Agenda

Mac Based Forwarding

Trunking on Non-Link Aggregation Links

MAC Moves & Layer2 Loops & Common L2 Loop Scenario

Changing the NSVLAN

High Availability - Best Practices

GARP & VMACs

Link Aggregation with LACP


Mac Based Forwarding (MBF) Mode

Design considerations for MBF Mode (Why turn it ON or OFF)

Special-use feature

L3 routing is not needed (e.g. Stub topology)

Used when “split-brain” or asymmetric routing condition is present

Common asymmetric routing conditions:

(a) Three arm – Third for management

(b) Lots of static routes to be maintained


Common “Three Arm” Deployment where MBF is used

Netscaler

ISP

Admin Enclave Server Farm

VIP

SNIPNSIP

VPN/LB traffic

DataAdmin


Common issues seen with MBF

- Intermittent performance issues

- Packet loss in high volume environments

- Same packet loss can result in dropped ICA connections for ICA-Proxy through Netscaler Gateway vserver

- Periodic UI access issues

- Effectively disables L3 routing in favor of L2-type forwarding


Trunking on Non-Link Aggregation Links

Common issues seen when misconfigured:

- Dropped packets due to promiscuous vlan drops.

nsconmsg -K newnslog -s disptime=1 -g nic_ -d current | grep 1/2 | grep drop | more 18 0 6874714506 35162 5033 nic_err_dropped_pkts interface(1/2) Tue May 19 02:58:17 2014 25 0 6862206790 35106 5025 nic_err_vlan_promisc_tag_drops interface(1/2) Tue May 19 02:58:17 2014 73 0 6874743016 28510 4072 nic_err_dropped_pkts interface(1/2) Tue May 19 02:58:24 2014 80 0 6862235244 28454 4064 nic_err_vlan_promisc_tag_drops interface(1/2) Tue May 19 02:58:24 2014 127 0 6874773155 30139 4306 nic_err_dropped_pkts interface(1/2) Tue May 19 02:58:31 2014 134 0 6862265285 30041 4292 nic_err_vlan_promisc_tag_drops interface(1/2) Tue May 19 02:58:31 2014 185 0 6874837857 64702 9243 nic_err_dropped_pkts interface(1/2) Tue May 19 02:58:38 2014

The solution in this case is to ensure that the switchport settings only reflect the same allowed vlans that the netscaler is configured to send or receive traffic for


Trunking on Non-Link Aggregation Links cont.

Design considerations – vlan number assignments and tagging:

Native vs. Default VLAN Trunking vs Tag All

Tagging the native vlan- For differentiation- Tagging single vlan- Network policy might dictate a different native vlan.


Changing the NSVLAN

Changing the nsvlan is considered an extreme case and should be used as a last resort

Common Reasons for changing nsvlan: a) Network operations policy b) Separate requirement for native non-tagged vlan

Pitfalls of changing the nsvlan: a) Requires reboot b) NSIP VLAN is impacted – special considerations needed c) Special order of operations may be needed


MAC Moves & Layer2 Loops“The Rule of Bridging” – ARPs will be flooded out all connected interfaces (on the same vlan) except the one on which the request was received.

- L2 Mode considerations – Frames forwarded. “Very” rarely used! (see next slide)- Common issues seen resulting from MAC moves /L2 loops. - Poor performance - Application traffic throughput on the Netscaler ADC may not be as timely as expected - Frequent packet retransmissions causing latency - Dropped packets on interface(s) - Packet drops incrementing on the interface stats (show screenshot/text output)

-Common topologies that cause these issues - Multiple interfaces connected into the same vlan

-Design considerations to alleviate MAC Moves/L2 loops - Disable multiple interfaces that are connected to the same vlan - Implement Link-Aggregation (LA) channel to manage the traffic for multiple interfaces in same vlan


L2 or L3?


Common L2 Loop Scenario

Network Switch

Netscaler

Int 1/1

Int 1/2

fa 0/3

fa 0/4

1

2

3

1. Netscaler sends ARP for destination MAC for a given IP. Since both int 1/1 and 1/2 are connected, the request is flooded out of both interfaces

2. Switch/End device receives both requests and responds with destination MAC for given IP via both connected switchports

3. Netscaler receives both responses and caches the first response, then sends the request to the destination MAC for the given IP in that first response.

4 4. The Network switch, since it has the MAC address in it’s ARP table for the Netscaler SNIP as being accessible via fa0/4 due to a previous transaction and ARP table entry, it responds via this interface. The Netscaler drops the packet in the response since it is aware of the destination having been mapped via int 1/1 (Bridge Table)


Detecting L2 Loops - Overview

The scenario in the previous slide is very common when more than one interface in the same vlan is connected to the switch with both switch ports also configured on the same vlan.

The result from the received packet arriving on the assumed incorrect interface is a packet drop. Packet drops are detectable in a few ways through debugging the Netscaler logs.


Detecting L2 Loops – From Shell debuggingThe NSCONMSG utility is very powerful and frequently used by Citrix Support in gathering useful information to help determine root cause for system issues. In this example we see that we have used ‘grep’ to parse the current newnslog file for the strings “nic_err” and “drop” simultaneously. *Note that even in a healthy environment, packet drops occur all the time however when interesting counters such as ‘nic_err_vlan_promisc_tag_drop’ increment, ‘so do nic_err_dropped_pkts’

root@ns# nsconmsg -K newnslog -d stats | grep nic_err | grep drop 1854 0 91 nic_err_dropped_pkts interface(1/1) 1855 0 91 nic_err_dropped_pkts interface(1/2) 1866 0 0 nic_err_tx_dropped 1886 0 0 nic_err_congested_pkts_dropped 1887 0 0 nic_err_congestionlimit_pkts_dropped 1914 0 0 nic_err_la_frame_collect_drops 1915 0 0 nic_err_la_tagged_bpdu_drops 1916 0 73 nic_err_vlan_promisc_tag_drops 1917 0 0 nic_err_la_untagged_pkt_drops 1919 0 0 nic_err_rl_pkt_drops 1920 0 0 nic_err_rl_rate_pkt_drops 1921 0 0 nic_err_rl_pps_pkt_drops 4438 0 0 allnic_err_rl_rate_pkt_drops 4439 0 0 allnic_err_rl_pps_pkt_dropsroot@ns#


More L2 Issues – From the Shell debugging

The below counter tracks how often a given MAC is being detected via different ports. In normal operation a MAC address should only be seen only on one port. A MAC moving often is a sign of a loop – This can cause both failovers and performance issues.

nsconmsg -K newnslog -d statswt0 | grep nic_tot_bdg30433 0 25681657 nic_tot_bdg_mac_moved interface(1/3)30434 0 25725446 nic_tot_bdg_mac_moved interface(1/2)30435 0 25556085 nic_tot_bdg_mac_moved interface(1/1)30436 0 28858167 nic_tot_bdg_mac_moved interface(0/1)30437 0 27641933 nic_tot_bdg_mac_moved interface(0/2)

This then results in the NetScaler eventually muting the interfaces to try and end the loop.nsconmsg -K newnslog -d statswt0 | grep nic_err30438 0 467 nic_err_bdg_muted interface(1/3)30439 0 445 nic_err_bdg_muted interface(1/2)30440 0 447 nic_err_bdg_muted interface(1/1)30441 0 605 nic_err_bdg_muted interface(0/1)30442 0 578 nic_err_bdg_muted interface(0/2)

*Generally issues can be seen with the ‘-d statswt0 | grep nic_err’ switch in nsconmsg utility


High Availability

What events trigger (an unexpected) failover?Network issues = Missed Heartbeats = failover

Tight loops on peer/crash/reboot

SSL Card failures

Interface failures

Heartbeats are sent/received on all interfaces by defaultRegardless of hamon setting

Use UDP port 3003

Common network issues leading to missed heartbeats:VLAN Tagging Mismatch

nsVLAN mismatch – Native vlan for heartbeats and NSIP


What is exchanged in a Heartbeat?


Verifying Heartbeat exchange

root@ns# nsconmsg -K newnslog -g ha_tot_pkt -s disptime=1 -d current | more Index rtim totalcount-val delta rate/sec symbol-name&device-no&time 0 3560 13 13 3 ha_tot_pkt_rx Sat May 17 13:48:48 2014 1 0 12 12 3 ha_tot_pkt_tx Sat May 17 13:48:48 2014 2 7000 50 37 5 ha_tot_pkt_rx Sat May 17 13:48:55 2014 3 0 49 37 5 ha_tot_pkt_tx Sat May 17 13:48:55 2014 4 6998 101 35 5 ha_tot_pkt_rx Sat May 17 13:49:06 2014 5 0 100 35 5 ha_tot_pkt_tx Sat May 17 13:49:06 2014 6 7000 136 35 4 ha_tot_pkt_rx Sat May 17 13:49:13 2014 7 0 135 35 4 ha_tot_pkt_tx Sat May 17 13:49:13 2014 8 7001 171 35 4 ha_tot_pkt_rx Sat May 17 13:49:20 2014 9 0 170 35 4 ha_tot_pkt_tx Sat May 17 13:49:20 2014 10 7000 206 35 5 ha_tot_pkt_rx Sat May 17 13:49:27 2014


Tagging of HA packets - Scenarios

Scenario 1NSVLAN is 1 (default)interface 1/2 is bound to VLAN 2Interface 1/3 is bound to VLAN 3Result: High Availability packets flow as untagged on the 1/2 and 1/3 interfaces on the native VLAN.

Scenario 2NSVLAN is 1 (default)interface 1/2 is bound to VLAN 2, which is configured with –tagall ONInterface 1/3 is bound to VLAN 3, which is configured with –tagall OFF (the default)Results: HA packets flow on 1/2 as tagged with a VLAN ID of 2, and untagged on the 1/3 interface.

Scenario 3NSVLAN is 3 (non default)interface 1/2 is bound to VLAN 2interface 1/3 is bound to VLAN 3Result: HA packets flow as tagged packets on interface 1/3 only and don’t flow on Interface 1/2


HA related events

Secondary Primaryroot@70# nsconmsg -K newnslog -d event | grep node 0031 0 PPE-0 remote node 10.72.137.50: Primary Tue May 20 21:48:12 2014 1032 0 PPE-0 self node 10.72.137.70: SYNC start Tue May 20 21:48:15 2014 1203 0 PPE-0 self node 10.72.137.70: SYNC complete Tue May 20 21:48:20 2014 1210 0 PPE-0 remote node 10.72.137.50: DOWN Tue May 20 21:51:51 2014 1211 0 PPE-0 self node 10.72.137.70: Claiming Tue May 20 21:51:51 2014 1212 0 PPE-0 self node 10.72.137.70: Primary Tue May 20 21:51:51 2014 1221 0 PPE-0 remote node 10.72.137.50: INIT Tue May 20 21:53:01 2014 1223 0 PPE-0 remote node 10.72.137.50: UP Tue May 20 21:53:02 2014

root@50# nsconmsg -K newnslog -d event | grep node 946 0 PPE-0 remote node 10.72.137.70: UP Tue May 20 21:53:02 2014 1017 0 PPE-0 self node 10.72.137.50: UP Tue May 20 21:53:03 2014 1018 0 PPE-0 self node 10.72.137.50: HA license check result: MATCH Tue May 20 21:53:03 2014 1019 0 PPE-0 remote node 10.72.137.70: Primary Tue May 20 21:53:03 2014 1020 5 PPE-0 self node 10.72.137.50: SYNC start Tue May 20 21:53:06 2014 1191 0 PPE-0 self node 10.72.137.50: SYNC complete Tue May 20 21:53:10 2014

This is why Tech support asks for Collectors from both Primary and Secondary units for HA related issues.Having the correct date and time set on the devices is important to simplify correlation.

#date +val YYMMDDHHMM


HA Best Practices

Use VMACs Disable unused interfaces (not just hamon on those interfaces)rpc passwords should match – Mismatch leads to sync/prop failures

“Warning: Unable to establish connection with the secondary. Command propagation failed”

Ensure certs and files match on peers – to avoid certificate binding failures>sync ha files

Ensure both devices are running same Versions and use the same hardware/license

Enable failsafe mode to avoid a complete outage when both nodes have an issue> set ha node -failSafe ON

Use Failover Interface Sets (FIS) - if interface bandwidth is not an issue>add fis testfis>bind fis testfis 1/1 1/2


Gratuitous ARP

Possible Firewall issues?

> set network L2param –garpReply enabled


VMACs - A better solution

– Simple config> add vrID 1> bind vrID 1 -ifnum 1/1

– Instantaneous failovers– Configuration propagated to secondary

Considerations:Just be sure switch supports it

You might need to flush arp once

> send arp all


LACP

Aggregating Multiple ports into a High Speed Link• 802.3ad Compliant• Aggregation of throughput• Better Resiliency• Channels are dynamically created using the port key when lacp is enabled on an interface

Sample configuration: >set interface 1/1 -lacpmode active -lacpKey 1 – lacpTimeout long

lacpkey value forms the basis for the LA/x notationactive/passive decides whether NetScaler sends LACPDUs or notlacptimeout decides how often they are sent

Note:- LACP Configs are neither propagated nor synchronized- Once an interface is bound to a channel, channel parameters take precedence i.e. interface VLAN config no longer applies


LACP continued..

LACP on SDX Relies on “NIC Bonding” feature of XS Switch and SDX are aware of the bond LACP PDUs are exchanged

Recommended VersionsSDX: 10.1 or higherNetScaler: 10.1 or higherXenServer: 6.1 or higher

What is now possible:• Sharing across instances• LA for Management Interfaces

Channel creation will require reboot on instances if the member interfaces are not already provisioned.

Best Practices – Only create Channels from SVM – Not from XenServer not from within Instance.Ensure of equal speed – or lowest speed will be chosen.Create channel first before plugging in cables.Similarly disconnect the cable before removing a port from the channel.


Logs on XenServer

tail –f /var/log/messages | grep bond

LACP going down[root@localhost ~]# cat /var/log/messages | grep bondMay 17 09:38:00 netscaler-sdx ovs-vswitchd: 03991|bond|INFO|interface eth8: link state downMay 17 09:38:00 netscaler-sdx ovs-vswitchd: 03992|bond|WARN|interface eth8: disabledMay 17 09:38:00 netscaler-sdx ovs-vswitchd: 03993|bond|INFO|bond bond0: active interface is now eth6May 17 09:38:01 netscaler-sdx ovs-vswitchd: 03994|bond|INFO|interface eth6: link state downMay 17 09:38:01 netscaler-sdx ovs-vswitchd: 03995|bond|WARN|interface eth6: disabledMay 17 09:38:01 netscaler-sdx ovs-vswitchd: 03996|bond|WARN|bond bond0: all interfaces disabledMay 17 09:38:02 netscaler-sdx xcp-networkd: [ info|netscaler-sdx|1||network_monitor_thread] Bonds status changed: bond0 nb_links 2 up 0 up_old 2

LACP back up[root@localhost ~]# cat /var/log/messages | grep bondMay 17 09:38:55 netscaler-sdx ovs-vswitchd: 03997|bond|INFO|interface eth6: link state upMay 17 09:38:55 netscaler-sdx ovs-vswitchd: 03998|bond|INFO|interface eth6: will be enabled if it stays up for 31000 msMay 17 09:38:55 netscaler-sdx ovs-vswitchd: 03999|bond|INFO|bond bond0: active interface is now eth6, skipping remaining 31000 ms updelay (since no interface was enabled)May 17 09:38:55 netscaler-sdx ovs-vswitchd: 04000|bond|WARN|interface eth6: enabledMay 17 09:38:55 netscaler-sdx ovs-vswitchd: 04001|bond|INFO|interface eth8: link state upMay 17 09:38:55 netscaler-sdx ovs-vswitchd: 04002|bond|INFO|interface eth8: will be enabled if it stays up for 31000 msMay 17 09:38:57 netscaler-sdx xcp-networkd: [ info|netscaler-sdx|1||network_monitor_thread] Bonds status changed: bond0 nb_links 2 up 1 up_old 0


On the SVM

/var/mps/log/mps_service.log

LA/1 configured in the SVM consists of 10/1 and 10/3:

Tuesday, 14 Jan 14 14:36:24.988 +0100 [Debug] Json Web Content: {"host_interface":{"mapped_port":"LA/1","sync_operation":false,"channel_type":"LACP","channel_throughput":"0","channel_bandwidth_high":"0","channel_bandwidth_normal":"0","channel_interface_list":"10/1,10/3","static_channel_state":true,"lacp_channel_time":"SLOW","channel_ha_monitoring":true,"channel_tag_all_vlans":true,"channel_alias":""}}


Logs – on Instance/vpx/mpx

# tail -f ns.log | grep interfaceMay 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT NICMIGRATE 55384 0 : NIC "interface(10/1)" State MigratedMay 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT NICLACPSC 55385 0 : Device "interface(10/1)" - RX state PORT_DISABLEDMay 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT NICLACPSC 55386 0 : Device "interface(10/1)" - RX state LACP_DISABLEDMay 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT DEVICEDOWN 55387 0 : Device "interface(10/2)" - State DOWNMay 17 09:41:11 <local0.notice> 10.90.196.77 05/17/2014:09:41:11 GMT VPX01 0-PPE-0 : EVENT DEVICEDOWN 55388 0 : Device "interface(LA/1)" - State DOWN

In a healthy case:root@VPX01# nsconmsg -K newnslog -g rx_lacp -s disptime=1 -d currentIndex rtime totalcount-val delta rate/sec symbol-name&device-no&time 0 1462998 737728 8 1 nic_tot_rx_lacpdus interface(10/1) Sat May 17 12:09:13 2014 1 0 737664 7 1 nic_tot_rx_lacpdus interface(10/2) Sat May 17 12:09:13 2014 2 7000 737735 7 1 nic_tot_rx_lacpdus interface(10/1) Sat May 17 12:09:20 2014

When it goes down:36 77000 737913 6 0 nic_tot_rx_lacpdus interface(10/1) Sat May 17 12:13:25 2014


Show channel – what to look at?

> show channel LA/11) Interface LA/1 (802.3ad Link Aggregate) #5 flags=0x4100c020 <ENABLED, UP, AGGREGATE, UP, 802.1q> MTU=1500, native vlan=1, MAC=e2:08:d2:15:b2:05, uptime 0h00m07s Requested: media NONE, speed NONE, duplex NONE, fctl NONE, throughput 0 Actual: throughput 2000

RX: Pkts(5099181) Bytes(1302893502) Errs(0) Drops(115221) Stalls(0) TX: Pkts(7816907) Bytes(1266490416) Errs(0) Drops(742) Stalls(0) NIC: InDisc(0) OutDisc(0) Fctls(0) Stalls(0) Hangs(0) Muted(0) Bandwidth thresholds are not set.

LA mode: AUTO

10/1: FIBER-1000-FULL-NONE UP 0h00m14s PortID=(32768,1), Mux=DISTRIBUTING, Rx=CURRENT, SELECTED <Active, Long timeout, Agg, Sync, Collecting, Distributing> Partner: SysID=(32768,b4:14:89:81:5f:00), Key=5, PortID=(32768, 266) <Active, Long timeout, Agg, Sync, Collecting, Distributing>

10/2: FIBER-1000-FULL-NONE UP 0h00m14s PortID=(32768,2), Mux=DISTRIBUTING, Rx=CURRENT, SELECTED <Active, Long timeout, Agg, Sync, Collecting, Distributing> Partner: SysID=(32768,b4:14:89:81:5f:00), Key=5, PortID=(32768, 267) <Active, Long timeout, Agg, Sync, Collecting, Distributing>


Trace

> show lacpActor SystemID: (32768, 00:e0:ed:26:dc:cd)

Verify if PDUs are being correctly exchanged

Question: do keys need to match on both devices?

No – But on a given devices all interfaces that part of the same LA channel need to have the same key


Next Webinar: June 2014

Title: Best practices for implementing, administering, and troubleshooting XenDesktop 7.5

Description: Citrix XenDesktop introduced a number of new concepts and processes for Desktop Administrators. Understanding these advancements and their effect on is key to a stable XenDesktop environment.

This session will discuss core deployment and configuration concepts and considerations and provide proven practices for troubleshooting the top three XenDesktop issues.

When: June 24th & 25th

June 24th – Register Now

June 25th – Register Now

https://www1.gotomeeting.com/register/581298641

https://www1.gotomeeting.com/register/764180648


Fuel your talent with continuous learning.

Citrix Education offers the following technical training for Networking professionals:

CNS-205: Citrix Netscaler 10 Essentials and Networking

CPE-350: Citrix NetScaler 10 Essentials and Networking Practice Exam

CNS-301: Citrix NetScaler 10 Advanced Implementation

Visit (bit.ly/05Webinar) to save 10% off through June 30*

*Not valid with any other promotions, packages, discounts or practice exams.. Applies only to new purchases. Regional limitations may apply.

http://training.citrix.com/mod/ctxcatalog/course.php?id=497

http://training.citrix.com/mod/ctxcatalog/course.php?id=794&utm_source=webinar&utm_medium=webinar&utm_campaign=NS%20Practice%20Exam

http://training.citrix.com/mod/ctxcatalog/course.php?id=794&utm_source=webinar&utm_medium=webinar&utm_campaign=NS%20Practice%20Exam

http://training.citrix.com/mod/ctxcatalog/course.php?id=695

http://bit.ly/05Webinar


Build your Citrix skills in your personal virtual sandbox

Play in your own Virtual Sandbox with Learning Labs from Citrix Education. With your purchase, you’ll receive your own dedicated server with access to the seven most popular Learning Labs from Synergy. Featured labs include:

• NetScaler, the Enterprise Security Swiss Army Knife

• Front-Ending and Load Balancing XenDesktop and XenApp with NetScaler

• Enhancing Visibility of Applications with NetScaler Insight Center

http://training.citrix.com/cms/education/promotions/learninglabs/




New Citrix Practice ExamsAccelerate Your Path to Certification

Available on training.citrix.com ($39 each):

CPE-350 – Citrix NetScaler 10 Essentials and Networking Practice Exam

CPE-300 – Deploying XenDesktop 7 Solutions Practice Exam

CPE-A22 – Citrix XenApp 6.5 Advanced Administration Practice Exam

http://training.citrix.com/cms/index.php/promotions/practiceexams/





WORK BETTER. LIVE BETTER.

troubleshooting common network related issues with netscaler

Technology

promiscuous vlan

vlan design considerations

different native vlan

destination mac

slide common issues

single vlan network

netscaler snip

common network related