validated reference design netscaler sdx clustering - citrix.com · validated reference design...

1Citrix.com

Solution Guide

Solution Guide

Validated Reference Design NetScaler SDX Clustering

This guide focuses on providing guidelines to customers on implementing SDX Clustering in NetScaler based on their use cases.

2Citrix.com | Solution Guide | Validated Reference Guide for NetScaler SDX Clustering

Solution GuideValidated Reference Design Guide for NetScaler SDX Clustering

Table of Contents

Section 1: SDX Clustering Overview 3 Pre-Requisites: NetScaler SDX Clustering 3 Layer 2 and Layer 3 Connectivity 3 ClusterTrafficDistributionMethods 4 Inter-nodeCommunications 4 InternalTrafficDistributionMethods 4 Cluster Coordinator (CCO) 5 IP Topology Requirements 6 IP Requirements 7 StaticRouteManagement 7 SVMandNSIPRequirements 8 DynamicRoutingRequirements 8Section 2: Design - NetScaler SDX Cluster 9 NetScaler Physical Design 9Section 3: NetScaler Appflow for Clustering and HA 11 Introduction 11 Proxy User Use Case 11 LAN User Use Case 11 NetScalerLANConfiguration 12 StorefrontConfiguration 12 HDX Insight Requirements for Cluster Environment 12 Session Reliability Behavior 12 Cluster Information 13 HA Information 13 AppflowRecords 13 Configuration 14Section 4: Configuration: NetScaler SDX Cluster 14 SDXBuild 14 ClusterBuild 14 RouteManagement 15 Create Spotted SNIP(s) 15 ConfigureVirtualServer 15 ConfigureDynamicRouting 15 ConfigurationExample:NetScalerSDXCluster 16Appendix A: Cluster Troubleshooting 19Appendix B: SVM Management 22Appendix B: OSPF Reference 22



Section 1: SDX Clustering Overview

This handbook is intended to guide resources through the practical implementation of NetScaler SDX appliances configuredinaClusterMode.Thisconfigurationisbasedonamulti-tenantdeploymentscenariowhichrequiresdataplaneandtrafficplaneisolationbetweentenantsusinguniqueVLANID’spertenant.ThedetailsinthisdocumentareSDXmodelspecificandcaveatsmayapplytoVPXandMPXapplianceeditions.

Pre-Requisites: NetScaler SDX Clustering

• A NetScaler cluster can only include NetScaler nCore appliances. Clustering of NetScaler Classic

appliances is not supported.

• FIPS140-2Level2isnotsupportedinNetScalerClusteredappliances.ThalesnShieldConnect

appliances can be used for this requirement.

• Allappliancesmusthavethesamesoftwareversionandbuild.

• All appliances must be of the same platform type. This means that a cluster must have either all

hardwareappliances(MPX)orvirtualappliances(VPX)orSDXNetScalerinstances.

• Allappliancesmustbeonthesamenetwork.

• All appliances must have the same licenses. Also, depending on the NetScaler version, there are some

additional aspects to address:

• Beinitiallyconfiguredandconnectedtoacommonclient-sideandserver-sidenetwork.

• PooledLicensingModeisnotsupported.

PleaserefertotheNetScalerProductDocumentationforConfigurationSupportofClusters:https://docs.citrix.com/en-us/netscaler/11-1/clustering/cluster-features-supported.html

Layer 2 and Layer 3 Connectivity

Dynamic Routing Requirements

IftheNetScalerSDXClusterrequiresthatmorethanoneVLANbecarriedoveranetworkinterfaceontheSDXappliance,theECMPtopologyisrequired.(Asaruleofthumb,alwaysuseECMProutingwithSDXclustering.)TheECMPtopologyusesthedynamicroutingmethodtoachieveequalcosttrafficdistributionacrossthemem-bernodesoftheSDXclusterforClienttrafficdistribution.

Nexus Switch(s) Requirements

NOTE:

• SomeNexusSwitchmodelsdonotsupportDynamicRoutingprotocolsoverVPC.

• Nexus7KSeriesswitchesrequireaminimumfirmwarebuildinordertosupportdynamicrouting

protocols over VPC.

• Nexus9KseriesswitchesdonotsupportdynamicroutingprotocolsoverVPC.

Jumbo Frame Support

JumboframesupportisrequiredwhenanSDXclusterhasbackplanedatathattraversesmorethanonephysi-cal SDX chassis. This requirement is due to the additional tuple information that is added to the Ethernet header

https://docs.citrix.com/en-us/netscaler/11-1/clustering/cluster-features-supported.html



toincludetheclustermemberinterfaceid.TheresultisanEthernetframethatis1514bytes,thusexceedingthestandard1500MTUEthernetframe.

EXAMPLE: First node in cluster uses 2-tuple: i.eint1/1/1 MTU1500

Subsequent nodes use 3-tuple: i.eint1/1/2 MTU1514

ClusterTrafficDistributionMethods:

CLAG: Single VLAN per Interface

SDXcansupportClusterLinkAggregationGroupsonaperinterfacebasis,solongasthereisonlyonenetworkperSDXinterface.(ThismethodisrarelyusedwithSDXclusteringduetothelimitationtoonlyoneVLANperinterface.)CLAGisanL2ChannelthatcanbeeitherStaticorDynamicandtheupstreamswitchseesasingleclusterMACaddressintheARPtable.

CLAG NOTES:

1. A separate physical medium is required for Client connection steering and node-to-node

communications.

2. Cluster Heartbeats cannot be exchanged over the CLAG interfaces.

3. VPXappliancesarenotsupportedwithCLAG,someESXandKVMversionscansupportCLAG.

ECMP Multiple VLAN’s per Interface

SDXrequirestheuseofEqualCostMultipathviadynamicroutingprotocolsifmorethanoneVLANistravers-inganSDXinterfaceandwillusetheFlowReceiver(FR)andFlowProcessor(FP)forClientconnectionsteeringacross the cluster backplane.

ECMPNOTES:

1. ECMPisanL3Routingmechanismthathasalimitationofthesupportingrouterregardingthenumber

ofECMProutesthatcanbesupported.

2. ClusterHeartbeatscanbeexchangedovertheECMPinterfaces(notrecommended).

Inter-node Communications:

ClusternodeswithinthesameClusterInstancecommunicatewitheachotherbyusingtheclusterbackplane.ThebackplaneisasetofconnectionsinwhichoneinterfaceofeachnodeisconnectedtoacommonswitchorVLAN,whichiscalledtheclusterbackplaneswitchorVLAN.EachnodeoftheclusterusesaspecialMACaddresstocommunicatewithothernodesthroughthebackplane.

InternalTrafficDistributionMethods:

AnupstreamL3devicewilluseaHashbasedalgorithmbasedontheroutesreceivedfromECMPtodeterminewhichclusternodewillreceivetheincomingclientconnection.Theclusternodethatreceivestheclientconnec-tionfromtheupstreamL3deviceisknownastheFlowReceiver(FR).



Flow Processor (FP):

TheFlowProcessoristheclusternodethatisassignedthepacketfortheinitialclientconnectionandestablish-esthetrafficflow.TheFlowProcessorwillprocesstheclienttrafficandrespondtotheclientusingitsownlink.

Flow Receiver (FR):

TheFlowReceiveristheclusterthatreceivestheinboundclientpacket.SometimestheFlowReceiverisalsotheFlowProcessorinwhichcasenofurtherpacketsteeringisrequired.WhentheFlowReceiverreceivesapacketthatispartofaflowforwhichitisnottheFlowProcessor,itdeterminestheFlowProcessor(FP)forthetrafficstreamandsteersthepackettotheclusternodethatownstheflow(FP).Allpacketsteeringhappensviathe data backplane, if the backplane is unavailable, the packet is dropped.

Spotted entity:AnyconfigurationelementthatisNodespecificanddoesnotbelongtoallnodesin the cluster. FP = node in processing set.

Striped entity: processing set = multiple nodes. Hash computed on the packet parameters. (TCPandUDPprotocolsuse-4tuplecomputation,anyotherprotocolsotherIP-2tuplecomputation)

An ACTIVE node from processing set is selected as FP based on the hash. Global Key synchronized across all the nodes to compute consistent hash.

Cluster Coordinator (CCO)

AllconfigurationsonaNetScalerclusterareperformedontheclusterIPaddress,whichisthemanagementaddressofthecluster.ThisclusterIPaddressisownedbyaclusternodethatisreferredtoastheclustercon-figurationcoordinator

Whenanodeisaddedtoacluster,theconfigurationsandthefiles(SSLcertificates,licenses,DNS,andsoon)thatareavailableontheclusterconfigurationcoordinatoraresynchronizedtothenewlyaddedclusternode.Whenanexistingclusternode,thatwasintentionallydisabledorthathadfailed,isonceagainadded,theclus-tercomparestheconfigurationsavailableonthenodewiththeconfigurationsavailableontheconfigurationcoordinator.Ifthereisamismatchinconfigurations,thenodeissynchronizedbyusingoneofthefollowing: Full synchronization.Ifthedifferencebetweenconfigurationsexceeds255commands,alltheconfigurationsoftheconfigurationcoordinatorareappliedtothenodethatisrejoiningthecluster.Thenoderemainsoperation-ally unavailable for the duration of the synchronization.

Incremental Synchronization.Ifthedifferencebetweenconfigurationsislessthanorequalto255commands,onlytheconfigurationsthatarenotavailableareappliedtothenodethatisrejoiningthecluster.Theopera-tional state of the node remains unaffected.

Determining the CCO:

EverycommandpropagatedinclusterwillbeassignedanumbercalledARUnumber.Itisasequentiallyincreas-ingnumberandisassignedbytheCCOnode.ARUnumberisusedtodetectconfigurationmismatchandtriggersync.

• ThenodewhichhasremainedCCOforalongertime(applicableonlyfor2-node)

• ThenodewhichhashighestARU(AllReceivedUpto)value(seesectionbelow)

• ThenodewhichhasbetterOperationalView–hasthemostvisibilityoftheclusternodeneighbors.

• Priority–(0-31)numericpriorityofthenodemembers,with0beingthehighestpriority.

• PreviousCCO–thenodeoftheclusterthatwasthepreviouselectedCCO.

• ThenodehavinglowestIP.



All Received Upto (ARU)

Whenweissueforcesynconanode,itresetsitslocalARUto0,triggersasyncfromtheCCOnodeandsetslocal ARU to highest ARU after sync is complete.

CounterstoviewARU:nsconmsg-garu-dstats

SCENARIO:Saywehaveaclusterwith3nodes{n1,n2,n3}andn1istheCCO.ARUstartsfrom0.

1. Commandc1isfiredontheclusterIPaddress.CCOassignsARU1tocommandc1andpropagatesthe

command to other nodes.

2. Commandc2getsfiredonclusterIPaddress.ARU2getsassignedtoc2andsoon.

3. Saynoden3isofflinewhencommandc3isfired.ARU3getsassignedtoc3andthecommandis

appliedontheonlinenodesn1andn2.Atthisstage,thelocalARUofn1andn2is3whereasthatof

node n3 is 2.

4. Whenn3comesbackonline,itseesthatthehighestARUnumberis3(ARUisadvertisedviaheart

beats, so highest ARU can be derived), and it triggers a sync to catch up.

IP Topology Requirements

Data Plane (Cluster Back Plane)

TheSDXclusterbackplaneisanisolatednetworkandrequiresNSIPtrafficonthethesamenetworkastheinter-clustercommunicationtrafficfortheSDXcluster.Thedataplanetrafficisspecifictotheclusterheartbeatandtrafficsteeringfromtheclustercoordinator(CCO).

BackplaneTrafficIncludes:

• NSIPtoNSIPtraffic-clusterheartbeats,clusterprotocol,configpropagation/synchronization,NNM

• Datatraffic/steering(srcmac:steeringmac,destmac:backplanemac,IPtupleremainssameas

client/server packet)

Traffic Plane (Client / Server Traffic Plane)

TheSDXtrafficplaneisanisolatedtrafficnetworkthatroutesdirectlytotheClientnetworkandiswheretheVirtualServerIP’swillbeterminatedforincomingclientrequests.TheClientandServernetworkscanbethesameordifferentnetworksinECMPmode.Asinglenetworkisusedforaonearmedconfigurationandtwoormorenetworksareusedforamultiplearmedconfiguration.

VLAN Separation

ThereisnotanylimitationonVLANseparationinregardstotheDataplaneortheTrafficplane.Jumboframes(MTUincrease)willstillberequiredonthedataplaneregardlessofVLANtaggingduetotheclusterinterfacetupleandthechangesmadetotheMTU(1514).

TheSDXappliancehasalimitationof63VLAN’sper10GpbsinterfaceifthefilteringmechanismisconfiguredintheSVM.IftherearenotanyVLANfiltersconfiguredintheSVM,theSDXappliancecanaccommodate4096VLAN’sperinterface.



IP Requirements

Cluster IP: (CLIP)TheCLIPisownedbytheclustercoordinator(CCO)andthisIPaddresswillfloattotheactiveCCOnode.TheCCOisresponsibleforconfigurationsynchronization,commandpropagation,andfilesynchroni-zation(certificates,CRL,etc…).ThinkoftheClusterIPassimilartotheNSIPforthewholecluster.

Striped Subnet IP: (SNIP)ThestripedSNIPaddressareownedbymultiplenodesmembersintheclusterandwillserviceallserverproxyconnectionstothebackendservices.Whentheclusterinitiatestrafficsuchasser-vicemonitors,itwillusuallybesourcedfromtheStripedClusterSNIP.

Spotted Subnet IP: (SNIP)ThespottedSNIPisthenetworkhostaddressthatisusedtoadvertiseroutestotheupstreamL3deviceforeachnodememberoftheclustertoallowforHashdistributionofincomingClientconnections.

Striped Virtual IP: (VIP)ThestripedIPaddressesareownedbymultiplenodesmembersintheclusterandwillservice all Client connetions based on the Hash table of the upstream distribution L3 device. This is the address thatisbeingadvertisedviaECMP.

Spotted Virtual IP: (VIP) ThespottedIPaddressisownedbyasinglenodeintheclusterandwillutilizetheclustersteeringoftheFlowReceiverviatheClusterDataplanetoprocessinboundClientconnectionstotheclusternodeownerofthespottedVIP.ASpottedClusterVIPwouldbeusedforfeatureswhichareavailableinnode only mode.

VirtualServerloadbalancingmethodscanbeappliedtotheVIP’sforservicedistributionaswell.

StaticRouteManagement

Overview

WhentheNetScalerinstancesareprovisioned,theywillonlyhaveanNSIPaddresspresentontheClusterBackplaneVLANandadefaultroutepointedtothegatewayontheClusterBackplaneVLAN.Aftertheclusterhasbeencreated,additionalVLANswillbeaddedtotheclusterandconfiguredondifferentinterfaces.Thiswillrequire static routing changes on the cluster.

Policy Based Routes

Aftertheclusteriscreated,aPBRneedstobecreatedontheclusterwithasourceIPrangestartingwiththeNSIPofeachmembernodeandincludingtheCLIP.ThenexthopgatewaywillbetheIPaddressofthegatewayon the cluster backplane VLAN.

Default Routes

After the PBR has been created and applied, the default route for the cluster needs to be changed. The default routeshouldbethroughthegatewayontheclienttrafficVLAN.

VLAN’s

VLAN Data / NSIP and CLIP (Cluster Backplane)VLANTraffic/StripedandSpottedSNIPs(Traffic)VLANSVM/XS(ChassisManagement)VLANLOM(ILOVLAN)



SVMandNSIPRequirements

SVM and NSIP Communication Requirements

Whileitispossibletorunnon-clusteredNetScalerinstancesonanSDXwheretheSVMdoesnothavecommunicationwiththeNSIPoftheinstances,itisnotrecommended.Whenclusteringisbeingused,thisisnotanoption.TheSVMmustbeabletocommunicatewiththeNSIPofallclusterinstancesatalltimes.FailuretomaintainSVMtoNSIPcommunicationwillleadtoorphanedclusterinstanceswhichwillexistinabadstate.Insomecases,thebestwaytoremedythesituationafterSVMtoNSIPcommunicationisrestored,istodeletetheinstance,recreateitandrejoinittothecluster.

Cluster Heartbeats

NodetoNodeclusterheartbeatsfromNSIPtoNSIPshouldbecarriedovertheclusterdataplanewheneverpossible.WhenECMPmodeisused,NodetoNodeclusterheartbeatscanbecarriedovertheTrafficplaneviarouting,however,specialcareshouldbetakeninthismodelasupstreamlayer3devicescanpotentiallyinter-ferewithclusterheartbeats.(Notrecommended)

SVM for Provisioning and Management of Cluster Node Groups

Do not attempt to provision the cluster from the administration GUI of the NetScaler instances. The Cluster must becreatedfromtheSVMandNodesmustbeaddedorremovedfromtheSVM.IfyoucreateaclusteroraddclusterelementsfromtheNetScalerinstanceadministrationGUI,theresultantclusterwillbeunstable.

Dynamic Routing Requirements

OSPF

OSPFroutingisrequiredtohavethecustomer’snetworkinfrastructuredistributetrafficbetweenthediffer-entnodesintheNetScalercluster.InordertoconfiguretheNetScalertoparticipateinOSPFroutingwiththecustomer, several steps are required. OSPF routing must be enabled for the cluster globally. The vServers that wewanttoadvertisetoOSPFmusthaveroutehealthinjectionconfigured.TheVIPmustbeconfiguredintheNetScaler System IP menus to participate in OSPF routing using Type 1 LSAs for the proper OSPF area assigned bythecustomer.ZebOSmustbeconfiguredfromVTYSHtoadvertiseOSPFrouteslearnedfromtheNetScalerkernel.EachnodeintheclustermustbegivenadistinctrouterIDinVTYSH.OSPFroutingcanbevalidatedbyrunningtheshowipospfneighborcommandfromVTYSH.OSPFrouteadvertisementcanbedisabledbyshow-ing the ip routes learned from the NetScalers on the customer infrastructure.



Physical To Virtual Example Diagram

Section 2: Design - NetScaler SDXCluster

NetScaler Physical Design

Physical Cabling Diagram:



Node Groups

Default Node Group

Member Nodes

N+1 Default Clusters:

• 1 node on Appliance (1)



NetScaler Logical Diagram



Section 3: NetScaler Appflow for Clustering and HA

Introduction

ANetScalerClustersolutionreferstoagroupofNetScalerswhichcanbeconfiguredandmanagedasasinglesystem. It provides scalability and availability.

ANetScalerHighAvaiulabilitysolution(Active-Passive)referstotwoNetScalersformingapairinwhichoneisactivelyprocessingthetrafficwhiletheotherisastand-by.Boththenetscalerareinsyncwithrespecttoconfiguration.Whentheprimarynetscalergoesdownforsomereason,thesecondaryimmediatelytakes-over,thus providing the user a minimal disruption.

TheHDXInsightsolutionconsistsofAppflow,whichisamethodofloggingprotocolandflowrelatedinfor-mation. The Netscaler exports this information and a collector (external to NetScaler) parses and stores this informationforsubsequentviewinginrequiredformat.

ThisdocumentdescribesthedesignforsupportingappflowonagroupofNetScalersthatbelongtoaclusterortwoNetScalersthatformaHApair.

Proxy User Use Case

Provides HDX Insight Deep Packet Inspection (DPI) for all ICA Client connections from the External Internet (WAN)whenscalingclusterdeploymentsinasingletierarchitecture,onceanHAPlatformhasreachedthemaximum number of ICA connections per location. Additionally,usingNetScalerClusterswillprovidecapacityforDynamicScalingrequirementswithouttheneedtorebuildthenetworktopologyorplacementoftheNetScalerappliancesastheenvironmentgrows.

LAN User Use Case

Provide HDX Insight Deep Packet Inspection (DPI) for all ICA Client connections from the Internal Intranet (LAN) whenscalingclusterdeploymentsinasingletierarchitecture,onceanHAPlatformhasreachedthemaximumnumber of ICA connections per location.



NetScalerLANConfiguration

• CreateaCRvserverwithtypeasHDX.

• BindanappflowpolicyofthetypeICA_REQ_DEFAULTglobally.

• Belowisanexampleconfiguration: addappflowcollectorcol1-IPAddress<collectorIP> addappflowactionact1-collectorscol1 addappflowpolicypol2trueact2 bindappflowglobalpol21END-typeICA_REQ_DEFAULT addcrvservercrvsHDX<crvserverIP><Port>-cacheTypeFORWARD-cltTimeout180

StorefrontConfiguration

• Edit“default.ica”fileofthestorethatisusedtolaunchapps/desktops.

• Thisfilewillbepresentin"C:\inetpub\wwwroot\Citrix\<Store>\App_Data”.

• FollowingconfigurationneedstobeaddedunderWFClientandApplicationsection ProxyType=Socks ProxyHost=<crvserverIP>:<Port> ICASOCKSProtocolVersion=0 ICASOCKSProxyHost=<crvserver> ICASOCKSProxyPortNumber=<Port>

HDX Insight Requirements for Cluster Environment

ThefollowingprerequisitesarerequiredwhenusingICAProxyand/HDXInsightfeaturesinaNetScalerCluster:

• AllNodesintheClustermustbeofthesameModelofNetScalerPlatform

• AllNodesintheClustermusthavethesameLicenseModeforClusterMembers

(NetScaler Platinum Edition is required for HDX Insight)

• StandardICAClusterissupportedwiththeEnterprise(EE)andStandard(STD)LicenseModes

Session Reliability Behavior

SessionReliabilityBehavior:(whenallNodesareworkingintheCluster)

• AllSessionContextsaresharedamongtheHealthyNodemembersandwillusetheBackPlanefor

connection communication.

• If, a Node member has died, the Session Reliability information becomes invalid, and Auto Client

ReconnectisthereconnectmethodfordisconnectedusersviatheCitrixReceiverClientforWindows

andMAC.

• IftheACRFeatureisnotsupportedforthatusersCitrixReceiverClient,theuserwillneedto

reconnect the session.

The Reconnect info is shared via the Back Plane, via the Inter-Node communicationThisisstandardtrafficoverheadforthereconnectoperation.TheBackplaneOverheadwillreduceoncetheconnectionsaremadeviadisconnectandreconnect



Cluster Information

WhenmultipleNetScalersbelongingtoaclusterexportappflowrecords,itbecomesnecessarytocorrelate/ag-gregatethemtogetasystem-levelview.SinceeachNetScalerexportsthedatarecordsindependently,allthenodesmustexportanadditionalinformationinalloftheirrecordsindicatingwhichclustertheybelongto.

Thisfield,calledtheobservationdomainID,willcontaintheclusterIDasconfiguredbytheadministrator.Ob-servationdomainIDispartoftheipfixheader.

Itisalsodesirabletodisplayanamewhentheaggregateddataispresented.Thisname,calledtheobservationdomainnameisassociatedwithanobservationdomainandexportedintheobservationdomaininformationrecord.

The IP address of each node in the observation domain is also exported in the observation domain information record.

HA Information

WhentwoNetScalersformaprimary-secondarypair,oneprocessestrafficandsendsappflowrecordswhiletheotherisstand-by.However,whenaswitch-overhappens,thenewprimarystartssendingappflowrecords.RecordsfromboththeNetScalersarerequiredtobecorrelated.WhenboththeNetScalersusethesameID,thecollectorcanmergetherecordsattheHA-pairlevelirrespectiveofwhichNSisprocessingthetrafficcurrently.ThisIDisconfiguredbytheadministratorandisexportedasobservationdomainIDinalltherecords.Itisalsorequired to send the NSIP of the netscaler in all the records because in case of SNIP, both the NSs use the same source IP. Similar to cluster, the observation domain name and the nodes (primary and secondary) IP addresses

are exported.

AppflowRecords

Anewrecordcalledtheobservationdomaininformationrecordwillbeaddedtoexportinformationsuchastheobservation domain id, observation domain name and the IP addresses of the nodes constituting the observa-tiondomain.Theobservationdomainidisalsopartofeveryrecordasitappearsintheipfixheader.Also,theobservationpointIDwillcarrytheNSIPoftherespectivenodethatisexportingtherecords.

Observation Domain Information (Appflow Template ID: 291):

IE-ID Enterprise ID Source of Inputs Size

149 0 observationDomainId 4-bytes

352 5951 numberOfObservationPoints 2-bytes

353 5951 observationPointId_1(NSIP) 4-bytes





300 0 observationDomainName String (variable)



Configuration

TheadministratorisrequiredtoconfigureadomainIDontheCCOorprimarynode.ThisispropagatedtoothernodesoftheclusterorsecondarynodeofHA-pair.Similarly,anobservationdomainnamecanalsobeconfig-uredviathefollowingSetcommands:

• setappflowparamobservationDomainID<num>

• setappflowparamobservationDomainName<name>

SampleAppFlowcommandsforanICAPRoxtCluster:

• addvpnvserverICAProxy_vipSSL192.168.x.x443-icaOnlyON-downStateFlushDISABLED-appflow

Log ENABLED

• addappflowcollectorCluster_Collector-IPAddress10.x.x.x

• setappflowparam-observationDomainId1020-observationDomainNameCluster_Domain

• addappflowactionappflow_act_cluster-collectorsCluster_Collector

• addappflowpolicyappflow_pol_clustertrueappflow_act_cluster

• bindappflowglobalappflow_pol_cluster1END-typeICA_REQ_DEFAULT

Bydefault,thisnumberis0,indicatingthatthenodedoesnotbelongtoanyclusterorHA-pair.Toforman

observationdomainofmorethanoneNetScalers,thisIDmustbemorethan1000.Thedefaultnameofthe

observation domain is “Default”.

Section 4: Configuration: NetScaler SDX Cluster

SDX Build

Configure NetScaler SVMs

InstalltheSVMbundle.

Configure NetScaler Instances

ProvisiontheNetScalerinstancesfromtheSVMwithNSIPanddefaultgatewayontheclusterbackplaneVLAN.

Cluster Build

Configure Cluster Instance

CreateaClusterinstancefromtheSVMonappliance(1)

Assign the unique Cluster IP (CLIP) Address

Configure Additional Cluster Nodes

JoinClusternodemembersfromeachSVMonwhichtheyarehostedintotheClusterGroup(Default_Group).

Configure the Cluster Node Priorities

ThedefaultSVMassignedpriorityforaclusternodeis31.Thisissetonallnodesandcanaffectthe

clustercoordinator(CCO)electionprocesswhenanodememberleavesorjoinsthecluster.Change



thepriorityforeachnodetogivefirst,secondandthirdpriorityforCCOelections.SettingtheCluster PriorityIDallowsformanagementoftheDefaultCCOprocess.

RouteManagement

Policy Based Routes

UsePolicyBasedRoutestomanageCLIPandSNIPcommunicationtotheSVMinstance.ThismayrequiremorethanonePBRruledependingonthenexthopdefinitionandhowmanynodesareinthecluster.

Adjust Default Route

ChangetheDefaultRoutetoforwardallVServertraffictotheproductionnetwork,fromthedefaultroutethatwasassignedbytheSVMattimeofprovisioning.

Create Spotted SNIP(s)

FromtheClusterIP,configureaspottedSNIPforeachmembernodeoftheclusterwithanIPaddressfromtheTrafficVLAN.TheseIPaddressesallneedtobeonthesamenetworksegment.EnableDynamicRouting.ASpottedSNIPisrequiredforeachclusternodethatwillparticipateinECMP.

ConfigureVirtualServer

Two steps required:

• Create LBVserver for HTTP service Enable RHI State on the LBVserver

• ConfiguretheStripedVIPtothecorrespondingVserver Set the LSA value = 1 Set the OSPF AREA value (xx) SetHostRouteGateway(StripedSNIP)

ConfigureDynamicRouting

Thedynamicroutingconfiguration(OSPFinthisexample)fortheSDXclustershouldbecompletedontheClus-ter Coordinator (CCO) via the Cluster IP address (CLIP).

OSPF ConfigurationDefiningthe“-ownerNode”commandforeachmembernodeoftheclusterwillallowtheCCOtodistributetheOSPF routes to all members of the cluster



Configuration Example: NetScaler SDX ClusterCluster Diagram

Configuration Example

CreatedInstancesfromSVM setnsconfig-IPAddress192.168.7.11-netmask255.255.255.0 addvlan301-sdxVlanYES setinterface10/1-lacpModeACTIVE-lacpKey2-ifnumLA/2 setinterface10/2-lacpModeACTIVE-lacpKey2-ifnumLA/2 setinterface10/3-lacpModeACTIVE-lacpKey3-ifnumLA/3 setinterface10/4-lacpModeACTIVE-lacpKey3-ifnumLA/3 set channel LA/2 -tagall ON setchannel0/LA/3 addroute0.0.0.00.0.0.0192.168.7.1-routeTypeSTATIC

setnsconfig-IPAddress192.168.7.12-netmask255.255.255.0 addvlan301-sdxVlanYES setinterface10/1-lacpModeACTIVE-lacpKey2-ifnumLA/2 setinterface10/2-lacpModeACTIVE-lacpKey2-ifnumLA/2 setinterface10/3-lacpModeACTIVE-lacpKey3-ifnumLA/3 setinterface10/4-lacpModeACTIVE-lacpKey3-ifnumLA/3 set channel LA/2 -tagall ON setchannel0/LA/3 addroute0.0.0.00.0.0.0192.168.7.1-routeTypeSTATIC



setnsconfig-IPAddress192.168.7.13-netmask255.255.255.0 addvlan301-sdxVlanYES setinterface10/1-lacpModeACTIVE-lacpKey2-ifnumLA/2 setinterface10/2-lacpModeACTIVE-lacpKey2-ifnumLA/2 setinterface10/3-lacpModeACTIVE-lacpKey3-ifnumLA/3 setinterface10/4-lacpModeACTIVE-lacpKey3-ifnumLA/3 set channel LA/2 -tagall ON setchannel0/LA/3 addroute0.0.0.00.0.0.0192.168.7.1-routeTypeSTATIC

InitialCreatedClusterConfig add cluster instance 1 addclusternode0192.168.7.11-stateACTIVE-backplane0/LA/3 addclusternode1192.168.7.12-stateACTIVE-backplane1/LA/3 addclusternode2192.168.7.13-stateACTIVE-backplane2/LA/3 bindclusternodegroupDEFAULT_NG-node0 bindclusternodegroupDEFAULT_NG-node1 bindclusternodegroupDEFAULT_NG-node2 enable cluster instance 1 addnsip192.168.7.14255.255.255.255-typeCLIP-vServerDISABLED-mgmtAccessENABLED addroute0.0.0.00.0.0.0192.168.7.1-routeTypeSTATIC

AddtrafficVLANtocluster addvlan401

Add spotted SNIPs: addnsip192.168.6.11255.255.255.0-vServerDISABLED-guiDISABLED-dynamicRoutingENABLED -ownerNode0 addnsip192.168.6.12255.255.255.0-vServerDISABLED-guiDISABLED-dynamicRoutingENABLED -ownerNode1 addnsip192.168.6.13255.255.255.0-vServerDISABLED-guiDISABLED-dynamicRoutingENABLED -ownerNode2

Add striped SNIP: addnsip192.168.6.14255.255.255.0-vServerDISABLED-guiDISABLED

BINDtrafficVLANtoclusterinterfacesandstripedSNIPaddress bindvlan401-ifnum0/LA/2-tagged bindvlan401-ifnum1/LA/2-tagged bindvlan401-ifnum2/LA/2-tagged bindvlan401-IPAddress192.168.6.14255.255.255.0

AddPBRforclusterbackplanetraffic addnspbrTraffic_PBRALLOW-srcIP"!="192.168.7.11-192.168.7.14-nextHop192.168.7.1-priority10 apply ns pbrs

ChangedefaultroutetousetrafficVLAN addroute0.0.0.00.0.0.0192.168.6.1-routeTypeSTATIC rmroute0.0.0.00.0.0.0192.168.7.1

Enable Dynamic Routing enable ns feature OSPF

Add Server: addserverHTTP-Test127.0.0.1



Create Service or Service Group: addserviceHTTP-Test-SvcHTTP-TestHTTP80-gslbNONE-maxClient0-maxReq0-cipDISABLED -usipNO-useproxyportYES-spON-cltTimeout180-svrTimeout360-CKANO-TCPBNO-CMPNO

Create LB vServer addlbvserverHTTP-Test-LBVSHTTP192.168.5.180-persistenceTypeNONE-cltTimeout180 -RHIstate ACTIVE

Bind Service to LB vServer bind lb vserver HTTP-Test-LBVS HTTP-Test-Svc

Enable Dynamic Routing on VIP for LB vServer addnsip192.168.5.1255.255.255.255-typeVIP-snmpDISABLED-hostRouteENABLED-hostRtGw 192.168.6.14-ospfLSATypeTYPE1-ospfArea51

ConfigureOSPFinVTYSH configt ! log syslog ! log record-priority ! interfacelo0 ! interfacevlan0 ! interfacevlan107 ipospfpriority0 !

router ospf 1 owner-node0 ospfrouter-id192.168.6.11 exit-owner-node owner-node1 ospfrouter-id192.168.6.12 exit-owner-node owner-node2 ospfrouter-id192.168.6.13 exit-owner-node auto-costreference-bandwidth100000 redistribute kernel passive-interfacevlan0 area 51 stub network192.168.6.0/24area51 ! exit ! writemem !



Appendix A: Cluster Troubleshooting

NetScalerprovidesseveraldiagnostictoolstoviewconsolemessages,eventmessagesanddownloadtraces.MostoftheseutilitiesareavailabledirectlyfromtheNetScalerGUI.Thissectionfocusesonexternaltools.

Nsclusterd: service daemon

Heartbeats:

NSIP:port7000UDPSenteveryhellointerval(default200ms),deadintervalis3secsARP resolution occurs on all UP interfaces, including the backplane interafces. If ARP is resolved, the cluster ARP cache is updated. HeartbeatsaresenttopeerviainterfaceswhereARPresolves.L3clusterusetheroutesforheatbeats,L2clusterfloodstheenabledinterfaces.

Cluster Communication:

UsesRPC(3011)orsecureRPC(3009)ClusterCommunicationhappensviabackplane.Ifthebackplaneisdown,anyotherinterfacewhereARPresolvesis chosen.

CLI Commands:

addclusterinstance<clid>-inc<ENABLED|DISABLED>-processLocal<ENABLED|DISABLED>addclusternode<nodeid><nodeip>-nodegroup<ng>addnspbr[-ownerGroup<string>]addnspbr6[-ownerGroup<string>]Showtechsupport–scopecluster

Counters:nsconmsg–dlogfromnfw(Logs)nsconmg–gclusterd_–gcl_-dcurrent(Counters)

Hearbeats:

ClusterARPcache(nsapimgr–yscall=nscl_ArpCacheDump)Trace.(UDPport7000)

NNM/config sync/propagation:

NodetoNodeMessagingforpacketenginecommunication• RPCNSIP:3011orsecureRPC3009

RPCnodemismatchbetweennodes(showrpcnode)Trace.(RPCport3011(non-secure)or3009(secure))nsapimgr–ystrace_nnm=1nsapimgr–dnnm_messagesnsconmg–gnnm–dcurrent



Cluster Counters:

clusterd_tot_cu_enqueuedcommand has been received by clusterd (incremented on CCO)clusterd_tot_cu_deliveredcommandisdeliveredtoconfigdbyclusterd(incrementedonallnodeswherecommandisdelivered)clusterd_tot_cu_timeoutpropagation timeout (incremented on CCO)

VTYSH:Ns#showiproutekernel• Showsallvipstobeadvertised

Ns#showiproute• ShowsbestroutesinRIB

Ns#sh ip interface brief• Showsallinterfacesandtheirprimaryipaddressalongwithup/downstatus

Ns#shownssync-status• Showslastsyncstatusns(config)#debugiminetscaler• Enableloggingofclusterevents,configcommandspropagationandsynchronizationevents:Alllogs willgoto/var/log/ns.logfile

ns(config)#debugnsmkernel• Logs all cluster state changes received by nsm module, route updates sent to or received from kernel

Tools

Wireshark:Wireshark(Version1.10.3andabove)isafreepacketsnifferandcapturetool;thatallowsfornetworktroubleshootingandanalysisinanetworkedenvironment.Wiresharkprovidesaverysimilarapproachtotcp-dumpbyusingaGUIfront-endtoviewnetworktraces.Wiresharkalsoprovidessophisticatedfilteringcapabili-ties;allowingnetworkadministratorstofilteronresponsecodes,HTTPmethodsandpayload.Wiresharkcanbedownloadedfrom:WireShark

NetScalerPluginforWireshark: http://engwiki.eng.citrite.net/mynswiki/NsWireshark

Putty:PuttyisafreetoolthatallowsWindowsuserstoconnecttoremotesystemsovertheInternetviaTelnetandSSH.WhilebothTelnetandSSHallowyoutoconnecttoremotesystems,SSH,supportedinPutty,providesfora"SecureShell",encryptinginformationbeforeitistransferred.Puttyistherecommendedtoolforconnect-ingviaSSHtotheNetScaler.Puttycanbedownloadedfrom:Putty

WinSCP:WinSCPisanopensourcefreeSFTPclientandFTPclientforWindows.WinSCPprovidesecuretransferoffilesbetweentheNetScalerandalocalhost.WinSCPprovidesagraphicalGUItothetraditionalSCP(securecopy)functionalityofferedby*NIXoperatingsystems.WinSCPcanbedownloadedfrom:WinSCP

PSCP:PSCPisthecommandlinealternativetoWinSCPthatisidenticaltothe“scp”functionalityofferedintraditional*NIXoperatingsystems.PSCPisalightweightDOSbasedapplicationthatcanbeusedtocopyfilesbetweentheNetScalerandthelocalhost.PSCPcanbedownloadedfrom: PSCP

https://www.citrix.com/blogs/2009/10/15/using-wireshark-to-analyze-netscaler-network-traces/

https://www.wireshark.org/download.html

http://engwiki.eng.citrite.net/mynswiki/NsWireshark

http://www.chiark.greenend.org.uk/~sgtatham/putty/

https://winscp.net/eng/download.php#download2

http://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html



Command Line Diagnostics

nstrace.sh:nstrace.shisaNetScalerutility(scriptfile)thatallowsnetworkadministratorstotakeNetScalertracesfromtheappliance.TheNetScalertracecapturesalltrafficgoingthroughtheboxatanygiventime.Thesyntaxfornstrace.shis/netscaler/nstrace.sh–sz0–tcpdump1.

Thissyntaxwillautomaticallycreateatracefileinthe/var/nstracedirectory.TheadministratorcanpressCtrl-Conhiskeyboardtostopthetrace.Thistracecanthenbedownloadedtoalocalhost(viaPSCP,WinSCP)andviewedonanypacketcaptureprogram(i.e.Wireshark).

nstcpdump.sh: The /netscaler/nstcpdump.sh script is a utility that emulates tcpdump syntax on NetScaler inter-faces.Themainbenefitusingnstcpdump.shincludesitsfilteringability.Usinganexamplebelow:/netscaler/nstcpdump.sh–w/var/nstrace/ftp.pcaphost192.168.1.1andhost192.168.1.2andtcp.port==21

TheabovefilterallowsforallFTPtrafficbetween192.168.1.1and192.168.1.2tobecapturedontheNetScaleranddownloadedto/var/nstrace/ftp.pcap.ThisfilecanbedownloadedtoalocalmachineandviewedonWiresharkfor easier analysis.

LocalSyslog:NetScalerstoresalllogfileslocallyontheapplianceunder/var/log/ns.log.DetailsregardingauthenticationerrorsorNetScalercanbeviewedbyrunningthefollowingcommand:

>shell

# cat /var/log/ns.log

tail–f/var/log/ns.log

Authentication Debugging Tools: NetScaler provides a debugging utility to check for authentication successes and failures. Group extraction can also be validated using this utility. User details including group extraction can beviewedonscreenbytypingthefollowingcommand,whileauserisloggingontotheVPN:

>shell

# cat /tmp/aaad.debug

Theoutputofthecommandwillprovidehelpfuldetailsontheauthenticationschemeused,success,failuresandcauseoffailures(i.e.incorrectpassword,badbindingsetc.).

ConsoleMessageDiagnostics:TheNetScalerprovidesusefulconsolemessagesthatcanshedlightonNetScal-erperformance.Forexample,usingthefollowingcommandbelowcaneasilyidentifyIPaddressconflictsandduplex mismatches:

>shell

#nsconmsg-Knewnslog-dconsmsg(live)

or

#nsconmsg-Knewnslog–K/var/nslog/newnslog-dconsmsg(fromlatestfile)

EventMessageDiagnostics:TheNetScalerprovidesusefuleventmessagesthatcanshedlightonstatusofconfiguredNetScalerservicesandhighavailability.Forexample,thestatusofaspecificconfiguredserviceornotificationofafailoverorarebootcanbeidentifiedbyrunningthecommandbelow:

>shell

#nsconmsg-Knewnslog-devent(live)

or

#nsconmsg-Knewnslog–K/var/nslog/newnslog-devent(fromlatestfile)



Appendix B: SVM Management

TheServiceVirtualMachine(SVM)andsubsequentManagementServiceshouldalwaysbeusedtoperformman-agementoperationsonNetScalerVPXinstances.OnceaNetScalerVPXinstanceisprovisioned,theManagementServicecreatespolicies,instanceadministrationprofiles,andotherconfigurationsontheVPXinstance(suchasCPU allocation, SSL chips, memory, etc.).

Aftertheconclusionofthisengagement,itwasfoundthattheNetScalerVPXinstanceseemedtobenameddifferentfromwhatwasdefinedontheSVM.AmemoryallocationchangeresultedintheSVMpushingdownthenamethatwasdefinedontheSVMandareboot(asexpectedtobringtheparametersintoeffect).Thiscausedamismatchwiththenameasdefinedinthelicensefileandthuspreventeduserconnections(CitrixTechnicalSupportCase#71253542).

WhileitispossibletoaccessmanyofthesamepropertiesdirectlyontheNetScalerVPXinstance,itisstronglyrecommendedtoonlyutilizetheSVMforconductingsuchchangesandconfiguration.

Appendix C: OSPF Command Reference

https://docs.citrix.com/content/dam/docs/en-us/netscaler/media/dynamic-routing-crgs/Citrix-ZebOSOSPFC-mdRef.pdf

https://docs.citrix.com/content/dam/docs/en-us/netscaler/media/dynamic-routing-crgs/Citrix-ZebOSOSPF

https://docs.citrix.com/content/dam/docs/en-us/netscaler/media/dynamic-routing-crgs/Citrix-ZebOSOSPF



About Citrix

Citrix(NASDAQ:CTXS)isaleaderinmobileworkspaces,providingvirtualization,mobilitymanagement,net-workingandcloudservicestoenablenewwaystoworkbetter.Citrixsolutionspowerbusinessmobilitythroughsecure,personalworkspacesthatprovidepeoplewithinstantaccesstoapps,desktops,dataandcommunica-tionsonanydevice,overanynetworkandcloud.ThisyearCitrixiscelebrating25yearsofinnovation,makingITsimplerandpeoplemoreproductive.Withannualrevenuein2013of$2.9billion,Citrixsolutionsareinuseatmorethan330,000organizationsandbyover100millionusersglobally.Learnmoreatwww.citrix.com.

Copyright©2014CitrixSystems,Inc.Allrightsreserved.Citrix,NetScalerMPX,NetScalerSDX,NetScaler,Cloud-BridgeandAppFlowaretrademarksofCitrixSystems,Inc.and/oroneofitssubsidiaries,andmayberegisteredin the U.S. and other countries. Other product and company names mentioned herein may be trademarks of their respective companies.

validated reference design netscaler sdx clustering - citrix.com · validated reference design...

Documents