netscaler vpx implementation and troubleshooting

SUM307: NetScaler VPX Implementation and Troubleshooting Harvey Miller Senior Escalation EngineerMay 11, 2010

Citrix Confidential - Do Not Distribute

Introduction to NetScaler VPXInstallation and LicensingTroubleshootingUse CasesCitrix Confidential - Do Not DistributeAgenda


Introduction to NetScaler VPX


App delivery without the expensive tin- Maxwell Cooter, TechWorld



Virtual NetScaler applianceHardware requirementsHypervisors SupportedDifferences between VPX and hardware

Citrix Confidential - Do Not DistributeIntroduction to NetScaler VPX


XenServerXenCenterXenConvert

Citrix Confidential - Do Not DistributeXenServer components


XenServer Architecture

Xen Hypervisor

DomU

Dom0

NS VPX

LinuxDriversPV DriversClientServerL2 /L3eth1eth0L2/L3

DomU

Guest OSCitrix XenServer

Citrix Confidential - Do Not DistributeVM MechanismBinary Translation Sensitive/Privileged CPU Instructions are replaced with hypervisor code or calls on the flyAdvantages: Unmodified Guest OS, No special hardwareDisadvantages: Performance

Paravirtualization The Guest OS is modified so that privileged/sensitive instructions are replaced with calls to the hypervisorAdvantages: Performance, no special hardware required, relatively easy for hypervisorDisadvantages: Guest OS must be modified

Hardware assisted Sensitive/Privileged CPU Instructions executed by the Guest OS trap out to the hypervisorEach processor vendor brands and implements this differently.Intel: VT-xAMD: AMD-VAdvantages: Unmodified Guest OS, Relatively easy for hypervisorDisadvantages: Special hardware required


Hypervisor Architecture - XenServer

Xen Hypervisor

DomainU

DomainU

Domain0XenCenterClient

GuestOS2

NS VPX

LinuxXen daemon(s)DriversPV DriversXen Tools Bare metal. Managed by Domain0 Domain 0 manages network and storage I/O of guest VMs Hardware drivers run in Domain0 Paravirtualized Guest OS or hardware assist VMs only no Binary Translation



Xen Hypervisor NS VPX: InternalsDom0 (Linux)DomU (NS VPX )

CPU SchedulerMemoryVirtual CPU Virtual CPU Virtual Memory Virtual Memory Xen ToolsXen daemon(s)Citrix Confidential - Do Not Distribute


Virtual hardware assistCPUsMemoryNICCitrix Confidential - Do Not DistributeHardware requirements


Differences between VPX and hardwareFeatures of VPXFeatures of HardwareCitrix Confidential - Do Not Distribute Tagged VLANs not supportedNo LACP No hardware assists No nCore support (yet)Only version 9.1 & up

Full L2 support Hardware assists nCore with MPX modelsAll versions compatible with hardware


Installation and Licensing


Setting the VM memory and VCPUs for the NetScaler VPX


Citrix Confidential - Do Not DistributeInstallation


Identifying the VPX from the CLI and GUI


1 Mbps20 Mbps1000 MbpsStandardEnterprisePlatinumhttp://support.citrix.com/article/ctx122426

Citrix Confidential - Do Not DistributeNetScaler VPX


Free license Unlimited VMsXenConvert (P2V and V2V)Centralized multi-server manangement consoleLive motionVirtual infrastructure patch managementIntelligent server maintenance mode

Citrix Confidential - Do Not DistributeXenServer licenses


Citrix Essentials

High AvailabilityAdvanced server virtualization managementIntelligent workload placementRapid provisioning of new VMs

Citrix Confidential - Do Not DistributeXenServer licenses (continued)


Troubleshooting


Issues dealing with NetworkingTracingLog file locations

Citrix Confidential - Do Not DistributeTroubleShooting VPX


CPU from XenCenter is 100%VPX shows lower valuesWhich is right?Citrix Confidential - Do Not DistributeWhy is my CPU so high?


Citrix Confidential - Do Not DistributeXenCenter View


Citrix Confidential - Do Not DistributeNetScaler VPX view


Lack of proper license while adding SSL certificates


NetScaler VPX missing a valid license

Some examples of problems with the license file(s)The shell command `cat /var/log/license.log` reveals a missing license:The shell command `cat /var/log/license.log` reveals an invalid license:The shell command `cat /var/log/license.log` reveals an expired license:


Using Tagged VLANs With the NetScaler VPXCitrix Confidential - Do Not DistributeXenServer Host (Dom0)Virtual Machines (DomU)Trunk Port(tagged VLANs)ExternalSwitchVirtual SwitchesVlan 1Vlan 53Vlan 128NetScaler VPXUntaggedVLANs1/21/31/4Virtual Interfaces


Identifying the NetScaler VPX interfaces


Reboot messages in the logsCitrix Confidential - Do Not Distribute


Use Cases


Lab EnvironmentProof of ConceptSeparation of trafficReal world simulationCitrix Confidential - Do Not DistributeUse Cases


Real World SimulationCitrix Confidential - Do Not DistributeVirtual NetScalersVirtual ServerVirtual RouterNorth AmericaEuropeAsiaAfricaHyperVisorReal World


NetScaler Licensing: CTX122426NetScaler VPX 9.1 FAQ CTX12191NetScaler setup and configuring CTX124306How to video: importing and configuring CTX122721Importing VPX on ESX CTX123683Support.citrix.com (search for VPX)ForumsCitrix Confidential - Do Not DistributeAdditional Resources


Before you leaveSession surveys are available online at www.citrixsummit.com starting Thursday, May 13Provide your feedback and pick up your complimentary Starbucks or iTunes giftcard at the registration deskDownload presentations starting Friday, May 21, from your My Schedule Tool located in your My Synergy Microsite event account

Questions/Comments?Citrix Confidential - Do Not Distribute



Hello, and welcome to NetScaler VPX Implementation and Troubleshooting. My name is Harvey Miller, and Im a senior escalation Engineering from the Santa Clara office. Ive been working for Citrix for the past 3 years, and with supporting Network appliances for 5. My colleague, Gregor Visconty is a TRM based in the Redmond Washington office. He has been working for Citrix for the past 2 1/2 years, and has been supporting networks and application delivery for the last 12.

*Have you ever made an seemingly innocuous change that brought down production resources? Or needed to validate that a particular change would work as needed without impacting production? Sometimes simple changes can have unintended consequences, but having enough hardware dedicated to a lab is sometimes not feasible. Well talk today about a way to enable you to validate configuration changes without any impact to your production systems.

Well take a look at several aspects of the NetScaler including what the product is and can do, how to license it, and how to troubleshoot any issues that might arise. Well also look at some good use cases for the product, based on some customer requirements, and dealing with some of the issues Ive touched upon. We feel that customers can really benefit from using the VPX product. One use case that I find exciting was done for a customer that needed to recreate the world for testing. Well talk about that in detail later today.

*First, lets make sure everyone understands what VPX is, so that were all on the same page. Well go a bit into the architecture of XenServer and NetScaler VPX to give you a better understanding of how they fit together.

*This is an apt description of NetScaler VPX. The product is software only, so you provide your own hardware. As a software-only product, we can deliver this at a fraction the cost of hardware solutions, and still keep most of the functionality of a hardware NetScaler.

*NetScaler VPX is software that runs under XenServer, VMWare, or soon, HyperV that emulates a hardware NetScaler. Naturally some minor compromises have had to be made, but almost all the functionality of a hardware NetScaler works in the virtual appliance.Hardware requirements include Virtual Machine support in the CPU and BIOS, enough memory for all the planned virtual machines, and a large hard drive. As I said before, we currently support our own XenServer as well as VMWare and Microsofts HyperV.

*Id like to take a minute to go over some terminology used for XenServer and its component parts, since Ill be using these terms in the rest of the presentation. XenServer is the hypervisor, which is the native OS of the hardware. Its responsible for any IO as well as managing the guest Oses running. XenCenter is the GUI that allows configuration and access to the virtual machines running. And XenConvert will allow the conversion of a running system to a file suitable to import into a XenServer, or from one virtual format into another. For example XenConvert allows the conversion of a VMWare virtual system into a XenServer format. While well be using XenServer for purposes of this presentation, the other hypervisors have similar functionality.

*XenServer software will be the native OS that runs at the bare hardware level. Once XenServer is installed, virtual machines can be defined to run on top of XenServer. In the graphic, Dom0 is the virtual machine that will be in charge of the hardware and controlling the other guest machines.

*There are 3 methods to provide virtual machines. You see the general descriptions here. Binary translation is the brute-force method of watching the instructions that need to be executed, and dealing with them as necessary. Naturally this is very slow. Paravirtualization can be used if the base operating system is aware that it is running as a virtual machine, and has adapted to the environment. So drivers that normally would interact with real hardware understand that they need to invoke the hypervisors drivers instead. Finally we have hardware assisted virtual machines. This allows the guest operating system to run unmodified, but have the hardware trap out instructions that need to be virtualized.*Here we can see the XenCenter client communicates with the Xen daemon running on a Linux VM on Domain0. The Linux VM on Domain0 manages the hypervisor, as well as manages the network and storage I/O of the guest VMs. Based on the resource requirements and the available resource capacity of the XenServer, other virtual machines, including other NetScaler VPX virtual machines, can be running on the same XenServer

From an architectural point of view, we see NetScaler VPX runs as a paravirtualized vm. Its drivers have been written to send hardware requests to dom0s backend drivers which will get the IO requests finally to the hardware. This allows us to keep performance as close to native as possible.*Lets dive into the hardware requirements for a XenServer in a bit more detail. Both AMD and Intel have their recent processors like the Intel i5 and i7 or AMDs Opteron line that support virtual assist mode. This is called Intel VT or AMD-V, and more information can be found on the Intel and AMD websites on which processors include this. Note that the CPU must be a 64-bit processor. In addition, the BIOS for the system needs to also support the Virtual assist mode. As a practical matter, most machines manufactured recently should satisfy this requirement.Since there could be multiple virtual machines running simultaneously, a CPU with multiple cores is recommended. For even modest configurations with NetScaler VPX and a few Windows machines, for example, a 4 core system would be needed. For the ability to run more virtual systems simultaneously, you would need additional cores.Each virtual system will have dedicated memory, so plan to have as much memory as needed for all the systems to be run simultaneously. We recommend that each VPX system running have a dedicated core, and the hypervisor itself should have its own core as well.In practice a small lab system could be configured with a 4-core system with 8GB of memory and a 1TB hard drive. The resources dedicated to the NetScaler would typically be 1 CPU, and at least 1GB of memory. The default install of NetScaler VPX generates a 20GB drive. With the 1GB of dedicated memory, packet processing will have a bit under 600MB to use. For environments that need more memory for the NetScaler, this can easily be adjusted, so plan the hardware memory size accordingly.At least a 1 Gbps NIC is required, but we recommend two.*While the vast majority of the functionality of a hardware NetScaler is supported, there naturally are a few things that are different between the virtual and real implementations of NetScaler.

Tagged VLANs and LACP channels are not supported, but can be defined at the hypervisor level. Well see what that looks like in the troubleshooting section. Essentially since the interfaces are not real, the layer 2 support has to be done by the hypervisor. So some of the settings for interfaces arent allowed.

SSL cards to accelerate the encryption and decryption are naturally not available, since the hardware isnt present. However since the encryption and decryption can also be done in the software, the only impact is that SSL transactions flowing through the unit will be somewhat slower compared to using a hardware NetScaler.

NetScaler VPX only supports one CPU currently, so we have the equivalent of the classic builds, but nCore VPX could be available in the future.

Each hardware model has a minimum version and build that are required for it to work properly. With the VPX the minimum version is 9.1.*Now lets see what it takes to install NetScaler VPX on your hypervisor (using XenServer as the example). Well also show the different licensing options for NetScaler VPX and XenServer.*Here we see the interface from XenCenter to import the NetScaler VPX virtual machine. Once you have the VPX XVA file downloaded from Citrix, installation is a simple matter of importing it*And then defining the characteristics you want for the machine. Choose the XEN home server, storage repository, and the interfaces. The imported VM will be available to be started after a few minutes. Then proceed with the configuration steps as you would for a hardware NetScaler. Give it the NSIP, mask, configure the appropriate SNIPs and any resources. Copy the appropriate license file to /nsconfig/license and reboot. The system will then be available to configure using the normal GUI or CLI commands youre used to.*This is where we can set the memory and VCPUs for the NetScaler VPX in XenCenter. Note that 1GB is the minimum RAM setting for NetScaler VPX, and although we could add more CPUs, NetScaler VPX wont use them at this time. Its a simple matter to increase the RAM if more is needed.*This is the console view you will see when the NetScaler is up and logged in. You can use this as the equivalent of the hardware console, and comes in handy for command line processes. Naturally you can use puTTY or another SSH client to access the CLI through the network once the NSIP is established, but when the system is first installed the console is the preferred way to initially configure it.*And the management GUI looks very much like the GUI for other 9.x versions. Note the NetScaler VPX title to distinguish this as the VPX version of the code. At this point you can do whatever configuration you need to using either the GUI or the CLI. Notice that the interface is exactly what you would see from the a hardware system running the same build.*The CLI of show hardware and the GUI equivalent found by selecting the System folder shows that this is a NetScaler VPX rather than a hardware appliance. In the case of the GUI, you can also see that NetScaler VPX is in the title bar. In the case of the CLI, the prompt is configurable by the user, and in this case it has been set to NetScaler VPX*Standard, Enterprise, and Platinum licenses are available to determine the set of features that are available. A license for one of the family edition licenses is necessary for proper activation of the system. In addition to the features, there are 3 levels of bandwidth depending on your requirements. A license will be the combination of the bandwidth supported along with the set of features. The NetScaler license can be downloaded from the mycitrix site, and will require the MAC address of the first interface. This process is fully explained in KB article CTX122426. You will need to remember that the license will be tied to that MAC, so if the MAC changes you will need to either adjust it, or get a new license for the new MAC address. The lmhost command will give you the information you need to complete the licensing. This is explained quite well in the KB article, however the lmhost command essentially identifies the appropriate interface and pulls its MAC address.*The basic XenServer license is free, and includes unlimited VMs, using Windows and Linux guests. Physical 2 Virtual and Virtual 2 Virtual conversion is done using XenConvert. I wont read the list of features, but certainly with the free XenServer license you have a fully functional virtual environment that can be used in a number of useful ways.*The basic XenServer license is free, and includes unlimited servers, VMs, using Windows and Linux guests. Physical 2 Virtual and Virtual 2 Virtual conversion is done using XenConvert. Beyond the VMWare ESX features Citrix XenServer also includes a centralized multi-server management console, live motion, virtual infrastructure patch management and intelligent server maintenance mode. For a more robust implementation, Citrix Essentials adds HighAvailability, advanced server virtualization management, intelligent workload placement and rapid provisioning of new virtual machines.*Now well turn to see how to troubleshoot any issues that can arise with a VPX system. Although the basic techniques for troubleshooting are the same in VPX as in a hardware appliance, there are some issues unique to the virtual environment.*Getting the networking right is probably the trickiest part of defining a virtual machine. The interfaces can be either virtual or real and can be exposed outside the appliance, or strictly within the hypervisor. Note that the license will be tied to one of the MAC addresses, so if the MACs change, or the order changes, the license could become invalid. If that happens youll notice that the features that were enabled will no longer be enabled. Well see an example of this later.In general troubleshooting an issue on a VPX appliance is the same process as on real hardware NetScalers. If the issue involves the need to examine network packets, a trace can be taken just as on a real NetScaler. All the logs are in the same places, and have the same types of messages. Lets look at some types of issues that are unique to the VPX environment.*Lets take a look at a CPU issue that has come up.

When the VPX system is imported into XenServer, the XenServer view shows the CPU as always being 100%, but looking at CPU from the perspective of the NetScaler we see more normal values. So who is right, and why is this happening? First lets take a quick look at what this looks like in XenCenter.*Notice that XenCenter thinks the CPU is pegged, and the value never drops from 100%*However from the NetScalers perspective (in this case from the monitoring view), we see that the CPU is a changing, more realistic value. Given that the NetScaler in this case is not really passing much traffic, wed expect to see that its not really running at maximum capacity.Whats happening is that XenCenter is reporting what it thinks the underlying OS CPU is using. Since the NetScaler is using all spare cycles for polling, XenCenter is reporting a false value. This has been corrected in the later VPX builds, but you might see it if you are running an older build, as in this example.In addition to using the monitor view, we could also see the CPU from newnslog, SNMP, or the stat cpu detail command.*Here is another case that some have run into.

When you attempt to add an SSL certkey greater that 512 bytes, you are unable to, and receive the error Certificate with key size greater than RSA512 or DSA512 bits not supported. This can happen for two reasons; you are running the beta release of the NetScaler VPX, or the VPX license is missing or invalid. In the former case, you must upgrade the VPX to a post-beta release. In the latter case, you must install a valid license on the NetScaler VPX. Lets look a bit further at what is happening to determine why the license might not be correct.*Here is the CLI output of `show license` and the GUI version of the same. Notice that Web Logging, SSL Offloading, Dynamic Routing, Access Gateway, Rewrite, and Responder are all shown as licensed. This doesnt indicate that a valid license has been loaded, but is actually a symptom of a system without a license. In fact the system can be run without a license, but itll show strange symptoms. For example even though Access Gateway is licensed, youll find that it wont really respond to a configured AGEE VIP.*The shell command `cat /var/log/license.log` can be used to determine the problem with the VPX license. In the first case, we can see that the license file, which should be in the /nsconfig/license directory is missing, so check for the existence of that file. The file name is arbitrary, but the file extension must be .lic, so make sure that the file exists and that the extension is correct. In the second case, we see that the license file was generated with the wrong hostid, so it is invalid. In this case, a new license file must be created and installed. This could have happened if the MAC address of the first interface has changed, as we noted earlier. In the last case, we can see that the license file has expired, so check the system clock and the expiration date of the license file. Remember that whenever a license file is modified or added, the VPX must be rebooted for the changes to take effect. *One issue that can be a bit confusing in the VPX environment is how to deal with tagged VLANs. VPX doesnt support tagged VLANs, so youll have to work around that by using the hypervisors VLAN support.

This diagram shows how an environment that needs to have tagged VLANs could be set up. Note that the NetScaler can have untagged VLANs, but the tagged VLANs would need to be set up between the external network and the hypervisor. The box in dark blue represents the VPX virtual machine. In this example, we have 3 interfaces, all defined on different VLANs, which are untagged to the virtual switches in the light blue box. The untagged VLANs are carried through to the hypervisors interface, where it will tag the packets going to the external switch.*The top window is the Network tab of the VPX window in XenCenter. The bottom window is the NetScaler VPX CLI output of `show interfaces`. Match the Device number in the VPX Network configuration with the interface device number shown at the end of the first line of each interface in the output of the CLI command `show interfaces`. This will help you ensure that you have bound the correct VLANs and IPs to the correct virtual interfaces in the VPX, which will map to the correct VLANs and physical interface(s) on the XenServer host.*In the case of a hardware appliance or the VPX, when the NetScaler is rebooted through the shell or NSCLI, once the NetScaler has completed the reboot, you can see details about the reboot command that was issued. This can be seen in the file /var/log/ns.log in the case of a reboot issued from the NSCLI, and in the file /var/log/messages in the case of a reboot issued from the shell.*However, if you reboot the VPX through the XenCenter console, there will not be reboot messages in the VPX logs, but will be in the Xen logs.*In this case, the reboot can be seen in the file /var/log/xensource.log on the XenServer, or through the logs tab of the virtual machine through XenCenter.*What sort of cases can the virtual NetScaler be used for? Lets examine some possibilities.*The NetScaler VPX is a natural for a lab environment. You can take your production configuration with minimal changes and build a robust place to test configuration changes, upgrades, or examine how things interact. Along those same lines, a quick proof-of-concept can be built to configure a new feature or web site and determine the optimal settings in an environment that wont impact the production use of your servers.With the low cost of a XenServer/NetScaler VPX system, customers can use these appliances to separate special types of traffic. For example if you have many VIPs on one NetScaler, but one or more has special characteristics, a new NetScaler VPX system could be deployed just for that traffic. That would allow more flexibility to tune the configuration to the specific needs of the traffic. So global settings could be different as necessary.

*Id like to talk about a situation we had recently with a customer who had very specific requirements for the GSLB site returned for their customers throughout the world. They had GSLB sites scattered around the world, so it was important that a user would reliably be directed to the site closest to them. Additionally they required that when sites or resources from a particular site were unavailable, the user would be directed to the next closest site. However since the production environment couldnt be impacted, and testing different scenarios would be impossible, we turned to XenServer and NetScaler VPX to encapsulate the entire world.We set up XenServer with 4 GSLB VPX appliances, and 1 Linux-based router. The production configurations were used with minimal changes and put on the various VPX systems. All the VPX systems used the Linux router as their default gateway to complete the picture. The router was configured to route all the traffic in this virtual world to the appropriate interfaces. Scripts were defined to simulate down circuits or systems to evaluate what impact that had on the load balancing decisions from around the world. DIG command simulates DNS requests from various locations, and was used in the scripts to validate the GSLB DNS decisions that were made in the normal case as well as when specific parts of the environment were unavailable. Notice that all the traffic weve talked about is contained within the XenServer, so there was no possibility of any testing to impact their production. Talk about a virtual world! *Here Ive highlighted a few of the important Knowledge Base articles that are available from support.citrix.com. In addition to these, there is a wealth of information in the product documentation that comes up on a quick search of the site. There are forums specifically devoted to NetScaler VPX. While some of the threads are dealing with beta code, there is still a lot of good information in the forums. Questions can be quickly posted to get other opinions in the community and perhaps a quick answer.The first article goes into detail to walk you through getting your NetScaler license from mycitrite, including retail, partner and evaluation licenses (90 days) as well as the free NetScaler VPX Express license for standard edition, 1Mbps bandwidth.The FAQ answers many of the questions that have come up over the past few months. It deals with licensing, XenServer and VMWare ESX, and talks about what can and cant be done. This is a good article to read through to see if your plans are reasonable or not.CTX124306 is a video showing the installation of XenCenter followed by importing a VPX image, and then doing the initial configuration of the NSIP, SNIP, and one VIP. This is the first video of a 3 part series.Finally CTX123683 shows the correct way to import a VPX image into ESX.

*That concludes my prepared comments. Please feel free to ask questions now, or find me in the hall or booth afterwards.*

netscaler vpx implementation and troubleshooting

Documents

distributecitrix confidential

hardwarecitrix confidential

licensingcitrix confidential

techworld citrix confidential

unmodified guest os

netscaler vpx implementation

netscaler vpxinstallation

hardware assist vms