xenapp load balancing

Load Balancing and Intelligent Load BalancingLoad Balancing and Intelligent Load Balancing

Ivan LorenteEscalation EngineerIvan LorenteEscalation Engineer

2© 2007 Citrix Systems, Inc. — All rights reserved

Objectives

• Deep understanding of load balancing

• Understanding Intelligent load balancing

• Typical scenarios

• Troubleshooting techniques


Agenda

Load Balancing on an Application Delivery Scenario

Architecture

Where can it go wrong?

Troubleshooting



Agenda item number 1

Architecture


Troubleshooting




Citrix® NetScaler®

Deliver WebApps

Citrix Presentation Server™

Deliver Windows Apps

Citrix Desktop Server™

Deliver Windows Desktops

Users AppsCitrix

EdgeSight™

Monitor Real-Time User Experience

Citrix WANScaler™

Accelerate Apps to Branch Offices

Citrix Access Gateway™

Enable Secure App Access



Users Apps

PS Farm


Architecture

Agenda item number 1Load Balancing on an Application Delivery Scenario


Troubleshooting

Architecture


Architecture

• Data Collector Updates

• Understanding Bias

• Load Balancing during logon

• Performance Data Helper

• Citrix and the PDH

• Load balancing in the Data Store


Architecture

Data Collector DCDynamic Store DS

LmsSS.dllIMA Subsystem

LMS20Rules.dllSystem counters

MFRules.dllMetaFrame counters

PDH.dllPerformance Data Helper


Data Collector Updates

• Connection logon or logoff

• Change in the load evaluator +/- 500

• DC(DS) is updated every 30 s

• If nothing changes in 1 minute the DC(DS) sends 40 byte ping, to make sure they are alive.

• If nothing changes in 5 minutes the DC(DS) request a server load update.


Understanding Bias

• Definition of Bias in Science

Bias is defined as the difference between the measurement result and its unknown 'true value'


Understanding Bias

queryds /table:LMS_ServerLoadTable

5: Scheduling

6: Page Swaps

7: Disk Data I/O

8: Disk operations

9: IP Range

a: Application user load

b: Server User load

1: CPU Utilization

2: Context switches

3: Memory Usage

4: Page Faults

Bias = 10000/(40 USERS) = 250 DEC = fa HEX

Load = 3E8/FA HEX = 4 USERS


Load Balancing during logon

IMA

Ser

ver

BCtxnotif.dll

LMSSS.dll

ICA Client

Data Collector

Server A

Server B

MFSrvSS.dll

Dynamic Store

Ser

ver

A

WEVENT_LOGON


Performance Data Helper

• The performance data helper (PDH) interface calls the registry interface to retrieve performance data

• Uses the PDH.DLL to access the PDH API

• http://msdn.microsoft.com/archive/default.asp?url=/archive/en-us/dnarperfmo/html/msdn_pdhlib.asp

• http://msdn2.microsoft.com/en-us/library/aa373083.aspx

http://msdn.microsoft.com/archive/default.asp?url=/archive/en-us/dnarperfmo/html/msdn_pdhlib.asp

http://msdn.microsoft.com/archive/default.asp?url=/archive/en-us/dnarperfmo/html/msdn_pdhlib.asp



• Code examples

http://msdn2.microsoft.com/en-us/library/aa373169.aspx

…

TCHAR szCounterPath[45] = TEXT("\\Processor(0)\\% Processor Time");

…

pdhStatus = PdhAddCounter (hQuery, szCounterPath, 0, &hCounter);

…




• The performance counters information is stored under:

• HKLM\SOFTWARE\Microsoft\WindowsNT\CurrentVersion\Perflib\009\Counter

• HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Perflib\009\Help

…

6

% Processor Time

8

…

…

7

% Processor Time is the percentage of elapsed time that the processor

spends to execute a non-Idle thread. It is calculated by measuring the

duration of the idle thread is active in the sample interval, and

subtracting that time from interval duration. (Each processor has an

idle thread that consumes cycles when no other threads are ready

to run). This counter is the primary indicator of processor activity, and

displays the average percentage of busy time observed during the

sample interval. It is calculated by monitoring the time that the service

is inactive, and subtracting that value from 100%.

9

…



• The same information can be found in the following files

• %Systemroot%\System32\Perfc009.dat

• %Systemroot%\System32\Perfh009.dat

• For instance those files will be used to recreate the performance counters again

• http://support.microsoft.com/kb/300956/en-us

http://support.microsoft.com/kb/300956/en-us



• Perfmon.exe reads the registry keys

• IMA reads the Perfc009.dat

• Only once when IMA is starting


Citrix and the PDH

• We also provide some performance counters

• %Systemroot%\inf\009\IMAService\imaperf.ini

• %Systemroot%\inf\009\MetaFrameXP\mfxpperf.ini


Citrix and the PDH

• HKLM\SYSTEM\CurrentControlSet\Services\IMAService\Performance


Citrix and the PDH

• List of counters that LMS20Rules.dll uses

• \\Processor(_Total)\\% Processor Time

• \\System\\Context Switches/sec

• \\Memory\\% Committed Bytes In Use

• \\Memory\\Page Faults/sec

• \\Memory\\Pages/sec

• \\PhysicalDisk(_Total)\\Disk Bytes/sec

• \\PhysicalDisk(_Total)\\Disk Reads/sec

• \\PhysicalDisk(_Total)\\Disk Writes/sec


Citrix and the PDH

• Same counter with different name depending on the OS version

• English W2K

\\System\\% Total Processor Time

• English W2K3\\Processor(_Total)\\% Processor Time


Load balancing - Data Store

012C Hex = 300 dec = 5 min

3C Hex = 60 dec = 1 min


Load balancing - Data Store




Architecture

Troubleshooting




• Black hole effect• Load Throttling - Intelligent Load Balancing

• Failing to read performance counters

• English OS with and MUI


Black hole effect

• Problem

• At Peak logon times a recently boot up server will get all connections and might become unresponsive

• Description

• When a server comes into a farm its load will be 0 and it will be getting all new connections to the farm. At peak logon times this could make the server unresponsive if overloaded


Black hole effect

• Solution

Load Throttling

???


Load Throttling

• Load Throttling uses Intelligent Load Biasing (ILB)

• HKLM\SOFTWARE\Citrix\IMA\LMS\UseILB

Default =1 (0,1)

• HKLM\SOFTWARE\Citrix\IMA\LMS\ILBMultiplier

Default =2 (2, 3, 4…)

Current Resolution Load += [(Max Load – Current Resolution Load) / ILBMultiplier]


Load Throttling - Low logon rate

• 2 server farm (server A and server B) using the Default load evaluator



ICA Client

Data Collector

Server A

Server BDynamic Store

Server B

Server A

Server Load BIAS

0

0 0

0(10000-0)/2+0 = 5000100

(10000-0)/2+0 = 5000100



• Server load if Low logon rate

0

1000

2000

3000

4000

5000

6000

ILB=1

ILB=0


Load Throttling - Black hole effect

• 2 server farm (server A and server B) using the Default load evaluator

• Server B is running normally (60 users) while at Peak logon times the server A is booted up


Load Throttling - Black hole effect

ICA Client

Data Collector

Server A

Server BDynamic Store

Server B

Server A

Server Load BIAS

0

6000 0

0(10000-0)/2+0 = 5000(10000-5000)/2+5000 = 7500

(10000-6000)/2+6000 = 8000

(10000-7500)/2+7500 = 8750

x60


Load Throttling

• Server load at Peak logon times

0

2000

4000

6000

8000

10000

12000

1 8

15

22

29

36

43

50

57

64

71

78

85

92

99

ILBMultiplier=2

ILBMultiplier = 10

ILBMultiplier = 20

ILB=0


Failing to read performance counters

• Problem

One server in the farm gets all the connections

• Cause

If a farm is balanced using advance load evaluators and we fail to read those performance counters in one server, it will report 0 load and so it will get all the connections and will get unresponsive, even crashing eventually


Failing to read performance counters

• Solution

• From CPS4.0 HRP02 we have change the way the load is calculated

• If we fail to read some performance counters we will report full load

Note: If you were using custom load evaluators like Advance + Default after HRP02 you might report 10000 load while with HRP01 seemed to be OK


English OS with and MUI

• Problem

Advance load evaluators report full load when an MUI is installed

• Cause

We were expecting English counters only

Note: To know if your problem is due to the MUI and not to a problem reading performance counters you can swap the files perfc009.dat and perfc007.dat. If that resolves the issue you are in the right direction


English OS with and MUI

• Solution

• %Program Files%/citrix/system32/citrix/ima/lms/lms20rules.dll

• HKLM\Software\Citrix\IMA\LMS\EnableTranslation=1 (dword)

• To be manually added

• First introduced in PSE400R02W2k3016


Architecture


Architecture


Troubleshooting


Troubleshooting

• Failing to read performance counters => full load

• Use filemon while rebooting IMA to see possible problems accessing perf*.dat files. (leave it running for a while if they don't come up)

• Consider rebuilding performance counters

• Check that no performance counters are disabled

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfDisk\Performance]

"Disable Performance Counters“ = 1 (DWORD)


Troubleshooting

• Failing to read performance counters => full load

• CDF traceslms20rules ERROR!!!!! _CreatePdhQueryHelper() failed for

m_hDiskWriteCounter,error = -1073738823

lms20rules LMSRuleDll_Interface::CreatePDHQuery() ERROR!!!!! rc =

-1073738823

• -1073738823 in HEX is FFFFFFFFC0000BB9


0xC0000BB9 (PDH_CSTATUS_NO_COUNTER) The specified counter could not be found.


Summary

• Deep understanding of load balancing• BIAS, PDH…

• Understanding Intelligent load balancing• Load Throttling

• Typical scenarios• Full load versus 0 load, MUI…

• Troubleshooting techniques• So far we are resolving all the problems with those tips


Q&A

?

xenapp load balancing

Documents

load balancing data

pdh load balancing

bias load balancing

agenda load balancing

load evaluator

server user load

server load update

fa hex load