Load Balancing and Intelligent Load BalancingJesús González Escalation Engineer

Objectives

• Deep understanding of load balancing architecture

• Troubleshooting techniques

• Identify the root cause of any LB problem

Agenda

• Architecture• Data Collector

• Updates - Examples• Performance Data Helper• Citrix and the PDH

• Where can it go wrong?• Black hole effect (Load Throttling - Intelligent Load Balancing)• Some servers get all the connections• Full load after installing an MUI

• Troubleshooting

Architecture

Architecture

Data Collector DC - Dynamic Store DSData Collector DC - Dynamic Store DS

LmsSS.dll - IMA SubsystemLmsSS.dll - IMA Subsystem

MFRules.dll - XA countersMFRules.dll - XA countersLMS20Rules.dll - System countersLMS20Rules.dll - System counters

PDH.dll - Performance Data HelperPDH.dll - Performance Data Helper

Data Collector Updates

• Data Collector is updated every 30 s

• Only when the change in the load evaluator is bigger than 5%• Every 5 minutes we send a full update

• Connection logon or logoff

Data Collector UpdatesServer load - CPU Load example

• Data Collector

XenApp

Time

30s 60s

15 20 23 25 23 50 98 34 45 56

X 20 23 25 23 50 98 34 45 56 40

X 23 25 23 50 98 34 45 56 40 12

38 41 40Server Load % :

5 minutes

Data Collector UpdatesXA CPU Load != Task Manager CPU Load

• CPU load

• XA = 25%

• Task Manager = 1%

Ctxnotif.dll

MFSrvSS.dll

LMSSS.DLL

Data Collector UpdatesAt Logon time - Default load evaluator

Data Collector

ICA Client

Dynamic Store

Server A

Server B

IMAS

erve

r B

Ser

ver

AS

erve

r A

BIAS

WI/XML

IMA

X

Performance Data Helper – API

The performance data helper interface calls the registry interface to retrieve performance data

Uses the PDH.DLL to access the PDH API

http://msdn2.microsoft.com/en-us/library/aa373083.aspx

Performance Data Helper – C++ Example

pdhStatus = PdhAddCounter ( hQuery,

"\\Processor(0)\\% Processor Time",

0,

&hCounter);

YOU MUST CALL THE PERFORMANCE COUNTER BY ITS NAME

Citrix and the PDH

• \\Processor(_Total)\\% Processor Time

• \\System\\Context Switches/sec

• \\Memory\\% Committed Bytes In Use

• \\Memory\\Page Faults/sec

• \\Memory\\Pages/sec

• \\PhysicalDisk(_Total)\\Disk Bytes/sec

• \\PhysicalDisk(_Total)\\Disk Reads/sec

• \\PhysicalDisk(_Total)\\Disk Writes/sec

Performance Data Helper - Registry

Performance Data Helper – File system

EnglishPerfc009.dat

Perfh009.dat

GermanPerfc007.dat

Perfh007.dat

Performance Data Helper - perfmon

Performance Data Helper - IMA

IMA(At start up)

IMA(At start up)

Citrix and the PDH

We also provide Performance counters

Citrix and the PDHHKLM\SYSTEM\CurrentControlSet\Services\IMAService\Performance

Where can it go wrong?

Black hole effect

Problem Solution

• At Peak logon times a recently boot up server will get all connections and might become unresponsive

• Cause: Server is unable to update the DC

• Load Throttling

• Intelligent load balancing

HKLM\SOFTWARE\Citrix\IMA\LMS\

(DWORD ) UseILB = 1

(DWORD ) ILBMultiplier = 2

Before Intelligent Load Biasing

BIASBIASCurrent Load (0)

Max Load (10000)

0 sessions

Current Load (100)

Max Load (10000)

1 session

BIASBIAS

Max Load (10000)

2 sessions

Current Load (200)

1 session comes 2nd session comes

Default BIAS = 10000/100 = 100

Intelligent Load Biasing

BIASBIAS

Current Load (0)

Max Load (10000)

ILBMultiplier (2)

0 sessions

Current Load (5000)

Max Load (10000)

1 session

BIASBIAS

Max Load (10000)

2 sessions

Current Load (7500)ILBMultiplier (2)

1 session comes 2nd session comes

Load = [(Max Load – Current Load) / ILBMultiplier] + Load

Load Throttling- Low logon rate

Data CollectorICA Client

Dynamic Store

Server A

Server B

Server B

Server A

Server Load BIAS

0

0 0

0(10000-0)/2+0 = 5000100

(10000-0)/2+0 = 5000100

Load Throttling- Low logon rate

Server load if Low logon rate

0

1000

2000

3000

4000

5000

6000

ILB=1

ILB=0

Load Throttling- Black hole effect

Data CollectorICA Client

Dynamic Store

Server A

Server B

Server B

Server A

Server Load BIAS

0

6000 0

0(10000-0)/2+0 = 5000(10000-5000)/2+5000 = 7500

(10000-6000)/2+6000 = 8000

(10000-7500)/2+7500 = 8750

Load Throttling

Server load at Peak logon times

0

2000

4000

6000

8000

10000

12000

1 8

15

22

29

36

43

50

57

64

71

78

85

92

99

ILBMultiplier=2

ILBMultiplier = 10

ILBMultiplier = 20

ILB=0

Upgrade to W2K3 causes full load

Problem Solution• English W2K

\\System\\% Total Processor Time

• English W2K3

\\Processor(_Total)\\% Processor Time

• In W2K3, use

\\Processor(_Total)\\% Processor Time

Some servers get all the connections

Problem Solution

• 1 server all the connections

• Cause: Fail to read performance counters causes Load = 0

• Fail to read performance counters causes Load = 10000

Full load after installing an MUI

Problem Solution

• After installing MUI with advance load evaluators it results in load = 10000

• Cause:

-> perfc007.dat (German) perfc009.dat (English)

• HKLM\Software\Citrix\IMA\LMS\

EnableTranslation=1 (dword)

Full load after installing an MUI

• Jesús González

in a German restaurant

English Menu (perfc009.dat)

22 . Vegetable Soup

German Menu (perfc007.dat)

22. Gemüsensuppe

Troubleshooting

Troubleshooting

Data Collector DCData Collector DC LmsSS.dllLmsSS.dll

LMS20Rules.dllLMS20Rules.dll

PDH.dllPDH.dll

\\Processor(_Total)\\% Processor TimeXXXXXXXXXXXXXXXX

Full Load

TroubleshootingFailing to read performance counters => full load

• Use procmon (filemon) while restarting IMAMake sure the correct perfcXXX.dat file can be accessed.

• Consider rebuilding performance countershttp://support.microsoft.com/kb/300956/en-us

• Check that no performance counter are disabled[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfDisk\Performance]"Disable Performance Counters"=dword:00000001

• Use CDFControl to gather CDF traceshttp://support.citrix.com/article/CTX111961

•LMSRuleDll_Interface::CreatePDHQuery() ERROR!!!!! rc = -1073738823

•-1073738823 -> FFFFFFFFC0000BB9 (HEX)

•http://msdn2.microsoft.com/en-us/library/aa373046.aspx

“The specified counter could not be found”

TroubleshootingCDF Traces example

Summary

• Deep understanding of load balancing• BIAS, PDH…

• Understanding Intelligent load balancing• Load Throttling

• Typical scenarios• Full load versus 0 load, MUI…

• Troubleshooting techniques• So far we are able to resolve all the problems that arrive to EMEA tech support with those tips

Before you leave…

• Session surveys are available online at www.citrixsynergy.com starting Thursday, 7 October• Provide your feedback and pick up a complimentary gift card at the registration desk

• Download presentations starting Friday, 15 October, from your My Organiser Tool located in your My Synergy Microsite event account