load balancing and intelligent load balancing

37
Load Balancing and Intelligent Load Balancing Jesús González Escalation Engineer

Upload: armando-carver

Post on 31-Dec-2015

124 views

Category:

Documents


0 download

DESCRIPTION

Load Balancing and Intelligent Load Balancing. Jesús González Escalation Engineer. Deep understanding of load balancing architecture Troubleshooting techniques Identify the root cause of any LB problem. Objectives. Architecture Data Collector Updates - Examples Performance Data Helper - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Load Balancing and Intelligent Load Balancing

Load Balancing and Intelligent Load BalancingJesús González Escalation Engineer

Page 2: Load Balancing and Intelligent Load Balancing

Objectives

• Deep understanding of load balancing architecture

• Troubleshooting techniques

• Identify the root cause of any LB problem

Page 3: Load Balancing and Intelligent Load Balancing

Agenda

• Architecture• Data Collector

• Updates - Examples• Performance Data Helper• Citrix and the PDH

• Where can it go wrong?• Black hole effect (Load Throttling - Intelligent Load Balancing)• Some servers get all the connections• Full load after installing an MUI

• Troubleshooting

Page 4: Load Balancing and Intelligent Load Balancing

Architecture

Page 5: Load Balancing and Intelligent Load Balancing

Architecture

Data Collector DC - Dynamic Store DSData Collector DC - Dynamic Store DS

LmsSS.dll - IMA SubsystemLmsSS.dll - IMA Subsystem

MFRules.dll - XA countersMFRules.dll - XA countersLMS20Rules.dll - System countersLMS20Rules.dll - System counters

PDH.dll - Performance Data HelperPDH.dll - Performance Data Helper

Page 6: Load Balancing and Intelligent Load Balancing

Data Collector Updates

• Data Collector is updated every 30 s

• Only when the change in the load evaluator is bigger than 5%• Every 5 minutes we send a full update

• Connection logon or logoff

Page 7: Load Balancing and Intelligent Load Balancing

Data Collector UpdatesServer load - CPU Load example

• Data Collector

XenApp

Time

30s 60s

15 20 23 25 23 50 98 34 45 56

X 20 23 25 23 50 98 34 45 56 40

X 23 25 23 50 98 34 45 56 40 12

38 41 40Server Load % :

5 minutes

Page 8: Load Balancing and Intelligent Load Balancing

Data Collector UpdatesXA CPU Load != Task Manager CPU Load

• CPU load

• XA = 25%

• Task Manager = 1%

Page 9: Load Balancing and Intelligent Load Balancing

Ctxnotif.dll

MFSrvSS.dll

LMSSS.DLL

Data Collector UpdatesAt Logon time - Default load evaluator

Data Collector

ICA Client

Dynamic Store

Server A

Server B

IMAS

erve

r B

Ser

ver

AS

erve

r A

BIAS

WI/XML

IMA

X

Page 10: Load Balancing and Intelligent Load Balancing

Performance Data Helper – API

The performance data helper interface calls the registry interface to retrieve performance data

Uses the PDH.DLL to access the PDH API

http://msdn2.microsoft.com/en-us/library/aa373083.aspx

Page 11: Load Balancing and Intelligent Load Balancing

Performance Data Helper – C++ Example

pdhStatus = PdhAddCounter ( hQuery,

"\\Processor(0)\\% Processor Time",

0,

&hCounter);

YOU MUST CALL THE PERFORMANCE COUNTER BY ITS NAME

Page 12: Load Balancing and Intelligent Load Balancing

Citrix and the PDH

• \\Processor(_Total)\\% Processor Time

• \\System\\Context Switches/sec

• \\Memory\\% Committed Bytes In Use

• \\Memory\\Page Faults/sec

• \\Memory\\Pages/sec

• \\PhysicalDisk(_Total)\\Disk Bytes/sec

• \\PhysicalDisk(_Total)\\Disk Reads/sec

• \\PhysicalDisk(_Total)\\Disk Writes/sec

Page 13: Load Balancing and Intelligent Load Balancing

Performance Data Helper - Registry

Page 14: Load Balancing and Intelligent Load Balancing

Performance Data Helper – File system

EnglishPerfc009.dat

Perfh009.dat

GermanPerfc007.dat

Perfh007.dat

Page 15: Load Balancing and Intelligent Load Balancing

Performance Data Helper - perfmon

Page 16: Load Balancing and Intelligent Load Balancing

Performance Data Helper - IMA

IMA(At start up)

IMA(At start up)

Page 17: Load Balancing and Intelligent Load Balancing

Citrix and the PDH

We also provide Performance counters

Page 18: Load Balancing and Intelligent Load Balancing

Citrix and the PDHHKLM\SYSTEM\CurrentControlSet\Services\IMAService\Performance

Page 19: Load Balancing and Intelligent Load Balancing

Where can it go wrong?

Page 20: Load Balancing and Intelligent Load Balancing

Black hole effect

Problem Solution

• At Peak logon times a recently boot up server will get all connections and might become unresponsive

• Cause: Server is unable to update the DC

• Load Throttling

• Intelligent load balancing

HKLM\SOFTWARE\Citrix\IMA\LMS\

(DWORD ) UseILB = 1

(DWORD ) ILBMultiplier = 2

Page 21: Load Balancing and Intelligent Load Balancing

Before Intelligent Load Biasing

BIASBIASCurrent Load (0)

Max Load (10000)

0 sessions

Current Load (100)

Max Load (10000)

1 session

BIASBIAS

Max Load (10000)

2 sessions

Current Load (200)

1 session comes 2nd session comes

Default BIAS = 10000/100 = 100

Page 22: Load Balancing and Intelligent Load Balancing

Intelligent Load Biasing

BIASBIAS

Current Load (0)

Max Load (10000)

ILBMultiplier (2)

0 sessions

Current Load (5000)

Max Load (10000)

1 session

BIASBIAS

Max Load (10000)

2 sessions

Current Load (7500)ILBMultiplier (2)

1 session comes 2nd session comes

Load = [(Max Load – Current Load) / ILBMultiplier] + Load

Page 23: Load Balancing and Intelligent Load Balancing

Load Throttling- Low logon rate

Data CollectorICA Client

Dynamic Store

Server A

Server B

Server B

Server A

Server Load BIAS

0

0 0

0(10000-0)/2+0 = 5000100

(10000-0)/2+0 = 5000100

Page 24: Load Balancing and Intelligent Load Balancing

Load Throttling- Low logon rate

Server load if Low logon rate

0

1000

2000

3000

4000

5000

6000

ILB=1

ILB=0

Page 25: Load Balancing and Intelligent Load Balancing

Load Throttling- Black hole effect

Data CollectorICA Client

Dynamic Store

Server A

Server B

Server B

Server A

Server Load BIAS

0

6000 0

0(10000-0)/2+0 = 5000(10000-5000)/2+5000 = 7500

(10000-6000)/2+6000 = 8000

(10000-7500)/2+7500 = 8750

Page 26: Load Balancing and Intelligent Load Balancing

Load Throttling

Server load at Peak logon times

0

2000

4000

6000

8000

10000

12000

1 8

15

22

29

36

43

50

57

64

71

78

85

92

99

ILBMultiplier=2

ILBMultiplier = 10

ILBMultiplier = 20

ILB=0

Page 27: Load Balancing and Intelligent Load Balancing

Upgrade to W2K3 causes full load

Problem Solution• English W2K

\\System\\% Total Processor Time

• English W2K3

\\Processor(_Total)\\% Processor Time

• In W2K3, use

\\Processor(_Total)\\% Processor Time

Page 28: Load Balancing and Intelligent Load Balancing

Some servers get all the connections

Problem Solution

• 1 server all the connections

• Cause: Fail to read performance counters causes Load = 0

• Fail to read performance counters causes Load = 10000

Page 29: Load Balancing and Intelligent Load Balancing

Full load after installing an MUI

Problem Solution

• After installing MUI with advance load evaluators it results in load = 10000

• Cause:

-> perfc007.dat (German) perfc009.dat (English)

• HKLM\Software\Citrix\IMA\LMS\

EnableTranslation=1 (dword)

Page 30: Load Balancing and Intelligent Load Balancing

Full load after installing an MUI

• Jesús González

in a German restaurant

English Menu (perfc009.dat)

22 . Vegetable Soup

German Menu (perfc007.dat)

22. Gemüsensuppe

Page 31: Load Balancing and Intelligent Load Balancing

Troubleshooting

Page 32: Load Balancing and Intelligent Load Balancing

Troubleshooting

Data Collector DCData Collector DC LmsSS.dllLmsSS.dll

LMS20Rules.dllLMS20Rules.dll

PDH.dllPDH.dll

\\Processor(_Total)\\% Processor TimeXXXXXXXXXXXXXXXX

Full Load

Page 33: Load Balancing and Intelligent Load Balancing

TroubleshootingFailing to read performance counters => full load

• Use procmon (filemon) while restarting IMAMake sure the correct perfcXXX.dat file can be accessed.

• Consider rebuilding performance countershttp://support.microsoft.com/kb/300956/en-us

• Check that no performance counter are disabled[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfDisk\Performance]"Disable Performance Counters"=dword:00000001

• Use CDFControl to gather CDF traceshttp://support.citrix.com/article/CTX111961

Page 34: Load Balancing and Intelligent Load Balancing

•LMSRuleDll_Interface::CreatePDHQuery() ERROR!!!!! rc = -1073738823

•-1073738823 -> FFFFFFFFC0000BB9 (HEX)

•http://msdn2.microsoft.com/en-us/library/aa373046.aspx

“The specified counter could not be found”

TroubleshootingCDF Traces example

Page 35: Load Balancing and Intelligent Load Balancing

Summary

• Deep understanding of load balancing• BIAS, PDH…

• Understanding Intelligent load balancing• Load Throttling

• Typical scenarios• Full load versus 0 load, MUI…

• Troubleshooting techniques• So far we are able to resolve all the problems that arrive to EMEA tech support with those tips

Page 36: Load Balancing and Intelligent Load Balancing

Before you leave…

• Session surveys are available online at www.citrixsynergy.com starting Thursday, 7 October• Provide your feedback and pick up a complimentary gift card at the registration desk

• Download presentations starting Friday, 15 October, from your My Organiser Tool located in your My Synergy Microsite event account

Page 37: Load Balancing and Intelligent Load Balancing