capacity & performance management in solaris virtualized ... · pdf filecapacity &...

Capacity & Performance Managementin Solaris Virtualized Environment

Neeraj Bhatia | Capacity & Performance Management in Solaris Virtualized Environment

Capacity & Performance Managementin Solaris Virtualized Environment

Neeraj Bhatia

About Me

~ 10 years of experience in Oracle Database Performance Optimization and IT

Services/Infrastructure Capacity Planning.

Certified Oracle DBA – OCP 9i, 10g, 11g, RAC Expert

Other Certifications – ISO/20000 Certified Auditor, ITIL V3 Practitioner, Base

SAS

Currently working with Barclays Bank PLC as a Capacity Manager

When time allows I blog at http://neerajbhatia.wordpress.com/


~ 10 years of experience in Oracle Database Performance Optimization and IT

Services/Infrastructure Capacity Planning.

Certified Oracle DBA – OCP 9i, 10g, 11g, RAC Expert

Other Certifications – ISO/20000 Certified Auditor, ITIL V3 Practitioner, Base

SAS

Currently working with Barclays Bank PLC as a Capacity Manager

When time allows I blog at http://neerajbhatia.wordpress.com/

AgendaSection - 1

• Virtualization Basics• Different Oracle Virtualization techniques• When to use which• Basics – Zones, Containers etc

Section - 2• Capacity Planning Goals and Basics• Challenges to Capacity & Perf Management in Solaris Zones Environment• Resource Management techniques

Section - 3• Implementing Foundation for Effective Capacity and Perf Management• Case Study -1: Applications Consolidation with Advance Resource UtilizationTechniques• Case Study -2: Tweaking configuration for additional application load.


Section - 1• Virtualization Basics• Different Oracle Virtualization techniques• When to use which• Basics – Zones, Containers etc

Section - 2• Capacity Planning Goals and Basics• Challenges to Capacity & Perf Management in Solaris Zones Environment• Resource Management techniques

Section - 3• Implementing Foundation for Effective Capacity and Perf Management• Case Study -1: Applications Consolidation with Advance Resource UtilizationTechniques• Case Study -2: Tweaking configuration for additional application load.

Virtualization: Basics Virtualization is a technology which enables multiple applications/operating systemsto run on one computer system as if they have exclusive access to the computer. Three commonly used Virtualization technologies are:

1. Operating System Virtualizationo This provides multiple isolated execution environments in one OS instance.o In this model environment thinks that it got an exclusive access to the OS.o Suitable in case of application Consolidation.

2. Server Virtualizationo In this technology, multiple operating system instances (also different OS) run ona common system.o This is helpful in case of Server consolidation.

3. Hardware Virtualizationo In this model, server hardware is partitioned and provides multiple independentcomputers within one computer.o Suitable in case of Server consolidation with high degree of isolation andsecurity.


Virtualization is a technology which enables multiple applications/operating systemsto run on one computer system as if they have exclusive access to the computer. Three commonly used Virtualization technologies are:

1. Operating System Virtualizationo This provides multiple isolated execution environments in one OS instance.o In this model environment thinks that it got an exclusive access to the OS.o Suitable in case of application Consolidation.

2. Server Virtualizationo In this technology, multiple operating system instances (also different OS) run ona common system.o This is helpful in case of Server consolidation.

3. Hardware Virtualizationo In this model, server hardware is partitioned and provides multiple independentcomputers within one computer.o Suitable in case of Server consolidation with high degree of isolation andsecurity.

Oracle Virtualization Technologies


Source: Oracle VM Server for SPARC Overview @OTN

Trend to Flexibility Trend to Isolation

What is Capacity Planning

“Capacity Planning is the process of predicting when future loadlevels will saturate the system and determining the most cost-effecting way of delaying system saturation as much as possible”

- by Daniel A. Menasce and Virgilio A.F. Almeida(Authors of Capacity Planning for Web Services: Metrics, Models & Methods)


“Capacity Planning is the process of predicting when future loadlevels will saturate the system and determining the most cost-effecting way of delaying system saturation as much as possible”

- by Daniel A. Menasce and Virgilio A.F. Almeida(Authors of Capacity Planning for Web Services: Metrics, Models & Methods)

6

Capacity Management Methodology

1. State Goals2. Define the System

Configuration3. Classify Workload4. Quantify & Control

Resource Usage5. Analyze and Model Data

and Interpret the Results


1. State Goals2. Define the System

Configuration3. Classify Workload4. Quantify & Control

Resource Usage5. Analyze and Model Data

and Interpret the Results

Resource Management: Why?• In Consolidation/Virtualization Scenarios, each workload may have different priorityand resource Demand.• One workload may hog all the resources of underlying server and thus affecting theperformance of other workloads.• Resource Management allows to create limits for Resource Usage.• Three Important Roles of Resource Management Framework:

1. Classify Workload: Achieved with the help of Projects and Zones2. Quantify Resource Usage: Measure how much Resources a particular

Workload is consuming3. Control the Resource Usage: To eliminate cross-workload interference using 3

techniques: Constraints - This defines the upper bound on specific resources Scheduling - This defines run-time priorities among the Workloads Partitioning - Isolate workload from each other


• In Consolidation/Virtualization Scenarios, each workload may have different priorityand resource Demand.• One workload may hog all the resources of underlying server and thus affecting theperformance of other workloads.• Resource Management allows to create limits for Resource Usage.• Three Important Roles of Resource Management Framework:

1. Classify Workload: Achieved with the help of Projects and Zones2. Quantify Resource Usage: Measure how much Resources a particular

Workload is consuming3. Control the Resource Usage: To eliminate cross-workload interference using 3

techniques: Constraints - This defines the upper bound on specific resources Scheduling - This defines run-time priorities among the Workloads Partitioning - Isolate workload from each other

Workload Classification• The first step in laying down effective Capacity & Performance Management is toidentify the workloads running on the System.• A Conventional and often challenging way is to identify Workloads by Username ORProcess Name.• Oracle Solaris Containers like Zones and Projects provide a systematic way to Identifyand Manage Workloads.• For example – Multiple instances of the same Application can run on the samesystem and provided separate Containers are defined for each, one would be able toultimately control the workloads .


Zones Basics• Solaris Zones is a Software-partitioning technology introduced in Solaris 10 OperatingSystem.• Each Zone acts like a separate Operating System although each shares a singleinstance of Solaris Operating system.• Provides an isolated and Secured Environment for running Applications; processes ina zone can’t interact with processes in other Zones.• Some Possible Use cases:

o To consolidate several small Solaris/Legacy serverso To consolidate many lowly-utilized serverso To enable the migration of Solaris 8 Application Environmentso To provide Maximum Isolation and Security by integrating with Oracle VMServer with SPARC


• Solaris Zones is a Software-partitioning technology introduced in Solaris 10 OperatingSystem.• Each Zone acts like a separate Operating System although each shares a singleinstance of Solaris Operating system.• Provides an isolated and Secured Environment for running Applications; processes ina zone can’t interact with processes in other Zones.• Some Possible Use cases:

o To consolidate several small Solaris/Legacy serverso To consolidate many lowly-utilized serverso To enable the migration of Solaris 8 Application Environmentso To provide Maximum Isolation and Security by integrating with Oracle VMServer with SPARC

Resource Capping Techniques• Limits the amount of CPU and Memory resources for a Workload.• Prevents a Workload from using excess CPU/Memory and interfere with otherWorkloads.• Restrict a Workload to excess additional CPU even if other Workloads are Idle.• When used with Processor sets, it limits CPU usage within a set.

zonecfg:my-zone> add capped-cpuzonecfg:my-zone:capped-cpu> set ncpus=3.5zonecfg:my-zone:capped-cpu> end

Note: ncpus = 3.5 will limit the workload to 3.5 CPUs

zonecfg:my-zone> add capped-memoryzonecfg:my-zone:capped-memory> set physical=512mzonecfg:my-zone:capped-memory> set swap=1024mzonecfg:my-zone:capped-memory> set locked=100mzonecfg:my-zone:capped-memory> end


• Limits the amount of CPU and Memory resources for a Workload.• Prevents a Workload from using excess CPU/Memory and interfere with otherWorkloads.• Restrict a Workload to excess additional CPU even if other Workloads are Idle.• When used with Processor sets, it limits CPU usage within a set.

zonecfg:my-zone> add capped-cpuzonecfg:my-zone:capped-cpu> set ncpus=3.5zonecfg:my-zone:capped-cpu> end

Note: ncpus = 3.5 will limit the workload to 3.5 CPUs

zonecfg:my-zone> add capped-memoryzonecfg:my-zone:capped-memory> set physical=512mzonecfg:my-zone:capped-memory> set swap=1024mzonecfg:my-zone:capped-memory> set locked=100mzonecfg:my-zone:capped-memory> end

Scheduling Classes (FSS)• By default Solaris dispatcher uses TS scheduler which gives relatively equal access toCPU resources.

• Priority among workloads for CPU can be set using FSS (Fair Share Scheduler).

• FSS achieves Resource Efficiency & Fairness by:o Increasing workload entitlement in case of low usageo Reducing workload’s entitlement in the case of heavy CPU usage.

• Workload’s relative importance is expressed in number of CPU shares and is notequivalent to percentages of CPU resources.

• NO meaning in absolute terms; define the relative importance of workloads.

• To optimally set CPU Shares to a workload – understand total number of workloads ona system, their relative importance and usage profile.

• Can be configured with Processor Sets for granular control over resources.


• By default Solaris dispatcher uses TS scheduler which gives relatively equal access toCPU resources.

• Priority among workloads for CPU can be set using FSS (Fair Share Scheduler).

• FSS achieves Resource Efficiency & Fairness by:o Increasing workload entitlement in case of low usageo Reducing workload’s entitlement in the case of heavy CPU usage.

• Workload’s relative importance is expressed in number of CPU shares and is notequivalent to percentages of CPU resources.

• NO meaning in absolute terms; define the relative importance of workloads.

• To optimally set CPU Shares to a workload – understand total number of workloads ona system, their relative importance and usage profile.

• Can be configured with Processor Sets for granular control over resources.

Resource Partitioning Techniques• Oracle Solaris Resource Pools partition the System resources and limit Workloads tospecific subsets of the resources.• This helps in avoiding system-wide overcommitment. However may result in lowerutilization of system resources.• Possible Use cases:

o Mixed Workloadso Phased approach for Applications releaseo Resource allocation based on user roleo Implementing chargeback model


Resource Partitioning Techniques• Resource Pool Configuration have following parameters:

• Pool: Example - pool.importance• Pset: Example - pset.min and pset.max• CPU: Example - cpu.pinned

• Dynamic resource pools provide a mechanism for dynamically adjusting each pool'sresource allocation in response to system events and application load changes.• DRPs simplify and reduce the number of decisions required from an administrator.• Adjustments are automatically made through system daemon poold.• Poold daemon acts on Constraints and Objectives provided by an Administrator.

• Constraints:o Min/Max CPU Allocation: pset.min and pset.max place limits on the number ofprocessors that can be allocated to a processor set.o CPU Pinning: cpu.pinned constraint indicates that a particular CPU should notbe moved by DRP from the processor set in which it is located. This is to achievemaximum cache utilization for a particular application that is executing within aprocessor set.o Pool Importance: The pool.importance property describes the relativeimportance of a pool.


• Resource Pool Configuration have following parameters:• Pool: Example - pool.importance• Pset: Example - pset.min and pset.max• CPU: Example - cpu.pinned

• Dynamic resource pools provide a mechanism for dynamically adjusting each pool'sresource allocation in response to system events and application load changes.• DRPs simplify and reduce the number of decisions required from an administrator.• Adjustments are automatically made through system daemon poold.• Poold daemon acts on Constraints and Objectives provided by an Administrator.

• Constraints:o Min/Max CPU Allocation: pset.min and pset.max place limits on the number ofprocessors that can be allocated to a processor set.o CPU Pinning: cpu.pinned constraint indicates that a particular CPU should notbe moved by DRP from the processor set in which it is located. This is to achievemaximum cache utilization for a particular application that is executing within aprocessor set.o Pool Importance: The pool.importance property describes the relativeimportance of a pool.

Resource Partitioning Techniques• Objectives:• Weighted Load Objective: The wt-load objective favors configurations that matchresource allocations to resource utilizations.• Useful in situations where you are satisfied with the minimum and maximumproperties constraints.• Utilization Objective (0-100%)•The utilization objective favors configurations that allocate resources to partitionsthat are not meeting the specified utilization objective.

oThis objective is specified by using operators and values.o The “less than” operator (<) indicates maximum target value.o The “greater than” operator (>) indicates minimum target value.o The “about” operator (~) indicates a target value with some acceptablefluctuation.


• Objectives:• Weighted Load Objective: The wt-load objective favors configurations that matchresource allocations to resource utilizations.• Useful in situations where you are satisfied with the minimum and maximumproperties constraints.• Utilization Objective (0-100%)•The utilization objective favors configurations that allocate resources to partitionsthat are not meeting the specified utilization objective.

oThis objective is specified by using operators and values.o The “less than” operator (<) indicates maximum target value.o The “greater than” operator (>) indicates minimum target value.o The “about” operator (~) indicates a target value with some acceptablefluctuation.

Resource Partitioning Techniques• The dedicated-cpu resource specifies that a subset of the System’s processors shouldbe dedicated to a Workload.• System on runtime creates a temporary pool till the Workload’s lifetime.• The dedicated-cpu resource has 2 properties; ncpus and importance• ncpus: It specifies the number of CPUs or a range, such as 2-4 CPUs.• Importance: An optional property, defines the relative importance of the workload,in case a range is provided for ncpus.

zonecfg:my-zone> add dedicated-cpuzonecfg:my-zone:dedicated-cpu> set ncpus=1-3zonecfg:my-zone:dedicated-cpu> set importance=2zonecfg:my-zone:dedicated-cpu> end


• The dedicated-cpu resource specifies that a subset of the System’s processors shouldbe dedicated to a Workload.• System on runtime creates a temporary pool till the Workload’s lifetime.• The dedicated-cpu resource has 2 properties; ncpus and importance• ncpus: It specifies the number of CPUs or a range, such as 2-4 CPUs.• Importance: An optional property, defines the relative importance of the workload,in case a range is provided for ncpus.

zonecfg:my-zone> add dedicated-cpuzonecfg:my-zone:dedicated-cpu> set ncpus=1-3zonecfg:my-zone:dedicated-cpu> set importance=2zonecfg:my-zone:dedicated-cpu> end

Case Study -1: Application Consolidation• Three Applications are being consolidated onto a single system.

• Applications have different resource requirements.o Batch Workload: Heavy batch programs, primary concern is throughput.o OLTP Workload: Heavy transaction processing, primary concern is response time.Driven from Web, headroom should be there to accommodate transaction spikes.o Admin Workload: It provides admin console to manage tools and applications.

• Due to business priority you want to make sure a fixed Capacity is always available foran Application.

• Also unused Capacity by less-important Applications should be available to otherapplications.


• Three Applications are being consolidated onto a single system.

• Applications have different resource requirements.o Batch Workload: Heavy batch programs, primary concern is throughput.o OLTP Workload: Heavy transaction processing, primary concern is response time.Driven from Web, headroom should be there to accommodate transaction spikes.o Admin Workload: It provides admin console to manage tools and applications.

• Due to business priority you want to make sure a fixed Capacity is always available foran Application.

• Also unused Capacity by less-important Applications should be available to otherapplications.

Process Sets and CPU Shares - Examples• Total CPU on the Target Server = 8• Three Processor Sets:

o Processor Set 1 = CPU numbers { 0, 1 }o Processor Set 2 = CPU numbers { 2, 3 }o Processor Set 3 = CPU numbers { 4, 5, 6, 7 }

• Three Workloads on the System:o W1 (Admin Processes), CPU Shares =1, on Processor Set -1o W2 (Batch Processes), CPU Shares =2, on Processor Set -1, 2o W3 (OLTP Application Processes), CPU Shares =3, on Processor Set- 1, 2, 3


• Total CPU on the Target Server = 8• Three Processor Sets:

o Processor Set 1 = CPU numbers { 0, 1 }o Processor Set 2 = CPU numbers { 2, 3 }o Processor Set 3 = CPU numbers { 4, 5, 6, 7 }

• Three Workloads on the System:o W1 (Admin Processes), CPU Shares =1, on Processor Set -1o W2 (Batch Processes), CPU Shares =2, on Processor Set -1, 2o W3 (OLTP Application Processes), CPU Shares =3, on Processor Set- 1, 2, 3

Process Sets and CPU Shares - ExamplesWorkload’s CPU Allocation = For Each Processor Set

Total system-wide CPU allocation for workloads can be calculated as follows:

Total Admin Workload CPU Allocation:(1/6 X 2/8)pset1 = 4.17%

Total Batch Workload CPU Allocation:(2/6 X 2/8)pset1 + (2/5 X 2/8)pset2 = 18.33%

Total OLTP Workload CPU Allocation:(3/6 X 2/8 )pset1 + (3/5 X 2/8)pset2 + (3/3 X 4/8)pset3 = 77.50%

These figures represent minimum guaranteed CPU shares assuming all workloadsare CPU bound. Each Workload could use 100% of CPU resources in their processor sets providedother Workloads are idle.


Workload’s CPU Allocation = For Each Processor Set

Total system-wide CPU allocation for workloads can be calculated as follows:

Total Admin Workload CPU Allocation:(1/6 X 2/8)pset1 = 4.17%

Total Batch Workload CPU Allocation:(2/6 X 2/8)pset1 + (2/5 X 2/8)pset2 = 18.33%

Total OLTP Workload CPU Allocation:(3/6 X 2/8 )pset1 + (3/5 X 2/8)pset2 + (3/3 X 4/8)pset3 = 77.50%

These figures represent minimum guaranteed CPU shares assuming all workloadsare CPU bound. Each Workload could use 100% of CPU resources in their processor sets providedother Workloads are idle.

Process Sets and CPU Shares - Examples

Figure: CPU Utilization in theabsence of sophisticatedResource Utilization techniques


Figure: CPU Utilization in case ofProcessor Sets and FSS

The Future lies in the History

"I have seen the future and it is very much like the present,only longer."--Kehlog Albran, The Profit

This philosophy is actually a concise description of statistical forecasting. We searchfor statistical properties of a time series that are constant in time - trends, seasonalpatterns, correlations etc. We then predict that these properties will also describe thefuture.


"I have seen the future and it is very much like the present,only longer."--Kehlog Albran, The Profit

This philosophy is actually a concise description of statistical forecasting. We searchfor statistical properties of a time series that are constant in time - trends, seasonalpatterns, correlations etc. We then predict that these properties will also describe thefuture.

Case Study -2: Capacity ForecastingScenario: There are 2 applications running on a system with 8 CPU cores. One of theapplication (Oracle Database) is anticipating additional workload. The objective is toassess if the system has sufficient Capacity to sustain the workload and is there anyscope to tweak the configuration to reduce the pressure on additional CPUs.

Current Configuration:• Total CPUs on the System: 8• Total Zones running on system: 2• Zone-1 (Oracle Database) CPU Shares: 3• Zone-2 (Test Environment) CPU Shares: 2


Scenario: There are 2 applications running on a system with 8 CPU cores. One of theapplication (Oracle Database) is anticipating additional workload. The objective is toassess if the system has sufficient Capacity to sustain the workload and is there anyscope to tweak the configuration to reduce the pressure on additional CPUs.

Current Configuration:• Total CPUs on the System: 8• Total Zones running on system: 2• Zone-1 (Oracle Database) CPU Shares: 3• Zone-2 (Test Environment) CPU Shares: 2

Case Study -2: Capacity ForecastingUsage profile of Zones are depicted below:

Test Env Zone usingunused Capacity


With current Resource Management settings of 3 and 2 CPU Shares, Databaseapplication can access up to 60% CPU and Test environment Zone up to 40% CPUCapacity.

Case Study -2: Observations/Questions• System has some headroom for additional which can be used by additionalload on Oracle Database Zone.• With current configuration, there would be no chance of Test environmentgetting idle Capacity if Database Zone happens to be using all the Capacity at it’sdisposal.• Is that a Concern?• Consider the intrinsic nature of the Test environments.• Now what If we reduce the Capacity that can be accessed by Test environmentby reducing the CPU shares?• This will increase the guaranteed Capacity that database Zone can have at anygiven point of time.• The Key is to strike a balance between both Zones.• If still Capacity asks of database application can’t be satisfied withoutcompromising with the Test environment, Capacity upgrade is inevitable.


• System has some headroom for additional which can be used by additionalload on Oracle Database Zone.• With current configuration, there would be no chance of Test environmentgetting idle Capacity if Database Zone happens to be using all the Capacity at it’sdisposal.• Is that a Concern?• Consider the intrinsic nature of the Test environments.• Now what If we reduce the Capacity that can be accessed by Test environmentby reducing the CPU shares?• This will increase the guaranteed Capacity that database Zone can have at anygiven point of time.• The Key is to strike a balance between both Zones.• If still Capacity asks of database application can’t be satisfied withoutcompromising with the Test environment, Capacity upgrade is inevitable.

Case Study -2: Capacity Forecasting

Figure: Effect of additional Capacity usage byDatabase Zone


Figure: CPU Shares changed to 4 and 1respectively, and database is able toprocess additional load.Also, Test zone can go beyond 20% in caseof idle Capacity is available.

Case Study -2: ConclusionsProvided Database workload is going to increases in phased manner, we candelay the configuration change/Capacity upgradation.

o In Phase-1 additional workload can be supported by leveraging theavailable headroom.o In Phase-2, further increase in the database workload can be sustained byputting constraint on Test environment.o If further increase in the database workload is expected, Capacityupgradation is inevitable.


Provided Database workload is going to increases in phased manner, we candelay the configuration change/Capacity upgradation.

o In Phase-1 additional workload can be supported by leveraging theavailable headroom.o In Phase-2, further increase in the database workload can be sustained byputting constraint on Test environment.o If further increase in the database workload is expected, Capacityupgradation is inevitable.

Quotation

"I hear and I forget. I see and I remember. I do and I understand."-- Chinese Proverb

The best way to learn a subject is to apply the concepts to a real system. Thetechniques presented in this presentation may appear simple on the surface, theirapplications to real world may offer a different experience.


Neeraj Bhatia | Capacity & Performance Management in Solaris Virtualized Environment 28

capacity & performance management in solaris virtualized ... · pdf filecapacity &...

Documents