lab overview - hol-1704-chg-2

Table of ContentsLab Overview - HOL-1704-CHG-2 - vSphere 6: Challenge Lab ..........................................2

Lab Guidance .......................................................................................................... 3Module 1: Get Your Feet Wet (30 minutes)........................................................................ 9

Clipboard copy and paste not working? ................................................................ 10Poorly Performing VM............................................................................................ 20

Module 2: Operations Challenge, Basic Level (30 minutes) ............................................36Unable to login to vSphere Web Client.................................................................. 37Empty inventory in vSphere Web Client. Help! ..................................................... 43VMRC is not allowing me to connect to my virtual machine..................................46Why is SSH not working on ESXi host? .................................................................. 51

Module 3: Performance Challenge, Basic Level (30 minutes)..........................................55Why is my VM slow?.............................................................................................. 56Troubleshoot a poorly performing VM ................................................................... 79

Module 4: Operations Challenge, Advanced Level (30 minutes) .....................................97Why does my ESXi host keeps disconnecting?! .................................................... 98Unable to login to vCenter using SSO Admin account to assign permission(SSOAdministrator account locked) ............................................................................ 108Unable to put server in maintenance mode ........................................................ 111

Module 5: Performance Challenge, Advanced Level (30 minutes) ................................121Poor Storage performance .................................................................................. 122vNUMA Troubleshooting ...................................................................................... 152

Module Hints ................................................................................................................. 164Module 1: Get Your Feet Wet ............................................................................... 165Module 2: Operations Challenge, Basic Level...................................................... 167Module 3: Performance Challenge, Basic Level ................................................... 169Module 4, Operations Challenge, Advanced Level .............................................. 170Module 5, Performance Challenge, Advanced Level............................................172Conclusion........................................................................................................... 173

HOL-1704-CHG-2

HOL-1704-CHG-2

Lab Overview -HOL-1704-CHG-2 -

vSphere 6: Challenge Lab

HOL-1704-CHG-2

HOL-1704-CHG-2

Lab GuidanceThe vSphere 6 Challenge asks you to put on your thinking cap to save the day! Eachmodule places you in a different fictional scenario to fix common vSphere operationaland performance problems. Have you ever had a disconnected host or a VM which youcould not launch remote console into? How about a mysterious poorly performing VM? Inthis lab, you will learn to solve these and more.

As you walk into a tall glass building, you glance at your watch: 9:00AM. It's thebeginning of another workday at one of the most desirable businesses in the year 2096:PanContinental Teleportation Corporation.

PanContinental Teleportation moves people and things instantaneously across the globe.After a quick payment, anyone can step into a booth-like "Waypoint™," located in everymajor city across the globe, select their destination, and instantaneously emerge at anyother Waypoint™. But PanContinental technology isn't magic. No, it's powered by theprimary datacenter, which is the gleaming building you just entered, and by vSphere6.0.

As the largest and most trusted global teleportation network, PanContinental's highestpriorities are the safety of its customers and continuity of operations. That responsibilityrests on your and your colleagues' shoulders.

The modules below test your mettle and keep the teleportation network runningsmoothly. This lab is best suited for those with some vSphere experience, or who havealready taken labs such as HOL-1710-SDC-1: vSphere 6 with OperationsManagement or HOL-1704-SDC-1: vSphere 6 Performance Optimization. This labincludes over two hours of content, so you might not be able to complete it all in onesitting. Because the vSphere 6 Challenge lab requires troubleshooting, your completiontime could vary from those listed below. We provide optional hints along the way as wellas providing the Challenges' final solutions.

We recommend that you start with Module 1 and then proceed to any module whichinterests you most. If you've already taken Module 1, proceed to any module.

Click on any module below to begin.

• Module 1 - Get Your Feet Wet (30 minutes) First day on the job? No problem.Start here.

• Module 2 - Operations Challenge, Basic Level(30 minutes) This modulechallenges you to solve common vSphere operational issues, including an emptyinventory in the vSphere Web Client and a non-functional VM Remote Console.

• Module 3 - Performance Challenge, Basic Level(30 minutes) Can yourecognize and resolve common vSphere performance issues?

HOL-1704-CHG-2

HOL-1704-CHG-2

• Module 4 - Operations Challenge, Advanced Level (30 minutes) Bring thedatacenter back to health by fixing operational issues in vSphere such as adisconnecting ESXi host, vCenter Server health alert, and others.

• Module 5 - Performance Challenge, Advanced Level(30 minutes) We tip ourhat to you if you can resolve these lesser-known storage performance and vNUMAissues.

Lab Captains: Nick Fritsch and Tony Okwechime.

This lab manual can be downloaded from the Hands-on Labs Document site found here:

http://docs.hol.vmware.com/HOL-2017/hol-1704-chg-2_pdf_en.pdf

This lab may be available in other languages. To set your language preference and havea localized manual deployed with your lab, you may utilize this document to help guideyou through the process:

http://docs.hol.vmware.com/announcements/nee-default-language.pdf

HOL-1704-CHG-2

HOL-1704-CHG-2

http://docs.hol.vmware.com/HOL-2017/hol-1704-chg-2_pdf_en.pdf

http://docs.hol.vmware.com/announcements/nee-default-language.pdf

Location of the Main Console

1. The area in the RED box contains the Main Console. The Lab Manual is on the tabto the Right of the Main Console.

2. A particular lab may have additional consoles found on separate tabs in the upperleft. You will be directed to open another specific console if needed.

3. Your lab starts with 90 minutes on the timer. The lab can not be saved. All yourwork must be done during the lab session. But you can click the EXTEND toincrease your time. If you are at a VMware event, you can extend your lab timetwice, for up to 30 minutes. Each click gives you an additional 15 minutes.Outside of VMware events, you can extend your lab time up to 9 hours and 30

minutes. Each click gives you an additional hour.

Activation Prompt or Watermark

When you first start your lab, you may notice a watermark on the desktop indicatingthat Windows is not activated.

One of the major benefits of virtualization is that virtual machines can be moved andrun on any platform. The Hands-on Labs utilizes this benefit and we are able to run thelabs out of multiple datacenters. However, these datacenters may not have identicalprocessors, which triggers a Microsoft activation check through the Internet.

Rest assured, VMware and the Hands-on Labs are in full compliance with Microsoftlicensing requirements. The lab that you are using is a self-contained pod and does nothave full access to the Internet, which is required for Windows to verify the activation.

HOL-1704-CHG-2

HOL-1704-CHG-2

Without full access to the Internet, this automated process fails and you see thiswatermark.

This cosmetic issue has no effect on your lab.

Alternate Methods of Keyboard Data Entry

During this module, you will input text into the Main Console. Besides directly typing itin, there are two very helpful methods of entering data which make it easier to entercomplex data.

Click and Drag Lab Manual Content Into Console ActiveWindow

You can also click and drag text and Command Line Interface (CLI) commands directlyfrom the Lab Manual into the active window in the Main Console.

Accessing the Online International Keyboard

You can also use the Online International Keyboard found in the Main Console.

<div class="player-unavailable"><h1 class="message">An error occurred.</h1><div class="submessage"><ahref="http://www.youtube.com/watch?v=xS07n6GzGuo" target="_blank">Try watching this video on www.youtube.com</a>, or enableJavaScript if it is disabled in your browser.</div></div>

HOL-1704-CHG-2

HOL-1704-CHG-2

1. Click on the Keyboard Icon found on the Windows Quick Launch Task Bar.

Click once in active console window

In this example, you will use the Online Keyboard to enter the "@" sign used in emailaddresses. The "@" sign is Shift-2 on US keyboard layouts.

1. Click once in the active console window.2. Click on the Shift key.

Click on the @ key

1. Click on the "@" key.

Notice the @ sign entered in the active console window.

HOL-1704-CHG-2

HOL-1704-CHG-2

Look at the lower right portion of the screen

Please check to see that your lab is finished all the startup routines and is ready for youto start. If you see anything other than "Ready", please wait a few minutes. If after 5minutes you lab has not changed to "Ready", please ask for assistance.

HOL-1704-CHG-2

HOL-1704-CHG-2

Module 1: Get Your FeetWet (30 minutes)

HOL-1704-CHG-2

HOL-1704-CHG-2

Clipboard copy and paste not working?Your new colleague, Murphy, asked you to help with some issues that he is facing withhis brand new virtual machine, linux-base-01. This is a crucial virtual machine asMurphy uses it to transfer key information from his research laboratory into the virtualmachine using clipboard copy and paste. After receiving his brand new virtual machine,linux-base-01, Murphy has not been able copy/paste clipboard information from hisresearch laptop into the virtual machine. He is relying on you, the expert in VMwarevSphere to help with troubleshooting this issue.

Guidelines for troubleshooting:

To troubleshoot the challenge, please use the internet to research the resolution ifneeded.

If you are viewing this Lab in Full Screen Mode, you may press the F11 key to disableFull Screen Mode and access the internet.

For users with non-US keyboards

If you are using a non-US keyboard, it might be challenging to enter CLI commands oruser names and passwords throughout this lab. There are several options to facilitatethis.

Option 1: On the taskbar, click on the keyboard icon to open the clickable On-ScreenKeyboard.

HOL-1704-CHG-2

HOL-1704-CHG-2


Option 2: Open README.txt on the desktop. This file contains all credentials andwritten input commands for the vSphere 6 Challenge Lab. You can then copy and pastethem where needed.

HOL-1704-CHG-2

HOL-1704-CHG-2


Option 3: Lastly, the Send Text option via the VMware Learning Platform allows you toinput text using your native keyboard and send it into the Lab desktop.

Click the Send Text button at the top left corner of the Lab console. A window will openallowing you to enter text; then click Send.

Wait for Lab Status: Ready

In the bottom right corner of the desktop console, you will see the text Lab Statusdisplayed in either green or red. Please find this on your Hands-on Lab desktop now.

If Lab Status is Green, please proceed to the next page to begin the lab now.

If Lab Status is Red, do not start the lab. We are still preparing your virtualenvironment. Instead, please wait for the Lab Status to become green and read Ready.In some cases, this might take over 10 minutes. When Lab Status reads Ready, pleaseproceed to the next page to begin the lab.

HOL-1704-CHG-2

HOL-1704-CHG-2




Rest assured, VMware and the Hands-on Labs are in full compliance with Microsoftlicensing requirements. The lab that you are using is a self-contained pod and does nothave full access to the Internet, which is required for Windows to verify the activation.Without full access to the Internet, this automated process fails and you see this

watermark.

This cosmetic issue has no effect on your lab. If you have any questions or concerns,please feel free to use the support made available to you either at VMworld in theHands-on Labs area, in your Expert-led Workshop, or online via the survey comments aswe are always looking for ways to improve your hands on lab experience.

Open the vSphere Web Client

On the desktop, click on the Chrome browser desktop shortcut. Locate the bookmarkRegionB Web Client. Click on it.

HOL-1704-CHG-2

HOL-1704-CHG-2

Login to vSphere Web Client

Select "Use Windows session authentication" and click Login.

Navigate to Hosts and Clusters

1. Click on Home.2. Click on Hosts and Clusters.

Launch Remote Console

1. Expand the quetips on vcsa-01b.2. Click on the virtual machine, linux-base-01.

HOL-1704-CHG-2

HOL-1704-CHG-2

3. Click on Summary.4. Launch Remote Console.

Note: Ensure that linux-base-01 is powered-on before launching Remote Console.

HOL-1704-CHG-2

HOL-1704-CHG-2

Logging into linux-base-01

From the Remote Console application, login to linux-base-01 using these credentials:

• User: holuser (you do not need to key in these)• Password: VMware1!

NOTE: Click on the HOL User for the Password field to populate and then type inVMware1!

Copy text into linux-base-01

After logging into Murphy's machine, you are supposed to solve his problem copyingsome of his research information into a document that he has created. To access thedocument:

HOL-1704-CHG-2

HOL-1704-CHG-2

1. Open the quantum_teleportation folder2. Open the protocol.txt file

Murphy has given you a snippet of his research under Module 1 of the README.txtfile which you can find on the Main Console desktop.

NOTE:Minimize your Chrome window to access the Main Console desktop README.txt.

Your task:

1. Fix the clipboard copy/paste issue on linux-base-012. Copy the text snippet from the Main Console desktop README.txt, under

heading Module 13. Paste it into the protocol.txt file in the quantum_teleportation folder on

linux-base-01

Need a hint?

View Hint #1

View Hint #2

HOL-1704-CHG-2

HOL-1704-CHG-2

Solution

There may be times where you may want to enable clipboard copy/paste on a virtualmachine, or even the virtual machines that are residing on an ESXi host. However, bydefault this functionality has been disabled.

To enable Copy and Paste option for a specific virtual machine:

HOL-1704-CHG-2

HOL-1704-CHG-2

NOTE: VMware Tools must be installed for this to work.

1. Log in to vSphere Web Client2. Navigate to vcsa-01b and locate linux-base-013. Power off the virtual machine (linux-base-01).4. Select the virtual machine (linux-base-01) and click the Summary tab.5. Click Actions, then Edit Settings.6. Navigate to VM Options > Advanced > Configuration Parameters > Edit

Configuration.7. Click Add Row and add the configuration parameter

isolation.tools.copy.disable with value false.8. Click Add Row again and add the configuration parameter

isolation.tools.paste.disable with value false.

isolation.tools.copy.disable = false

isolation.tools.paste.disable = false

You may click OK twice to close the Configuration dialog and proceed to power on thevirtual machine.

Note: After making those changes above, remember to power on linux-base-01 beforelaunching Remote Console.

For more information on this, you may proceed to read this KB article.

HOL-1704-CHG-2

HOL-1704-CHG-2

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1026437

Poorly Performing VMAfter enabling clipboard copy and paste on Murphy's VM, you feel great. But like Murphyalways says, "Anything that can go wrong will go wrong," and there's another challengein store for you.

One of the Waypoint techs filed a high priority ticket stating that her waypoint is stallingwhen it tries to lock onto a phase. Because of the delay, there's a queue forming in frontof the Waypoint. This could be due to poor performance on the Waypoint phase selectorVMs. We need to get it fixed fast, so let's check it out.

Open Challenge Scripts Folder on Desktop

1. Back at the Hands-on Labs Main Console, open the Challenge Scripts folder on theDesktop.

Start Module 1 Script

1. Right click on Module1-start.ps1.

2. From the context menu, select Run with PowerShell.

Start Module 1 Script

The script starts running and will take a minute to complete. While the script is running,please proceed to the next page.

HOL-1704-CHG-2

HOL-1704-CHG-2

Challenge Guidelines

As in the last challenge, please use the internet to research the resolution if needed.

If you are viewing this Lab in Full Screen Mode, you may press the F11 key to disableFull Screen Mode and access the internet.

Of course, if you have a connected mobile device with you, you can use that as well.

Let's take a moment to consider ways to troubleshoot a performance problem. Pleaseproceed to the next page.

Troubleshooting methodology

The first step in troubleshooting a problem is pinpointing the symptoms. In the vSphere6 Challenge lab, we make this part easy by telling you exactly what behavior you willsee when the problem is solved.

In the real world, if you are comparing the performance of your system to its pastperformance or to a different system, ensure that it is an apples-to-apples comparison.If the physical or virtual infrastructure has changed, the comparison might not be valid.

HOL-1704-CHG-2

HOL-1704-CHG-2

Remember that not all guidelines or rules of thumb will apply to your environment.When you consider these factors, you might learn there's no performance problem afterall.

Many different subsystems could be responsible for a performance problem. The circlein the image above is a rough guideline of the likelihood that each subsystem isresponsible for the problem. Statistically, storage is the most frequent cause ofperformance problems, while network is an infrequent cause of performance problems.

Once you are focusing on a particular subsystem, brainstorm ideas for as manypotential solutions as you can.

Implement each potential solution one at a time and re-test to determine if the problemis fixed.

If not, return the implementation to its previous state. Repeat this pattern iteratively toeliminate potential problem sources.

HOL-1704-CHG-2

HOL-1704-CHG-2

Module 1 Script Finishes

When the script is finished running, it will read "You may now close this window".Your exact output may vary from that above.

Press Enter twice to close the window or use the close button in the top right handcorner.

Open challenge-01a Remote Desktop

On the desktop, double click on chg-01a.RDP to open a Remote Desktop session to thevirtual machine challenge-01a.

If you receive the popup "Remote Desktop can't connect to the remote computer...,"click OK, wait a moment, and try again to open Remote Desktop.

challenge-01a Desktop

The virtual machine challenge-01a has some tools installed on it to help you monitorits well-being.

1. On the desktop, you can see information about the virtual machine including itshostname, IP address, CPU usage, and memory usage. The CPU usage and memoryusage update in real time.

HOL-1704-CHG-2

HOL-1704-CHG-2

In the CPU usage chart you can see both the total CPU used ("Total:") and thepercentage of CPU usage that is consumed by the guest operating system's kernel("Krnl:").

2. Double click on SPECjbb to start the performance measurement application.

HOL-1704-CHG-2

HOL-1704-CHG-2

SPECjbb2005

A window titled "SPECjbb2005" opens. A copy of SPECjbb2005 is installed onchallenge-01a. SPECjbb2005 is a real SPEC java-based benchmark that can measureCPU and memory performance.

1. Click the "Start Benchmark" button to start the benchmark.

SPECjbb starts

It will take a few seconds for the benchmark to begin running. Please proceed to thenext page.

SPECjbb is running

1. When SPECjbb starts producing performance results, you will see the PerformanceScore displayed in the SPECjbb2005 window. A higher number indicates betterperformance.

HOL-1704-CHG-2

HOL-1704-CHG-2

2. The text file at the bottom shows the output of the benchmark in real time. Theperformance results, measured as business operations per second (BOPS), orthroughput, are highlighted in yellow. This window is just for viewing the execution ofthe benchmark and you can close this window without affecting the operation of thebenchmark.

If you close the SPECjbb2005 window, or hit "Stop Benchmark", the benchmark willstop. To analyze the performance results, you should leave the benchmark running.

Performance results update every 10 seconds. When you start SPECjbb2005 forthe first time or make changes to the environment, you might have to wait a fewupdate cycles for performance results to stabilize or reach steady-state.

Your results might differ from those shown here.

Although the SPECjbb2005 benchmark is now retired, it can still be useful in showing ourVM's performance characteristics.

Because SPECjbb2005 generates load on the environment, it's not appropriate as aperformance monitoring tool alongside production workloads. We use it fordemonstration purposes in the Hands-on Labs.

vRealize Operations Manager is designed to monitor the performance characteristics ofthe environment. To learn more about vRealize Operations Manager, see LabHOL-1710-SDC-3, vSphere with Operations Managment: Advanced Topics.

3. You can monitor CPU and memory usage here. Notice how they have changed sinceyou started the benchmark.

Note: SPECjbb2005 will stop running after 45 minutes. If this occurs, you must run theModule1-start.ps1 script again, and then restart SPECjbb on challenge-01a.

Unfortunately, the Performance Score from challenge-01a is far too low!challenge-01a controls the phase selection of the Waypoint network; if this isn'tcorrected, the Waypoints will continue stalling. Even worse, they might lock onto thewrong phase and passengers could be redirected to the parallel universe! Not good forrepeat business.

Your challenge is to increase the performance. Can you make challenge-01a doubleits stable-state performance?

But first... where is this VM hosted and how can you manage it?

HOL-1704-CHG-2

HOL-1704-CHG-2


Leave the remote session to challenge-01a open and navigate back to the vSphereWeb Client.

If needed, open Chrome and click on the RegionA Web Client bookmark in thebookmarks bar.

Hosts and Clusters

If needed, select "Use Windows session authentication" and click Login.

HOL-1704-CHG-2

HOL-1704-CHG-2


If needed, at the Home page, click the Hosts and Clusters icon.


Expand vcsa-01a as needed. challenge-01a is managed by vCenter Server vcsa-01a.It is hosted on esx-02a along with neighboring VM challenge-02a.

vSphere Performance Charts

For troubleshooting a performance issue in real time, the vSphere performance chartsare a first-line tool. Let's try using them now.

If you are familiar with vSphere performance charts, feel free to skip ahead.

HOL-1704-CHG-2

HOL-1704-CHG-2

Performance charts can be viewed at the datacenter, cluster, resource pool, ESXi host,or virtual machine level.

In the vSphere Web Client,

1. Click on the object of interest. In this case, click challenge-01a.

2. Click on the Monitor tab.

3. Click the Performance tab.

4. You can view the Overview or Advanced screens. Right now, click Advanced.

5. Click the pin icon to hide the left sidebar and give yourself more viewing room.

6. Click on the << arrows to hide the internal sidebar and give yourself even moreviewing room.


By default, the vSphere Web Client performance charts are updated every 20 secondsand display one hour of performance statistics.

HOL-1704-CHG-2

HOL-1704-CHG-2

You can switch views by clicking the drop down menu.

1. Now, click Chart Options to explore further.

HOL-1704-CHG-2

HOL-1704-CHG-2


The important parts of the Chart Options screen are highlighted. You don't need toclick anything here, as we are just outlining the important parts of this screen.

1. Select the set of metrics you would like to choose from.

2. Adjust the chart's timespan.

3. Objects vary by metric type. Here 0 refers to the VM's first vCPU, 1 refers to theVMs's second vCPU, and challenge-01a is the average of both.

4. Select or deselect chart metrics here.

5. Click Cancel to return without modifying the chart.

HOL-1704-CHG-2

HOL-1704-CHG-2


To bring back the left sidebar,

1. Click (0) Navigator.

2. Click the pin icon.

Your Challenge

Your challenge is to double the performance of challenge-01a!

If you make a change to the underlying infrastructure, please allow some time (30-60seconds) for the VM's performance to stabilize.

For this Challenge,

You are allowed to:

- Modify VM settings.

- vMotion VMs.

- Modify resource allocations.

- Restart SPECjbb. Sometimes if the available resources are very low on a VM, SPECjbbmight stop running, or SPECjbb will eventually stop running after 45 minutes. If theSPECjbb2005 window reports "Performance: Not Running", close the SPECjbb2005window and reopen it again from the VM desktop shortcut.

HOL-1704-CHG-2

HOL-1704-CHG-2

- If, for any reason, you need to restart this challenge, you can re-run the script C:\HOL-CHG\Module1-start.ps1. This will reboot all VMs and return some VM settings to the waythey were at the beginning of the challenge. Please keep in mind that not all settingsand virtual hardware will be reset, so if you make a change to the environment thatdoes not solve the challenge, please set it back.

You are not allowed to:

- Rename any virtual machines.

- Power off or reboot any virtual machines.

- Change the operation of SPECjbb by modifying any benchmark parameters onchallenge-01a or challenge-02a (which is running the Linux version of SPECjbb2005).

- Turn off SPECjbb on any VM.

- Look at the Module1-start.ps1 script (that would be cheating!)

If you do these things, it threatens the integrity of the teleporter network...and then youmight not have a way to get home tonight!

Need a hint?

View Hint #1

View Hint #2

When you feel you have found the answer, or if you give up, proceed to the next page.

Solutions

Just as in real life, there is more than one way to resolve this performance problem.

The root cause of the poor performance is CPU contention, or high CPU overcommitment. Both challenge-01a and challenge-02a are placing heavy CPU load onesx-02a. Further, the host esx-02a has only two cores, while challenge-01a andchallenge-02a have 3 vCPUs total which are each asking for 100% utilization. Toimprove VM performance, eliminate or mitigate the contention. The primary way todetect the presence of CPU contention is by monitoring the ESXi metric % Ready.Readiness is the percentage of time that the virtual machine was ready, but could not

HOL-1704-CHG-2

HOL-1704-CHG-2

get scheduled to run on the physical CPU. It happens when virtual machines' demandfor CPU time exceeds the availability of the physical CPU. As a rule of thumb, ifReadiness exceeds 10%, CPU contention could be negatively affecting virtual machineperformance, although the acceptable ready time will depend on your environment. Youmay have noticed that the Windows 2012 system kernel appears to consume a largerproportion of the VM's CPU time when there is heavy CPU contention on the host.Windows guest CPU time accounting can become skewed when there is resourcecontention on the host, so take these metrics with a grain of salt.

High percent ready can also occur if over conservative power management techniquesare slowing the CPU; see KB 1018206 (opens in a new window). Always ensure that yourhost's BIOS power management setting is set to "OS controlled" and that your ESXipower mangement profile is "Balanced" or "High Performance."

Solutions to this scenario:

1. Migrate challenge-01a from ESXi host esx-02a to ESXi host esx-01a. esx-01a isnot being used, so with fewer VMs contending for CPU resources, performanceincreases. Alternatively, you could migrate challenge-02a from esx-02a to esx-01a,for the same reason, esx-01a is not currently in use and has available CPU resources.

2. Use Resource Allocation to limit the CPU MHz given to challenge-02a. Using "EditResource Settings," place a CPU limit on challenge-02a.

3. Use Resource Allocation to guarantee challenge-01a will receive at least a certainnumber of CPU MHz. Using "Edit Resource Settings," create a CPU reservation forchallenge-01a.

If you solved this challenge some other way, good for you. However, please return thesetting to its prior state if you will take another module after this one.

For more resources to understand CPU performance,

• Take Lab HOL-1704-SDC-1, vSphere Performance Optimization.• Whitepaper: vSphere Resource Management for ESXi 6.0(opens in a new

window)• Whitepaper: The CPU Scheduler in VMware vSphere 5.1 (opens in a new

window)• Although it is useful to monitor CPU utilization on a per-host level, vRealize

Operations is the tool of choice to monitor the CPU utilization of your datacenter.Take the Hands-on Lab HOL-1710-SDC-1 or HOL-1710-SDC-3 to find out more.

You've finished Module 1

Congratulations on completing the first challenge, Module 1. The teleporters are still upand running! PanContinental thanks you. All in a day's work.

HOL-1704-CHG-2

HOL-1704-CHG-2


https://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-6-pubs.html

http://www.vmware.com/resources/techresources/10345

Proceed to any module below which interests you most. If you are relatively new tovSphere, choose the Basic challenges, and if you are an experienced administrator, theAdvanced challenges are more suited for you.

• Module 2 - Operations Challenge, Basic Level (30 minutes) This modulechallenges you to solve common vSphere operational issues, including an emptyinventory in the vSphere Web Client and a non-functional VM Remote Console.




HOL-1704-CHG-2

HOL-1704-CHG-2

Module 2: OperationsChallenge, Basic Level (30

minutes)

HOL-1704-CHG-2

HOL-1704-CHG-2

Unable to login to vSphere Web ClientYou've just reached office, about to grab a nice cup of coffee when your new colleague,Murphy called. He recently ran into problems logging into vSphere Web Client using hiscredentials. As a long-time VMware expert, you have volunteered to help him resolvethis issue.




HOL-1704-CHG-2

HOL-1704-CHG-2



HOL-1704-CHG-2

HOL-1704-CHG-2





watermark.


Launch Chrome

1. On your desktop, locate the Chrome shortcut.2. Double click to open it.

HOL-1704-CHG-2

HOL-1704-CHG-2

Launch vSphere Web Client

To launch the vSphere Web Client:

• Click on the bookmark RegionB Web Client


Your colleague's user name "[email protected]" is reporting login failure fromvCenter Single Sign On. This is your colleague's login credentials:

1. User name: [email protected]. Password: VMware1!

You are provided with the vCenter Single Sign On Administrator account to help with thistask:

1. User name: [email protected]. Password: VMware1!

For this challenge:

1. You are to investigate into the account "[email protected]" and identifyMurphy is not able to log into vSphere Web Client

2. Fix the issue and test to ensure [email protected] is able to log in intovSphere Web Client

Need hints?

View Hint #1

Solution

Many times, users can get logged out of the vSphere system if:

HOL-1704-CHG-2

HOL-1704-CHG-2

1. Their account has been disabled by the SSO administrator2. They have too many failed attempts that breach the Lockout policies set by the

SSO administrators.

To solve this challenge:

1. Log in to RegionB Web Client as a SSO administrator2. Click on Administration -> Single Sign-On -> Users and Groups3. Look for account name murphy.4. Right click and enable the user account murphy.

Congratulations!

Murphy wants to extend a huge thanks to you. He promised to bring you your favoritecoffee brew the next time he comes to office. Awesome to have you as part of the team!

HOL-1704-CHG-2

HOL-1704-CHG-2

Empty inventory in vSphere WebClient. Help!NOTE: In order to attempt this challenge, you would need to have completed theprevious challenge: Unable to login to vSphere Web Client. Please do not attempt thischallenge if you have not done so. Click here if you have not completed the "Unable tologin to vSphere Web Client." challenge.

Your colleague Murphy, was extremely happy that he can once again log in to vSphereWeb Client. Not long before, you've got a call from Murphy again. This time round he iscomplaining that although he can login to vSphere Web Client, he is not able to see anyvSphere inventory objects. He can't see vCenters, Datacenters, Clusters, ESXi host, VMs,etc.. He once again needs you, to help him resolve why is vSphere Web Client givinghim an "Empty Inventory" after logging in.

Launch Chrome

1. On your desktop, locate the Google Chrome shortcut.2. Double click to open it.


To launch the vSphere Web Client

• Click on the bookmark RegionB Web Client

HOL-1704-CHG-2

HOL-1704-CHG-2


Login to vSphere Web Client using this credentials:

1. User name: [email protected]. Password: VMware1!3. Click Login button

Navigate to Host and Clusters

1. Selecton Hosts and Clusters

Empty Inventory?

[email protected] is supposed to be granted Read-Only access permission onthe vcsa-01b vCenter server.

But you are seeing an "Empty Inventory" after logging into the RegionB vSphere WebClient. Your task is to fix [email protected] account permissions.

HOL-1704-CHG-2

HOL-1704-CHG-2

You may use the vCenter Single Sign On administrator account to do so:

• User name: [email protected]• Password: VMware1!

Need Hint?

View Hint #1

Solution

Users require a minimum of read-only permissions at the vCenter Server level. If thesepermissions are missing, users will receive the Empty Inventory status.

To resolve the issue, assign the affected users a minimum of read-only permissions atthe vCenter Server level.

To assign read-only permissions:

1. Log in to the vCenter Server as administrator with the vSphere Web Client.2. Open RegionB Web Client, click vCenter Inventory Lists, vCenter Servers,

vcsa-01b.corp.local3. Click Manage -> Permissions.4. Right click on VSPHERE.LOCAL\murphy, select Change Role...5. Change access to Read-only and hit Ok.

HOL-1704-CHG-2

HOL-1704-CHG-2

VMRC is not allowing me to connect tomy virtual machineMurphy is trying to use VMware's Remote Console to access one of his systems, linux-desk-01a, that manages the usage of the teleportation particles across the globe. TheESXi host, esx-01b, which is hosting the VM has recently gone through some securitytesting. It was after this activity, that Murphy realizes that he could no longer access hisVM using VMware Remote Console. However, despite multiple efforts to connect, he hassomehow given up and is looking at you, the VMware expert to solve it for him.

Launch Chrome

1. On your desktop, locate the chrome shortcut.2. Double click to launch it.


• From the chrome bookmarks, clickon RegionB Web Client.

HOL-1704-CHG-2

HOL-1704-CHG-2


To login to vSphere Web Client:

1. User name: CORP\Administrator2. Password: VMware1!3. Click Login

HOL-1704-CHG-2

HOL-1704-CHG-2


1. Select Hosts and Clusters

HOL-1704-CHG-2

HOL-1704-CHG-2

Launch Remote Console of Virtual Machine

To launch a Remote Console to the virtual machine (linux-desk-01):

1. Expand the vcsa-01b inventory to reveal the inventory of the vCenter.2. Click on the virtual machine linux-desk-01.3. Click on the Summary tab.4. Click on Launch Remote Console.

VMRC access issues?!

Once you have launched Remote Console from vSphere Web Client, you will bepresented with this error when trying to connect to Remote Console of virtual machinelinux-desk-01. Your task is to resolve this issue and allow user to access their virtualmachines using Remote Console.

Additional Information: This issue was discovered only after your security team changedmade changes to security settings on your ESXi host (esx-01b)

Need hint?

View Hint #1

HOL-1704-CHG-2

HOL-1704-CHG-2

View Hint #2

Solution

Before you can use VMware Remote Console (VMRC) to open a remote console to avirtual machine, you must enable SSL authentication on the ESX/ESXi host. SSLauthentication is enabled by default on ESX/ESXi hosts.

To enable SSL authentication from GUI:

1. Connect to the vCenter that is managing the host using the vSphere Web Client.2. Select esx-01b.3. Click the Manage tab.4. Under Settings - > System, click Advanced System Settings. Search for the name

Config.Defaults.security.host.ruissl option and set it to yes/true.

To find out more, you can read this KB article.

HOL-1704-CHG-2

HOL-1704-CHG-2


Why is SSH not working on ESXi host?Murphy has been really happy with your help till now. Good job! He recently raised someissue when connecting to an ESXi host (esx-01b) using SSH, despite multiple attemptsto connect using SSH protocol, Murphy still could not connect. He has now come back toyou, for help with this issue.

Launch PuTTY client

1. On your desktop taskbar, locate the PuTTY client shortcut2. Click on PuTTY to launch it.

HOL-1704-CHG-2

HOL-1704-CHG-2

Connect to ESXi host using SSH

To connect to ESXi host esx-01b using SSH:

1. Click on esx-01b from Saved Sessions.2. Click on Open button.

Additional info:

1. SSH user: root2. SSH password: VMware1!

Connection Refused?

Once you try to connect to ESXi host esx-01b through SSH, your PuTTY client is givingyou a "Network error: Connection refused" message. Murphy needs to have SSH accessto the ESXi host in order to perform certain troubleshooting activities.

Your challenge is to resolve the SSH communication issues with ESXi host esx-01b.

NOTE: You may want to login to the RegionB Web Client to check on esx-01b usingthese credentials:

HOL-1704-CHG-2

HOL-1704-CHG-2


Need a Hint?

View Hint #1

View Hint #2

Solution

During most ESXi troubleshooting activities, many administrators do get into issuestrying to establish a remote SSH session with a ESXi host. This is because SSH is oftendisabled to prevent remote access to the ESXi host. It is only activated duringtroubleshooting activities by the administrators.

To enable SSH on the ESXi host using vSphere Web Client:

1. Login to RegionB Web Client2. Search and locate the ESXi host (esx-01b)3. Go to Manage tab4. Click on Settings -> Security Profile5. Under Services(scroll down below "Firewall"), check the status of SSH.6. Click Edit, SSH, and Start to change the service status to "Running"

HOL-1704-CHG-2

HOL-1704-CHG-2

7. Verify SSH works now by switching back to the failed PuTTY window and trying torestart the session.

Awesome!

Yes, you did it! Now that everything is running smoothly again, you've just ensured thatPanContinental's teleporters will not have any major issues.

Please proceed to the next page.


Congratulations on completing Module 2!






HOL-1704-CHG-2

HOL-1704-CHG-2

Module 3: PerformanceChallenge, Basic Level (30

minutes)

HOL-1704-CHG-2

HOL-1704-CHG-2

Why is my VM slow?You're sitting in your office, reading Performance Best Practices for VMware vSphere 6(opens in a new window), when you decide to double check the configuration for yourtwo transporter particle sender VMs, challenge-01a and challenge-04a. You want tomake sure they are in line with best practices. Maybe you should double check theirperformance numbers while you are at it.






1. Open the Challenge Scripts folder on the Desktop.

HOL-1704-CHG-2

HOL-1704-CHG-2

http://www.vmware.com/files/pdf/techpaper/VMware-PerfBest-Practices-vSphere6-0.pdf

Start Module 3 part 1 script

1. Right click on Module3-part1.ps1

2. From the context menu, select Run with PowerShell.

Module 3 part 1 script Starts

The script will take a minute to complete. In the meantime, proceed to the next stepand we will go over some challenge guidelines.

HOL-1704-CHG-2

HOL-1704-CHG-2



If you are viewing this Lab in Full Screen Mode, you can press the F11 key to disableFull Screen Mode and access the internet.


HOL-1704-CHG-2

HOL-1704-CHG-2




HOL-1704-CHG-2

HOL-1704-CHG-2


When you first start your lab, you might notice a watermark on the desktop indicatingthat Windows is not activated.



watermark.

This cosmetic issue has no effect on your lab. If you have any questions or concerns,please feel free to use the support made available to you either at VMworld in theHands-on Labs area, in your Expert-led Workshop, or online via the survey comments aswe are always looking for ways to improve your Hands-on Lab experience.

HOL-1704-CHG-2

HOL-1704-CHG-2

Module 3 part 1 Script Finishes

When the script is finished, it will read You may now close this window. Your exactoutput may vary from that above depending on which module you completed prior tothis one.


Open Remote Desktop for challenge-01a andchallenge-04a

On the Desktop, double click on chg-01a.RDP to open a Remote Desktop Session tothe virtual machine challenge-01a.

Next, do the same for chg-04a.RDP.

If you receive the popup "Remote Desktop can't connect to the remote computer...,"click OK, please wait a moment and try again to open Remote Desktop.

HOL-1704-CHG-2

HOL-1704-CHG-2

Position the windows so you can monitor both at the sametime

Position the Remote Desktop windows so it is comfortable for you to view both at thesame time.

HOL-1704-CHG-2

HOL-1704-CHG-2

Explanation of Desktop

The virtual machines challenge-01a and challenge-04a have some tools installed onthem to help you monitor their well-being. On the desktop, you can see informationabout each virtual machine including its hostname, IP address, CPU usage, and memoryusage. The CPU usage and memory usage update in real time.

In the CPU usage chart, you can see both the total CPU used ("Total:"), and thepercentage of CPU usage that is consumed by the guest operating system's kernel("Krnl:").

1. On challenge-01a, double click on SPECjbb to start the performance measurementapplication.

SPECjbb2005

A window titled "SPECjbb2005" opens. A copy of SPECjbb2005 is installed onchallenge-01a. SPECjbb2005 is a real SPEC java-based benchmark that can measureCPU and memory performance.

HOL-1704-CHG-2

HOL-1704-CHG-2

1. Click the Start Benchmark button to start the benchmark.

SPECjbb starts

It will take a few seconds for the benchmark to begin displaying performance results.

While SPECjbb is starting on challenge-01a, repeat the process on challenge-04a:Double click the SPECjbb icon on challenge-04a and click the Start Benchmarkbutton.

Proceed to the next page.

SPECjbb is running

1. When SPECjbb starts producing performance results, you will see the PerformanceScore displayed in the SPECjbb2005 window.

2. The text file at the bottom shows the output of the benchmark in real time. Theperformance results, measured as business operations per second (BOPS), or

HOL-1704-CHG-2

HOL-1704-CHG-2

throughput, are highlighted in yellow. This window is just for viewing the execution ofthe benchmark and you can close this window without affecting the operation of thebenchmark.

If you close the SPECjbb2005 window, or hit "Stop Benchmark," the benchmark will stop.To analyze the performance results, you should leave the benchmark running.

Performance results update every 10 seconds. When you start SPECjbb2005 forthe first time or make changes to the environment, you might have to wait a fewupdate cycles for performance results to stabilize or reach steady-state.



Compare challenge-01a and challenge-04a performance

As the benchmark reaches a steady performance, you can notice a difference in the waychallenge-01a and challenge-04a are performing. Check the SPECjbb Performancescore for both VMs.

Which VM has better performance, that is, a higher Performance score in SPECjbb2005?

Turn the page once you think you can see a consistent difference.

HOL-1704-CHG-2

HOL-1704-CHG-2

challenge-04a is performing better than challenge-01a

It soon becomes clear that challenge-04a is performing better than challenge-01a.Your results might not exactly match those above depending on the Hands-on Labenvironment.

The Performance Score is higher for challenge-04a, and you can also see a differenceif you look at the CPU usage over time. challenge-01a has much higher kernel time onaverage than challenge-04a.

Oh no! challenge-01a and challenge-04a replenish the flow of teleportation particlesto Waypoints™! If one is running faster than another, Waypoints that are supplied bychallenge-01a could run out of particles and become non-functional, while Waypointsmapped to challenge-04a could receive too many which would make the Waypointsradioactive and a health risk to customers.

But where are these VMs located and how can you manage them?

HOL-1704-CHG-2

HOL-1704-CHG-2

Open Chrome

Double click on the desktop icon to open Chrome.

HOL-1704-CHG-2

HOL-1704-CHG-2

Open Region A Web Client

On the bookmarks bar, click RegionA Web Client to open the vSphere Web Client.


Select Use Windows session authentication and click Login.


If needed, at the Home page, select the Hosts and Clusters icon.

Refresh the Web Client When Needed

Because this is the Hands-on Labs, you might need to manually refresh the Web Clientto see your changes. Click on the arrow to refresh.


Expand vcsa-01a.corp.local as necessary. challenge-01a is managed by vCenterServer vcsa-01a.corp.local. It is hosted on esx-03a.corp.local along with VMschallenge-04a and challenge-05a.

challenge-01a and challenge-04a are both Windows Server 2012 VMs, whilechallenge-05a is a Linux VM.

HOL-1704-CHG-2

HOL-1704-CHG-2

Your coworker Murphy is also an administrator on vcsa-01a.corp.local. challenge-01aand challenge-05a are two of Murphy's favorite VMs.

But why is challenge-01a running slowly if it's one of his favorite VMs?


For troubleshooting a performance issue in real time, the vSphere performance chartsare a first-line tool. Let's try using them now.

If you are familiar with vSphere performance charts, feel free to skip ahead.

Performance charts can be viewed at the datacenter, cluster, resource pool, ESXi host,or virtual machine level.


1. Click on the object of interest. In this case, click challenge-01a.



4. You can view the Overview or Advanced screens. Right now, click Advanced.

HOL-1704-CHG-2

HOL-1704-CHG-2

5. Click the pin icon to hide the left sidebar and give yourself more viewing room.

6. Click on the << arrows to hide the internal sidebar and give yourself even moreviewing room.

HOL-1704-CHG-2

HOL-1704-CHG-2


By default, the vSphere Web Client performance charts are updated every 20 secondsand display one hour of performance statistics.

You can switch views by clicking the drop down menu.

1. Now, click Chart Options to explore further.

HOL-1704-CHG-2

HOL-1704-CHG-2


The important parts of the Chart Options screen are highlighted. You don't need toclick anything here, as we are just outlining the important parts of this screen.

1. Select the set of metrics you would like to choose from.

2. Adjust the chart's timespan.

3. Objects vary by metric type. Here 0 refers to the VM's first vCPU, 1 refers to theVMs's second vCPU, and challenge-01a is the average of both.

4. Select or deselect chart metrics here.

5. Click Cancel to return without modifying the chart.

HOL-1704-CHG-2

HOL-1704-CHG-2


To bring back the left sidebar,

1. Click (0) Navigator.

2. Click the pin icon.

Your Challenge

Your challenge is to increase the performance of the VM challenge-01a so that theperformance of challenge-01a and challenge-04a are roughly equal. challenge-04amay show somewhat decreased performance as a result. For a roughly equal result, tryto make the difference in performance scores between challenge-01a andchallenge-04a a maximum of 2000. Additionally, each VM's kernel time should beroughly equivalent.


For this Challenge,

HOL-1704-CHG-2

HOL-1704-CHG-2

You are allowed to:


- Modify resource allocations.

- vMotion VMs.

- Restart SPECjbb. Sometimes if the available resources are very low on a VM, SPECjbbmight stop running, or SPECjbb will eventually stop running after 45 minutes. If theSPECjbb2005 window reports "Performance: Not Running", close the SPECjbb2005window and reopen it again from the VM desktop shortcut.

- If, for any reason, you need to restart this challenge, you can re-run the script C:\HOL-CHG\Module3-part1.ps1. This will reboot all VMs and return some VM settings to the waythey were at the beginning of the challenge. Please keep in mind that not all settingsand virtual hardware will be reset, so if you make a change to the environment thatdoes not solve the challenge, please set it back.




- Modify VM virtual hardware.

- Change the operation of SPECjbb by modifying any benchmark parameters onchallenge-01a, challenge-04a, or challenge-05a (which is running the Linux versionof SPECjbb2005).


If you do these things, it threatens the integrity of the teleporter network! Half of youmight end up in one place and half in the other, and that's not pretty.

Need a hint?

View Hint #1

View Hint #2


HOL-1704-CHG-2

HOL-1704-CHG-2

I have a question for you

Question: Did you set the FavoriteVMs resource pool to CPU shares of 4000?

If so, that's not the correct answer. Try again and proceed to the next page when youfeel you have found the answer, or if you give up.

Solutions

There are many ways you can resolve this.

Murphy created a FavoriteVMs resource pool, possibly to keep some VMs logicallydistinct from others (we'll never know why).

Resource pools always have performance implications for the VMs located inside them.The resources assigned to the pool are split among the VMs in the pool accordingto the VM-level resource allocation settings.

The resource pool had a total of 500 CPU shares. That means that:

challenge-01a had 166 CPU shares because the VM had 1000 shares, 1/3 of theresource pool.

challenge-05a had 333 CPU shares because the VM had 2000 CPU shares, 2/3 of theresource pool.

challenge-04a had "Normal" CPU shares which equaled 2000.

Because the shares for the powered-on VMs totaled 2500, challenge-01a had 7% ofthe total CPU shares, challenge-05a had 13%, and challenge-04a had 80%.

This resulted in CPU contention and therefore high %ready time for challenge-01a butnot challenge-04a. Did you see that challenge-01a had a higher amount of kerneltime than challenge-04a? This is equivalent to ready time, where kernel time is theway that the guest OS sees ESXi's ready time.

Solutions to this scenario:

1. Removing challenge-01a from the resource pool and setting its VM-level resourceallocation to 2000 CPU shares would give equal shares to challenge-01a andchallenge-04a.

2. You could migrate challenge-01a from ESXi host esx-03a to ESXi host esx-02a.esx-02a is not being used, so with fewer VMs contending for CPU resources,performance increases. Alternatively, you could migrate challenge-04a from esx-01ato esx-02a to accomplish the same goal.

HOL-1704-CHG-2

HOL-1704-CHG-2

PanContinental thanks you!

Murphy thanks you too. Next time he brings donuts to work, you will definitely get one.Thanks for being part of the team.

HOL-1704-CHG-2

HOL-1704-CHG-2

Troubleshoot a poorly performing VMAfter finishing lunch, you get back to your desk to get a phone call from Murphy.

"I powered on an extra particle reassembly VM, challenge-03a, and you should seewhat happened to challenge-01a. I powered challenge-03a off for now-- But can youtake a look at it?"

Let's go!

What's going on? And even more importantly...can you take a meditation break afterthis?

No matter, you will get things fixed up, because you are "ready for any"...anything, thatis!



HOL-1704-CHG-2

HOL-1704-CHG-2

Start Module 3 part 2 script

1. Right click on Module3-part2.ps1

2. From the context menu, click on Run with PowerShell.

Module 3 Part 2 Script Finishes

Wait for the script to complete. When it is finished, it will read You may now close thiswindow. Your exact output might vary from that above depending on which module youcompleted prior to this one.



From the desktop, double click on chg-01a.RDP to open a Remote Desktop session tothe virtual machine challenge-01a.

HOL-1704-CHG-2

HOL-1704-CHG-2


The virtual machine challenge-01a has some tools installed on it to help you monitorits well-being. On the desktop, you can see information about the virtual machineincluding its hostname, IP address, CPU usage, and memory usage. The CPU usage andmemory usage update in real time.

In the CPU usage chart, you can see both the total CPU used ("Total:") and thepercentage of CPU usage that is consumed by the guest operating system's kernel("Krnl:").

1. Double click on SPECjbb to start the performance measurement application.

Start SPECjbb

A window titled SPECjbb2005 opens. A copy of SPECjbb2005 is installed onchallenge-01a. SPECjbb2005 is a real SPEC java-based benchmark that can measureCPU and memory performance.

HOL-1704-CHG-2

HOL-1704-CHG-2

1. Click the Start Benchmark button to start the benchmark.

SPECjbb starts.

Wait a few seconds for the benchmark to begin running.

SPECjbb is running

1. Wow, looks like performance here is doing great. When you make changes to theenvironment, you may have to wait a few cycles for performance results tostabilize or reach steady-state.



CPU usage is high and steady, indicating the VM is getting the most it can out of itsvCPUs. Memory is only 62% used.

HOL-1704-CHG-2

HOL-1704-CHG-2

Remember this performance score because you will need to know it later in thechallenge.

Let's check our next VM, challenge-03a.

Open Chrome

Back on the desktop, double click on the desktop icon to open Chrome. If Chrome is stillopen and logged into the Web Client, you can skip this step.

HOL-1704-CHG-2

HOL-1704-CHG-2

Open Site A Web Client

If you are not automatically taken to the VMware vCenter Single Sign on Page, on thebookmarks bar, click the RegionA Web Client to open the vSphere Web Client.


1. Select Use Windows session authentication

2. Click Login.


If needed, at the Home page, select the Hosts and Clusters icon.

Refresh the Web Client When Needed

Because this is the Hands-on Labs, you may need to manually refresh the Web Client tosee your changes. Click on the arrow to refresh.

HOL-1704-CHG-2

HOL-1704-CHG-2


Expand vcsa-01a as necessary. challenge-01a is managed by vCenter Servervcsa-01a. It is hosted on esx-03a.

Power on challenge-03a

Let's power on Murphy's particle reassembly VM challenge-03a and see what happens.

1. Click challenge-03a.

2. Click the drop down Actions menu for challenge-03a.

3. Hover over Power.

4. Then click Power On.

HOL-1704-CHG-2

HOL-1704-CHG-2

View challenge-01a Remote Desktop session

Click back to challenge-01a to see how it was affected by powering onchallenge-03a.

What happened??

Your Remote Desktop connection got dropped? You might see this screen, or maybe not.What do you think caused it? Please proceed to the next page.

Disconnected Remote Desktop Session

Wait a bit and the Remote Desktop Session will reconnect. Probably had something todo with the VM you powered on.

HOL-1704-CHG-2

HOL-1704-CHG-2

challenge-01a Behavior Has Changed

Oh dear.

1. challenge-01a is back, but the Performance Score is a fraction of what it was before.

2. The Kernel's CPU usage has jumped, and memory usage has increased to 94%.

What happened to challenge-01a?

Your Challenge

Your challenge is to increase the performance score of challenge-01a! Can youincrease the performance score to what it was before you powered on challenge-03a?


For this Challenge,

You are allowed to:

- vMotion VMs.


- Restart SPECjbb. Sometimes if the available resources are very low on a VM, SPECjbbmight stop running, or SPECjbb will eventually stop running after 45 minutes. If the

HOL-1704-CHG-2

HOL-1704-CHG-2

SPECjbb2005 window reports "Performance: Not Running", close the SPECjbb2005window and reopen it again from the VM desktop shortcut.

- If, for any reason, you need to restart this challenge, you can re-run the script C:\HOL-CHG\Module3-part2.ps1. This will reboot all VMs and return some VM settings to the waythey were at the beginning of the challenge. Please keep in mind that not all settingsand virtual hardware will be reset, so if you make a change to the environment thatdoes not solve the challenge, please set it back.





- Change the operation of SPECjbb by modifying any benchmark parameters onchallenge-01a.


If you do these things it threatens the integrity of the teleporter network! Someonemight get their hat left behind.

Need a hint?

View Hint #1

View Hint #2


Solutions

When VM challenge-03a was powered on, it started executing a memory-intensiveworkload.

Memory on the esx-03a host was already overcommitted: the virtual memory allocatedto the virtual machines exceeded the physical memory capacity of the host. The

HOL-1704-CHG-2

HOL-1704-CHG-2

esx-03a host has 4 GB of physical memory. challenge-01a was assigned 2 GB ofvirtual memory and challenge-03a was assigned 3 GB of virtual memory, which totals5. Remember also that ESXi requires some memory overhead for itself and for each VMit is running.

This can be an acceptable scenario when each VM is not using all its available memory,however it does put your performance at risk if memory needs increase.

In this case, the performance of challenge-01a suffered when challenge-03a startedrunning because the memory demand exceeded the supply. Like most applications, theperformance of SPECjbb on challenge-01a will be negatively affected if memory isscarce.

To solve this issue, you might have:

1. vMotioned challenge-01a onto a different host where there is no contention.

2. Created a memory reservation for challenge-01a, or increased its memory shares.

3. Created a memory limit on challenge-03a, or decreased its memory shares.

HOL-1704-CHG-2

HOL-1704-CHG-2

View Guest Memory on challenge-01a

Whether or not your workloads are still running, let's take a look at the memory state onchallenge-01a in the web client.

1. In the Hosts and Clusters view, click on challenge-01a.


3. Click on the Utilization tab.

Active Guest Memory: This is the amount of memory that ESXi estimates the VM isactively using.

Not all of the 2GB of virtual memory is in the same state.

Private: This is the virtual memory which maps to physical memory. Only 25% of theVM's memory is private.

Shared, Compressed, Swapped, and Ballooned are all forms of memoryreclamation.


HOL-1704-CHG-2

HOL-1704-CHG-2

HOL-1704-CHG-2

HOL-1704-CHG-2

challenge-01a memory usage before memory pressure andafter memory pressure begins

The figure on the top shows challenge-01a guest memory with no memory pressure.Active memory is 1.86GB. All memory is either private or unaccessed.

After memory pressure begins, ESXi begins reclaiming memory from challenge-01a.ESXi uses different memory management mechanisms to dynamically reduce theamount of physical memory required for each virtual machine. In addition, the totalActive Guest Memory has decreased from 1.86GB to 1.40. Due to constraints, thebenchmark's use of memory has become limited.

Page Sharing: In vSphere 6.0, pages of memory are shared by default within a virtualmachine. Page sharing between virtual machines is disabled by default in vSphere 6.0,but can be enabled by setting the host advanced setting Mem.ShareForceSalting to0. See the book Performance Best Practices for VMware vSphere 6.0 for details.

HOL-1704-CHG-2

HOL-1704-CHG-2

Memory Management in ESXi

ESXi can use a number of memory management techniques:

Ballooning: The memory balloon driver (vmmemctl) collaborates with ESXi to reclaimpages that are considered the least valuable by the operating system.

The driver uses a ballooning technique that closely matches the behavior of a nativesystem under similar memory constraints. This technique increases or decreases

HOL-1704-CHG-2

HOL-1704-CHG-2

memory pressure on the guest operating system, causing the guest to use its ownnative memory management algorithms. When memory is tight, the guestoperating system decides which pages to reclaim and, if necessary, swaps them to itsown virtual disk.

Some ballooning is normal and is not indicative of a problem.

You may have noticed that the memory consumption of challnege-01a increased afterchallenge-03a was powered on. This was due to memory reclamation techniques,especially ballooning.

Compressed: ESXi compresses memory pages to reduce the number of pages needed.The time required to decompress a page of memory is less than that required to swap todisk.

Swapped: ESXi forcibly reclaims memory from the virtual machine by writing pages ofmemory to disk. This mechanism can cause virtual machine performance to degradesignificantly due to its high access latency.

These memory reclamation techniques are applied in a hierarchical manner according tothe degree of memory pressure in the environment. Page sharing is the first techniqueapplied, which has almost no performance impact. If this does not resolve the shortage,ballooning occurs. If needed, compression will be used, which carries a greaterperformance impact because the compressed page must be decompressed before it canbe read or written. Swapping is used as a last resort due to its high performanceimpact: the page will be read in from disk instead of from memory.

HOL-1704-CHG-2

HOL-1704-CHG-2

Great job!

Yes, everything is running smoothly again! No teleporter casualties except for somepocket lint that got left behind. I don't think anyone will miss that.


HOL-1704-CHG-2

HOL-1704-CHG-2


Congratulations on completing Module 3.






HOL-1704-CHG-2

HOL-1704-CHG-2

Module 4: OperationsChallenge, AdvancedLevel (30 minutes)

HOL-1704-CHG-2

HOL-1704-CHG-2

Why does my ESXi host keepsdisconnecting?!It is another day here at PanContinental, where a huge part of your job is to ensure thatthe datacenter runs smooth and is operational.

You have been kept busy recently by Murphy and it looks like he is not going to stopanytime soon. Just as you are about to look away from your email, you've got a pingfrom Murphy, describing that his ESXi host (esx-02b) has been disconnected. He is veryconcerned about this because the system that is charging the various teleportationwaypoints is being managed by a VM (linux-base-01) on this host. The VM is stilloperational even though the ESXi host (esx-02b) has been disconnected from vCenterand Murphy highlighted that this issue is preventing him from managing the ESXi hostand VM from vSphere Web Client.

Making things seem more urgent, he stressed that there is a coming system patch thathe needs to run that will affect the operational readiness of all teleportation waypointsacross the globe if he does not get access to his VM soon!




HOL-1704-CHG-2

HOL-1704-CHG-2



HOL-1704-CHG-2

HOL-1704-CHG-2





watermark.




HOL-1704-CHG-2

HOL-1704-CHG-2

Start Module 4 script

1. Right click on Module4-start.ps12. From the context menu, select Run with Powershell

Module 4 script finishes

Wait for the script to complete. When it is finished, it will read "Module 4 is ready".

Press Enter to close the window or use the close button in the top right hand corner.

Proceed to next step.

Open Chrome

Back on the desktop, double click on the desktop icon to open Chrome.

HOL-1704-CHG-2

HOL-1704-CHG-2

Open RegionB Web Client

If you are not automatically taken to the VMware vCenter Single Sign on Page, on thebookmarks bar, click the RegionB Web Client to open the vSphere Web Client.


1. Select Use Windows session authentication2. Click Login.


If needed, at the Home page, click the Hosts and Clusters icon.


Expand vcsa-01b as necessary. esx-02b is managed by vCenter Server vcsa-01b.

HOL-1704-CHG-2

HOL-1704-CHG-2

vCenter Alarm Details

To investigate further on this issue:

1. Click on the Alarms tab2. Click on the alarm esx-02b.corp.local3. Drill into the details of the alarm

HOL-1704-CHG-2

HOL-1704-CHG-2

Drilling into the alarm details and definitions

1. Click on esx-02b2. Click on the Monitor tab3. Click on Issues4. Navigate to Triggered Alarms5. Click on Host connection and power state

HOL-1704-CHG-2

HOL-1704-CHG-2

Drilling into the alarm details and definitions

1. Expand on Triggers

By now, you would be convinced that the ESXi host (esx-02b), has either lostconnection or not responding. Your task is to bring esx-02b back to normal state andmanaged by vCenter (vcsa-01b).

After asking around, you heard rumors from your team that some rogue administratormight have meddle with service(s) running on the ESXi host.

Need Hints?

View Hint #1

View Hint #2

Solution

This issue is very common if vCenter server could not maintain a heartbeat with an ESXihost, even though the ESXi host is still online, VMs are still running. There is a highchance that the management agents are either not running, or require a restart.

To check the status of the management agent on ESXi host using remote SSH:

1. Log in to ESXi host using PuTTy or your SSH client as root

HOL-1704-CHG-2

HOL-1704-CHG-2

2. Run these commands:

/etc/init.d/vpxa status

To restart the management agent on ESXi host using remote SSH:

1. Log in to ESXi host using PuTTy or your SSH client as root2. Run these commands:

/etc/init.d/vpxa start

For more information, you may want to check out this KB article.

HOL-1704-CHG-2

HOL-1704-CHG-2

http://kb.vmware.com/selfservice/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1003490

Unable to login to vCenter using SSOAdmin account to assignpermission(SSO Administrator accountlocked)Murphy has recently asked for special approval to allow himself to manage permissionson his own VMware environment. He has hired a good team of engineers to offloadsome of the administrative tasks. This will allow you to focus more on the upcomingexpansion of the teleportation waypoint manufacturing plant.

As such, Murphy has been given a SSO administrator account. So he can assignpermission rights to his own team members to help manage his environment. However,few days later, he starts pinging you again about not being able to access vSphere WebClient using the SSO administrator account that you've created for him.

He gave you two options, either you help him solve his login issue, or you can help himassign 100 new accounts to his new team members. Obviously, you have chosen to helphim with his login issue.

Open Chrome

On the desktop, double click on the desktop icon to open Chrome.


From the chrome bookmark bar, click on RegionB Web Client.


This is the SSO administrator account ([email protected]) that Murphy hasissues logging into. The login credentials are:

1. User name: [email protected]

HOL-1704-CHG-2

HOL-1704-CHG-2

2. Password: VMware1!3. Click Login

Login is incorrect!

You would have noticed by now that it is not like Murphy, to give you a wrong usernameor password. This issue is definitely more than just an "Incorrect username/password".

To help you with this task, you are given access to the default SSO administratoraccount:


For this task, you would need to:

1. Find out what is wrong with [email protected]. Find a way to fix the login issue3. Test that [email protected] can login once again to vcsa-01b.corp.local

Need a Hint?

View Hint #1

View Hint #2

Solution

It is very common for SSO administrator account to be locked due to a couple ofreasons:

HOL-1704-CHG-2

HOL-1704-CHG-2

1. Exceeded the number of failed login attempts2. Password has expired based on Password Policy

Therefore, it is very important to constantly check for expiring passwords for yourVMware Single Single-On accounts. In addition, it is also crucial to apply properPassword Policies and Account Lockout Policies. This will better ensure that you do notget into a situation where you get a total lockout from your VMware vSphereenvironment.

You can unlock an account using another account with SSO administrator privileges:

1. Click Home.2. Click Administration.3. Click Single Sign-On > Users and Groups.4. Click the Users tab.5. Right-click the affected user account, such as [email protected], and

click Unlock.

For more information, you can read this KB article.

HOL-1704-CHG-2

HOL-1704-CHG-2


Unable to put server in maintenancemodeGreat work resolving Murphy's login issue! Now that the challenge is behind you, you'dlike to proactively carry out maintenance on one of the ESXi hosts to ensure you dontget any more calls. In order to do this, you will need to put the host in maintenancemode. Log into the vSphere Web client and put esx-01b into maintenance mode.

Open Chrome

On the task bar, click on the icon to open Chrome.


From the chrome bookmark bar, click on RegionB Web Client.


1. Select "Use Windows session authentication"2. Click Login.

HOL-1704-CHG-2

HOL-1704-CHG-2


1. Click on Home.2. Click on Hosts and Clusters.

HOL-1704-CHG-2

HOL-1704-CHG-2

Put esx-01b.corp.local in maintenance mode

1. Expand the quetips on vcsa-01b.2. Select then right click esx-01b3. Click Maintenance Mode4. Select Enter Maintenance Mode

HOL-1704-CHG-2

HOL-1704-CHG-2

Confirm Maintenance Mode

Click OK in the Confirm Maintenance Mode dialog box

Complete the Maintenance Mode Wizard

Click OK in the Warning dialog box

HOL-1704-CHG-2

HOL-1704-CHG-2

Monitor the task progress

Click Recent Tasks to view the task progress

View Enter Maintenance Task Progress

After waiting a couple minutes, you can see that the Enter Maintenance Mode task hasstalled at 19% and isn't progressing any further even though there is no error messager.

Your task:

HOL-1704-CHG-2

HOL-1704-CHG-2

1. Determine why the maintenance mode task is failing to complete2. Ensure that future attempts to put servers in the cluster into maintenance mode

will not require manual intervention3. Complete the maintenance activity by successfully putting esx-01b into

maintenance mode

Need a Hint?

View Hint #1

View Hint #2

HOL-1704-CHG-2

HOL-1704-CHG-2

Solution

DRS has 3 Automation Levels. The table above lists the Automation Levels and theirassociated actions.In order to ensure that VMs are migrated off a host without manual intervention, thecluster DRS Automation Level must be set to Fully Automated.

HOL-1704-CHG-2

HOL-1704-CHG-2

View DRS Automation level

1. Ensure that the RegionB01-COMP01 cluster is selected2. Click the Manage tab3. Select vSphere DRS4. Note that DRS Automation is set to Partially Automated

Set DRS Automation level to Fully Automated

Click Edit to change the DRS Automation level

HOL-1704-CHG-2

HOL-1704-CHG-2

Change the DRS Automation level

1. Click on Partially Automated to reveal the other DRS Automation level settings2. Select Fully Automated3. Click OK to confirm the cluster settings configuration change

HOL-1704-CHG-2

HOL-1704-CHG-2

Problem Solved!

Note that esx-01b has now successfully entered Maintenance mode.

HOL-1704-CHG-2

HOL-1704-CHG-2

Module 5: PerformanceChallenge, AdvancedLevel (30 minutes)

HOL-1704-CHG-2

HOL-1704-CHG-2

Poor Storage performanceToday's a great day. You woke up in a good mood. You've recently migrated from avSphere 5.1 environment to vSphere 6.0 and you're looking forward to what it has tooffer.

Checking your email, you see that the Waypoint Payments Operations units in theSouthwest global quadrant are reporting abnormally slow customer lookup processing.The payments operations are disk heavy workloads, so that is where you will start.

Troubleshooting is always an exhilarating way to start the day.







HOL-1704-CHG-2

HOL-1704-CHG-2

Start Module 5 script

1. Right click on Module5-start.ps1.

2. From the context menu, click on Run with PowerShell.

Module 5 script starts

The script will take a minute to complete. In the meantime, proceed to the next stepand we will go over some challenge guidelines.

HOL-1704-CHG-2

HOL-1704-CHG-2



If you are viewing this Lab in Full Screen Mode, you can press the F11 key to disableFull Screen Mode and access the internet.


HOL-1704-CHG-2

HOL-1704-CHG-2




HOL-1704-CHG-2

HOL-1704-CHG-2


When you first start your lab, you might notice a watermark on the desktop indicatingthat Windows is not activated.



watermark.

This cosmetic issue has no effect on your lab. If you have any questions or concerns,please feel free to use the support made available to you either at VMworld in theHands-on Labs area, in your Expert-led Workshop, or online via the survey comments aswe are always looking for ways to improve your Hands-on Lab experience.

Module 5 script is in progress

After the script gets to this point, do not close the window. Instead, proceed to the nextpage.

HOL-1704-CHG-2

HOL-1704-CHG-2


Double click on the chg-04a.RDP shortcut on the desktop to open a Remote DesktopSession.

Open Iometer

Double click on the Iometer icon on the desktop to open it.

HOL-1704-CHG-2

HOL-1704-CHG-2

Iometer

The Iometer window opens. Iometer (rhymes with "thermometer") is a storagebenchmark.

1. Click the green flag to start the test.

It will take the benchmark a minute to start. In the meantime:

2. Go back to the Powershell window on the Hands-on Lab desktop.

HOL-1704-CHG-2

HOL-1704-CHG-2

Continue Module 5 script

At the Powershell window on the Hands-on Lab desktop, press Enter twice to allow thescript to complete.

While the script completes, please proceed to the next page.

Module 5 script completes

Wait until the script finishes and reads You may now close this window as displayedabove.

Then close the window by pressing Enter twice or clicking the close button at the topright of the window.

HOL-1704-CHG-2

HOL-1704-CHG-2

Iometer Display

Go back to the Iometer window in the challenge-04a remote desktop session.

1. Click on the Results Display tab.

Iometer is placing storage load on the VM challenge-04a.

How many Total I/Os per Second (IOPS) is displayed? Only five? PanContinental's SLAfor this VM states that challenge-04a should be getting at least 16 I/Os per second.

Average I/O Response Time (ms) is approximately 12000 milliseconds, or 12seconds. Typical real-life I/O response times, or latencies, are between 1 to 10milliseconds depending on storage and I/O load characteristics. A 12 second I/Oresponse time would be extraordinarily long for real life. However, this isn't real life, sodon't worry about the response time. For this exercise, you should focus on fixing the I/Oper second.

Murphy said that before he migrated this VM from an ESXi 5.1 host to its current ESXi6.0 host, it was meeting the SLA fine on the 5.1 host. And there's nothing wrong withthe 6.0 host. What changed?

Let's check out the VM in the Web Client.

HOL-1704-CHG-2

HOL-1704-CHG-2

Open Chrome

Double click on the desktop icon to open Chrome.

Open Site A Web Client

On the bookmarks bar, click the RegionAWeb Client to open the vSphere Web Client.


Select Use Windows session authentication and click Login.

It's up to you to find challenge-04a in the vSphere Web Client.

Proceed to the next page when you've located challenge-04a in the vSphere WebClient.

HOL-1704-CHG-2

HOL-1704-CHG-2

Overview of esxtop

Before we begin the challenge, let's do a short overview of a helpful performancemonitoring tool known as esxtop.

esxtop is the tool of choice to examine advanced host-level performance statistics ofany kind. You could consider esxtop as the ESXi analog of the UNIX "top" command.

It's not necessary to use esxtop to solve this challenge, but it is a useful tool fortroubleshooting. If you'd like, you can skip this section.

1. On the taskbar, click the PuTTY icon.

Open PuTTY session to esx-03a

Double click on esx-03a.corp.local to open a PuTTY session.

esxtop

Type

HOL-1704-CHG-2

HOL-1704-CHG-2

esxtop

and then press Enter.

HOL-1704-CHG-2

HOL-1704-CHG-2

esxtop

The default view, the "CPU screen," shows the running worlds on ESXi in real time. Thetable sorts the running worlds on their %USED value in descending order.

challenge-04a is at the top of the list because it is using the most CPU time out of allthe worlds. challenge-04a is not using very much CPU time (6.78%), but it is the onlyVM running on this host.

As you can see, the esxtop process itself also uses CPU time.

esxtop

Press

h

to view the esxtop help screen.

esxtop allows you to switch between a number of different specialized screens. Eachscreen reports a different set of metrics which are updated in real-time.

Only one screen can be viewed at a time, and a single keystroke switches you betweenscreens.

HOL-1704-CHG-2

HOL-1704-CHG-2

c takes you to the CPU screen, d takes you to the disk adapter screen, and so forth.

esxtop

1. Enlarge the PuTTY window by clicking the Maximize Window button.

2. Press

u

to view the disk device screen shown above.

naa.6589cfc000000cfd9bc58d680c24402e indicates the backing storage device forchallenge-04a (and indeed all of the VMs in this lab). You can see that the CMDS/s (I/Ooperations per second) highlighted above is similar to the operations per secondreported by Iometer. This statistic in esxtop shows the total number of I/O operationsper second received by the backing storage device. It's reasonable to conclude thatsome of these I/O operations are created by Iometer while others are created by thechallenge-04a guest operating system.

esxtop uses many different metric abbreviations. For an explanation of esxtop screensand metrics, see the document Interpreting esxtop Statistics (opens in a new window).

HOL-1704-CHG-2

HOL-1704-CHG-2

https://communities.vmware.com/docs/DOC-11812

There is much more to this advanced tool. If you are interested in learning more aboutesxtop, see HOL-1704-SDC-1, Module 10: Performance Monitoring Tool: esxtopCLI introduction.

If you are already familiar with esxtop, KB 1008205 (opens in a new window), Usingesxtop to identify storage performance issues for ESX / ESXi (multipleversions) is an excellent resource.

Close the PuTTY window.

Proceed to the next page to begin the challenge.

Your Challenge

Your challenge is to increase the I/O per second of challenge-04a so its Total I/Os perSecond is approximately 16.

For this Challenge,

You are allowed to:

- vMotion VMs

- Modify VM settings

- If, for any reason, you need to restart this challenge, you can re-run the script C:\HOL-CHG-1695\Module5-start.ps1. This will reboot all VMs and return some VM settings tothe way they were at the beginning of the challenge. Please keep in mind that not allsettings and virtual hardware will be reset, so if you make a change to the environmentthat does not solve the challenge, please set it back.


- Change the Iometer settings. You are allowed to stop and restart it if needed, but donot change any other settings.


- Power off any virtual machines.


HOL-1704-CHG-2

HOL-1704-CHG-2



If you do these things it threatens the integrity of the teleporter network! If the networkof Waypoints goes dark, our competitors will gain a lot of new customers.

Need a hint?

View Hint #1

View Hint #2


HOL-1704-CHG-2

HOL-1704-CHG-2

Solution

Solution to this scenario:

1. Increase the virtual disk I/O limit from 16 to 32.

Read on to find out why...

Solution

IOPs limits are one feature of Storage I/O Control. Before you can use I/O limits, StorageI/O control must be enabled.

In the Lab, Storage I/O Control was already enabled. If you need to do this in yourenvironment, here's how. In the vSphere Web Client, select a datastore, click theManage tab, click Settings, then under Datastore Capabilities, click Edit... EnsureEnable Storage I/O Control is checked.

HOL-1704-CHG-2

HOL-1704-CHG-2

Storage I/O control does not take effect unless the ESXi host detects that datastorecongestion is occurring. The congestion threshold is 30ms by default, and can be set asas low as 5ms.

HOL-1704-CHG-2

HOL-1704-CHG-2

Iometer

Browse back to the challenge-04a remote desktop session.

In Iometer, you can see that the Total I/Os per Second is now approximately 16. Youmay have noticed that the Average I/O Response Time (ms) has also decreased fromits previous average of 12 seconds to around 4 seconds. The virtual disk I/O limitcreated the high latency. If you use a virtual disk I/O limit, keep in mind that the I/Oresponse times for the VM in question will be negatively affected as well-- but theywon't be as high as you have seen in this lab. In this example, an I/O limit of 32 isunrealistically low, which translates into unrealistically high I/O response times. For thisexample, we wanted to keep storage load low for the Hands-on Labs.

You may be asking yourself, why, when I set a limit of 32 I/O per second, was the VMlimited to 16 I/Os per second?

That is a good question.

1. Back at challenge-04a, click the red stop button to stop testing storage.

HOL-1704-CHG-2

HOL-1704-CHG-2

HOL-1704-CHG-2

HOL-1704-CHG-2

Iometer

1. Choose the Access Specifications tab.

In Iometer, an Access Specification determines the type of I/O load that Iometer placeson the assigned target, which in this case is the challenge-04a disk.

HOL-1704-CHG-2

HOL-1704-CHG-2

Iometer

The left pane shows Access Specifications that are active. The right pane shows allpossible Access Specifications that may be used.

The I/O access specification that Iometer was using was a 64K block size, 100% readoperations, and 0% random I/O (that is, 100% sequential I/O).

HOL-1704-CHG-2

HOL-1704-CHG-2

Iometer

1. Select the Access Specification in the left hand pane.

2. Click Remove to remove it from use.

HOL-1704-CHG-2

HOL-1704-CHG-2

Iometer

Would the IOPS result be the same or different with a different access specification?

Next,

1. Select the 512B, 100% read, 0% random.

2. Click Add to make it active.


4. Click Results Display to see what is happening.

Iometer testing

What happened?

Try a different access specification next.

Just like when troubleshooting a problem, when you are testing, change one variable ata time and then retest to determine the effect of that change in isolation.

HOL-1704-CHG-2

HOL-1704-CHG-2

We just compared a 64k, 100% read, 100% sequential trace with a 512 bytes, 100%read, 100% sequential trace.

1. You choose an access specification.

2. Click Add to make it active.


4. Click Results Display to see what is happening.

There's no specific problem to solve here. Just play around and see what happens. Tobudget your time, remember that there is still one more challenge in this module tocomplete after this one.

If you want, you can go back to the vSphere Web Client and change the virtual disk IOPslimits too.

When you're ready to move on, proceed to the next page.

End Iometer testing

1. Click the red stop button to end Iometer testing.

2. Close the Iometer window.

IOPS limit algorithm

You probably noticed that when an IOPs limit is set, the way that the limit is enforceddepends on the I/O block size, that is, 512 bytes, 16KB, 64KB, and so forth.

vSphere 5.5 introduced a new IOPS limit scheduling algorithm which providesadditional controls to I/O activity and also changes the way that the IOPS limit isenforced. The new algorithm is colloquially known as "mclock" and weights the IOPSallowed for the VM by the I/O block size. In other words, setting a limit of 16 will notnecessarily result in a maximum of 16 IOPS from the VM. Instead, the IOPS seen willdepend on the size of the I/O the VM is generating and may be greater or less than thelimit setting. The larger the block size, the lower the IOPS will be.

This explains why Murphy's VM wasn't meeting the 16 IOPS SLA once it was migratedfrom an ESXi 5.1 host to an ESXi 6.0 host. If you are migrating to an ESXi 5.5 or 6.0 host,you should either reset the IOPS limit for the VM accordingly, or you can disable the newalgorithm and roll back to the pre-5.5 IOPS limit algorithm.

HOL-1704-CHG-2

HOL-1704-CHG-2


Advanced System Settings

If you would like to roll back to the pre-5.5 IOPS limit algorithm, simply set the hostadvanced setting Disk.SchedulerWithReservation to 0, where 1 is the default value.

Reset Virtual Disk Limit

Since we are finished playing, let's set the IOPS limit back to unlimited so it won'tinterfere with our next challenge.


1. Right click on challenge-04a.

2. Click on Edit Settings...

HOL-1704-CHG-2

HOL-1704-CHG-2

Reset Virtual Disk Limit

Under Edit Settings,

1. Click Hard disk 1 to expand.

2. Click the Limit - IOPS drop down menu and change to Unlimited.

3. Click OK.

Congratulations!

Your new knowledge of IOPS limits will really come in handy with some of those disk-heavy VMs in the future.

HOL-1704-CHG-2

HOL-1704-CHG-2

Onwards and upwards! Do you know what vNUMA is?

HOL-1704-CHG-2

HOL-1704-CHG-2

vNUMA TroubleshootingMurphy bursts into your office.

"There's no vNUMA on my VM challenge-04a.. I don't know why. I need a vNUMAtopology on this VM for some new testing."

vNUMA Troubleshooting

Time to put your thinking cap on. You can do this!

Let's take a look at challenge-04a.


There is no startup script for this challenge.

If the challenge-04a Remote Desktop session is not already open, then double click onthe desktop icon "chg-04a.RDP" to open a Remote Desktop session to the virtualmachine challenge-04a.

HOL-1704-CHG-2

HOL-1704-CHG-2


1. Click on the taskbar shortcut to open a Command Prompt.

HOL-1704-CHG-2

HOL-1704-CHG-2

Examine vNUMA on challenge-04a

1. In the command prompt, type or copy from the README.txt file:

coreinfo -n

and press Enter.

You can see the Logical Processor to NUMA Node map. From the guest's perspective, ithas four cores (denoted by *) located on one NUMA node, 0.

Examine vNUMA on challenge-04a

Coreinfo is a third party tool that has been installed on challenge-04a make it easy toview the NUMA architecture.

There is also a way to natively examine virtual NUMA architecture on Windows Server2012.

1. Click on the Server Manager taskbar shortcut.

HOL-1704-CHG-2

HOL-1704-CHG-2

Open Resource Monitor

1. Click on Tools.

2. Then click on Resource Monitor.

HOL-1704-CHG-2

HOL-1704-CHG-2

Resource Monitor

1. Click on the CPU tab.

2. Click on the Views drop down menu.

3. Select Small.

4. You can see here that Windows detects that it has four CPUs. There is no mention ofNUMA nodes.

HOL-1704-CHG-2

HOL-1704-CHG-2

vNUMA

When vNUMA is enabled, however, a VM with two sockets and two cores per socketwould look different.

In Resource Monitor, Windows aggregates the CPU usage by NUMA node, and you canclearly see which virtual cores map to which NUMA nodes.

Next, let's go find the VM in the Web Client.

Open Chrome

If the vSphere Web Client is not still open and logged in, then double click on the MainConsole desktop icon to open Chrome.

HOL-1704-CHG-2

HOL-1704-CHG-2

Open RegionA Web Client

If you are not automatically taken to the VMware vCenter Single Sign on Page, on thebookmarks bar, click RegionA Web Client to open the vSphere Web Client.

Login the vSphere Web Client

1. Select "Use Windows session authentication".

2. Click Login.

The VM in question is challenge-04a; you should already know where it is.

Proceed to the next page to begin your challenge.

Your Challenge

Your challenge is enable vNUMA on challenge-04a.

If you are not familiar with NUMA, you may want to take lab HOL-1704-SDC-1,vSphere Performance Optimization, Module 6: vNUMA with Memory Hot Add.However, a technical understanding of NUMA is not required for this challenge.Troubleshoot the VM so that, like above, it shows a topology of two NUMA nodes eachwith two cores denoted by "*".

For this Challenge,

You are allowed to:

- vMotion VMs.


- Reboot virtual machines.

HOL-1704-CHG-2

HOL-1704-CHG-2

- If, for any reason, you need to restart this challenge, you can re-run the script C:\HOL-CHG\Module5-start.ps1. This will reboot all VMs and return some VM settings to the waythey were at the beginning of the challenge. Please keep in mind that not all settingsand virtual hardware will be reset, so if you make a change to the environment thatdoes not solve the challenge, please set it back.





If you do these things it threatens the integrity of the teleporter network! Someonemight end up in Antarctica when they meant to go to Grandma's.

Need a hint?

View Hint #1

View Hint #2


Solutions

If the Hot Add vCPU virtual hardware setting is enabled, then the architecturepresented to the virtual machine will be uniform memory access (UMA) only.

To solve this issue:

HOL-1704-CHG-2

HOL-1704-CHG-2

1. Power off challenge-04a.2. In the Web Client, right click on challenge-04a, then click Edit Settings. Under

CPU, deselect Enable CPU Hot Add.3. Click OK.4. Power the VM back on and vNUMA architecture matching the VM's virtual

hardware settings is now visible.

You should keep the Enable CPU Hot Add setting disabled unless you know that youneed it.

Note that the VM is configured with 4 vCPUs and 2 cores per socket because this is theHands-on Labs environment. Typically, however, you should leave Cores per Socket= 1 which results in the best performance. When Cores per Socket is left at itsdefault value of 1, this allows ESXi to automatically choose the optimal vNUMAconfiguration for the VM based on the physical server, which generally results in thebest performance. Leave Cores per Socket = 1 unless you have a very specific reason todo otherwise, such as for licensing purposes.

If your guest OS does not see a vNUMA architecture and you think it should, check thefollowing:

By default, vNUMA is enabled only for virtual machines with more than eightvCPUs. You can enable vNUMA for virtual machines with eight or fewer vCPUs bychanging the VM setting numa.vcpu.min from its default value of 9 to the number ofvCPUs in the virtual machine. This setting controls the minimum number of vCPUsrequired to enable vNUMA. This setting was modified on challenge-04a.

vNUMA requires virtual hardware version 8 or later.

Other factors that may completely disable NUMA (for the host and the VM) are BIOSsettings such as enabling node interleaving. Node Interleaving is generally disabledby default, but if you enable it, it will present a UMA architecture to the ESXi host.

For more great information about vNUMA, take HOL-1704-SDC-1, vSpherePerformance Optimization, Module 6, vNUMA with Memory Hot Add.

HOL-1704-CHG-2

HOL-1704-CHG-2

HOL-1704-CHG-2

HOL-1704-CHG-2

The Result

If we take a look at challenge-04a after making the change, we see 2 vNUMA nodesnow.

Done for today!

Time for some well earned time off.


HOL-1704-CHG-2

HOL-1704-CHG-2


Congratulations on completing Module 5.




• Module 3 - Performance Challenge, Basic Level (30 minutes) Can yourecognize and resolve common vSphere performance issues?


HOL-1704-CHG-2

HOL-1704-CHG-2

Module Hints

HOL-1704-CHG-2

HOL-1704-CHG-2

Module 1: Get Your Feet WetThe following pages contain hints for Module 1, Get Your Feet Wet.

Module 1, First Challenge, Hint 1

linux-base-01 is not a critical VM to PanContinental. It is acceptable if the VM has to beshutdown or rebooted in order to apply configuration changes to the VM.

Click here to return


Check the advanced configuration parameters of linux-base-01, you might need toadd/ edit some configuration parameters to enable copy/paste on the VM.


Module 1, Second Challenge, Hint 1

challenge-02a is not a very important VM to PanContinental. It's okay for itsperformance to take a nosedive. However, you cannot power or reboot challenge-02a.



View the Percent Ready metric in vSphere Web Client.

1. Click on esx-01a.

2. Click the Monitor tab.


4. Click Advanced.

5. Click Chart Options.

[not shown] Under Select counters for this chart:, check the box Readiness, thenclick Ok.

Readiness is the percentage of time that the virtual machine was ready, but could notget scheduled to run on the physical CPU. It happens when a virtual machine's demand

HOL-1704-CHG-2

HOL-1704-CHG-2

for CPU time exceeds the availablity of the physical CPU. As a rule of thumb, ifReadiness exceeds 10%, CPU contention could be negatively a ecting virtual machineperformance.

6. Look for the Readiness line in the graph. What does it show?

7. What is the Latest value of Readiness?


HOL-1704-CHG-2

HOL-1704-CHG-2

Module 2: Operations Challenge, BasicLevelThe following pages contain hints for Module 2, Operations Challenge, Basic Level.


The SSO administrator account ([email protected]) has been given toyou; have you logged in as that user and checked if there are any misconfigurations tothe account "[email protected]"?



Have you checked under "Global Permissions" for user [email protected] usingthe SSO administrator ([email protected]) account?


Module 2, Third Challenge, Hint 1

There is nothing wrong with the VMRC plugin installed on Chrome or the desktop. Itmight be an issue with the ESXi host (esx-01b).



It is not a user permission issue, as the user "[email protected]" has been givenadministrative privileges. You might want to check the ESXi host's (esx-01b) "AdvancedSettings" and verify the requirements that must be in place in order to open a remoteconsole to a virtual machine.


Module 2, Fourth Challenge, Hint 1

There is nothing wrong with the user account (root), or its password. You suspect itmight be a misconfiguration done by Murphy on the ESXi host (esx-01b) that mighthave caused SSH to be failing.

HOL-1704-CHG-2

HOL-1704-CHG-2


Module 2, Fourth Challenge, Hint 2

This can be solved by logging in to vSphere Web Client and reconfiguring a setting onthe ESXi host (esx-01b).


HOL-1704-CHG-2

HOL-1704-CHG-2

Module 3: Performance Challenge,Basic LevelThe following pages contain hints for Module 3, Performance Challenge, Basic Level


To change performance, you do not need to change anything inside the VM. Makechanges at the infrastructure level only.



challenge-01a is not being treated fairly by its host. Make it fair by checking theresource allocation at the VM and resource pool level.



challenge-03a is consuming a great deal of one type of resource on the host esx-01a.Which resource?



1. Click on challenge-01a.

2. Click the Monitor tab.

3. Click Utilization.

What do you see in Guest Memory and what might it mean?


HOL-1704-CHG-2

HOL-1704-CHG-2

Module 4, Operations Challenge,Advanced LevelThe following pages contains hints for Module 4, Operations Challenge, Advanced Level


Have you tried pinging the ESXi host (esx-02b), to see if it is actually still online? Ormaybe connect to one of the VMs using PuTTY to see if the VM is still online?



Have you used PuTTy to SSH into esx-02b and run some common ESXi service statustests?



[email protected] is not tagged as a disabled account. There are also nopermission issues with [email protected].



Have you checked under Users and Groups to verify the status [email protected] using [email protected]?



Have you reviewed the task details for esx-01b? As there isn't an error messageviewing the details of the Enter Maintenance Mode task could provide usefulinformation.


HOL-1704-CHG-2

HOL-1704-CHG-2


Have you reviewed the DRS recommendations and configuration. DRS acts differentlydepending on the defined automation level.


HOL-1704-CHG-2

HOL-1704-CHG-2

Module 5, Performance Challenge,Advanced LevelThe following pages contain hints for Module 5, Performance Challenge, Advanced Level


Something is constraining the number of IOPS. It's not slow storage. We only have onedatastore anyway.



Under Edit settings, check the Virtual Disk IOPS limits.



The change you need to make cannot be done from the guest VM.



There's one VM setting you need to change, and it's related to adding vCPUs to a VM.


HOL-1704-CHG-2

HOL-1704-CHG-2

ConclusionThank you for participating in the VMware Hands-on Labs. Be sure to visit

http://hol.vmware.com/to continue your lab experience online. Lab SKU:HOL-1704-CHG-2

HOL-1704-CHG-2

HOL-1704-CHG-2

http://hol.vmware.com/

ConclusionThank you for participating in the VMware Hands-on Labs. Be sure to visithttp://hol.vmware.com/ to continue your lab experience online.

Lab SKU: HOL-1704-CHG-2

Version: 20161027-102854

HOL-1704-CHG-2

HOL-1704-CHG-2

http://hol.vmware.com/

lab overview - hol-1704-chg-2

Documents