Neumont UniversityDan Taylor -- IS Capstone

Project Horus

Troubleshooting Version 3.0

For Official

Use Only

P a g e | 1

Contents1.0 Introduction...........................................................................................................................................1

1.1 Change Log............................................................................................................................................1

2.0 Network Topology.................................................................................................................................1

2.01 Management IPs..............................................................................................................................1

2.02 Production – VLAN X, 10.0.X.0/24...................................................................................................1

2.03 Workstation IPs...............................................................................................................................1

2.1 Power on Order.....................................................................................................................................1

2.11 Physical Machines............................................................................................................................1

2.12 Virtual Machines (The next step).....................................................................................................1

2.2 Power off Order.....................................................................................................................................1

2.21 Virtual Machines..............................................................................................................................1

2.22 Physical Machines............................................................................................................................1

2.3 Network Connectivity Issues..................................................................................................................1

2.31 No Internet Connectivity on VMs....................................................................................................1

3.0 SAN Connectivity Errors.........................................................................................................................1

3.01 ERROR: Virtual Machines Become Inaccessible...............................................................................1

3.02 ERROR: Cannot connect to vCenter Single Sign-On server (vSphere Web Client)............................1

3.03 ERROR: Empty Inventory.................................................................................................................1

4.0 Spector 360 Server Configuration..........................................................................................................1

4.01 ERROR: Test email failed with the following message: Failure to send mail. Unable to read data from the transport connection: net_io_connectionclosed......................................................................1

5.0 Spector 360 Server Troubleshooting.....................................................................................................1

5.01 Unable to Start Spector Control Center Server Service: Logon Failure............................................1

5.02 Unable To Retrieve Server Settings – Unable to communicate.......................................................1

6.0 Spector 360 End-User Configuration.....................................................................................................1

6.01 Start Windows Remote Registry Service..........................................................................................1

6.02 ERROR: Cannot Open Service Control Manager on [Computer Name]............................................1

6.03 Hosts cannot ping one another.......................................................................................................1

P a g e | 2

1.0 IntroductionThe contents of this document are only to be viewed by those with sole authorization from the creator of this document. Any unauthorized individuals viewing this document could potentially face charges under 18 U.S. Code § 1905. This document is subject to change and will be maintained through a change log documenting what had changed per each instance.

1.1 Change LogVersion # Name Changes Made Sections Added/Modified

1.0 Daniel Taylor Initial implementation of Change Log. All changes made at this point will be documented here.

Reference DocumentsIntroductionChange Log

2.0 Daniel Taylor Added documentation on Active Directory configurations as well as setting up and connecting to a SAN using software iSCSI adapters.

3.0 (and subsections)4.0 (and subsections)

3.0 Daniel Taylor Split document into two different documents: “Installation and Configuration” and “Troubleshooting”.

P a g e | 3

2.0 Network Topology2.01 Management IPs

Object IP Configuration (All Subnets are /21)SAN Management 10.0.0.9

Firewall 10.0.0.10DHCP for IS Wireless Network 10.0.7.0 to 10.0.7.254

Wireless Management 10.0.0.27Chassis 1 Blade 8 10.0.1.8Chassis 2 Blade 8 10.0.2.8Chassis 3 Blade 8 10.0.3.8

ESXi; Chassis 1 Blade 8 10.0.1.24ESXi; Chassis 2 Blade 8 10.0.2.24ESXi; Chassis 3 Blade 8 10.0.3.24

vSphere/vCenter; Chassis 1 Blade 8 10.0.1.40vSphere/vCenter; Chassis 2 Blade 8 10.0.2.40vSphere/vCenter; Chassis 3 Blade 8 10.0.3.40

Web Client 10.0.2.40:9443

2.02 Production – VLAN X, 10.0.X.0/24VLAN 84, 10.0.84.XXX/24Public IP: 69.27.22.30Public Gateway: 69.27.22.1

2.03 Workstation IPsUser Category Project Horus IPs

Workers (two VMs) 10.0.84.20; 10.0.84.21Exploders (two VMs) 10.0.84.30; 10.0.84.31Slackers (two VMs) 10.0.84.40; 40.0.84.41

Active Directory (one VM) 10.0.84.10; 10.0.7.253Spector 360 Server (one VM) 10.0.84.11; 10.0.7.254

2.1 Power on OrderThe order in which to turn on these systems is hierarchical and is to be followed explicitly as it is listed, with no deviation or variation for the system to operate:

2.11 Physical Machines1. Chassis 2, Blade 82. Chassis 1, Blade 83. Chassis 3, Blade 8

Allow a period of 5 minutes before continuing to the next step.

P a g e | 4

2.12 Virtual Machines (The next step)1. On Chassis 2, Blade 8, turn on the vCenter Server VM.

a. Log in to the vCenter Server VM and wait 5 minutes before continuing to the next step for all of the services to fully activate.

b. Access this VM by using the vSphere Client and connecting to 10.0.2.24.2. On Chassis 1, Blade 8, turn on the AD Server VM.

a. Log in to the AD Server VM and wait 5 minutes before continuing to the next step for all of the services to fully activate.

b. Access this VM by using the vSphere WEB Client and connecting to https://10.0.2.40:9443

3. On Chassis 1, 2, and 3, turn on the two End-User VMs.

After this, turn on all the End-User VMs, of which there are 2 (TWO) on Blade 8 of Chassis 1, 2, and 3.

2.2 Power off OrderThe order in which to turn off these systems is hierarchical and is to be followed explicitly as it is listed, with no deviation or variation for the system to power off in the correct order:

2.21 Virtual Machines1. On Chassis 1, 2, and 3, all of Blade 8, turn off the End-User VMs.2. On Chassis 1, Blade 8, turn off the AD Server VM.

a. This can be done simply by right-clicking the VM and clicking “Power Off”.3. On Chassis 2, Blade 8, turn off the vCenter VM.

2.22 Physical MachinesNow that all of the VMs have been turned off, the physical machines can be powered off in any order through their associated iDRAC controls (10.0.1.8, 10.0.2.8, 10.0.3.8 respectively).

DO NOT TURN THE SERVERS OFF BY HOLDING THE POWER BUTTON DOWN ON THE PHYSICAL MACHINES.

2.3 Network Connectivity Issues2.31 No Internet Connectivity on VMsIf there is an issue where a VM is unable to connect to the internet, first make sure they have two NICs (Network Interface Cards) via the vSphere Web Client. NIC 1 MUST connect to DPortGroupVLAN10 and NIC 2 MUST connect to the Test.Note: DPortGroupVLAN10 is the Internet, while Production is the Intranet.Your VM Settings should look something like this:

P a g e | 5

3.0 SAN Connectivity Errors3.01 ERROR: Virtual Machines Become InaccessibleThis issue is most likely caused from the Data Source on which your VMs run has come across some problem, typically due to the Data Source being shut off or the servers being detached from the Data Source. To remediate this issue, typically you can rescan the adapter that is connected to the Data Source for any “newly discovered” storage devices, as shown below.

If that does not resolve your issue, the issue lies in the SAN configuration. Your SAN interface, upon selecting your server cluster, will most likely show some result comparable to Dell Compellent’s “Partially Connected” status, as shown below:

For this project, the IP of the Compellent Management Interface is https://10.0.0.9. The solution to this is very simple. First, navigate to the Mapping tab on any of the servers within your cluster in Compellent storage. Click on the Volume and then click Remove Mapping. This will remove the Volume from all of your servers simultaneously, provided you correctly implemented them into a Server Cluster. Now, to add the Volume back to the Server Cluster, navigate to Storage Volumes Big Brother Mapping and click on “Map Volume to Server”. You’ll be presented with a dialogue box on which server or server cluster to select for mapping. Click on 1984.

P a g e | 6

Once the Volume has been mapped back to the Server Cluster, go ahead and rescan the Storage Adapters as demonstrated in the first step, and you’ll see that the iSCSI Software Adapters have connected back to the SAN and the Virtual Machines have been discovered. The HBAs may still show partially connected status, though that is not a glaring issue to be concerned about.

3.02 ERROR: Cannot connect to vCenter Single Sign-On server (vSphere Web Client)This issue is caused by the vCenter Server Service (located on the vCenter Server) having difficulties on starting. Resolving this issue may vary in difficulty depending on the state of the vCenter Server and if the vCenter Server Service is in the midst of starting or not. The easiest solution, of course, is to restart the vCenter Server and not force start any processes; let VMWare and vSphere organize the start order for all of these processes.

3.03 ERROR: Empty InventoryIn the event of an Empty Inventory upon trying to access hosts on the vSphere Web Client, there is a service that doesn’t always automatically start, though it should. Firstly, navigate to Local Services on the vCenter Server and look for VMware vCenter Inventory Service and ensure that it is running. If it is, simply wait five to ten minutes for the service to inform the web server of its inventory.

P a g e | 7

4.0 Spector 360 Server Configuration4.01 ERROR: Test email failed with the following message: Failure to send mail. Unable to read data from the transport connection: net_io_connectionclosedShould this error occur, it is a firewall issue with the network-wide firewall. To resolve this, you must be able to gain administrative access to your internal Firewall. The following two commands will work for a Cisco ASA Firewall.

access-list smtp extended permit tcp any host 10.0.7.254 eq smtpaccess-group smtp in interface outside

5.0 Spector 360 Server Troubleshooting5.01 Unable to Start Spector Control Center Server Service: Logon FailureIn the event the below error is received, the issue lies in with Single Sign On capabilities.

To resolve this issue, open the Spector 360 Control Center (ignore all warnings/errors) and navigate to “Database”. From there, click on “Manage Database Logins” and finally click “Create a new Login Account”, as shown below.

P a g e | 8

From this screen, click the “Use SQL Server Authentication” radial button, then fill in user credentials as seen below:

Before saving this, make sure that under the “Select Events” and “Select Tools” tabs that all options are selected except for “Auditing” under “Select Tools”. Once that is confirmed, click “Save and Close”, close the Spector 360 Control Center application and log out of the current Administrator account. Log back in to the server as an account that does not have single sign on privileges and use the newly created logon credentials via Database Login. Alternatively, logging in as “sa” will also work provided the credentials are known.

5.02 Unable To Retrieve Server Settings – Unable to communicate.If the steps in the previous subsection (6.01) do not resolve this issue, then the issue lies in incorrect login credentials for the Spector Control Center Server Service itself. To resolve this issue, navigate to Local Services, locate the “Spector Control Center Server” service, right click it and click Properties. Now click on the Log-On tab. The following window will appear:

There are now two possibilities on how to remediate this issue. The first is (and easiest) is to click the “Local System account” radial button to allow the service to start by the “SYSTEM” user than an Administrator user account. The second choice for fixing this issue is to change the account credentials that activate the service. The most common problem here is that the password has changed on the associated account. Update the password and start the service.

P a g e | 9

6.0 Spector 360 End-User Configuration6.01 Start Windows Remote Registry ServiceIf the WRRS can’t be reached, log in to the end user machine as an Administrator and open services.msi. Navigate to the Remote Registry service, right click it and click on Properties. In the drop down box, click on Automatic, then click on start. It will look like this if done right:

In the event manually starting the service is proving impossible, click the Log On tab of the Remote Registry service. Make the Username field say “NT Authority\LocalService” and clear out both password fields. Hit apply and try restarting the service.

6.02 ERROR: Cannot Open Service Control Manager on [Computer Name]If such an error occurs while running diagnostics on any machine, log in to the end user machine as an Administrator and navigate to the Advanced Firewall Options. Make a new Inbound Rule for Port-based access on TCP and UDP ports 135. If done right, there will be two entries in the Inbound Rules section for Windows Firewall, as seen on the next page.

P a g e | 10

6.03 Hosts cannot ping one anotherResolving this issue is extremely simple. First, open up a Command Prompt on the VM you want to ping to and the VM you want to ping from. Next, type this command all on one line:

netsh advfirewall firewall add rule name="ICMP Allow incoming V4 echo request"protocol=icmpv4:8,any dir=in action=allow

This adjusts the firewall to allow ICMP packets to be sent to it.