aos troubleshooting tips march 25 2010
TRANSCRIPT
AX AOS TROUBLESHOOTING TIPS
AOS Troubleshooting Tips and Techniques
Agenda
AOS ConfigurationBest PracticesTroubleshooting AOS issuesFinding Sessions and SPIDSAOS Crash MaintenanceTools for Troubleshooting AOS issues
AOS Configuration
Application Object Server ConfigurationDatabase Tuning TabTracing TabPerformance Tab (Dynamics AX 2009)
AOS Configuration
Dynamics AX Server Configuration Utility
Configuration stored in Registry:HKLM\SYSTEM\CurrentControlSet\Dynamics Server\4.0\instance\config name
Multiple AOS instances and configurations
Server Configurations
Each AOS instance listedEach AOS has configurations
Application Object Server Tab
Configuration command to run at startupAllowing clients to print from AOS printersEnabling breakpointsFatal error reportingLoad Balancing
AOS Load BalancingNumber of client sessionsWhat is a load balancer How connections work when using load balancingFactors when deciding if you need a load balancer
AOS Load Balancing
Provides fail over abilityBalances number of clients between AOS’Control the number of clients per AOS
AOS without clustering
With no options marked in the load balancing section each client will connect to the AOS in their client configuration
Each AOS balancing load
As a client connects to an AOS that AOS queries the database to determine which AOS the client will connect to
AOS as Load Balancer
The load balancer makes the connection to the database and determines which AOS the client will connect toThis AOS will not accept any client connections nor do any processingThis AOS does not count against your AOS licenses
AOS Load Balancer - Client Configuration
In the Dynamics AX 4.0 Configuration utility you will want to specify the machine name and port number for the AOS load balancer instanceAdd AOS member instances after the load balancer
Database Tuning Tab
Max open cursorsMax buffer sizeAllow index hints in queries
Tracing Tab – Server Settings
RPC round trips to serverX++ method callsFunction callsStart and Stop Tracing
Tracing Tab – SQL Settings
SQL statementsRow FetchConnect and disconnectTransactions: TTSBegin,TTSCommit,TTSAbortBind variablesRow fetch summary
Dynamics AX 2009 featuresPerformance tab
64 bitBatchingTime Zones
Best Practices
Separate AOS’Separation of environmentsMonitoring performanceCode deploymentApplying latest Kernel hotfixes
Separate AOS’
Consider separating out the following processes / components based on load
Additional Dynamics AX componentsThird PartiesBatch
Separate out environments
Servers for production environment should be on dedicated boxes
Server resources dedicated to production performanceReduce risk of development code utilizing majority of Servers resourcesApplication of hotfixesAdditional components that need .Net BC
AOS Baselines
Performance MonitorMemory CPU
Memory usageDifferent times of dayWhen specific processes run
Proper code deployment
Code promotion schedulePromote code during down timeDetermine if you will use xpo’s or layers to move code from development/test to productionTest code in environment that matches production before deploying
Apply Kernel Hotfixes
Improved StabilityApply on all machines with a Dynamics AX components
Dynamics AX clientEnterprise PortalSSRS machineAny machine with .Net Business Connector
Troubleshooting AOS errors
Application event logRPC errorsConnection issuesAOS Crashing
Application event log
Windows default event logging toolDifferent types of events
AOS startingUser errorsCache flushingCrashes
Events related to connection
Lost connection to AOSCannot or lost connection to SQLLost connection to the AOS
Connection to AOS
Object Server 01: RPC error: Client provided an invalid session ID 75
This error message indicates that the AOS has crashed. After the AOS crashed a client that was open when the crash occurred tried to connect to the AOS. Since the AOS was restarted it does not recognize the existing client connection. To resolve this issue the client has to be shut down and restarted.
Connection to AOS
Object Server 01: No ping from 77. Terminating the session.
This error message indicates that the end user ended the client session before the AOS could return the requested results to the client. To prevent this error message end users shouldn’t end task (ctrl –alt – delete) their Dynamics AX session. Instead an end user should contact their AX administrator and have them end their session from the online users form.
Connection to AOS
Object Server 01: SPID 57 for Session Id 77, is still present in the database. Please delete the SPID from the database.
This indicates that the end user shut down their Dynamics AX client before SQL finished processing the request.You can delete the SPID or let SQL finish processingIf you don’t clean up the SPID it could cause issues such as blocking transactions, and open transactions which may lead to transaction log backup failures.
Connection to SQL
Object Server 03: Fatal SQL condition during login. Error message: "[Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for user 'NORTHAMERICA\MBSAXAPTA04$'.“
This typically indicates the AOS service account does not have permissions to the database.
*Typically followed by error on next slide
Object Server 03: SQL diagnostics: [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for user 'NORTHAMERICA\MBSAXAPTA04$'.. Connect information was: Userid = [], Database = [DAX4_SP1_Prod], Server = [server1], DSN = [], Other = []
This is caused by insufficient SQL permissions. To resolve this issue please use Knowledge base article 911380 method 2.
Connection to SQL
Object Server 03: Internal Error occurred executing stored procedure when creating session for the AOS.
This indicates the AOS Service account does not have adequate permissions to the stored procedures in the database.To resolve this issue please use Knowledge base article 911380 method 2..
Connection to application files
Error in file: C:\Program Files \DynamicsAX\ Application\appl\prod \axsys.aod while reading in record 215B1D0Windows Error: = Error Code: 64 = The specified network name is no longer available.Try again?
This error indicates the AOS has lost connection to the Application files.To resolve this error message you need to restart the AOS.
Finding Sessions and SPIDS
Find a user using a session ID in SQLFind a SPID using a session ID in SQLFinding User and computer from the AOT
Find User and Computer in SQL referenced in the Event Log
If the following event was logged on 6/1/2009 you could run the query below. Object Server 01: No ping from 77. Terminating the session.
select * from sysuserlog where sessionid = 77 and createddate > '2009/06/01' order by createddate desc
How to find an orphaned spid and associate it with an AX session ID
1. Check the status of the Microsoft Dynamics AX sessions, verify that these sessions have active SPIDs in the Microsoft Dynamics AX database where the client session is in the shut down process. To do this, run the following SQL statement:
Select * from SYSCLIENTSESSIONS where status = 2 or status = 3In the results of the statement, check the matching session ID from the warning message that appears in the Application log on the AOS server
2. Check the status of the orphaned SQL SPID. To do this, run the following SQL statement:
Select * from master..sysprocesses In the results of the statement, locate the SPID that is indicated in the warning message, and then verify the following:
If the SPID is blocked or if the SPID is blocking other SPIDs, and if the waittime column shows a high number, you can decide to delete the SPIDIf the open_trans column has a value of 1 for a SPID line, there is a pending transaction for the SPID line. Additionally, if the waittime column shows a high number for this SPID line, you should delete the SPID
Finding User and computer from the AOT
Users should NOT end task (CTRL–ALT-DEL) their Dynamics AX session. Instead your users should:
Contact their AX administrator and have them end their session from the (Administration>Online) users form
<Or>
Users should CTRL-BREAK the process.
*This allows the AOS to gracefully close network connections and recover resources
RPC errors
RPC 1726 – communication issue between client and server. Typically it doesn’t mean you have a network error, but rather that the AOS was overloadedRPC 5 – Access deniedRPC 1702 -The binding handle is invalid
AOS Crash Maintenance
Options for collecting dump filesOption in Dynamics AX Server ConfigurationHow to get information to Microsoft
AOS Crashing options
Automatically send reports to MicrosoftHow to use ADPlus to troubleshoot "hangs" and "crashes” http://support.microsoft.com/kb/286350/en-usDebugging Tools for Windows – Overview (Download) http://www.microsoft.com/whdc/devtools/debugging/default.mspx Debug Diagnostic Tool http://www.microsoft.com/downloads/details.aspx?FamilyID=28bd5941-c458-46f1-b24d-f60151d875a3&displaylang=en
Automatically Send Report to Microsoft
Mark the “For all AOS instances in this computer, automatically send reports about fatal errors to Microsoft” option on the Application Object Server tab in the Server Configuration Utility.
Informing Microsoft of Crash
When AOS crashes you receive an event id of 1001 in application event logHere is an example of the error:
Microsoft Analyzes the Crash Dump File(s)
Contact Microsoft Support with contents of the application event log event id 1001Support Technician passes bucket id on to Escalation Engineer so they can obtain fileTechnical Support may contact you for more information. Ie. Application files (*.AOD)
Tools to troubleshoot AOS
Performance MonitorApplication event logLoggingSQL Server toolsDebug Diagnostic Tool
Performance Monitor
PAL toolMemory
Typically greatest cause of issuesMonitor VIRTUAL BYTES in 32 bit environmentMonitor WORKING SET in 64 bit environment
CPUPrior to Dynamics AX mostly an issue for batch processingIn 2009 with 64 bit CPU may see more CPU usage than in the past
Application Event Log
Monitor itHave a schedule for monitoring
Sizing of itMake sure events don’t get overwritten
Schedule to clear it out
Logging Tools
SQL Server Tracing in Dynamics AXEasy way to get sql and X++
Client/Server TracingMore advanced requires some setup and use of another tool to review information
Trace ParserUsed to review Client/Server Tracing
SQL Server tools
SQL ProfilerSee log of what sql is doing
SQL Management StudioRunning queriesChanging settingsMaintenance
SQL DMV’s
Debugging tools
Debug Diagnostic ToolHave it installed on AOS machinesHave Hang collection setup
Debugging tools for WindowsIf you want to review memory dump files
Q & A
© 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after
the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.