troubleshooting asp.net and iis scalability hotspots

Please Standby: Webcast to Begin Shortly

Unable to listen via your speakers? Dial in: Toll Free: (877) 309-2074 Access Code: 119-063-034

International: Click “telephone” in audio section,then click “additional numbers”

Troubleshooting ASP.NET and IIS Scalability Hotspots

Confidential, Dynatrace, LLC

Asad AliDirector, Product SpecialistDynatrace

Troubleshooting ASP.NET and IIS Scalability Hotspots

• Real Use Cases• IIS/ASP.net Metrics• High Latency Application Example• Synchronization Issues• Sharepoint Example

Agenda

Confidential, Dynatrace LLC

Real Use Cases

Online Room Service

22% spent in IIS

Online Room Service

Tip: Elapsed Time tells us WHEN a Method was executed!

Online Banking: Slow Balance Check1.69m (=101s!) To

Check Balance!

87% spent in IIS

Time really spent in IIS

Tip: Elapsed Time tells us WHEN a Method was executed!

Finding: Thread 32 in IIS waited 87s to pass control to Thread 30 in ASP.NET

Tip: Thread# gives us insight on Thread Queues / Switches

Confidential, Dynatrace LLC

• Modules Installed

• IIS/ASP.net Metrics

What To Look

Confidential, Dynatrace LLC

IIS Modules

IIS/ASP.net Metrics

Confidential, Dynatrace LLC

Arrival Rate• Rate at which requests are arriving in the queue

• Perfmon Counter: Http Service Request Queues\ArrivalRate

• Should be >= W3WP_W3SVC\Requests / sec

IIS/ASP.net Metrics

Confidential, Dynatrace LLC

MaxQueueItemAge• Incoming requests are waiting at this long to be processed

• Perfmon Counter: Http Service Request Queues\MaxQueueItemAge

IIS/ASP.net Metrics

Confidential, Dynatrace LLC

CLR ThreadPool Queue•Requests are queued if all the CLR threads are busy

• Perfmon Counter: ASP.NET v4.0.30319/Requests Queued

• Symptom: ASP.net returns 503 Service Unavailable

IIS/ASP.net Metrics

Confidential, Dynatrace LLC

Number of Free DB Connections

• Number of free ADO.net connections available

• Perfmon Counter: .Net Data Provider for SqlServer/NumberOfFreeConnections

• Alert: If value == 0

ADO.Net Metrics

Confidential, Dynatrace LLC

Number of Active DB Connections

•Number of Active ADO.net connections

•Perfmon Counter: .Net Data Provider for SqlServer/NumberOfActiveConnections

ADO.Net Metrics

Confidential, Dynatrace LLC

High Latency Application Example

Confidential, Dynatrace LLC

Too Many Queries crash

Undersized Pools

Online Banking: Slow Balance Check1.69m (=101s!) To

Check Balance!

87% spent in IIS 600! SQL Executions

SQL ExecutionsFinding: EVERY SQL

statement is executed on ITS OWN Connection!

Tip: Look at “GetConnection”

Lesson Learned

ASP.NET Worker Thread Pool Sizing!

DB Connection PoolsMore Efficient SQL

SQL ExecutionsFinding: EVERY SQL

statement is executed on ITS OWN Connection!

Tip: Look at “GetConnection”

Confidential, Dynatrace LLC

SQL Executions

#1: Same SQL is executed 67! times

#2: NO PREPARATION because everything

executed on new Connection

Confidential, Dynatrace LLC

• Idle vs. Busy Threads• # SQLs / Request• # GetConnection

Tips To Apply

Confidential, Dynatrace LLC

SynchronizationIssues

Confidential, Dynatrace LLC

Synchronization Issues

Load vs. Memory vs Response Time vs. Threads

Confidential, Dynatrace LLC

Increasing MemoryAvg Response Time: Increases over Time

Max Response Time correlates with # Threads

Max Response Time correlates with # Threads

Load: System can’t handle constant load over time

# of Worker Threads increase with constant load

Layer Breakdown: Performance Problem is in their own code (Sync Time)

Confidential, Dynatrace LLC

96% Synchronization TimeMonitor.Enter: By far biggest

performance contributor

Called from GetStatus and _GetDebugStatus. Why also from Debug method? Bug?

Confidential, Dynatrace LLC

Excessive file attribute access second hotspot

Can FileExist check be cached to reduce

File Access?

File Access next hotspots

Confidential, Dynatrace LLC

• # of Threads vs # Incoming Requests

• Time Spent in Sync• Time Spent in I/O

Tips To Apply

Confidential, Dynatrace LLC

Sharepoint Example

Confidential, Dynatrace LLC

Frustrated user report bad response times

Frustrated User

Slow Page Load caused by Browser JS Time

Slow Page Load caused by Server-Side Processing

Confidential, Dynatrace LLC

Really slow page6.8s to deliver Default.aspx page

Involved Web Parts

Most of the Time spentIn waiting

Confidential, Dynatrace LLC

WebPart uses multiple parallel threads

Async Threads are busy with I/O

Confidential, Dynatrace LLC

First remote call is very slow

Web Service call by ContentEditorWebPart

HttpWebRequests uses ServicePoint internally

First Web Serivce Requests takes 5.8s to return

Confidential, Dynatrace LLC

Thread limit lets all other threads wait!

We have 10 parallel calls in our background threads

The other background threads spend their time“waiting” in the ServicePoint

Confidential, Dynatrace LLC

Solution: Change Defaults

Confidential, Dynatrace LLC

• Review installed IIS modules• Pay attention to IIS/ASP.net metrics• Optimize thread synchronization• Reduce I/O • Analyze SQL statements• Use DB Connection Pool Optimally

Tips To Apply

Confidential, Dynatrace LLC

Free Trial: bit.ly/dttrial

Thank you!

Time for Q&A