top causes for poor application performance case studies
DESCRIPTION
Top Causes for Poor Application Performance Case Studies. Mike Canney. Bad Packets = Bad Analysis = Engineer that is wrong!. Where do you capture? Dedicated appliance vs. move when you need it What Tools do you use? How much money do you have? How can I do this on the cheap? - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/1.jpg)
Top Causes for Poor Application PerformanceCase StudiesMike Canney
![Page 2: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/2.jpg)
Sharkfest 2014
Bad Packets = Bad Analysis = Engineer that is wrong!
• Where do you capture?• Dedicated appliance vs. move when you need it
• What Tools do you use?• How much money do you have?• How can I do this on the cheap?• Do I TAP the network or do I SPAN?
![Page 3: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/3.jpg)
Sharkfest 2014
Design your own capture Strategy• What does your network look like?• What are your goals with capturing the packets?
• Troubleshooting?• Statistical information? (Who, what, when and why.)• Long term solution vs. turning it on and off when
needed?• Do multiple engineers need to view this capture data?
![Page 4: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/4.jpg)
Sharkfest 2014
Rethink the use SPANs…
![Page 5: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/5.jpg)
Sharkfest 2014
Creating your own Capture to Disk Appliance (CDA)
• What is needed?• Wireshark full download (dumpcap)• Decommissioned PC or standalone “server” with
multiple NICs.• Simple batch file to launch the captures.• Pilot!
![Page 6: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/6.jpg)
Sharkfest 2014
The Batch File
cd \program files (x86)\wiresharkdumpcap -i 1 -b files:1000 -b filesize:2000000 –w c:\traces\internet\capture1.pcap
• This batch file creates 1,000, 2 Gigabyte trace files in a ring buffer. When it reaches file 1,000 it starts rewriting the trace files.
![Page 7: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/7.jpg)
Sharkfest 2014
So why 2 GB files?
• Pilot!• I have analyzed traces in excess of 2 TB with Pilot in
a matter of minutes.
![Page 8: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/8.jpg)
Sharkfest 2014
Trace File sizes are out of control
10 Mbps 1.25 MBps100Mbps 12.5 MBps1 Gbps 120.5 MBps10 Gbps 1.25 GBps100 Gbps 12.5 GBps
![Page 9: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/9.jpg)
Sharkfest 2014
The case of the SLOW Internet…
Case Study 1
![Page 10: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/10.jpg)
Sharkfest 2014
The best way to create a CDA and still be cost effective• vShark• Up to 2 TB of rolling capture disk• Can install it on a stand alone ESXi server and use
the physical NICs for capture.• Allows multiple people to access the Shark.• Can set up alerts (watches)
![Page 11: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/11.jpg)
Sharkfest 2014
TCP and my Love/Hate relationship…
![Page 12: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/12.jpg)
Sharkfest 2014
TCP Windowing
• The TCP Window Size defines the host’s receive buffer.
• Large Window Sizes can sometimes help overcome the impact of latency.
• Depending on how the application was written, advertised TCP Window Size may not have an impact at all (more on this later).
![Page 13: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/13.jpg)
Sharkfest 2014
Flow Control…
• Slow’s the sender down if the receiver cannot handle the load
• Uses a “window update”• Typically along with an ACK
• Sender is allowed to inject X amount of data onto the network before receiving an ACK
• Set by the receiver
![Page 14: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/14.jpg)
Sharkfest 2014
When are TCP ACKs sent?
15
RFC 1122
![Page 15: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/15.jpg)
Sharkfest 201416
Server Client
5,840 Byte Block
5,840 Bytes ACKed
5,840 Byte Block
![Page 16: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/16.jpg)
Sharkfest 2014
Mystical Slow File Transfer…
Case Study 2
![Page 17: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/17.jpg)
Sharkfest 2014 18
Server Client
2,944 Byte Block
2,944 Bytes ACKedWith ~200ms delayedACK
2,944 Byte Block
![Page 18: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/18.jpg)
Sharkfest 2014
Application Turns and Application Block Size
![Page 19: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/19.jpg)
Sharkfest 2014
Application Turns
• An Application Turn is a request/response pair• For each “turn” the application must wait the full
round trip delay.• The greater the number of turns, the worse the
application will perform over a WAN (Classic “Chatty” application).
20
![Page 20: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/20.jpg)
Sharkfest 2014
App Turn Example
![Page 21: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/21.jpg)
Sharkfest 2014
![Page 22: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/22.jpg)
Sharkfest 2014
How does this impact response time?
• It is easy to determine Application Turns impact on end user response time
• Multiply the number of App Turns by the round trip delay:
• 100,000 turns * .040 ms delay = 4,0000 seconds due to latency
![Page 23: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/23.jpg)
Sharkfest 2014
How do we make the most of our Bandwidth?
• Bandwidth Delay Product (BDP)• Bandwidth * Round Trip Latency/8 = offered load
needed to fill the pipe
• (44,000,000 * .04)/8 = 220,000 bits per second to fill a DS3
![Page 24: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/24.jpg)
Pop Quiz!!
• We have an application that uses a 61,440 byte block size
• This application has been deployed to Hawaii over a DS3 with 200 ms of round trip latency.
• What kind of throughput should I expect?
Sharkfest 2014
![Page 25: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/25.jpg)
Sharkfest 2014
How fast can we go?
• Throughput = Offered Load (Block Size)/Round Trip Latency
• Throughput for this deployment:
• (61,440/.200)*8 = 2,457,600 bits per second!!
![Page 26: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/26.jpg)
Sharkfest 2014
Microsoft file transfer
Using SMBv1 in a WAN environment, is it faster to copy a file to the server or copy a file from the server?
![Page 27: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/27.jpg)
Sharkfest 2014
The File Copy is Killing us!Case Study 3
![Page 28: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/28.jpg)
Sharkfest 2014
Multiple Capture points fill in the picture!
Case Study 4
![Page 29: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/29.jpg)
Sharkfest 2014
Lay of the Land…E
nd
Use
r
WEB Gateway
Internet
Users complaining that the Internet is slow!
![Page 30: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/30.jpg)
Sharkfest 2014
Domain Users Login Nightmare!Case Study 5
![Page 31: Top Causes for Poor Application Performance Case Studies](https://reader036.vdocument.in/reader036/viewer/2022070403/56813982550346895da11623/html5/thumbnails/31.jpg)
Sharkfest 2014
Thank You!Mike Canney