pitfalls in performance testing axxescc 06/2015
TRANSCRIPT
Technical point of view: Single user performance Acceptable slowdown under load/stress
Not only websites! Every project should consider performance explicitly
5
What is (applica/on) performance?
#AxxesCC
hWp://survey.neuronsparks.be
hWp://splunk.neuronsparks.be/en-‐US/app/search/speed_survey
6
Experiment
#AxxesCC
“The site needs to be sufficiently fast” “Page must load in 4s” PDF genera/on should not take > 30s The site must be able to handle 1000 users Trading system must be able to do 1000 TPS
7
Performance Requirements
#AxxesCC
Specific: 200 concurrent users, scenario walkthrough, 4s @ 95 percen/le response all pages upon first /me visit, pc on dsl Measurable: at the client side, with preload Achievable: ‘heavy’ page in 200ms? Relevant
Above the fold load /me “Onload” event versus speed index
Time-‐bound: golive, gradual adop/on, campaign start
9
“SMART” Performance Requirements
#AxxesCC
Throughput, load envelope Response /me User behaviour (think /me, flow) Device, connec/on Context: /cket sale, tax deadline
10
Aspects of Performance Reqs
#AxxesCC
Scalability needs to be built-‐in (and tested) What is the uncertainty of your expected load? Is scalability important? How fast can you provision? Business impact of out of spec performance?
13
Avoiding Surprises
#AxxesCC
Simple > complex Understand what the tool is doing
What do the metrics mean What is measured exactly
Tools are no subs/tute for thinking, but… Good repor/ng helps Intui/ve interface++
Find similar use cases – how did they do it?
19
Choose Your Weapon Wisely
#AxxesCC
Generate load Measure end-‐user perf “RUM”/synthe/c
Transac/on breakdown Code-‐level profiling Infrastructure monitoring Business analy/cs
22
Tool Categories
} APM
#AxxesCC
SaaS ‘cloud’ load genera/on Blazemeter, loadstorm,…
APM suites New Relic AppDynamics Dynatrace …
Big (machine) data Splunk, Loggly, logentries, Sumologic, Logstash/elk
Monitoring infrastructure 23
Tool Convergence
#AxxesCC
Domain knowledge System administra/on Networking exper/se Database adminstra/on Applica/on stack knowledge: frameworks Tools knowledge Performance tes/ng knowhow (+ some sta/s/cs)
24
Needed Skills
#AxxesCC
End-‐user performance sufficient?
Transac/on drilldown
Iden/fy & eliminate boWleneck
29
Chasing the BoWleneck
#AxxesCC
Week before golive Performance tes/ng completed, green light Func/onal freeze Cri/cal bugfixes only Launch goes
30
Recipe for Disaster
#AxxesCC
Regression tes/ng performance CI tools: Jenkins, Atlassian Bamboo, TFS, … Webpagetest/sitespeed.io Single-‐user performance Performance budget Speed index
Demo: Jmeter/Jenkins
32 #AxxesCC
Integra/on environment Not produc/on representa/ve Regression detec/on Rule out startup effects
Iden/fy cri/cal measurements HTTP APIs DB transac/ons XML processing
33
Jenkins and JMeter
#AxxesCC
Annota/on overlays on metrics hWp://play.grafana.org/dashboard/db/annota/ons What to annotate:
Deploys Config changes Anything with poten/al performance impact
Newrelic deployments demo
35
What introduced the problem?
#AxxesCC