Download - Performance Monitoring at Spreadshirt
Performance Monitoringat Spreadshirt
Why is performance relevant?
Performance has an impact on conversion rate ...
http://de.slideshare.net/devonauerswald/walmart-pagespeedslide
… and other business metrics
● bounce rate
● cart size
● revenue
● time on site
● page views
http://www.soasta.com/whitepapers/time-is-money-the-business-value-of-web-performance/Time is Money, The Business Value of Web Performance, Tammy Everts
● user satisfaction
● user retention
● organic search traffic
● brand perception
● productivity
How can performance be measured?
Core Metrics Overview
time to first byte
Core Metrics Overview
time to first byte start render time
Core Metrics Overview
time to first byte start render time
above the fold time
Core Metrics Overview
time to first byte start render time
above the fold time page load time
Metric Detailstime to first byte start render time page load time fully loaded
above the fold time
Metric Detailsbackend frontend
Metric DetailsDNS lookup
Init TCP connectionServer processing
Content download
Metric DetailsNumber of resources and transfered bytes +
order of resources
Which performance tools can be useful in software development lifecycle?
Software Development Lifecycle
Lifecycle image taken from SOASTA tool overview
Data Analysis &Optimization Tools
Synthetic Performance Monitoring &Real User Monitoring (RUM) &
Application Performance Monitoring (APM)
Load Testing & Optimization Tools
Performance Tools
Performance Tools at Spreadshirt
Grafana/Graphite
Performance Tools at SpreadshirtData Analysis
Synthetic Performance Monitoring
Real User Monitoring (RUM)
Grafana/GraphiteApplication PerformanceMonitoring (APM)
Why do we have anApplication Performance Monitoring tool
and why isn’t it enough?
Machine
Application Performance Monitoring (APM)
Service
Monitoring Server
Monitoring Data
Send Metrics
Write Monitoring Data Read Monitoring Data and Aggregations
Browser
Developer Request
Send Metrics
ServiceAgent
MachineAgent
Monitoring Website/
Application
Pros
● Good overview on performance of monitored machines and services
● Machine level information on cpu, memory, disk usage
● Service level information like JVM heap usage, number of threads, etc.
● Resource level information on response times, throughput, error rates
● Analysis tools allow to quickly find performance problems
● End-to-end drill down on a per request basis allows to quickly find root cause of
problems
Cons
● Complex to set up in a microservice environment as each service and machine needs
to be monitored with an agent
● No information on how pages load in real browser environments
● No information on how my DNS provider or CDN works
● No information on how 3rd party libraries behave in browser
Application Performance Monitoringprovides view on internal service and
machine performancesynthetic and real user monitoring
and why do we need both?
What is the difference between Synthetic and Real User Monitoring
and why do we need both?
Synthetic Performance Monitoring
SpreadshirtWebsite
Monitoring Server
Monitoring Website
Monitoring Data
Poll Page
Write Monitoring Data Read Monitoring Data and Aggregations
BrowserDeveloper Request
Pros
● Provides information about page load times in real browsers
● Controlled test environment
● Defined test locations
● Defined frequency
● Traffic independent
● Allows to check availability, speed, errors
● Allows to monitor 3rd party libraries, DNS provider and CDN
Cons
● Only information about configured pages
● No information about speed, errors on different customer environments like○ browser
○ device
○ country
○ operating system
○ ...
● No information about traffic distribution
● No connection to business metrics (like conversion rate)
Real User Monitoring (RUM)
SpreadshirtWebsite
BrowserMonitoringWebsite
Monitoring Data
Request Page
Write Monitoring Data
Read Monitoring Data and Aggregations
BrowserDeveloper Request
MonitoringJavascript
MonitoringServer
Send Beacon
Pros
● Runs on end users browsers and collects information about speed and errors on
different○ browsers
○ devices
○ countries
○ operating systems
○ …
● Allows to track real user traffic in real time
● Allows to track business metrics (conversion rate) in connection to performance
metrics
● Allows to track 3rd party libraries, DNS provider and CDN
Cons
● Provides data on pages with traffic only
● Uncontrolled environment
● Monitoring Javascript needs to be added to each page
● Single page apps (SPA) need to be monitored individually
Synthetic and Real User Monitoringcomplement each other
and provide outside view on performancesynthetic and real user monitoring
and why do we need both?
Which features do the available monitoring tools provide and how do
they work?
Application Performance Monitoring light with Grafana/Graphite
Grafana/Graphite Features
● Available on https://grafana.sprd.net resp. https://graph.server.lej.sprd.net
● Light version of Application Performance Monitoring
● Allows to track all kinds of metrics (not only performance related ones)
● Allows to create custom dashboards
● Allows to visualize metrics in different ways (time lines, numbers, tables, ...)
● Allows to run more complex analysis with the available Graphite functions (like top
10 slowest resources, ...)
● Allows to define and visualize thresholds
● Does not allow to define alerts yet
Grafana Dashboard
Favorite dashboards Available dashboards
Search dashboard Create new dashboard
Grafana Shopping Overview
Throughput metric as number
Response time metric with exceeded threshold
Linked to dashboard with details (on click)
Add panel
Grafana UMP Details
Response time metric p99 as timeline
Response time metric mean as timeline
Stacked throughput as timeline
Edit panel
Grafana UMP Details Configuration
Response time p99 metric
Response time mean metric
Synthetic Performance Monitoring with Rigor
Rigor Features
● Available on https://monitoring.rigor.com
● Allows to configure uptime, real browser and even selenium-style checks
● Allows to measure from different locations with different frequencies
● Allows to define thresholds on measured metrics
● Provides dashboards to visualize performance data
● Provides http archive on measurement and visualization as waterfall diagram
● Collects all kinds of performance metrics
● Allows to create and distribute performance reports
● Provides tags to be used in dashboards and for conducting actions
● Allows to alert via e-mail, phone, sms, chat, web hook
● Provides status page
● Provides API
Rigor Dashboard
Real browser checks Access check
Rigor Real Browser Checks
Add check
Configure check Access check
Rigor Real Browser Configuration
Configure URL
Configure frequency
Configure alert
Rigor Real Browser Check Configuration
Configure locations
Configure thresholds
Configure tags
Rigor Real Browser Check Dashboard
Switch view
Access measurement
Adjust timeDisplay events
System event
Rigor Real Browser Check Page Performance Dashboard
Select metric
Access measurement
Rigor Real Browser Check Domain Performance Dashboard
Select metricAccess measurement
Rigor Real Browser Check Waterfall Diagram
Open details
Rigor Executive Performance Dashboard
Access check
Rigor Operations Performance Dashboard
Access check
Rigor Custom Tag Performance Dashboard
Select tag
Rigor Custom Tag Performance Dashboard
Access check
Rigor Status Page - https://monitoring.rigor.com/status/218
Current load time
History
Rigor Custom Report Configuration
Select checks
Select date range
Select metric
Configure E-Mail
Rigor Custom Report Visualization
Access check
Rigor E-Mail Alert
Access check
Current state
Rigor E-Mail Report
Access check
Rigor Rocket.Chat Integration
Access check
Real User Monitoring with SOASTA mPulse
SOASTA mPulse Features
● Available on https://mpulse.soasta.com/concerto/Central
● Allows to configure applications per domain (e.g. for spreadshirt.de)
● Allows to configure page groups, metrics, timers and dimensions
● Collects performance metrics available to browsers
● Displays real-time performance data
● Provides dashboards to visualize performance data for different roles
● Allows to create custom dashboards
● Allows to filter data by all kinds of dimensions, e.g. browser, device, country
● Provides means to visualize complex correlations
● Allows to integrate data from other sources, like Rigor
● Allows to create and distribute reports
● Allows to alert via e-mail, chat, web hook …
SOASTA Applications
Create application
Access application
SOASTA Application Configuration
Access page groups
SOASTA Page Group Configuration
Configure page groups
Test page
SOASTA Metrics Configuration
Configure metric
SOASTA Timers Configuration
Configure timer
SOASTA Dimensions Configuration
Configure dimension
SOASTA Javascript Plugin
Add Boomerang Javascript
Add custom Javascript code for handling SPA
SOASTA Javascript Beacon
Loaded Boomerang Javascript
Sent beacon
SOASTA Dashboards
Pre-built dashboards
Custom dashboards
Add dashboard
Access dashboard
SOASTA Summary DashboardFilters for drill down
Page load vs page views
Key metrics and timersFilter statistics
Page load by country
SOASTA Summary Dashboard Configuration
Show available widgetsShow dashboard configuration
Drag & drop to add to dashboard
Select to configure widget
Configure widget
Configure default filter
SOASTA Showcase Dashboard
Live beacons
Live page load and back-end time
Live statistics
SOASTA DevOps Dashboard
SOASTA Custom Revenue Dashboard
Business relevant timers and metrics
Revenue vs. bounce rate
Page views vs. page load
Orders
SOASTA Custom Page Load Times Dashboard
Page load mean
Page load percentiles
SOASTA Custom Shopping Team Dashboard
Backend time vs. frontend time vs. page views
Page load time by page type
Load times from Rigor (external data)
SOASTA What-If Dashboard
Move sliders to see impact of improvements on revenue
Sessions with average load time vs.conversion rate per average load time
SOASTA Waterfall Dashboard
Found beacons for selected filters
Resource timings for selected beacon
Access to beacon data
SOASTA Waterfall Dashboard Performance Data
SOASTA Waterfall Dashboard Request Data
SOASTA Waterfall Dashboard Page Construction Data
SOASTA Reports
Add report
Custom reports
SOASTA Report Configuration
Configure schedule
Configure recipient
Drag & drop dashboard
SOASTA Report E-Mail
Configured dashboard as image or link
SOASTA Alerts
Add Alert
Custom Alerts
SOASTA Alert Configuration
Configure data event
Configure recipient
Configure message
Drag & drop elements
SOASTA Alert E-Mail
Link to dashboard that provides more details
SOASTA Rocket.Chat Integration
Link to dashboard that provides more details
Alert from SOASTA
What if there are more complex questions to answer?
Questions like ...
● On which page groups do I have performance problems?
● What are the most used path through my application regarding the defined
conversion goal?
● Which page groups are relevant regarding my defined conversion goal (e.g. order)?
● On which environments do I have performance problems (devices, browsers,
countries, operating systems)?
● Which page resources on my site are slow?
● Which page groups should I optimize first?
Data Analysis with SOASTA Data Science Workbench
(DSWB)
Data Science Workbench (DSWB)
Soasta DSWB
Importer
Monitoring Data
BrowserAnalyst
Request
AWS Redshift Cluster
Page Data
Resource Data
SoastaDSWB(Julia)Workbench
RequestData
Import
Data Science Workbench Features● Available on https://spreadshirt.soasta-dswb.com● Soasta DSWB is an environment that allows to analyze and visualize
performance data● Runs on AWS Redshift Cluster with Julia as programming language● Packages for Julia available to access databases, work with data, etc. ● Soasta DSWB provides additional packages for data analysis, drawing and
charting
SOASTA DSWB Overview
Select running notebook
SOASTA DSWB Notebook
Cell with default connection configuration for spreadshirt.de data
Execute selected cell (or Shift + Enter)
Tables with page and resource data for spreadshirt.de
Selected time frame
Documentation
SOASTA DSWB Page Load Co-occurrencesHistogram that shows page views vs. page load for selected time frame
SOASTA DSWB Page Load Co-occurrences
Visualization of factors that influence load times
SOASTA DSWB Page Load Treemap
Treemap that visualizes load times in relationship to page views for selected filter hierarchy
SOASTA DSWB User Path Sunburst Chart
Visualization of all user click paths towards checkout success page (means measured conversion goal)
SOASTA DSWB Conversion Impact Chart
Visualization of load time relevance of page group in order to reach conversion goal (means reaching checkout success page)
SOASTA DSWB Conversion vs Load Time Chart
Visualization of average session load time for converted and unconverted sessions (means sessions that reached checkout success page or not) vs conversion rate for average session load time
SOASTA DSWB Data AccessVisualization of plain beacon data for spreadshirt.de basedon an SQL executed against AWS Redshift Cluster
So, which tool should I use for what again?
Conclusion
● Use Grafana/Graphite (and Icinga) to get information about internal services like
response times, throughput and error rates
● Use tools like JProfiler to debug performance problems
● Use Rigor to get outside view on selected pages regarding page response times and
availability
● Use SOASTA mPulse to get extended view on page load time behavior in real user
environments
● Use SOASTA DSWB as planning tool to answer more complex questions regarding
performance and plan the next steps