performance monitoring at spreadshirt

Performance Monitoringat Spreadshirt

Why is performance relevant?

Performance has an impact on conversion rate ...

http://de.slideshare.net/devonauerswald/walmart-pagespeedslide

… and other business metrics

● bounce rate

● cart size

● revenue

● time on site

● page views

http://www.soasta.com/whitepapers/time-is-money-the-business-value-of-web-performance/Time is Money, The Business Value of Web Performance, Tammy Everts

● user satisfaction

● user retention

● organic search traffic

● brand perception

● productivity

How can performance be measured?

Core Metrics Overview

time to first byte

Core Metrics Overview

time to first byte start render time

Core Metrics Overview

time to first byte start render time

above the fold time

Core Metrics Overview

time to first byte start render time

above the fold time page load time

Metric Detailstime to first byte start render time page load time fully loaded

above the fold time

Metric Detailsbackend frontend

Metric DetailsDNS lookup

Init TCP connectionServer processing

Content download

Metric DetailsNumber of resources and transfered bytes +

order of resources

Which performance tools can be useful in software development lifecycle?

Software Development Lifecycle

Lifecycle image taken from SOASTA tool overview

Data Analysis &Optimization Tools

Synthetic Performance Monitoring &Real User Monitoring (RUM) &

Application Performance Monitoring (APM)

Load Testing & Optimization Tools

Performance Tools

Performance Tools at Spreadshirt

Grafana/Graphite

Performance Tools at SpreadshirtData Analysis

Synthetic Performance Monitoring

Real User Monitoring (RUM)

Grafana/GraphiteApplication PerformanceMonitoring (APM)

Why do we have anApplication Performance Monitoring tool

and why isn’t it enough?

Machine

Application Performance Monitoring (APM)

Service

Monitoring Server

Monitoring Data

Send Metrics

Write Monitoring Data Read Monitoring Data and Aggregations

Browser

Developer Request

Send Metrics

ServiceAgent

MachineAgent

Monitoring Website/

Application

Pros

● Good overview on performance of monitored machines and services

● Machine level information on cpu, memory, disk usage

● Service level information like JVM heap usage, number of threads, etc.

● Resource level information on response times, throughput, error rates

● Analysis tools allow to quickly find performance problems

● End-to-end drill down on a per request basis allows to quickly find root cause of

problems

Cons

● Complex to set up in a microservice environment as each service and machine needs

to be monitored with an agent

● No information on how pages load in real browser environments

● No information on how my DNS provider or CDN works

● No information on how 3rd party libraries behave in browser

Application Performance Monitoringprovides view on internal service and

machine performancesynthetic and real user monitoring

and why do we need both?

What is the difference between Synthetic and Real User Monitoring

and why do we need both?

Synthetic Performance Monitoring

SpreadshirtWebsite

Monitoring Server

Monitoring Website

Monitoring Data

Poll Page

Write Monitoring Data Read Monitoring Data and Aggregations

BrowserDeveloper Request

Pros

● Provides information about page load times in real browsers

● Controlled test environment

● Defined test locations

● Defined frequency

● Traffic independent

● Allows to check availability, speed, errors

● Allows to monitor 3rd party libraries, DNS provider and CDN

Cons

● Only information about configured pages

● No information about speed, errors on different customer environments like○ browser

○ device

○ country

○ operating system

○ ...

● No information about traffic distribution

● No connection to business metrics (like conversion rate)

Real User Monitoring (RUM)

SpreadshirtWebsite

BrowserMonitoringWebsite

Monitoring Data

Request Page

Write Monitoring Data

Read Monitoring Data and Aggregations

BrowserDeveloper Request

MonitoringJavascript

MonitoringServer

Send Beacon

Pros

● Runs on end users browsers and collects information about speed and errors on

different○ browsers

○ devices

○ countries

○ operating systems

○ …

● Allows to track real user traffic in real time

● Allows to track business metrics (conversion rate) in connection to performance

metrics

● Allows to track 3rd party libraries, DNS provider and CDN

Cons

● Provides data on pages with traffic only

● Uncontrolled environment

● Monitoring Javascript needs to be added to each page

● Single page apps (SPA) need to be monitored individually

Synthetic and Real User Monitoringcomplement each other

and provide outside view on performancesynthetic and real user monitoring

and why do we need both?

Which features do the available monitoring tools provide and how do

they work?

Application Performance Monitoring light with Grafana/Graphite

Grafana/Graphite Features

● Available on https://grafana.sprd.net resp. https://graph.server.lej.sprd.net

● Light version of Application Performance Monitoring

● Allows to track all kinds of metrics (not only performance related ones)

● Allows to create custom dashboards

● Allows to visualize metrics in different ways (time lines, numbers, tables, ...)

● Allows to run more complex analysis with the available Graphite functions (like top

10 slowest resources, ...)

● Allows to define and visualize thresholds

● Does not allow to define alerts yet

Grafana Dashboard

Favorite dashboards Available dashboards

Search dashboard Create new dashboard

Grafana Shopping Overview

Throughput metric as number

Response time metric with exceeded threshold

Linked to dashboard with details (on click)

Add panel

Grafana UMP Details

Response time metric p99 as timeline

Response time metric mean as timeline

Stacked throughput as timeline

Edit panel

Grafana UMP Details Configuration

Response time p99 metric

Response time mean metric

Synthetic Performance Monitoring with Rigor

Rigor Features

● Available on https://monitoring.rigor.com

● Allows to configure uptime, real browser and even selenium-style checks

● Allows to measure from different locations with different frequencies

● Allows to define thresholds on measured metrics

● Provides dashboards to visualize performance data

● Provides http archive on measurement and visualization as waterfall diagram

● Collects all kinds of performance metrics

● Allows to create and distribute performance reports

● Provides tags to be used in dashboards and for conducting actions

● Allows to alert via e-mail, phone, sms, chat, web hook

● Provides status page

● Provides API

Rigor Dashboard

Real browser checks Access check

Rigor Real Browser Checks

Add check

Configure check Access check

Rigor Real Browser Configuration

Configure URL

Configure frequency

Configure alert

Rigor Real Browser Check Configuration

Configure locations

Configure thresholds

Configure tags

Rigor Real Browser Check Dashboard

Switch view

Access measurement

Adjust timeDisplay events

System event

Rigor Real Browser Check Page Performance Dashboard

Select metric

Access measurement

Rigor Real Browser Check Domain Performance Dashboard

Select metricAccess measurement

Rigor Real Browser Check Waterfall Diagram

Open details

Rigor Executive Performance Dashboard

Access check

Rigor Operations Performance Dashboard

Access check

Rigor Custom Tag Performance Dashboard

Select tag

Rigor Custom Tag Performance Dashboard

Access check

Rigor Status Page - https://monitoring.rigor.com/status/218

Current load time

History

Rigor Custom Report Configuration

Select checks

Select date range

Select metric

Configure E-Mail

Rigor Custom Report Visualization

Access check

Rigor E-Mail Alert

Access check

Current state

Rigor E-Mail Report

Access check

Rigor Rocket.Chat Integration

Access check

Real User Monitoring with SOASTA mPulse

SOASTA mPulse Features

● Available on https://mpulse.soasta.com/concerto/Central

● Allows to configure applications per domain (e.g. for spreadshirt.de)

● Allows to configure page groups, metrics, timers and dimensions

● Collects performance metrics available to browsers

● Displays real-time performance data

● Provides dashboards to visualize performance data for different roles

● Allows to create custom dashboards

● Allows to filter data by all kinds of dimensions, e.g. browser, device, country

● Provides means to visualize complex correlations

● Allows to integrate data from other sources, like Rigor

● Allows to create and distribute reports

● Allows to alert via e-mail, chat, web hook …

SOASTA Applications

Create application

Access application

SOASTA Application Configuration

Access page groups

SOASTA Page Group Configuration

Configure page groups

Test page

SOASTA Metrics Configuration

Configure metric

SOASTA Timers Configuration

Configure timer

SOASTA Dimensions Configuration

Configure dimension

SOASTA Javascript Plugin

Add Boomerang Javascript

Add custom Javascript code for handling SPA

SOASTA Javascript Beacon

Loaded Boomerang Javascript

Sent beacon

SOASTA Dashboards

Pre-built dashboards

Custom dashboards

Add dashboard

Access dashboard

SOASTA Summary DashboardFilters for drill down

Page load vs page views

Key metrics and timersFilter statistics

Page load by country

SOASTA Summary Dashboard Configuration

Show available widgetsShow dashboard configuration

Drag & drop to add to dashboard

Select to configure widget

Configure widget

Configure default filter

SOASTA Showcase Dashboard

Live beacons

Live page load and back-end time

Live statistics

SOASTA DevOps Dashboard

SOASTA Custom Revenue Dashboard

Business relevant timers and metrics

Revenue vs. bounce rate

Page views vs. page load

Orders

SOASTA Custom Page Load Times Dashboard

Page load mean

Page load percentiles

SOASTA Custom Shopping Team Dashboard

Backend time vs. frontend time vs. page views

Page load time by page type

Load times from Rigor (external data)

SOASTA What-If Dashboard

Move sliders to see impact of improvements on revenue

Sessions with average load time vs.conversion rate per average load time

SOASTA Waterfall Dashboard

Found beacons for selected filters

Resource timings for selected beacon

Access to beacon data

SOASTA Waterfall Dashboard Performance Data

SOASTA Waterfall Dashboard Request Data

SOASTA Waterfall Dashboard Page Construction Data

SOASTA Reports

Add report

Custom reports

SOASTA Report Configuration

Configure schedule

Configure recipient

Drag & drop dashboard

SOASTA Report E-Mail

Configured dashboard as image or link

SOASTA Alerts

Add Alert

Custom Alerts

SOASTA Alert Configuration

Configure data event

Configure recipient

Configure message

Drag & drop elements

SOASTA Alert E-Mail

Link to dashboard that provides more details

SOASTA Rocket.Chat Integration

Link to dashboard that provides more details

Alert from SOASTA

What if there are more complex questions to answer?

Questions like ...

● On which page groups do I have performance problems?

● What are the most used path through my application regarding the defined

conversion goal?

● Which page groups are relevant regarding my defined conversion goal (e.g. order)?

● On which environments do I have performance problems (devices, browsers,

countries, operating systems)?

● Which page resources on my site are slow?

● Which page groups should I optimize first?

Data Analysis with SOASTA Data Science Workbench

(DSWB)

Data Science Workbench (DSWB)

Soasta DSWB

Importer

Monitoring Data

BrowserAnalyst

Request

AWS Redshift Cluster

Page Data

Resource Data

SoastaDSWB(Julia)Workbench

RequestData

Import

Data Science Workbench Features● Available on https://spreadshirt.soasta-dswb.com● Soasta DSWB is an environment that allows to analyze and visualize

performance data● Runs on AWS Redshift Cluster with Julia as programming language● Packages for Julia available to access databases, work with data, etc. ● Soasta DSWB provides additional packages for data analysis, drawing and

charting

SOASTA DSWB Overview

Select running notebook

SOASTA DSWB Notebook

Cell with default connection configuration for spreadshirt.de data

Execute selected cell (or Shift + Enter)

Tables with page and resource data for spreadshirt.de

Selected time frame

Documentation

SOASTA DSWB Page Load Co-occurrencesHistogram that shows page views vs. page load for selected time frame

SOASTA DSWB Page Load Co-occurrences

Visualization of factors that influence load times

SOASTA DSWB Page Load Treemap

Treemap that visualizes load times in relationship to page views for selected filter hierarchy

SOASTA DSWB User Path Sunburst Chart

Visualization of all user click paths towards checkout success page (means measured conversion goal)

SOASTA DSWB Conversion Impact Chart

Visualization of load time relevance of page group in order to reach conversion goal (means reaching checkout success page)

SOASTA DSWB Conversion vs Load Time Chart

Visualization of average session load time for converted and unconverted sessions (means sessions that reached checkout success page or not) vs conversion rate for average session load time

SOASTA DSWB Data AccessVisualization of plain beacon data for spreadshirt.de basedon an SQL executed against AWS Redshift Cluster

So, which tool should I use for what again?

Conclusion

● Use Grafana/Graphite (and Icinga) to get information about internal services like

response times, throughput and error rates

● Use tools like JProfiler to debug performance problems

● Use Rigor to get outside view on selected pages regarding page response times and

availability

● Use SOASTA mPulse to get extended view on page load time behavior in real user

environments

● Use SOASTA DSWB as planning tool to answer more complex questions regarding

performance and plan the next steps