see your users in hi-def: synthetic and real-user ... › perform2015 › ...1 company confidential...

1 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015

David Jones, Director Sales Engineering

Gary Scott, Performance Analyst, Bell Canada

See Your Users in Hi-Def: Synthetic and Real-User Monitoring Best Practices


Insert

image hereThe better we get at getting

better, the faster we will get better

Douglas Engelbart


What’s Hi Def?

2015 -2020 Digital Visibility

• Anticipate and prevent Issues

• Correlate Business & Performance metrics

• Customer behaviors centric

2010-2015 Application Monitoring at

Code Level Depth

• Rapid troubleshooting

• High-fidelity data at scale

• End-user performance centric

2005-2010 Application Components

Monitoring

• Focus on Health and

availability

• Data sampling

• Data Center centric


• Business Challenges

• Synthetics

• Real User

• Best of Both Worlds

• Gary Scott, Bell Canada

• Summary

• Q/A

Agenda


Digital is transforming every industry

Believe that digital will disrupt

their business in the next 12

months

93% of CxOsRetail

Banking

EntertainmentConsumer

Goods

Travel

Health

find it hard to understand customer

behavior and needs

53% of CxOs


Business Challenge: Complexity Abounds

Outsourced & Hosted Services

GTS, BT

Legacy & Back-Office

CICS, IMS, zOS, SOAP

3rd Party Web APIs

Social, Ad Serving,, Web Analytics, Search, Maps ,

CDNs

SaaSSalesforce.com,

Marketo, NetSiuteServiceNow, …

Web and Mobile Services

iOS, Android, eCommerce platform

Home-grown Apps

Java, .NET, php, C++, node.js,

Enterprise Applications

SAP, Sharepoint, Oracle Business Suite, lync

http://www.ricbret.com/


Operations Development

• Customers are saying that the site is down

• Why is the site so slow today?

• Money’s tight this year. If I give you the $250K for those new servers, what’s in it for us?

• Customers are complaining about their account lookups. Where should we start?

• Is that’s your phone alert?

• I tried using the site from my mom’s house and I wanted to jump out the window

• What can we do to speed up the app

• Did the app get any faster after our last release?

• The new site looks great, but why doesn’t this function work?

Business Challenges What Keeps You Up At Night?


Ops Management Business

• I’m meeting with our board of directors tomorrow. Is there anything I should know about?

• I know we don’t have much budget for this project, but we’ve got to improve soon. How do you do more with less?

• We’ve got to get off of this Cloud Vendor. Who else is out there?

• The right data is there, but we really need to get it in a different way

• We’ve got good data, but it’s all over the place. It just doesn’t line up very well

• I’m seeing a lot of complaints about our booking process. A lot of customers are bailing at exactly the wrong time

• Our new survey shows 40% of our customers are using this new mobile device. Is that a problem?

• Industry Benchmarks says that we are dead last. Can I see you in my office?

• How can we determine how much money we are losing during downtimes?

• Are slow response times or errors impacting conversion rates? How do improve conversion?

Business Challenges What Keeps You Up At Night?


Delay User reaction

0 - 100 ms Instant

100 - 300 ms Slight perceptible delay

300 - 1000 ms Task focus, perceptible delay

1000+ ms Mental context switch

10,000+ ms+ I'll come back later...

Speed, performance and human perception

• Human Perception-Reaction Times

How Performance Impacts Users

Ilya Grigorik, Google, Performance Rails

http://chimera.labs.oreilly.com/books/1230000000545/ch10.html#ANATOMY_OF_WEB_APPLICATION


• Google: 2% slower = 2% less searching per user

• Yahoo: 400 milliseconds faster = 9% more traffic

• AOL: Faster pages = more page views

• Amazon: 100 milliseconds faster = 1% more revenue

• Shopzilla: 5 seconds faster = 25% more page views, 7 to 12% more revenue

• Aberdeen : 1 second slower = 11% fewer page views, 7 less conversion

How Performance Impacts Business

Ilya Grigorik, Google, Performance Rails

http://assets.en.oreilly.com/1/event/29/Keynote Presentation 2.pdf

http://www.slideshare.net/stoyan/dont-make-me-wait-or-building-highperformance-web-applications#btnNext

http://assets.en.oreilly.com/1/event/29/The Secret Weapons of the AOL Optimization Team Presentation.pdf

http://radar.oreilly.com/2008/08/radar-theme-web-ops.html

http://www.scribd.com/doc/16877317/Shopzillas-Site-Redo-You-Get-What-You-Measure

http://www.gomez.com/wp-content/downloads/Aberdeen_WebApps.pdf

11 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE11

SyntheticsProactive Performance Monitoring


Synthetics


What Has Synthetics Traditionally Provided?

Enhance/Protect

Revenue

Cost

Containment

Brand

Protection


KDI’s (Key Delivery Indicators)


Trending and DPI


Industry KDI Comparison

Year over Year Comparison of Industry KDI's (Key Delivery Indicators) by Dynatrace Benchmarks

Response Time

Response Time Last Year

Byte Count Byte CountLast Year

# Objects # ObjectsLast Year

# Connections

# ConnectionsLast Year

# Hosts # HostsLast Year

Alexa Top 50 5.35 3.55 2,479,414 1,632,867 131 117 55 46 35 31

Automotive 3.53 2.73 2,649,730 2,240,708 163 126 78 56 49 38

Banking 3.49 3.39 1,313,998 1,057,548 102 85 42 31 22 16

Brokerage 2.97 2.04 1,018,460 767,074 68 58 28 23 16 18

Finance 4.21 3.56 1,234,043 1,126,901 117 101 55 42 30 24

Healthcare 3.36 3.15 1,941,397 1,587,506 92 79 31 22 16 11

Insurance 3.22 2.74 866,288 891,645 108 94 52 42 28 23

Media 5.11 3.6 2,875,168 2,472,056 183 136 77 57 47 32

News 6.85 5.57 2,532,887 2,510,779 239 215 110 100 67 61

Social 2.99 2.98 1,690,921 1,547,001 93 93 47 48 26 28

Sports 7.81 3.95 3,074,240 2,315,087 219 175 96 75 54 45

Retail 5.07 3.23 2,529,206 2,255,401 170 150 67 57 42 36

Airlines 5.07 3.54 1,828,275 1,359,763 121 116 50 37 32 23

Travel 6.3 2.82 1,105,638 1,176,164 79 93 45 43 24 25

@davidlewisjones


6. Baseline and analyze performance trends across geographies and over time.

7. Benchmark your performance versus the competition.

8. Manage Change Aggressively, test performance in advance of making site changes.

9. Quantify the performance and business benefits of technology investments.

10. Align organization with a common set of baseline metrics generating controlled purepaths.

1. Test from the end-user perspective at every stage of development: Unify your application performance lifecycle.

2. Monitor your site’s availability and end-to-end performance 24/7 — even during low-traffic periods.

3. Understand the performance impact of third parties.

4. Objectively measure and enforce service-level agreements (SLAs).

5. Troubleshoot problems precisely using actionable, detailed, object-level metrics and diagnostics.

Top 10 Best Practices: Synthetics


Real UserPassive Performance Management


Web and Mobile Web Mobile App

• Javascript based instrumentation

• Native and Hybrid App based instrumentation

Real User


• #3 Be certain that every revenue-generating customer is a happy one!

• User Experience Index

• #4 Are your business critical actions successful, erroneous or slow?

• Failures (server and client*)

• Response and Perceived Render Times

• #1: Make sure that your online business is actually generating revenue

• Conversion

• Revenue

• #2: Make sure that your infrastructure is available to generate revenue

• Availability

• Traffic/visits

Top 4 Digital Performance Metrics


6. Track Client Side Errors which are impacting user most often

7. Understand what Devices, OSs, Geographies, Connection Types, etc. are contributing to poor user experience.

8. Manage CDN and Third Party impact on end users

9. Correlate user experience to Web Server and Host Performance

10. Leverage OOTB Use Cases

1. Focus on the user journey by managing Visits vs. simple Pages

2. Use Business Transactions to manage what matters

3. Manage User Actions (page loads, clicks, touches, etc…) as opposed to just Pages

4. Leverage User Action PurePathto diagnose problem areas

5. Leverage real time data collection, use incidents to be notified when users are having problems.

Top 10 Best Practices: Real User


Real User: User Experience Management

Use the User Experience Index to understand when real users are not receiving optimal experience or poor performance.

Leverage Purelytics to filter which devices, geographies, connection types (or combination) are impacting users. Manage Omni Channel

touch points which you users rely on most. Understand how performance impacts landing/exit pages in relation to bounces and conversion.

Track where users are entering and exiting the site

Track where users are entering and exiting the site


User Experience Index and Purelytics


Best of Both Worlds


Combining The Best of Both Worlds

Synthetic

Droids

Real User

Jedi Warriors


Synthetics Real User

• Pro-active event notification

• Automated root cause analysis

• External and internal SLA management

• Third party management

• Competitive benchmarking

• Real time user analytics

• Complete user visibility (every page, click, touch, swipe, etc…)

• Client side error visibility

• Third party and CDN impact

• Real time bounce, conversion and revenue

Benefits of Combining Synthetics and Real User


• Use real user data to identify key business steps, user paths and transactions

• Create synthetic scripts to monitor key business transactions

• Use Real User to monitor transaction steps that are impractical for Synthethic monitoring

• Use synthetic traffic as a “clean room” baseline for comparing real user results. (useful for tracking expected third parties)

Tips and Tricks


Real User View Synthetic View

Why a clean room view is necessary

16 Hosts62 Hosts


Synthetic Tile Synthetic Tile Detail

RoadMap

Tiles will link to Dynatrace or Keynote Synthetic Portal as required for additional diagnostics


Combined Synthetics and Real User

Synthetic Business Transaction

Rea User Traffic

Synthetic Availability

Real User Real Time Revenue

Real User Satisfaction


Real World Scenarios


Dynatrace & Bell Canada

A Success Story

Gary Scott

Bell Canada


Objective: Able to track the complete flow of a transactional process, up to the submitted order.

APM Synthetic (Gomez) is able to track the flow but we are unable to monitor the last step.

We can’t submit 280 orders per day!


Constraint: We have a “generic” URL that is used by several flows.


Our solution: Inject a header in the page that dynaTrace will be able to track.

A simple header (“dtflowtracker”) with 2 values: name of the flow & name of the step

Our guardians : Daniele D’Antoni & Christopher André


With dynatrace, we were able to get that header and its value.


What do we do with that info and how do we report it?

After Business Transaction and Dashboards.


Best part of this solution is that any new flows with headers will automatically be captured by dynaTrace.


Performance matters and no single point of view is the answer

Businesses need BOTH SyntheticAND Real User monitoring

Synthetic and Real User monitoring enhance each others value.

Summary

41 COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE #Perform2015COMPANY CONFIDENTIAL – DO NOT DISTRIBUTE

see your users in hi-def: synthetic and real-user ... › perform2015 › ...1 company confidential...

Documents