how to avoid epic web failure... lessons learned from healthcare.gov
DESCRIPTION
Web and mobile performance has become business critical in today’s world. Website failures are costly in terms of lost revenue and tarnished reputation. Quality user experience has taken center stage. Are you ready to deliver? To help prepare for web traffic spikes associated with seasonal readiness, promotions or special online events, proper planning and execution is a must. This means that the combination of technology tools, implementation best practices and expertise needs to be carefully managed and optimized. Since 2008, SOASTA has conducted tens-of-thousands of tests, and logged over a million hours in the cloud. During this time we have accelerated web and mobile testing with scale and affordability for the world’s leading brands. During our upcoming webinar, we will share key SOASTA testing strategies needed to avoid epic web fails. They include the need to: - Test continuously - Test end-to-end - And, test at scale Join SOASTA experts Peter Galvin, SVP SOASTA Marketing and Brad Johnson, SOASTA VP Product Marketing as they share the best practices you need to follow to ensure your web and mobile apps are real-world ready. We will also show you how SOASTA’s industry leading CloudTest platform enables you to streamline and automate performance tests, so you can deliver quality user experiences every time.TRANSCRIPT
1© 2013 SOASTA CONFIDENTIAL - All rights reserved.
How To Avoid Epic Web Failure
Lessons Learned from Healthcare.gov
Webinar
8
2© 2013 SOASTA CONFIDENTIAL - All rights reserved.
What have we learned from recent epic fails?In This Webinar
TODAY’S SOASTA PRESENTERS
Peter Galvin: Brad Johnson:SVP Marketing VP Product Marketing
Agenda: • Poll question• Lessons Learned• Some Best Practices
Questions: Submit in chat box during event
Poll Question
4© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Acceptable Failure?
5© 2013 SOASTA CONFIDENTIAL - All rights reserved.
This October…
6© 2013 SOASTA CONFIDENTIAL - All rights reserved.
This October…
“We didn’t have enough testing for a very complicated project”
US Health and Human Services Secretary, Kathleen Sebelius
7© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Impact of Healthcare Site Failures
• Millions of uninsured citizens• Healthcare program mistrust• President’s approval rating dropped• Congress tied up in legislation• Public cynicism• Press and social media circus• Uncertainty for the future
Brought QUALITY and TESTING to the public forefront!
8© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Industry Best Practices
RECOMMENDATIONS
“Name a single implementation leader”
“Lock down” requirements by April 30
“Streamline decision making”
“End-to-end integrated operations and IT testing”
Top down, Home Health Agencies Missed the Mark
http://www.businessweek.com/news/2013-11-19/obamacare-consultants-warned-of-health-website-failure-in-march
WHAT HAPPENED
“No single leader”
“Shifting requirements”
“Multiple definitions of success
“Insufficient time and scope of end-to-end testing”
http://www.npr.org/blogs/alltechconsidered/2013/11/19/246132770/this-slide-shows-why-healthcare-gov-wouldnt-work-at-launch
In March 2013
9© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Outages Are Serious Business
10© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Outages Are Serious Business
11© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Outages Are Serious Business
12© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Outages Are Serious Business
Source: Venture Beat & Ray Wang at Constellation Research 2012
“Twitter’s down…and it has cost the economy $1 billion already?”
13© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Outages Are Serious Business
Source: Venture Beat & Ray Wang at Constellation Research 2012
OK...reduce Twitter usage assumptions 50%.
Then, only $400,641,025.00 was lost.
“Twitter’s down…and it has cost the economy $1 billion already?”
14© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Last WeekTWO DAYS! (is “too long”)
15© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Extra October Stress for College Applicants 1 million students, 517 colleges, high anxiety
“Mass Panic As Common App Crashes”
16© 2013 SOASTA CONFIDENTIAL - All rights reserved.
But Wait, There’s Moreo Aug. 14: New York Times
• 2 hours - Stock dropped - opened paywall (lost $$) – Twitter storm
o Aug 17: Google
• Less than 5 minutes – lost $500,000 – Internet traffic dropped 40%
o Sept 2: HSBC (UK)
• Internal and external systems – “Weekend of disruption”
o Nov 12: Kayazoonga.com (India)
• Last cricket match of Sachin Tendulkar - 19.7M hits/hour – crashed all day – “Chaos”
o Nov 18: YouTube
• 35 minutes – est. 100,000 users affected
PR Nightmares…
17© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Every Site is SusceptibleEven BatKid’s
18© 2013 SOASTA CONFIDENTIAL - All rights reserved.
SOASTA Helps
19© 2013 SOASTA CONFIDENTIAL - All rights reserved.
London 2012 Results
500 Tests
400,000 concurrent users
431 Million Visitors
4.73 Billion Page Views
Access from 201 Countries
“We began working with SOASTA CloudTest six months prior to the
games.” Paul Bunnell,
Lead Architect, London 2012
20© 2013 SOASTA CONFIDENTIAL - All rights reserved.
SOASTA Helps
Prep for…
21© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Verbalizeit Results “ We were on very
tight timelines…
the SOASTA team
was able to run
multiple tests very
quickly over a two-
day period and get
us going on issues
to immediately
fix.”
Kunal Sarda,
VerbalizeIt
COO and Co-
Founder
Plan based on 6-8 million TV viewers
Tested & tuned for 10 million hits
250 to 30,000 visits/day
7x customer growth
Zero downtime
22© 2013 SOASTA CONFIDENTIAL - All rights reserved.
SOASTA Helps
23© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Dillard’s Results
Tested to 7x Initial load test goals
Reached 9x previous year’s peak
Validated dynamic caching investment
Reduced hardware capacity needs
Validated Akamai to 4x beyond peak
“One of the best investments to our
website this year .” Nancy Jordan,
IT Middleware & Integration
Manager
24© 2013 SOASTA CONFIDENTIAL - All rights reserved.
SOASTA Helps
25© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Hallmark Results“Website performance really
affects your brand image
when you have problems…
The inherent savings to our
brand by using CloudTest is
priceless.”
—Cheryl Davis,
digital technical
operations manager
Hallmark Cards, Inc.
Tested to 120% of expected load
Saved 50% on testing expenses
Reduced staff costs by 66%
Eliminated test software and hw costs
26© 2013 SOASTA CONFIDENTIAL - All rights reserved.
“No amount of testing within reasonable time limits can adequately replicate a live environment of this nature”
About that Healthcare.gov Testing…
27© 2013 SOASTA CONFIDENTIAL - All rights reserved.
“No amount of testing within reasonable time limits can adequately replicate a live environment of this nature”
About that Healthcare.gov Testing…
OBJECTION!
28© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Speed AND QualityThere is no need to compromise
29© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Testing Issues
o 55 Contractors
o Not enough time in schedule to conduct adequate performance testing
o No Integration testing between systems
o No end-to-end testing simulating real life experiences of end users
o Followed waterfall development with parallel/disconnected projects
o Locked at 2000 users at the 1st step
From the HHA press
http://www.washingtonpost.com/politics/house-panel-grills-contractors-on-troubled-health-insurance-web-site/2013/10/24/8f42c748-3ca7-11e3-b7ba-503fb5822c3e_story.html
http://www.cnn.com/2013/10/29/politics/obamacare-warning/
30© 2013 SOASTA CONFIDENTIAL - All rights reserved.
SOASTA Best Practices
o Test Continuously
From ten of thousands of tests
31© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Continuous PerformanceThe path to a performance culture
I JUST MADE A CODE CHANGE
CI system runs load tests against a pre-prod site
…while unit, functional and mobile device tests run in parallel
Daily (or more frequent) updates keep teams “performance focused”
Identify bottlenecks & inform teams, just like unit and functional tests in CI
32© 2013 SOASTA CONFIDENTIAL - All rights reserved.
SOASTA Best Practices
o Test End To End
From ten of thousands of tests
33© 2013 SOASTA CONFIDENTIAL - All rights reserved.
End-To-End Cloud Testing
Customers
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
CDNCredit Card
Web analyti
cs
Other 3rd pty svcs
34© 2013 SOASTA CONFIDENTIAL - All rights reserved.
“Conductor”
Analytics
CloudTest
Database
Main
Perf Tester 1
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
End-To-End Cloud Testing
CDNCredit Card
Web analyti
cs
Other 3rd pty svcs
35© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Analytics
CloudTest
Database
Main
Perf Tester 1
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
End-To-End Cloud Testing
CDNCredit Card
Web analyti
cs
Other 3rd pty svcs
36© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Analytics
CloudTest
Database
Main
Perf Tester 1
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
End-To-End Cloud Testing
CDNCredit Card
Web analyti
cs
Other 3rd pty svcs
37© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Analytics
CloudTest
Database
Main
Perf Tester 1
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
Data sources: Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
“Conductor”
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
East Coast (AWS)
End-To-End Cloud Testing
CDNCredit Card
Web analyti
cs
Other 3rd pty svcs
38© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Analytics
CloudTest
Database
Main
Perf Tester 1
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
East Coast (AWS)
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
San Francisco (GoGrid)
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
Chicago (Rackspace)
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
Tokyo (AWS)
AnalyticsLoad
GeneratorLoad Generator
Amsterdam (Azure)
AnalyticsLoad
GeneratorLoad Generator
Virginia (IBM)
SOASTA Global Test
Cloud
End-To-End Cloud Testing
CDNCredit Card
Web analyti
cs
Other 3rd pty svcs
39© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Analytics
CloudTest
Database
Main
Perf Tester 1
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
East Coast (AWS)
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
San Francisco (GoGrid)
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
Tokyo (AWS)
AnalyticsLoad
GeneratorLoad Generator
Amsterdam (Azure)
AnalyticsLoad
GeneratorLoad Generator
Virginia (IBM)SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
SOASTA Global Test
Cloud
Data sources: Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
“Conductor”
End-To-End Cloud Testing
CDNCredit Card
Web analyti
cs
Other 3rd pty svcs
Chicago (Rackspace)
40© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Analytics
CloudTest
Database
Main
Perf Tester 1
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
San Francisco (GoGrid)
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
Tokyo (AWS)
AnalyticsLoad
GeneratorLoad Generator
Amsterdam (Azure)
Data sources: Native (SSH), JMX, PerfMon, CA Introscope,
AppDynamics, New Relic, CloudWatch,….
SUT/AUT
Web Server
Web Server
App Server
Web Server
Web Server
App Server
App Server
App Server
Load Balancer
Cache
Cache
Database
AnalyticsLoad
GeneratorLoad Generator
Virginia (IBM)
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
SOASTA Global Test
Cloud
Access for all
AnalyticsLoad
GeneratorLoad GeneratorLoad
Generator
East Coast (AWS)
“Conductor”
End-To-End Cloud Testing
CDNCredit Card
Web analyti
cs
Other 3rd pty svcs
Chicago (Rackspace)
41© 2013 SOASTA CONFIDENTIAL - All rights reserved.
SOASTA Best Practices
o Test to Any Scale
From tens of thousands of tests
42© 2013 SOASTA CONFIDENTIAL - All rights reserved.
s
Memory leaks
Scale of Test
Sta
ge /
Team
Dev & Test
Release& Deploy
Network& Ops
CDN file placement
Load Balancer configuration
Network bandwidth
Network configuration
DNS routing
Inadequate server resources
Default configuration settings
Unbalanced web serversAuto-scaling failures
Latency between systems
Slow third-party plug-ins
Garbage collection
Database thread counts
Inefficient database queriesSlow pages
Conflict with other apps
Test Lab
Search technology limits
Method-level tuning
Max sockets exceeded
Firewall max capacity
Global latency variance
Security bottlenecks
Scale is relative – bugs are everywhere
Tuning for full-scale…and well beyond
Frequent Testing from Lab to Production
Staging Production (100% +++)
43© 2013 SOASTA CONFIDENTIAL - All rights reserved.
s
Scale of Test
Sta
ge /
Team
Dev & Test
Release& Deploy
Network& Ops
Test Lab Staging Production (100% +++)
Where do we typically find the issues?
Top 10 Performance Problems
Architecture
Configuration Settings
Load Balancer
Database Performance
Bandwidth
Third Party Services
Culture and Mindset
Application Issues
Shared Environments
Connectivity
44© 2013 SOASTA CONFIDENTIAL - All rights reserved.
Summary
o HHS mistakes have quality at the forefront (again)
o Risks to revenue and reputation are measurable and high profile
o Speed and quality do coexist
o Best practices are well established (follow them)
o There IS time, and you CAN test realistically
o Start now
Thanks
Contact SOASTA:www.soasta.com/cloudtest/
Follow us:
twitter.com/cloudtest
facebook.com/cloudtest
RESOURCES
Case studies: soasta.com/customers
Upcoming webinars: soasta.com/knowledge-center- This Friday, Nov. 22: “Getting Started with CloudTest”- Dec. 11: “Continuous Testing for Optimal Performance”
Need a test, fast? CloudTest On-Demand.- email [email protected] You’ll hear from us by tomorrow
www.SOASTA.com