W14
Test Techniques
5/7/2014 3:00:00 PM
Testing in the Wild: Practices for
Testing Beyond the Lab
Presented by:
Matt Johnston
Applause
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Matt Johnston
Applause
With more than a decade of experience at companies ranging from early-stage startups to publicly-traded enterprises, Matt Johnston leads the uTest and Applause organizational strategy efforts. Since joining uTest as employee #8 in late 2008, Matt has provided a technical sensibility to uTest’s engagement of testers and engineers. As CMO/CSO, he leads uTest’s large-scale initiatives, such as partnership formation and helping to define the company’s expansion strategy. Matt is a frequent speaker at top technology and mobile conferences, including STARWEST, CTIA, and Mobile App World. A renowned expert on crowdsourcing, Matt’s work has appeared in the bestseller Crowdsourcing for Dummies.
4/26/2014
1
|
Testing In the Wild: Practices for Testing Beyond the Lab
Matt Johnston | CMO/CSO, uTest | [email protected] | @matjohnstonApril, 2014
The Challenge
||
App Quality
Then vs. Now
4/26/2014
2
|
The Way Things Were
You Tell Them What To Think
|
The Way Things Are TODAY
They Tell You What They Think
4/26/2014
3
|
Mobile App Quality Is Judged More
Harshly
• Users less tolerant of
spotty quality
• Social & app stores
give every user a
megaphone
• Switching costs
lower than ever
• Cost of imperfect app
quality has spiked
|
So What Could Possibly Go Wrong?
4/26/2014
4
|
Cars
E-ReadersHousehold Appliances
Fragmentation of users, devices, OSes, browsers, locations
Oh, And It’s Not Just Mobile
Phones & Tablets
Laptops & Desktops
Connected TVs
Gaming Consoles
|
• Inside-out quality perspective is valuable but
incomplete
– Regardless of mobile app type or dev & test methodologies
• Replaced by outside-in quality worldview
– Quality signals come in many forms, and from USERS
- Bug reports
- Test cases executed
- Crashes
- User feedback
- Analytics
– And at many times (continuous)
No Turning Back Now
4/26/2014
5
|
• External Signals Inform Internal Decisions:
– What you think matters less than what users think
– How YOU measure app quality matters less
– What users feel and say matters MORE
– Court of public opinion rules
- App stores
- Social media
What’s That Mean?
The Challenge
|| 10
Ummmm7
So What Do We Do?
4/26/2014
6
|
Stuff Gets Built; Then It Breaks
1. Marketing identifies a need
2. Product specifies a solution
3. Engineering builds the product
4. QA tests the application
5. Help desk listens to complaints
6. Sales & execs blame everyone
7. Then we do it all over again and
hope for better results7
11
Vicious Cycle
| 12
Lab-Based Testing Innovation
Agile vs.
Waterfall
Manual vs.
Automation Managing vs.
Leading Teams
Exploratory vs.
Test Cases
Testers vs.
Engineers
In-House vs.
Outsourced
Offshore vs.
Nearshore
4/26/2014
7
|
Mob Mentality
BUT crowds often look (and act) like unruly mobs
13
|
From Mob to Community
And mobs don’t work in every category
Example: a skilled service like software testing or
requires an orderly and professional “community”
capable of consistently delivering the desired results
14
4/26/2014
8
|
The challenge is to turn this –
15
|
Into this
16
4/26/2014
9
The Challenge
|| 17
The Crowd:
Testing In-the-Wild
|
Community Management 101
• What Good CM Isn’t
– It’s not customer service
– It’s not marketing or social media
• What Good CM Is
– 1st gen � reactive & narrowcast (customer service)
– 2nd gen � proactive & broadcast (marketing)
– 3rd gen � independent & self-serve (program management)
– 4th gen � interdependent & vetting, teaching and policing one
another (community management)
18
4/26/2014
10
|
• CM goals – to build relationships, develop loyalty and nurture desired behavior
– Get beyond 1:1 discussions
– Get beyond reactive corrections
• CM Challenges – to do the following at scale
– Recruiting � Vetting � Engaging � Incenting � Managing
• Tool Kit
– Forums & message boards > email responses
– Help topics & tutorials > repeatedly answering same questions
– Webinars > phone calls
– Profiles
– Matching
– Ratings
– Reputations
– Payments
19
Community Management Tool Kit
|
• Top c-sourcing firms use two forms of compensation
– Monetary
– Reputation
• Performance-based ratings based upon a dozen factors:
– Participation Level:
- Lifetime & recent participation: # active test cycles, # reported issues
– Quality of Participation
- Approval percentage for bugs, test cases and usability surveys
- Accuracy of bug type & severity classifications
- Ability to write test cases & create automated test scripts
20
Rating & Reputation System
4/26/2014
11
The Challenge
||
uTest Case Study:
USA Today
|
• Timeline of USA Today Pioneering
– 1982: 1st nat’l paper to use full color
– 1996: Early adopter of web
– 2008: 1st major news brand with native apps
- iOS, Android, Windows, Kindle, web TVs, gaming consoles
– 2012: Re-designed & re-launched entirely new UI/UX across all
- Print + website + 14 native apps
- First web exceeded print; now apps exceed web
At the Forefront of Innovation
4/26/2014
12
|
• Great for USA Today’s readers
• Great for USA Today’s sales & marketing
• Vast pressure on tech execs to deliver quality across:
– Device makers & models
– OSes & browsers
– Carriers
– Languages (localized content)
– Locations (geo-based intelligence)
Impact on Testing
|
USA Today’s Testing Portfolio
In-House
Manual
Testing
Outsourced
Manual
Testing
In-House
Test
Automation
Outsourced
Test
Automation
In-The-Lab
Testing
4/26/2014
13
The Challenge
||
The Goal:
High Bar For App Quality
|
• Maintain Quality
– Launch nothing less than a 4.5 star app
– Each new version, across 14 native apps
• Increase Test Coverage
– Real-world conditions
– Cost-effective
– Scalable
• Meet Tight Deadlines
– Must have apps ready to win
- For each new platform that gains traction
- For each new device that gains traction
The Goals
4/26/2014
14
The Challenge
||
The Approach:
Achieve In-the-Wild Testing
|
• Traditional Outsourcing
– Lives in a lab
- Further removed from users
- Geo + demo
– Cheaper vs. in-house
– Hidden costs
- Time zones & language
- Scaling concerns and costs
– High noise-to-signal ratio
- No transparency
Two Distinct Choices
• Crowdsourcing & expertsourcing
– Lives beyond the lab, in the wild
- Professional testers + real-world conditions
– Mirrors USA Today’s user base
- Real-world conditions
- Tech + geo + demo
– Blended team
- Recurring testers (incl on-site) + crowd testers
- Dedicated uTest project manager
– Scales to match shifting needs
4/26/2014
15
|
• In-the-wild testing solution for USA Today
– Provide on-demand complement to in-house QA team
– Reduce escape rate by testing beyond the lab
- Across OS, browsers, devices & carriers
- Across locations, languages, industries & hobbies
– Blended test team built into USA Today’s dev & test lifecycle
- Dedicated uTest project manager
– Tester management
– Triaging services
- Team of dedicated testers (remote or on-site) + large team of crowd testers
Their Choice: Partner with uTest
The Challenge
||
Results:
Apps That Win
4/26/2014
16
|
• Reduced escape rate
– ID’d and documented 1,000+ bugs across 6 apps in first 90 days
• Improved usability
– Several iterations of usability testing & analysis
• Crowd details
– Testers from US, Canada, Brazil, UK, France, Germany, China & Japan
– Some projects (eg: brand new iPad app) involved 80+ testers
– Others with tighter scope required smaller teams of 10-12 testers
• Hit the ground running
– Launching Kindle Fire app same day the device launched
– Hand-picked testers pre-purchased various models of device
– Tested their app with all over the world, almost instantly
• Bottom line: better apps
– Enabled USA Today to keep innovating, while maintaining superior quality
– USA Today able to consistently meet 4.5 star app-store rating goal
Results
|
Customer Satisfaction
The Final Word
“uTest gives our organization a much better sense of where we
stand in terms of application stability and performance. For a group
that iterates as often as we do, that type of insight is critical”
-- Tim Carlson,
Director of Mobile Product Development, USA Today
4/26/2014
17
The Challenge
||
In-the-Wild Testing
Specific Takeaways
|
Key Takeaways: In-the-Wild Testing
• Apps universe has forever changed testing
– Exponentially more diverse user environments
– Devices & configurations
– Locations
– User demographics
• In-the-wild testing is a necessity
– Some testing must be done beyond the lab
– In-the-wild where app users live, work & play
• Agility is crucial to app success
– Test team size and tester location � enterprise-friendly mobile testing
• Partner selection is vital
– Not all created equal
– Alignment of capabilities, needs and philosophy
– Track record of relevant enterprise success
4/26/2014
18
The Challenge
|| 35
Questions?
Answers
Matt Johnston | CMO @ [email protected] | @matjohnston
