utest stpcon spring 2012 presentation
DESCRIPTION
TRANSCRIPT
|
The SoLoMo RevolutionAnd The Resulting Tectonic Shift In Testing
Matt Johnston | @matjohnston | CMO @ uTest | March 2012
The Challenge
||2
SoLoMoHuh?
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization 3
SoLoMo Breakdown
So Lo MoSocial Local Mobile
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization 4
SoLoMo Coming Fast
In the past six months…
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization 5
• Media & analysts aren’t the only ones talking about it– Marketing– Product– Engineering– IT– Sales– C-suite
• Want a seat at the table?– Get informed– Get involved– Develop an opinion– Advocate strongly
QA Leaders Better Sit Down
The Challenge
||6
Trends InSocial Apps
|7In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Not Just For B2C Anymore
• The social revolution isn’t coming… it’s here– Nothing is disconnected– No industry is immune
• The stats on social media adoption in Fortune 500 firms:– 77% integrate Twitter– 84% incorporate LinkedIn– 93% utilize Facebook– 57% publish on YouTube
• Gartner: By 2016, social
integrated w/ nearly all apps– Websites & apps– Mobile apps & mobile web– Desktop clients
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization 8
Goes Soooo Much Further
The Challenge
||9
Trends InLocal Apps
|10In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Not Just For Marketing
• Businesses use location for more than just marketing
• Growing trend toward location-based intelligence in apps– Asset tracking
– Maps & directions
– Location finders
– Check-in services
– Geo-based personalization
– Location-based ads
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization 11
Good news for users (and marketers)… good luck to testers.
Increasing Granularity
The Challenge
||12
Trends InMobile Apps
|13In-The-Wild Testing for Functional + Security + Load + Usability + Localization
• ABI: mobile revenues will hit $133 billion in 2014• B2B mobility no longer just for email
– Business apps: CRM, ERP, HR systems– Productivity apps: docs, spreadsheets, presentations– Collaboration apps: email, IM, publishing – Medical apps: health records, patient education
• B2C mobility growth even steeper– Retail: location-intelligent m-commerce– Media: magazines & newspapers going purely digital– Travel: mobile bookings, check-ins, maps, deals– Education: tablets in every classroom– Healthcare: patient records, physician notes– Social: no explanation required– Gaming: ditto
Mobile Apps Boom Market
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Mobile’s Just Getting Started
14
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Did I Mention…
15
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Cars
Gaming Consoles
E-Readers
Smart Homes
Household Appliances
16
• Smartphones & tablets make QA immeasurably tougher• Now imagine a whole world of apps:
Connected TVs
Psst… It’s Not Just Mobile
The Challenge
||17
SoLoMo ImpactOn Testing
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization 18
• Impact on design, dev & IT well-recognized & resourced• Sadly, but not at all surprisingly, not true for testing
– Want proof? Specialized designers & devs… but who tests?
Brave New World
|19In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Social: Impact On Testing
• Social presents functional test challenges
– Testing code & integrations that aren’t yours…
– That constantly changes…
– And you get the blame
• As well as security & privacy pitfalls
– Internally, a channel for spam, malware and data breaches
– Confidential data may be posted and trigger regulatory penalties
– Externally, increasingly used for user authentication on your products
|20In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Social: Impact On Testing
• New challenges include how will you:
– Regression test multiple social integrations for web & mobile apps?
– Write (and execute) test cases with zero visibility into social’s roadmaps?
– Automate anything?
– Test sharing/liking in a QA environment without publishing via social?
– Test your social footprint?
- FB page (and apps)
- LinkedIn page
- Twitter page
– Ensure security/privacy if you enable users to use social sign-on?
|21In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Local: Impact On Testing
• Not just a mobile problem– Localization testing– Geo-based personalization
• But it is a mobile problem too– Apps used outside the confines of the
QA lab, under in-the-wild conditions• Impossible to test effectively from
only inside the lab
|22In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Local: Impact On Testing
• New challenges include, how will you:– Replicate location (answer: you don’t)
– Test in NYC, SF, LA, DFW, CHI?
– Test abroad- SIM cards
- Carriers
– Test GPS-based intelligence?
– Test IP-based localization?
|23In-The-Wild Testing for Functional + Security + Load + Usability + Localization
• New challenges include how will you:– Native apps: Test across ever-changing OS, devices and carriers?– Mobile web: Test across wide range of OS and mobile browsers?– Either way, how does your testing replicate real-world conditions
- Specific and varied locales
- Tower-to-tower hopping
- Roaming
- Low battery
- Touch screen actions (pinch, swipe, pan, etc)
- Portrait vs. landscape
- Display definition
Mobile: Impact On Testing
|24In-The-Wild Testing for Functional + Security + Load + Usability + Localization
• Mobile web vs. native apps– 2009: 5-horse race– 2010: 4-horse race– 2011: 2-horse race (depending upon your industry)– 2012: 3 horses (including HTML5-driven mobile web)
Mobile: Impact On Testing
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization
F | R | A | G | M | E | N | T | E | D
25
• Android alone (and that’s only by OS version)
|26In-The-Wild Testing for Functional + Security + Load + Usability + Localization
• Form factors matter again– Phones (feature & smart)– Tablets– Hybrids
• Prohibitive HW & SW costs• Here we go again
– Late-90s web, part deux- Lack of mature tools- Lack of usability standards- Lack of security standards- Extremely fluid landscape
Mobile Testing Landscape
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Multiple Launch Schedules
Multiple products, roadmaps & launch schedules
27
|28In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Web Testing Matrix Is Tough
OS & Browser
Features
Test Coverage
|29In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Mobile Testing Matrix Is Insane
OS & Browser
Features
Handset Makers& Models
Test Coverage
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Mobile Testing Matrix Is Insane
30
OS & Browser
Features
Handset Makers& Models
WirelessCarriers
Test Coverage
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Mobile Testing Matrix Is Insane
31
OS & Browser
Features
Handset Makers& Models
WirelessCarriers
Location
Test Coverage
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization
Increasingly Complex Testing
32
• The laughably simplistic notion of a “test matrix”
The Challenge
||33
Ummmm…So What Do We Do?
|34In-The-Wild Testing for Functional + Security + Load + Usability + Localization
• Don’t be… the good news is that testing ain’t dead!– Chance for QA leaders to step up…– To lead the charge…– To defend users and app quality
Sufficiently Freaked Out?!?
|35In-The-Wild Testing for Functional + Security + Load + Usability + Localization
• You should be scared… need to completely re-think testing– Can’t automate our way out of this– Can’t hire or train our way out – Can’t solve this problem from soley within the lab
• More distributed problem than ever before– Need a new testing mindset– Need to bridge the growing gap
- Between testing & users- Between artificial and real- Between in-the-lab & in-the-wild testing
…And The Bad News…
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization 36
Lab-Based Testing Innovation
Agile vs.Waterfall
Manual vs. Automation Managing vs.
Leading Teams
Exploratory vs.Test Cases
Testers vs. Engineers
In-House vs. Outsourced
Offshore vs.Nearshore
In T
he L
ab
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization 37
In T
he L
ab
InThe
Wild
Users Work & Live In The Wild
|38In-The-Wild Testing for Functional + Security + Load + Usability + Localization
To Make Matters Worse
• Users are less tolerant of quality failures
• Social media & app stores give everyone a megaphone
• And their problem… is your problem
The Challenge
||39
A Call To Action ForTest Leaders (Two, Actually)
|40In-The-Wild Testing for Functional + Security + Load + Usability + Localization
• Yes, keep testing in-the lab– Keep training in-house– Keep outsourcing– Keep automating– Keep innovating
• Recognize it will never again be sufficient on its own• Build your company’s plan for testing in-the-wild
– Real-world conditions: Live testers, real devices, imperfect connectivity– Mirror your user base
- Technologically: OS, browser, anti-virus, device, carrier- Geographically: Continent, country, city, language- Demographically: Age, gender, education, employment, industry, hobby
• In-house QA leadership is vitally important– You have the business and product context– You know what your firm cares about– You need to be the orchestrator who leads this symphony
- What to test in the wild, how to test in the wild, holistic test planning
1. Look Beyond The Lab
|In-The-Wild Testing for Functional + Security + Load + Usability + Localization 41
• SoLoMo is coming – with or without QA’s buy-in• Want QA to be heard? Want quality to be built in?
– Don’t wait: No one else is coming to the rescue– Get informed: Develop an thoughtful opinion– Get involved: Find out when/where decisions are being made– Be proactive: Build a plan for quality from the ground up– Be protective: You’re a protector of the user… of the brand
2. Be The Voice For QA
The Challenge
||42
SoLoMoWoStoA Few Stories (if there’s time)
