integrating crowd testing into ci
TRANSCRIPT
QualiTest was recognized as the
only visionary pure play QA and testing Service Company in the “Gartner Magic
Quadrant”
2
Intro
3
Source: Nelson Hall Report, 2013
QualiTest is the world’s second largest testing
services specialist by revenue
Intro
Test Outsourcing Engagement Evolvement
4
Staff Augmentation
TCoE / Offshore
Managed Services
Crowd Testing
•Time & Materials
•Mainly in-house
contractors
•Cost deduction of
price per role
•Time & Materials
•Blended Rate
•Cost deduction of
average hourly
rate
•Time & Materials
and Fixed Price
•Responsibility
shitted to vendor
•Long Term
engagement
•Results Based
Testing: Service
Level Agreement /
KPIs
•Advanced pricing
models (pay per
bug/cycle)
•Test In The Wild
•Task based
•Results focus
Intro
• “Crowdsourcing is the process of obtaining needed services,ideas, or content by soliciting contributions from a large groupof people, and especially from an online community, ratherthan from traditional employees or suppliers”
• CrowdTesting is an emerging trend in software testing whichexploits the benefits, effectiveness, and efficiency ofcrowdsourcing and the Wisdom of the Crowd.
• It differs from traditional testing methods in that the testing iscarried out by a number of different testers from differentplaces, and not by hired consultants and professionals” Wikipedia
5
CrowdTesting is an approach
7
CrowdTesting
Access to over2.5 million testers
through over40 platforms
Top 5 reasons to use CrowdTesting
1. Access to specific talents
2. Ability to perform tasks quickly and efficiently
3. Test in the wild – different opinions, environments
4. The right incentive - reduce cost of xxx
5. It’s cool
8
Test Types1. Exploratory Testing – testing the entire functionality
2. Feature Testing – focus on specific feature/functionality
3. Localization Testing – focus on various languages
4. Compatibility – testing various Mobile devices
5. Usability – User Experience and Ease of use done by a mixture of real users and UI experts
6. Inter-operability Testing – running different applications to ensure proper performance
7. Configuration Testing – run device on different configuration settings
8. User specific testing – run testing on specific type of users (existing clients, designers etc.)
9. Test Data – create massive test data
10. Test Planning – convert design into test cases
11. Beta Testing – test the system using real users
9
Business Case
12
• Testers are usually paid by the bug
• Customers usually pay by cycle or subscription
• Pricing defined based on:
– Complexity of testing
– Amount of testers
– Crowd testers qualifications required
• ROI
– Cost of CrowdTesting vs. Traditional
– Cost of finding a defect
Moment of Involvement
• Before each release (Test Environment)
• After each release (Development Environment)
• Weekly basis – in parallel to internal testing
• After each sprint
13
CrowdTesting & DevOps
• DevOps is a software development method that stresses communication, collaboration, integration, automation and measurement…
• Testers becoming closer to the clients/end users and thus required to communicate with them
• CrowdTesting helps testers to get the real end-user experience and feedback
14
Example of CrowdTesting and Devops integration
• CrowdTesting cycles is performed after initial deployment of any new feature
• Companies with very frequent releases require on-going feedback from Crowd. Using Crowd as “human monitoring system”
• Communicate and get feedback from crowd before defining sprints backlog: Surveys, UI Review, Heuristic Evaluation
• Testing as a Service (TaaS):
15
1 hour
Target
• Reinforce the test team. Beta testing for specific SIM cards and locations
Achievements
• 10 private crowd testers
• Completed full cycle in South Africa within 3 days
16
Target
• Build Test Data used for regression test automation
• Repetitive work of tagging webpages and images
Achievements
• Categorize 157.000 URLs and images
• Completion within 3 weeks (compared to 6-12 months)
• Reduce the cost by over 80%
17
Target
• Testing web game in 52 countries and 23 languages
• Testing real life conditions
Achievements
• Full coverage delivered in 30 days
• Over 800 defects detected
18
Target
• Testing 20-30 mobile apps
• Running over 200 cycles each year
• Continuous integration SDLC (release each week)
• Functional, Usability and localization
Achievements
• Reduced cost per defect from 244£ to 48£
19