10 Point Success Strategy

in

Crowd Testing

Sree Ram Prasad Bhatiprolu, ManagerNilesh Kumar, Sr. Consultant

Deloitte Consulting India Pvt Ltd

IndexAbstract....................................................................................................................................................3

What is Crowd Testing: A brief introduction.............................................................................4

5 Major Drivers for Crowd Testing..................................................................................................5

Why Crowd Testing?.........................................................................................................................6

Types of Testing through Crowd Testing:.................................................................................8

Major Challenges in Crowd Testing:............................................................................................910 Point Definitive Success Strategy........................................................................................11

1. Train the Tester....................................................................................................................11

2. Focus on Coverage..............................................................................................................12

3. Communicate to Succeed.................................................................................................12

4. Maintaining Confidentiality and Building Trust..........................................................13

5. Motivated Crowd-testers...................................................................................................14

6. Address Technological Challenges –.............................................................................15

7. Mute False alarms:..............................................................................................................16

8. Deriving value from Defects............................................................................................17

9. Intelligent Crowd Sampling...............................................................................................17

10. Cohesive Team, not just crowd.......................................................................................18

When not to go for Crowd Testing?...........................................................................................19

Conclusion..........................................................................................................................................20

References.............................................................................................................20Author Biography...................................................................................................21

Abstract

With increasing digitization in every aspect of life, breakneck pace of development, ever shortening shelf-life of mobile applications, and more focus on user experience, it’s more than ever pertinent to look at Crowd Testing as our best ally in Quality Control.

This paper briefly touches upon Crowd testing in general, benefits of crowd testing and types of testing which can be targeted through Crowd – to set the context.

Paper lists some of the major challenges realized when engaged in crowdsourced testing projects. Challenges if not met, results in less than expected benefits from the crowd engagement. This paper presents a 10 point agenda to overcome all the major challenges, and introduces innovative techniques to reap definite success from Crowd Testing projects.

Detailed discussion on Crowd Sourcing is out of context of this paper.

What is Crowd Testing: A brief introduction

Crowd Testing or Crowd-sourced testing a trend which leverages the benefits of crowdsourcing and capability of cloud platform to accomplish the testing goal.

If is different from traditional testing teams in terms of its diversity, where testing is carried out by a number of testers spread across geographical locations, against the traditional test lab setup, where testing is carried out by hired in-house testing resources.

For big enterprises, awaiting project on-boarding can be a healthy crowd that can be leveraged very well for Crowd Testing.

It essentially comprised of three major parts -

In crowdsourced testing, application is put under diverse realistic testing scenarios and hence is a very reliable way of testing software, especially those which have a diverse target customer base across diverse localization settings.Crowd testing often employs payment per bug pay-out model, which helps in reducing the cost of quality to company whose software is being tested.“Having this diversity of staffing allows you to scale your resources up and down in a fluid manner, meeting tight deadlines during peak periods of development and testing, while controlling costs during slow periods.”

5 Major Drivers for Crowd Testing

“By 2018, crowdsourcing will constitute 20% of all enterprise application development sourcing initiatives.” - Gartner

“Gartner inquires about using crowdsourcing have grown by around 40% in the past year.”

Crowd testing has been in practice for years, however it is gaining more relevance due to fast transformations in digital landscape.

5 major drivers for Crowd Testing are –

Why Crowd Testing?

Benefits realized from Crowd testing -

1) Diversification of user base –Crowd testing can leverage a diverse tester base dispersed across the globe, having different skillsets, language proficiencies, locales and schedule. Software application based on localization can benefit greatly from this aspect of crowd testing. Issues related to geographical security, bandwidth issues across carriers can be uncovered easily.

2) Coverage of device fragmentation –The core testing team may not have all the possible resources to test the software in different environments, and situations. Android market for that matter is highly fragmented with 25,000+ devices from 1200+ brands. Even with a highly optimized test suit, it’s very difficult to achieve coverage across all possible devices, and operators.

3) Least setup cost, low Quality cost – Most of the crowdsourced testing projects have per bug payout, and, hence a sourcing agency has to pay only when a valid defect is uncovered. Companies need not maintain a big testing lab setup or pay for a large groups of testers who may not be utilized to their potential every time. This greatly reduces the Quality cost to a company.

4) Faster turnover – A large tester base, spread across the globe, working in different schedules results in a faster turn -around on

BUG TYPE BASE PAYOUT VALUABLE EXCEPTIONAL

GUI $4.00 $12.00 $20.00

FUNCTIONAL $4.00 $12.00 $20.00

TECHNICAL $4.00 $12.00 $20.00

SECURITY $6.00 $15.00 $25.00

bugs. More often the not, first bug is discovered within an hour, and in couple of days then entire bug hunting can reach plateau, for medium sized projects.

5) Collective cognizance of end users – Owing to a diverse and enormous base of testers, software companies can reap great benefits from the collective cognizance of tester base having specific skill-sets, which otherwise would be very cost intensive to maintain. Adding to that companies can benefit from the fresh and unbiased perspective of skilled testers which is closest to real world user testing.

6) Feedback – Aggregating Crowd Feedback while running the testing effort can yield invaluable insights into direction, scope and usability of the application.

Types of Testing through Crowd Testing:

Types of testing that can be done by Crowd Testing -

1) Web/Mobile – Crowd testing can greatly endow in Web testing and Mobile Testing. Testing across heavily fragmented mobile eco system, diverse browser settings coupled with a great number of locale, language and security settings, is big challenge in traditional testing lab settings, but is easily achievable in crowd testing.

2) Usability and Functional – Usability testing reveals how comfortable user is with a software solution. Crowd brings different perspectives in flow, navigation, speed, look-feel and content. In Crowd Testing, expose only the functionalities which are not-core to the application, so that the business edge is not compromised.

3) Localization – Localization testing aims to find defects on Software that rely heavily on language and locale setting. Across the globe dispersed tester crowd is match to none other test setting when it comes to localization testing.

4) Performance – Performance testing aims to test load, stress, & endurance of an application, when it subjected to concurrent user access. Crowd testing can give real world performance and behavior statistics through distributed user base across various data bandwidths.

5) Security – Security testing provides insights into how potentially vulnerable a software solution is. Crowd has different and enormous system security settings across different carriers and can easily uncover potential security threats which would otherwise need a complex lab setup to achieve.

6) Beta Testing – Beta Testing is conducted by end users to validate the usability, functionality, compatibility, and reliability of the application. Crowd testing is the most suitable setup for Beta Testing, because the crowd is essentially a diverse pool of potential end users.

Major Challenges in Crowd Testing:

1) Communication gap – Communication gap can be a major challenges in crowd testing. Since the crowd is delocalized, and linguistically diverse, establishing a proper and conductive communication challenge is a difficult task.a) Multiple Languages b) Non-communicative test managementc) Ambiguous scope documentationd) Cultural constraints

2) Lack of focus on Coverage – Crowd often starts with not a much focused agenda, and this, more often than not, results in skewed testing coverage of functional aspects.a) Low level of alignment to the scopeb) High autonomy of crowd

3) Lack of Motivation – Crowd is not always motivated and enthuse can go down if testers don’t taste initial success in bug finding. Unmotivated crowd can rarely yield value in proportion.a) Superficial level of testingb) Complex areas untestedc) More UI bugs than functional bugs

4) Confidentiality woes – Exposing confidential technology, workflows, or idea involves a risk of losing business edge. a) Security issuesb) Leak of confidential informationc) Non Adherence to NDA, Policy guidelines

5) Technological challenges – Technical challenges get more difficult to tame with a growing tester base. Technical challenges can be like -a) Improper version controlb) Improper App distribution c) Ineffective Test Cycle Management

6) False alarms – A large volume of false alarms is one of the major challenges in defect management in Crowd testing. False alarms for a large number of reported defects, which are not real issues. They can be one or many of the following –a) Improper understanding of the requirementb) Incorrect Data Setupc) Incorrect Environment setupd) Out of Scope defects

e) Duplicate defects

7) Non valuable bugs, enhancements masqueraded as bugs – Mainly because of (bug) quantity based payouts, testers have the incentive to log more and more bugs, which results in influx of a good number of least priority bugs, possible enhancements categorized as bugs.a) Large volume of P3,P4 defectsb) Plethora of UI issues but no security issuec) Enhancements masqueraded as defects

8) Incorrect tester sampling – Incorrect mix of tester cloud could defeat the purpose of crowd testing. Unskilled or improperly selected crowd may fail to uncover potential bugs.a) Unskilled tester baseb) Tester sample space doesn’t represent real end user space

9) Falsification of user information to gain more projects – A large number of testers exaggerate their skill sets and add false personal and/or device details with an intent to be suitable for more and more projects. Incorrect, outdated user data would result in improper crowd selection.a) Incorrect skills summaryb) Incorrect experience summaryc) Incorrect device profile

10) Legal Issues – One the most common legal issue is Tax avoidance but the crowd.

10 Point Definitive Success Strategy

1. Train the TesterAn aware tester is an early start in the game.Testers should be aware with the application under test. Testes should be educated around the functional, non-functional and other aspects of the application, based on the scope of testing. In addition to a traditional textual design document, other methods should be leveraged, like –

a) Recorded Demos – A recorded demo of the application can give a fair idea about the application in a very quick time, and can address most of tester queries then and there.

b) Mock Screens in design document – End user mock screens can significantly give the crowd a fair idea about what is least expected in terms of user interface and usability.

c) Micro-learning – Short and relevant micro learning videos can add greatly to the design specification of the application under test. Videos can either be distributed, or can be hosted to sites like YouTube, and can in-turn embedded in the design document itself, depicting functionality/page/module wise information. Embedding videos can be easily achieved in MS word like documents, or SWAY.e.g.Req: 1.01 When the user clicks on “||” button on the camera interface, video recording should pause.Demo video:

2. Focus on CoverageGoal based approach in paramount in crowd testing.

Crowd can easily loose the focus from the right objective and project can miss the coverage goals.Like a traditional testing project setup, a test-manager should take the reins of orienting the crowd to right aspects of application under test.Test Project manager’s role in critical in keeping the focus in the center of all the effort. Project Manager should, a) Crowd Re-orientation -

Judiciously and periodically reorient testers to right and critical area during the testing cycle. An agile and flexible approach would keep crowd focused on a specific area. PM can orient different pool of testers to different aspects/module of the application by ensuring more incentivized payouts.

b) Real time coverage report – Given that crowdsourced testing starts to yield defects very soon and hence a real time test coverage report would aid to test management decisions.

c) Crowd feedback to Drive – Real-time, non-intrusive feedback forms embedded on every major part of the application should be used to get feedback from the crowd, while the testing is still on. Crowd Recommendation can further drive the development effort and can result in more value realized than an in-house testing approach.

3. Communicate to SucceedCommunication is the key to success and this applies even to crowd testing projects. Communication challenges can range from asking for scope to clarification on functionality.A proper and conductive communication channel can add significant value to the entire testing process.Communication challenges can be mitigated by following, but not limited to, methods mentioned below -

a) Chat-bots to Answer generic queries – Chat-bots’ can range from being powered by simple inference engines to AI powered (like AWS, Azure, IBM Watson). Chat-bots can be empowered to address - i) General FAQs.ii) Page or Module Specific FAQsiii) Scope and specification related queriesiv) Payout schemes

b) Busting Language Barriers – Crowd testers are dispersed across the globe and follow different language and culture. A multilingual management environment is essential to collaborate with testers in different languages.

c) Multiple channels to disseminate information - Traditional channels like mail notifications are a good way to disseminate information to a larger audience in one go. Additionally, instant messenger will come to aide when a one to one connect is required. Additionally, leveraging mobile technology, in-app notifications can be a good way to share information quickly.

4. Maintaining Confidentiality and Building TrustBuilding trust and maintaining confidentiality is vital for any project.

a) Non-Disclosure Agreements – NDAs are kind of tried and tested methods for enforcing confidentiality, and prevent the leakage of critical information.

b) Meetups – Formal meet-ups, though sound traditional, can go long in building relations. All the involved parties get a chances to associate faces to the names.

c) Four Color Ethics Meter: While evaluating testers, a Four Color Ethics Meter can be adopted to showcase the ethical standards and policy adherence of testers.4 Color Policy -

5. Motivated Crowd-testersMotivated crowd is more than just a skilled crowd. Motivated testers can work enthusiastically, can connect well with the project and devote more time to find valuable defects. Money is not that only incentive that always motivates the crowd.PM’s role is important in motivating the crowd to achieve a collective success.Few methodologies that can be adapted to motivate -a) Interactive and engaging test management

– A responsive, and interactive test management provides a better engagement platform for Crowd testers.

b) Reward bonus point for active users – Rewarding active users in terms of bonus points, maintaining a leaderboard sort of thing would encourage crowd to participate more actively.

c) Crowd Contests, Bug Bounty - One of the simplest and proven way to engage is crowd is Contests. Popup quizzes, centered on the application, coupled with financial or non-financial incentives would keep crowd alert and active. Incentives can range from additional payout to bonus points to priority for next test cycle.

6. Address Technological Challenges – a) Version control – Proper care should be taken while maintaining

version control of the application under test. More often than not, it might be required to verify a defect in different version, or language, or locale settings.

b) Test Management – An effective test management is key to a project’s success. Like any traditional testing team, crowd would also need a proper test management and governance team. Building a pool of in-house testers to aid, support and verify crowd activities would yield much value in overall project outcome.

c) Confidentiality and security – Though crowd is supposed to adhere to NDA and policy guidelines, at times additional security measures can be implementedi) Feature Toggle – Some critical, and security compromising

features can to toggled off, while exposing the code, or app to the crowd.

ii) Exposing App vs exposing APIs :Based on security, complexity, readiness, and confidentiality, a judgement should be made as how much of application can be exposed to the crowd.

iii) Web-based App Host – For most of the web-based applications, a java-servlet type container can be implemented which will host the application to be tested. Users will interact with the application via

inside the container, which would hide the source code from the end user. Additionally, the left over space on the webpage can be utilized to get constant feedback from the crowd, to drive the development focus.

7. Mute False alarms:False alarms waste valuable time in negotiating worthless effort. Reducing the noise of false alarms can be achieved by following strategies -

a) Crowd Awareness Surveys - Using quick check surveys, with product awareness questionnaire, to ensure that crowd reads the design specification before embarking on the testing, along with known bugs, if any.

b) Crowd Evaluation Matrices and Tolerance limit – Set right matrices to evaluate the performance of crowd. Not more than, say, 10% invalid defects per cycle per tester; and provision of -Ve reward for invalid defects/ known bugs would deter users from logging defects without proper due diligence.

c) Right Sample space for Test Data – Defining what qualifies as the right test data is paramount to prevent data setup issues. More often than not, data issues crop up because of lacunae in understating the environment setup, and functional specifications.

d) Defect logs and Diagnostic report - Defect reporting to mandate collection of execution logs in addition to diagnostics information of the machine on which test case was executed. Using small utilities to automatically collect diagnostic information, crash reports, hardware and software profile can come as a handy solution.

e) Mandatory Execution Evidence – Defects should be supplemented with screenshots, test data, request-response logs for all the test cases.

8. Deriving value from Defectsa) Defect Guidelines – Properly set guidelines for defects should be in

place to clearly indicate what qualifies as a real bug. Guidelines list should contain, but not limited to, environment specifications, constraints, test data requirements, with a clear demarcation of expected vs actual output.

b) Pro-active exception handling – Known issues can be handled in the coding itself; known issues, with brief informative instruction, can be coded inside the exception handling process or service request/response code-module to make tester aware of the known bugs and limitations. This would prevent crowd from putting un-necessary focus on known issues or less valuable defects.

c) Bug Aware defect Management – Defect or Test management tool employed to manage testing efforts should have the facility to present a list of already discovered defects, filtered module wise. So that when a tester attempts to log a new defect, and selects some specific module, he/she should mandatorily check if any existing bug is already logged on the same issue. This would prevent duplicate bus to a large extent.

d) Bug Revalidation – All fixed and resolved bugs should be retested – wither by Crowd or in-house testing team. Active services from Sauce Labs/Browser stack can be utilized by in-house test team to revalidate high priority-severity bugs discovered by crowd, to get a detail understanding of the defect and its impacts.

e) Value Proposition of bug – Clear benchmarks should be set to define value proposition of the bug. Low priority bugs should earn low rewards, and significantly high payout for critical bugs, or security flaws, and so on.

f) Analytics based Defect Assessment – Analytical insights should be brought in defect impact assessment. Modules giving highest number of critical defects can be filtered, which in turn would direct the focus of development team.

9. Intelligent Crowd Samplinga) Qualifier round for crowd – A qualifier

round/online-contest can set in motion to select the right set of crowd based on performance in the qualifier. A quantitative measure could be as a simple as a sample application, or sandbox with purposefully injected defects can be offered to prospective crowd. Based on quantitative and qualitative value of defects found by testers, a right mix of crowd can be formed.

b) Analytics based Crowd Sampling – Analytics can be leveraged to find the most suited crowd based. Sampling criteria can be, but not limited to, following -

a. Industry Specific experience – FSI, LSHC, CMTb. Testing type experience – Functional, Security, Performancec. Demographics – Canada, UK, Russiad. Maturity – 6 years’ experience in penetration testing.e. Age, Genderf. Hardware Profile

For instance, for a first person shooter game targeted for teenagers, the target crowd base can be filtered to the level of mouse sensitivity and dpi. Setting right matrix to right filtering algorithm would result in most suitable crowd for the application under test.

c) Frequent Crowd Skills survey - Sourcing agency should keep the crowd skills data base updated with frequent and periodic skills survey. More often than not, updating skills is left to the crowd, which results in targeting of not so fit crowd for the projects. To avoid this, sourcing agency should encourage and at time stir the crowd to keep updating skills and profile as of often as possible.

d) Registered Device verification – Sourcing agency should ensure that user has entered correct information for hardware/software profile. Periodic device diagnostic information can be leveraged to keep a verified record of hardware/software profile of the crowd.

10. Cohesive Team, not just crowdFrom diversity to cohesiveness.

a) Inclusion – Crows inherently is a pool of individuals with diverse interests and mindsets. It’s essential for the project management to instill a sense of inclusion. Offering a transparent and unbiased test management is the starting point of inclusion.

b) Brand Loyalty – Building a brand loyalty can go a long way in making a crowdsourcing project and company succeed. Local meetups can also help in making crowd connected to the brand.

c) Long term relations – Rarely a project gets completed in one single cycle. So it’s very important to make long term relations with the crowd, or at least the best performing part of your outsourced cloud, so that if

ever a new testing cycle comes in, you would know whom to look for. It’s imperative build and maintain a database of star testers, as well as black-listed testers, so that the list can be leveraged in subsequent cycles.

d) Crowd Satisfaction Survey – Active and constructive feedback on various phases, from crowd would help tailor the process for greater good.

When not to go for Crowd Testing?

Crowd testing is not an ultimate solution to all the testing needs.It is not a substitute to traditional testing approach and should never be considered as one, for several factors.Engagement in Crowd Testing should be exercised with utmost caution, or be rather avoided when

Dealing with highly sensitive and confidential data Possibility of security compromise and intellectual property

theft Possibility of Exposure of critical and core business logic Unrealistic expectations from Crowd Testing Automation has better ROI than Crowd Testing

Conclusion

Technological advancements and ever increasing pace of innovation and delivery demands puts quality management under tremendous pressure to generate more and more value in a shorter time. With shortening shelf life of applications, organizations can tap into potential of crowd to achieve success.

Above mentioned 10 points would guide a project in right direction to get more value from the crowd testing.

Crowd testing has its own limitations, coupled with restrictions on what should be exposed to public, and is not a replacement of in-house testing team.

However, in-house testing can nowhere match the scope that crowdsourcing offers. A good strategy is to have a mix of crowdsourcing with in-house testing efforts.

References

1. https://hbr.org/2013/04/using-the-crowd-as-an-innovation-partner2. https://opensignal.com/reports/2015/08/android-fragmentation/3. http://blog.harbinger-systems.com/2016/02/crowdsourced-testing-an-

overview/4. http://www.qualitestgroup.com5. https://www.utest.com/6. https://www.techgig.com/testingcontest7. https://www2.deloitte.com/8. http://www.base2.com.br/wp-content/uploads/2016/11/

Author Biography

Sree Ram Prasad is a Manager with Deloitte Consulting USI, Hyderabad, India. Sree Ram is certified scrum master with 14 years of IT experience, worked across LSHC (Life sciences and Health care), TMT (Technology, Media & Telecommunications) and FSI (finance, securities and insurance) industries. Sree Ram holds Masters degree in software engineering from BITS Pilani. He has vast experience in leading, managing and delivering testing engagements. He has extensive experience in Automation, Integration Testing, Functional Testing and Performance and Testing.

Nilesh Kumar is a Sr. Consultant with Deloitte Consulting USI, Hyderabad, India. He has over 7 years of experience in Software Quality and Assurance. Nilesh has worked on various automation testing tools like QTP, Selenium, RFT and various test management tools. He is ISTQB and PAHM certified professional.

THANK YOU!