cloudteams playbook · adapt — remix, transform, and build upon the material for any purpose, ......

www.CloudTeams.eu

CloudTeams Playbook

A guide for Product Managers in Software Development supported by the CloudTeams Platform

Version 1, February 2017

http://www.cloudteams.eu/

The CloudTeams Playbook

2

Disclaimer: This e-book is published under the project CloudTeams, co-funded by the European Commission, under the Horizon

2020 funding scheme (Grant Agreement No: 644617). Under this project, different partners from Fraunhofer Institute for

Applied Information Technology FIT, National Technical University of Athens (NTUA), Singular Logic (SILO), FZI, Booreiland

(BOOR) and YASAD came together in order to realize the CloudTeams platform and validate the CloudTeams methodology, as

developed in the project. In this document, as authors and editors are considered the persons that actively participated in the

production of the current document solely, without forgetting the important role and contribution of every partner involved in

the project, individually.

Project Partners: FIT (DE), NTUA (GR), FZI (DE), SILO (GR), YASAD (TR), BOOR (NL)

Every effort has been made to ensure that all statements and information contained herein are accurate, however the Partners

accept no liability for any error or omission in the same.

This project has received funding from The European Union's Framework Programme for Research and Innovation - Horizon

2020, under Grant Agreement No: 644617. Action full title: 'Collaborative Software Development Framework based on Trusted,

Secure Cloud-based Pool of Users'. ICT-07-2014: Advanced Cloud Infrastructures and Services


3

This book is available under Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)

You are free to:

Share — copy and redistribute the material in any medium or format

Adapt — remix, transform, and build upon the material for any purpose, even commercially.

The licensor cannot revoke these freedoms as long as you follow the license terms.

Under the following terms:

Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.

ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.

No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

Notices:

You do not have to comply with the license for elements of the material in the public domain or where your use is permitted by an applicable exception or limitation.

No warranties are given. The license may not give you all of the permissions necessary for your intended use. For example, other rights such as publicity, privacy, or moral rights may limit how you use the material.

https://creativecommons.org/licenses/by-sa/4.0/








4

Main Authors & Contributors Main author

Iosif Alvertis

MSc in Electrical & Computer Engineering, National Technical University of Athens (NTUA), Greece

MBA in Entrepreneurship & Innovation, Athens University of Economics & Business, Greece

PhD Candidate in Electrical & Computer Engineering, NTUA, Greece

Emails: [email protected], [email protected]

Decision Support Systems Laboratory, NTUA, Greece

Main Contributors

Giannis Ledakis,

Dipl. Eng. Software Engineer, CEID University of Patras, Greece

[email protected]

SingularLogic S.A.

Dimitris Panopoulos, Dr.

MSc in Mechanical Engineering, National Technical University of Athens (NTUA), Greece

MBA in Technoeconomic Systems, NTUA & University of Piraeus, Greece

PhD in Electrical & Computer Engineering, NTUA, Greece

Emails: [email protected], [email protected]


Editor

Sotiris Koussouris, Dr.

MSc in Electrical & Computer Engineering, National Technical University of Athens (NTUA), Greece

MBA in Technoeconomic Systems, NTUA & University of Piraeus, Greece

PhD in IT Business Process Modelling and Re-engineering, School of Electrical & Computer

Engineering, NTUA, Greece

Email: [email protected]


Illustrated by

Iosif Alvertis

Booreiland

Strategic Interaction Design

Email: [email protected]

Website: http://booreiland.nl/

mailto:[email protected]






http://booreiland.nl/


5

Table of Contents

1 Introduction 8

2 What a Product Manager is 9

3 The CloudTeams Methodology 10

4 The CloudTeams Platform 14

5 Running the CloudTeams methodology step by step 17

5.1 Step 1: Ideation & Idea validation 17

5.2 Step 2: User Experience & Scenarios Validation 23

5.3 Step 3: System Backlog Definition & User Stories Validation 31

5.4 Step 4: Sprint Backlog Definition 38

5.5 Step 5: Design and Visual Modelling 40

5.6 Step 6: Coding 44

5.7 Step 7: Automated Unit Testing 46

5.8 Step 8: Continuous Integration Testing 47

5.9 Step 9: Automated Acceptance Testing 49

5.10 Step 10: Regression & Functional Testing 50

5.11 Step 11: User Acceptance Testing – UAT 54

5.12 Step 12: Market Test 58

6 Summary 61


6

Table of Figures Figure 3-1. The ideation and idea validation as visualized in the CloudTeams methodology 10 Figure 3-2. The CloudTeams methodology expanded with an Agile Software Development methodology 11 Figure 3-3. Using CloudTeams Methodology under different Software Engineering Methodologies 13 Figure 4-1. The general architecture of the CloudTeams platform 14 Figure 4-2. The main offerings of the CloudTeams platform for Software Teams (left sidebar) 15 Figure 4-3. The CloudTeams Personas as a layer of Privacy to enable CloudTeams campaigns 16 Figure 5-1. The ideation and idea validation as visualized in the CloudTeams methodology 17 Figure 5-2. Step 1 - Ideation & Idea Validation workflow 18 Figure 5-3. Documenting Business Model Canvases in CloudTeams Platform 20 Figure 5-4. The user experience and scenarios validation as visualized in the CloudTeams methodology 24 Figure 5-5. Step 2 - User Experience & Scenarios Validation workflow 24 Figure 5-6. The System Backlog Definition and User Stories Validation steps, as visualized in the CloudTeams

methodology 32 Figure 5-7. System backlog definition and User Stories Validation 33 Figure 5-8. Hierarchy in User Stories, relation with the goal of each level, and relative examples 34 Figure 5-9. Putting the user stories in priority 35 Figure 5-10. An example of a workflow that completes an interaction (i.e. a user map) 36 Figure 5-11. CloudTeams Platform allows the documentation of both Scenarios and User Stories 37 Figure 5-12. The position of Step 4 (Sprint Backlog Definition) in the implementation phase 39 Figure 5-13. Connecting external services to a CloudTeams project 40 Figure 5-14. The complete diagram of Step 4 “Design and Visual Modelling” 41 Figure 5-15. The “Snape” application helps software teams manage their UML diagrams 44 Figure 5-16. Suggestion on Product Manager engagement in Code Collaboration through CloudTeams 46 Figure 5-18. Suggested workflow for Continuous Integration and Automated Deployment 49 Figure 5-20. The relation of step 10 (regression and functional testing) with step 3 (system backlog definition and

update) 51 Figure 5-21. The suggested workflow for regression and functional testing 52 Figure 5-22. The relation of step 11 (user acceptance testing) with step 2 (user experience) 54 Figure 5-23. Running the User Acceptance Testing. 55 Figure 5-24. Setting up a UEQ (or other) questionnaire through CloudTeams platform 57 Figure 5-25. Validating the product based on market tests 58 Figure 5-26. Running the “final” step of the Market Test 59


7

Table of Tables Table 5-1. The criteria based on which an idea should be evaluated (i.e. first column), related to the business where

the ideas lay, interrelated based on the level of importance. N/A=important, *=very important, **=extremely

important 22 Table 5-2. The criteria based on which an idea should be evaluated (rows), related to the business where the ideas

lays, interrelated based on the level of importance (columns). N/I= not important (low), *= important

(medium), **=very important (high) 27 Table 5-3. Relation of strategies and criteria’s score (at least reaching the suggested score) 28 Table 5-4. Well-known User Acceptance Testing models 30 Table 5-5. An example of documenting a Use Case 37 Table 5-6. 7. The software development cycle in Software Development, Testi ng and Verification 45 Table 5-8. The template for unsupervised use case auditing (evaluating use cases) 52 Table 5-9. Different metrics for running supervised testing 53 Table 5-10. Comparing well-known evaluation frameworks where partners are experienced in 55 Table 5-11. Evaluation Metrics selected for the CloudTeams Pilots Operation 56


8

1 Introduction

This Playbook combines best practices described in various guides and methodologies, in order to deliver a

handbook for every Product Manager (PM), a series of steps that should follow in order to increase the

chances of launching a successful software product. This handbook is accompanied with real-life scenarios

that our team faced during the development phase of CloudTeams itself, and showcases the techniques we

used to meet our goals. Nevertheless, the innovation that this Playbook brings is that it is accompanied by a

web tool, the CloudTeams platform, which is designed to support the PM throughout different steps of the

product development phase; this is the reason it is called a “Playbook” and not a “Handbook”, as it brings a

playground for PMs to apply the theory, with a great level of flexibility and customization.

This Playbook should be read and mastered by people working as Product Managers, or people who want to

become ones. However, it is expected to be shared with the team members, as well as with high levels of

management, in order to help an organization to align its goals under a common mentality. The team that

applies this Playbook may declared itself as “rigid”, “agile” or “lean”, but it may use parts or complete steps

of the methodology and tools of the platform. To increase the chances to succeed on your goals, if you like

what you read we would suggest you to share it within your organization with people that have similar

“pains” as you; this is why it is available as a free e-book. The only thing we want is not to forget to attribute

our work, and the work of other people we reference in our book; this is way we have chosen to publish this

book under Creative Commons.


9

2 What a Product Manager is

There are different approaches and descriptions of what a Project Manager (PM) is as a role, what tasks she

should handle and how she collaborates with different team members. The most prominent describes that “a

Product Manager (PM) is responsible for making sure that a team ships a great product”1, and in some way,

she is “like a mini-CEO of their product”. Of course, this does not mean that a PM is responsible for driving

the strategy of the company, but rather aligning the company’s goals with the product’s goals, negotiating

the budget allocated to the product and managing the product lifecycle. In some cases, a PM may even have

to manage the innovation cycle of her product. In extreme cases, where the business matches a product (e.g.

in an early-stage or growth-stage startup), the CEO and the PM may be the same role.

Depending on the company, a Product Manager collaborates with a Technical Manager who has deeper

technical background (if not member of the team), while it is possible that even a Project or Team manager

may support the PM, to remove some roles from her and help her focus on the product. Everything is a

matter of the team culture and organization, nevertheless the PM herself should handle such organizational

issues that help the team improve its performance.

Typically, a Product Manager is an experienced and well-trained person that focuses on the product and the

team that she has to manage. For that reason, she should combine Business, Engineering, Design and

Communication skills. There is no curriculum that covers all these dimensions of a PM, for that reason the

person that takes this role is a team member with advanced business and leadership skill promoted to

manage the product itself, a Project Manager that extends her business and technical background, or a

(former) entrepreneur who builds practical experience. Additionally, because of the lack of proper, well-

established courses, the PM should extend her knowledge through reading books and attending seminars or

courses from different topics. At the end of this book you will find a list of suggested books that a PM should

have read to build a modern, lean mentality aligned with the needs of the business environment.

1 “Cracking the PM Interview: How to Land a Product Manager Job in Technology.”


10

3 The CloudTeams Methodology

CloudTeams aims at providing an environment to support the whole lifecycle of software development with

social features, starting from the ideation phase and ending at the final market release of a software product.

The CloudTeams methodology provides a path to the development lifecycle and defines the points and ways

of interaction among the software teams and the customers, as well as a set of recommended verification

and validation activities. It is not an obligatory process, but a recommended methodology to increase the

effectiveness of the tools that CloudTeams project delivers. Of course the CloudTeams methodology may be

adopted without the usage of the CloudTeams platform, and its integrated tools, but such an option would

require the team to find and choose its own tools to conclude those steps.

Figure 3-1. The ideation and idea validation as visualized in the CloudTeams methodology

The CloudTeams methodology (Figure 3-1) provides an iterative process starting from the Ideation Phase

(step IA) that aims not only at connecting development teams to potential customers, but also at supporting

the validation of any given idea, based on the customers’ opinions provided in several alternative ways. The

second step of the methodology focuses on the User (i.e. Customer) Experience (step IIA) in terms of

collecting feedback from targeted customer groups (represented by different personas), leading to the

specification of the complete customer requirements list of the software to be developed. The third step

targets at the definition of the functional and non-functional requirements of the software, expressed as

User Stories; With the responsibility of the Product Manager, the User Stories backlog is being filtered

according to the validation results and the stories are being classified in the System Backlog (step IIIA). The

next steps of the methodology refer to core Development of the Software (grey box in the bottom of the

Figure), and cover the sprints/iteration cycles required for the release of the software. The exact flow within

the sprints can be customized according to the exact development methodology that a software

development team follows, as long as (semi-)automated testing processes – both in unit and in sprint output

level – are incorporated to the overall process and monitored in each cycle with the help of the CloudTeams

platform. After the completion of each build of the software, the CloudTeams methodology supposes


11

Functional/ Regression Testing (step IIIB) of the release, taking advantage of the testing plan delivered in the

third step of the methodology. Following the verification of each software build and before the release of the

software, the CloudTeams methodology suggests the establishment of a validation process (step IIB), based

on the User Acceptance Testing plan, delivered in the second step. The involvement of the Customers in this

process and their feedback are both important aspects. The final step of the CloudTeams cycle refers to

identifying the Market Acceptance level (step IB) of each release in practice.

Moreover, as already illustrated in Figure 3-1, the initial 3 steps (IA, IIA, IIA) of the methodology are

supported directly by the CloudTeams platform tools, while the code developing and testing tasks may be

supported through the establishment of monitoring and management interfaces with existing, popular third

party solutions. The final step (IB) may be validated through the community acceptance and participation of

the CloudTeams community, like following the project, acquiring rewards and pre-ordering a product.

Figure 3-2. The CloudTeams methodology expanded with an Agile Software Development methodology

So far the software development part was described as a “black box” in the CloudTeams methodology. The

main reason for that decision was that every team is used to another Software Engineering methodology,

while the problem is identified on the way the technical team is synced with the business operations and

needs of the organization; this role is typically part of the job of a Product Manager, to synchronize the

software development cycle with the product development cycle. In CloudTeams we have chosen to go with

an Agile approach, and in particular with the Scrum methodology to describe how the technical team will

align its actions with the ones of the business team. Additionally, as seen in the “unfolded” view of the

methodology in Figure 3-2, the highest steps of the methodology require validating by users and customers,

while the lowest steps need verification by the technical team. Overall, the lower the steps of the

methodology in the V-path, the more technical skills are required by the team members; the Product

Manager gets less involved as the process goes down while the role of the Development Team becomes


12

more crucial. On the other side, for the steps on the top of the methodology more business skills are

required, thus the involvement of the Product Manager and the Team of Developers (i.e. in many companies

there is a discrete role of a Technical Manager) reverse.

Regarding the utilisation in software products development, the CloudTeams methodology is agile-oriented,

supporting popular agile-driven software methodologies, like Scrum, Kanban, Scrum-ban, etc. On the other

hand, this does not exclude teams that wish to follow the waterfall model, as long as the methodological

steps in the grey background are amended respectively. A relative example is given, presenting how three

popular methodologies (i.e. Waterfall, Agile, Lean) may be based in the CloudTeams methodology.

The main point that needs clarification is why and how Agile is separated from Lean; Lean has strong Agile

mentality, but the idea is to run early (business) customer experiments, even without an actual

implementation. It has many common characteristics with a prototyping phase, but it needs to measure

customers’ intention to acquire the product. The difference may seem quite vague, as for example the

Dropbox experiment with a YouTube video is considered by many an MVP (i.e. Minimum Viable Product)

rather than a prototype. Figure 3-3 tries to clarify those differences, without asking for strong separations, as

we consider that an Agile team can easily become Lean, and the opposite, depending of the product phase

that is positioned (e.g. Lean in early stages or major updates, Agile during progressive improvements and

market growth phases). As seen in the Figure, with the relevant steps of the CloudTeams methodology,

initially a Waterfall methodology (i.e. grey line) will go through all of the steps to release after some months a

new product. On the contrary, an Agile methodology (e.g. Scrum), will try to prioritize meaningful features

using early-step validation by customers, will validate the initial version with users (not customers) in order to

improve any issues identified, and then after any improvement it will launch the product (i.e. this may need

multiple testing releases of course). On the contrary, a Lean methodology will go faster to market, even

without or a partially developed solution, to see what is the MVP, run multiple experiments and then speed

up the last cycle of the development with the necessary features. Conceptually, the Agile methodology

should know what the market wants but does not how to deliver it, the Lean methodology should not know

what the market is but considers it undisrupted and sees many opportunities even with modest solutions,

and a Waterfall solution runs in highly-competitive markets, with strict regulations and low-differentiation

capabilities.


13

Figure 3-3. Using CloudTeams Methodology under different Software Engineering Methodologies

In the next chapter, an introduction of the CloudTeams platform is given, to understand how it may empower

a team in the Software Lifecycle.


14

4 The CloudTeams Platform

CloudTeams conceptually is separated in two main offerings for Product Managers (Figure 4-1): (a) a

collaborative platform for software teams where they can exchange knowledge, manage their product

lifecycle with intuitive tools, connect with third party solutions to monitor team’s progress, and (b) a user

community platform where users can find interesting projects, connect their cloud services to fit better in

recommended projects, and engage in campaigns that projects launch in order to get feedback on new

products developed. The incentives for customers to enrol a campaign are the coins (i.e. CloudCoins or CC)

collected for the time spend in teams’ campaigns, which later may be redeemed in offerings by the projects

(i.e. rewards). Teams are incentivised to offer interesting campaigns and rewards, by paying for the coins that

are needed to launch a campaign and reach the required number of participants. Nevertheless, during the

testing period of the platform and for one more year, there is no plan for charging teams for participating,

rather than controlling how many coins each team is allocated with; thus, more engaging teams will be

allocated more CloudCoins.

Figure 4-1. The general architecture of the CloudTeams platform

In practice, the Software Teams gets the following list of services (see Figure 4-2):

Link the following External Services: GitHub, BitBucket, SonarQube, JIRA, PaaSport, Trello,

and Snape.

Setting up and managing campaigns with questionnaires, through a dedicated Persona

Builder Tool developed by CloudTeams to target specific customer segments.

A Requirements library with scenarios and user stories.

A Business Model library where teams can store and link business models with personas (i.e.

customer segments).


15

project management and team collaboration.

Document management.

Project blog as a project diary.

Tool to manage the project rewards offered to customers.

Messaging, Slack-like, service for the team members.

Invitation mechanism to bring existing customers to the project to test the current project.

Customer Ideas, where teams can interact with customers on direct feedback.

The CloudCoins service to manage the project’s wallet.

Notification centre to monitor project’s activities.

Figure 4-2. The main offerings of the CloudTeams platform for Software Teams (left sidebar)


16

In order to bring those offerings to the Software Teams and link them with the prospective Customers of the

platform, CloudTeams developed the concept of the Persona Builder. In detail, as seen in Figure 4-3, each

time a Product Manager or a Software Team is about to launch a new campaign, they should choose for a set

of personas where the interaction will be targeted. CloudTeams handles these filters provided by the team

member, and invited a set of users of the Customer platform to the campaign that match those criteria.

Customers will participate in the campaign through a rotating pseudo-name, which protects them from

exposing their real identities.

Figure 4-3. The CloudTeams Personas as a layer of Privacy to enable CloudTeams campaigns

By the time this book is written, CloudTeams is provided as a SaaS platform hosted in www.cloudteams.eu.

Nevertheless, private installations have been explored as a possibility for enterprises that have already an

adequate customer base and would like to drive their feedback mechanism through a white-labelled solution.

In case you are interested in such enterprise solutions, you should contact our project representative2.

In the next chapter, a detailed manual on implementing the CloudTeams methodology, step by step, by using

the CloudTeams platform as well as popular other tools and techniques, is described.

2 Project Coordinator: Prof. Wolfgang Prinz, PhD, [email protected]

http://www.cloudteams.eu/



17

5 Running the CloudTeams methodology step by step

Someone may be already in the position to adapt the CloudTeams methodology on his own engineering

process, and combine it with his own tools. In this chapter, we go deeper into the methodology and

demonstrate how the CloudTeams methodology may be realised in action, with the help of the CloudTeams

platform and other popular and well-established tools or techniques.

In the given detailed workflows, with grey colour it indicates the processes that a team has to run, while with

purple colour we indicate the objects or outcomes of a process. In case that the CloudTeams platform may be

used to support or conclude a step, or document an object, there is a small icon with the logo of CloudTeams

next to each step.

Step 1: Ideation & Idea validation The first step in the CloudTeams methodology is about ideation and idea validation. As along as these parts

involve different stakeholders, we split them in two sub-steps, 1a (ideation) and 1b (validation) accordingly,

and we describe them in detail.

Figure 5-1. The ideation and idea validation as visualized in the CloudTeams methodology

Step 1a – Ideation (Figure 5-2 left side):

1. Ideas may pop up from different directions; epiphany from a team member, outcomes of

brainstorming activities, or the analysed feedback collected from existing and prospective customers.

It is important as an organization to enable all these different pools of incentives, and fine tune the

way input is collected and documented in the organisation; to make them useful, properly diffused


18

and reusable by different groups in the organisation. A useful framework to generate and validate

your ideas is the theory of “Jobs to be Done”3.

2. Every idea should be available in a backlog of ideas4, and may be potentially the upcoming solutions

that the organisation produces. Before starting discussing an idea with the team to start developing

it, the business context of the organisation should be defined. For that reason, an updated repository

of the business goals and business needs, derived from the business plan and the yearly planning

activity of the company, should be available to and accessible by all team members. Everyone

internally should be aware of the context where the ideas are generated, produced and delivered.

For “fresh”, early-stage teams that may have not been incorporated yet (i.e. prospective startups),

the business context is the vision and the values of the team, or what the team members want to

cope with, as well as where they want to get to.

Figure 5-2. Step 1 - Ideation & Idea Validation workflow

3. Then, the team members should come into meetings to discuss the promoted ideas in context, share

opinions and knowledge on the area, hopes and visions, filter out ideas and describe details to end

3 Jobs to be Done: http://www.christenseninstitute.org/key-concepts/jobs-to-be-done/ 4 CloudTeams platform can be used as a place to document and archive ideas

http://www.christenseninstitute.org/key-concepts/jobs-to-be-done/


19

up with the most prevailing ideas. Synthesis of different ideas, even generation of new ones, may be

possible too.

External advisors with experience on the relative market may need to be invited, people either

coming from the organization or external experts; they can give greater insights of the feasibility, the

viability and the customer value of each idea, but mostly to describe the current landscape and

customer behaviour. This role may be allocated to a team member with the greatest relative

experience. Caution, the advisor should not drive the team to drop an idea without further

exploration, or to build something they do not believe on. Additionally, the advisor should be trustful

and give-backs schemes may be needed to keep him/her involved in different steps of the product

development.

4. If the meetings have been concluded and the team has promoted one or more ideas, a further

process is needed before presenting the idea(s) to the management team and to decide what is

going to be the next product/service or the new feature of the company that will be first tested and

then developed. The team may break into two parts the analysis, one allocated to a technical team

and one to a marketing team:

a. The technical (sub-)team should do desk research on proprietary (e.g. R&D inventory,

patents etc.) and external solutions related on the idea explored, and if it is something new

in technical perspective a feasibility study should provide the team with all the alternative

technical solutions, from manual, to semi-automated and completely automated by

technology solutions. Early estimation of the time schedule and the cost for each solution

should be given, in order to allow the team decide if it will take on the new idea

development line later.

Of course, the less experienced the team is in the area of the idea, the less accurate those

estimations will be, and a proper fault rate should be given. For inexperienced teams this

fault may reach even 3 times up the normal estimations (in time and costs of development).

b. The marketing (sub-)team should use common marketing analysis techniques to estimate

the size of the market, by triangulating top-down with bottom-up and break-even analyses.

On this size of the market, the competition analysis should take place, to estimate the

portion of the market that the idea may possibly capture, and evaluate the quality of the

market and the level of the competition (e.g. through SWOT analysis and Porter’s 5 forces).

5. Provided that both sub-teams have completed their work, the team is ready to call a new

brainstorming meeting to cross-check findings and end up with a smaller set of ideas. For every idea

that qualifies, a draft business model canvas should be generated; both the marketing and the

technical analyses should allow the team cover every important aspect of the proposed business

model, like costs, required resources, value proposition and activities required. The channels, the

relationships and the pricing model for revenues may be part of further analysis, if they are not the

core differentiation factor of the idea. Gaps in the business model should be clear to all the team

members, and meetings may continue until there is a complete idea.


20

In case that there are some ideas that raise worries but seem promising, the team should take into

account that the main power that will drive the success or the failure of a new concept is the

customer; any marketing or technical worries may be overcome by changing slightly the production

roadmap, or by mocking some of the offerings until the market or a technology is more mature. For

that reason, do not hesitate to promote an idea to the “Idea Validation” phase, as long as you have

the management approval in the next step.

The CloudTeams platform allows the documentation and collaboration of business model canvases,

for each project (Figure 5-3).

Figure 5-3. Documenting Business Model Canvases in CloudTeams Platform

6. At the end of these brainstorming meetings, for each qualified idea the team should assign to the

team members the development of a 10-slide pitch deck5 and a two-page report, where the business

idea will be explained and presented as if investors should buy-in. It is important to include both

documents, as there are management teams that are prone to visual presentations and

management teams that prefer to read the ideas documented in advance and do their own research.

Additionally, the produced material may support future decisions, or may be reused in next steps, for

example if there is the need of an external fund.

In case of a startup, where the management team is the development team in most of the cases, the

audience may be the experts/advisors of the team, existing investors, prospective investors (i.e.

when you know that you will not burn the bridges for future communication) or even innovation

competitions where business people participate to.

5 http://guykawasaki.com/the-only-10-slides-you-need-in-your-pitch/

http://guykawasaki.com/the-only-10-slides-you-need-in-your-pitch/


21

7. The phase of ideation ends with the internal pitching of the idea(s) to the management team, which

should approve the idea that qualifies to be validated by the team. There are historic examples that a

strict management team dropped down nice ideas that have been a success in the future (e.g. Kodak

or Polaroid), but the business context and limitations where a product development team operates

should not be neglected, and it is preferable for an idea to be rejected in early stages rather than

after resources have been spent on development and marketing. On the other hand, there is no

reason for strict judgment of the ideas, as the next step should validate the idea based on early

customers’ feedback with small costs (step 1b).

Thus, the methodology under which the idea will be validated executed should be available to the

management team, especially the fact that there are multiple steps of validation before the idea

goes into the production phase.

By the end of this step, it is suggested that the team ends up with one idea where it focuses on. If multiple

ideas have been qualified, the idea validation step may run from different teams, may run in consecutive

cycles of validation, or more preferably the idea returns to the idea backlog for any future iterations.

Step 1b – Idea validation (Figure 5-2 right side):

8. After the team has ended up with the idea, it is time to validate it in an abstract level, with the

prospective customers. Based on the segmentation of the market research that the team has done

already in step 4b, and the business model canvas that has defined the selected segments and the

value propositions (step 5), the team may use CloudTeams persona builder6 to identify and describe

the target customers by personas that should use the described product/service (if the team does

not want to use CloudTeams, traditional methods for persona building should be used, but then the

team should have an agenda of contacts already to interview in next steps). After running the

queries in persona builder, the team has created a library of personas that are relevant to their

business model.

9. Having defined those personas, the team is ready for the first contact with the customers. In that

step, the team should select the customers that it will interview. If the team uses CloudTeams that

step is faster and easier, as recommended and anonymous users are linked to the constructed

personas and be invited directly. Before inviting them, the team should know that on this stage the

problem hypotheses (i.e. what the value proposition solves and for whom) are validated, and choose

if it goes with face to face interviews, online questionnaires, or a mixture of them. Of course,

questions related with the selected idea may be included, and glimpses of the solution may be tested

especially in the face to face interviews, and the team should pay attention not to direct

interviewees to the desired result. Demographics should be also collected, in order to look for any

correlations in later analysis, even if the sample in that stage is expected to be too small.

There are many rules and guidelines for running an interview or building a proper questionnaire. The

book “Running Lean: Iterate from Plan A to a Plan that Works” by Ash Maurya gives a detailed

6 https://about.cloudteams.eu/hey-teams-meet-your-customers/

https://about.cloudteams.eu/hey-teams-meet-your-customers/


22

example and proper guidelines for problem-oriented interviews; The relative advice is to focus on

three main problems as hypotheses of what the offering addresses, and ask the interviewees about

alternatives they use, and rank them based on the urgency and importance for them. For more

detailed guidelines (e.g. using open questions, driving questions from general to specific), you may

search for online resources or books about Market Research.

10. Having collected the results of the tests, an analysis on the data should be done, in order to generate

raw data analysis or look either for possible correlations among problems or even demographics and

problems. If the results are conflicted, the interviews should continue. If the results are contradicting

to the initial hypotheses the team should think even to drop the idea and start over, or look for new

customer segments to serve and start the interviews over.

11. When the team considers that the interviews have been concluded, and the report is ready, a

meeting should be called to discuss all together the outcomes. Based on the selected business model

and the interview report, the team should refine any parts of the business model. In particular, by

the time the meeting ends the team should have stated clearly what business the idea is into.

Generally, software industry may be summarized in 7 main business models: User generated content

where customers create and consume content and business makes money on ads, Gaming where

customers pay upfront or in-app for gaming entertainment, E-commerce when the business sells

goods online (or over mobile apps), Content production where the business produces high-quality

content and gets paid directly (subscription) or indirectly (ads) for that, Utilities/productivity when

native software or online services are provided to customers to help them solve their problems or be

more productive, and Two-sided platforms when the business allows two different stakeholders to

exchange goods for a fee.

12. Having decided what exactly is the business model of the idea, and having interviewed an initial set

of customers, the team should be able to evaluate the developed offering or features based on that

feedback. The evaluation may be done on the criteria defined in D2.1, but as long as the problem

only has been tested a subset will be used in this step and the rest will be used in step 2. The

selected criteria are: Clarity of the idea as it is explained to the customers, Value that the idea adds

to them or their lives, Need Level for expressing the importance or the relevance of the idea for the

customers, Urgency for describing how fast they say they want the particular offering. It should be

pointed out that every business model should not care equally for every criterion;

13. shows our recommendation about how each business model should measure some of the criteria.

In case the software team is in a feature development cycle, the business model should be taken into

consideration to see if the new added features improve the existing offering significantly for the existing

customers, or for the new targeted segment.

14. Based on that work, the team should update both the pitch and the report, to include results from

the interviews and relative metrics, as well as state clearly the business model that the product will

use. Then the team should present the results to the addressing management team to decide if it

continues to the next step, the user experience and the scenario validation, where the solution


23

should be tested in a high-level, or if the team should fold and use the customer feedback to start

over with a new idea.

Table 5-1. The criteria based on which an idea should be evaluated (i.e. first column), related to the business

where the ideas lay, interrelated based on the level of importance. N/A=important, *=very important,

**=extremely important

User-

generated

Content

Gaming E-commerce Content

production

Utilities/productivity Two-sided

platforms

Clarity * ** ** ** ** **

Value - - ** - ** *

Need

Level

** * ** - ** **

Urgency - ** * - * *

The described approach is “business first, validation with customers after”, or in other words the ideation

starts with internal incentives and in the business context where the business operates, and then the

user/customer’s context is used to validate the idea. That flow is more convenient for a software

organization that wants to standardise its operations, as the ideation (i.e. especially epiphany) includes

already some usage context. It is expected that after multiple cycles of ideation and idea validation, the

business (or organizational) and usage goals have been aligned. Nevertheless, if it is convenient for someone,

the usage context may be part of the ideation process, and the following steps change accordingly.

There are many methodologies for stating and validating a business idea in early stages. The Customer

Development methodology as described in the book “Four Steps to the Epiphany” by Steven Blank, “Running

Lean” by Ash Maurya which has been recommended already, are two of the many books that can be found in

that area of interest.

Step 2: User Experience & Scenarios Validation The second step in the CloudTeams methodology is about User Experience and Scenarios Validation.

In that step, you may want to go with fast and tested prototyping methodologies, such as the “Sprint”

framework introduced by Google Ventures, or you may want a less time-pressuring procedure as you cannot

allocate so many resources for intensive weekly iterations. In every case, you should both read the “Sprint7”

book to learn some UX/UI design and brainstorming techniques, but also read the following steps. Then you

may decide to follow the more convenient approach or design another procedure at your will. Don’t forget

that innovating is about building new procedures and take nothing for granted… But if you are inexperienced,

stick to a plan and follow it until you realise that it is inconvenient for you. 7 Sprint: How to Solve Big Problems and Test New Ideas in Just Five Days http://www.thesprintbook.com/

http://www.thesprintbook.com/


24

The CloudTeams Playbook suggests two sub-steps in that phase, step 2a where team collaborates internally

to build a meaningful scenario and even prototype for the customers, and 2b where the customers

participate to evaluate those scenarios. Those steps are explained in detail below.

Figure 5-4. The user experience and scenarios validation as visualized in the CloudTeams methodology

Figure 5-5. Step 2 - User Experience & Scenarios Validation workflow


25

Step 2a – Design User Experience (see Figure 5-5 left side):

This step includes internal collaboration for the team, after the idea has been finalized, in order to create

material that can be used both to get early feedback from customers and to drive the further technical

development of the idea.

1. Having ended up with a single business idea and the relative personas that are involved in it, the

software team should start collaboration sessions to transform these business documents into

concrete scenarios. These scenarios should describe how the business idea will add value to each one

of the selected personas, in a complete story that covers mainly the business value of the offering

and the business context where the software team wants to move. The software team may use

CloudTeams to store these scenarios and facilitate collaboration, enable commenting and voting on

these scenarios, and end up with one complete scenario for every persona (i.e. the scenario may

cover a full day or even some interrelated days of the life for a persona).

2. Based on the nature of the offering and the business model, the maturity of the development (e.g. a

team is in a second or third iteration of the product, to design next features based on customers’

feedback) and the technical skills of the team, the team should choose among three alternatives in

order to move forward and make the scenarios usable for early feedback:

a. Wireframes are easy to be produced and necessary as a visualization of the software. There

are many wireframe tools in the market that someone can use, free or paid services, like

Balsamiq, Proto.io or Ninja Mock. Even someone with no technical skills may use a series of

similar solutions to get ideas and understand the UI and UX formalities, in order to produce

a proper initial wireframe. Either way, the product manager or the head of product should

drive that process and this is the first step towards implementation. The output may be pure

images or linked html files, depending on the used software.

b. Videos are useful to engage customers, by putting emphasis on the scenario and the

problem that the software is about to solve, rather than the solution itself; as long as the

offer is not yet finalized, it is not recommended to use a real product in first plan, unless it is

easy and useful to mock some functionalities and hide them through visual effects or proper

video editing. Videos are sure more engaging than some text, may reach more users easily,

but they can test mainly the problems of the customers rather than the offering itself.

c. Mockups. There are multiple definitions for mockups, there are people that perceive

wireframes as mockups and others that describe mockups as dummy, not complete

applications. We use the second term here, and we perceive as mockups screens in the

application (i.e. web, mobile, desktop etc.) that have no logic behind, they are not

connected with databased and show some static information to give the user the perception

of what will be produced. Templates, especially in web application, may decrease very much

the time and the cost of mockups development.


26

A team may develop all three alternatives if necessary. Not having the proper budget to develop mockups or

videos should not deter the team from testing the idea before moving to the implementation; wireframes are

cheap and easy to be produced.

3. At the end of the “Design User Experience” phase, the team should come together to evaluate the

quality of the wireframes, videos or mockups, give some early feedback, and decide if it is ok to

move forward and start the validation phase with the customers. Low quality content may defer or

annoy some customers, based on the industry where the team operates in, so the team should be

cautious and rerun the process if any serious flaws are identified.

Step 2b – Scenarios Validation (see Figure 5-5 right side):

4. Having prepared the material to interact with the customers, the team should meet together to

discuss alternatives on the available options for testing the initial hypotheses. A validation planning

meeting should allow the team to decide based on its expertise and the iteration that the product is

on, which one or ones of the following options will be used:

a. Running Solution Interviews is always the simplest option to validate the scenarios. The team

should present the scenarios (i.e. wireframes, videos, mockups) to a subgroup of customers

that have shown interest during the first round of interviews, and get qualitative feedback

on what they think about the recommended solution. Also asking for pricing is a vital part of

this step, as the price is a major component of the offering to the customers. This step may

look time wasting for some teams, but should never be neglected especially in case that the

changes are quite vast compared to the previous releases. This technique validates mainly

the need coverage and in some cases the innovation intensity of the suggested solution, at

least in a qualitative way (see next steps for statistically-correct techniques).

b. Testing Ad Campaign with a Landing page is an option for quantitative feedback, for

offerings in their initial iterations, to test the concept and not the solution (e.g. not the

performance of a new algorithm). The goal of this technique is to test the problem

hypothesis, and use the scenarios developed to build a landing website to describe a

solution, including some visual content if available. Then this web address is used to build an

advertising campaign for Search Ads, in selected markets, and see how web users search

under specific term and how they behave when landing on the page. Action calls -like email

subscription- may be used, but also mock-up functionalities may be added -like a “buy now”

button that leads to a “coming soon” page- in order to collect more information on the

users’ behaviour through the analytics. This test validates mainly the need coverage of the

idea (see next step).

c. Crowdsourcing with a Video applies in cases the team needs early buzz for its project, either

to build a pre-subscribed community or get funds to start building the offering. The

scenarios are described in video(s), a demo of the usage flow or the problem that the

product wants to solve, and are pushed to the selected channel, e.g. YouTube or Kickstarter.


27

Metrics are based on views and subscriptions, and this type of tests is validating mainly the

virality of the concept.

d. A/B Testing with Landing Pages (mainly) applies on early concepts or major changes. As long

as this step does not include the development of the prototype, as at least two iterations

should have been concluded to have to versions to split and test A/B Testing, this A/B

Testing technique tries to test the preference of user between two different versions. A/B

Testing uses two groups of validation customers, the first one gets the initial solution and

the latter one gets the new offering. This technique is very similar with the “Testing with Ad

Campaigns”, but it tries to test the uniqueness/innovation of the hypothesis compared with

the existing offering (see next step). This technique is difficult to be used in that phase, as

explained already because of the lack of prototypes, it is mainly a “User Acceptance Testing”

step; nevertheless a team may decide to experiment even on a scenario basis.

5. When the team considers that the scenario validation process has been concluded, a meeting should

be called to discuss all together the outcomes. Based on the initial (see step 1) selected business

model and the interview report, the team should refine any parts of the business model, and then

re-evaluate the idea based on any qualitative and quantitative feedback collected.

Table 5-2. The criteria based on which an idea should be evaluated (rows), related to the business where the

ideas lay, interrelated based on the level of importance (columns). N/I= not important (low), *= important

(medium), **=very important (high)

User

generated

Content

Gaming E-commerce Content

production

Utilities/

productivity

Two-

sided

platforms

Need Level * * ** N/I ** **

Urgency N/I N/I * N/I ** *

Innovation/Uniqueness * * N/I ** * *

Virality ** ** * ** * **

Every type of test covers a different criterion as already discussed, so in general the team should evaluate the

idea based on the following 4 criteria: (a) Need Level for describing the degree that the product/service

covers their needs; (b) Innovation/Uniqueness as perceived by the customers or found in the market analysis,

and (c) Virality based on the customers’ tendency to share the idea and talk about it openly to others. Those

criteria have different importance for different industries, based on the forces that drive the decisions of the

customers; see Table 5-2 for an empirical weighting algorithm in order to understand the importance of each

criterion far different software industries where you may apply.


28

6. The team should be able now to decide whether the idea is worth building or not; if it is not clear,

the validation process should continue, or alternative techniques should be explored to evaluate the

scenarios (see step 2b.4). A new brainstorming meeting should let the team decide on that. If the

team decides to move forward with the implementation, a positioning strategy should be developed.

According to the three general strategies of Porter, the team may choose among the following

options, based on what feedback has collected and where it wants to go (Table 5-3):

Table 5-3. Relation of strategies and criteria’s score (at least reaching the suggested score)

Low Cost High Cost

Broad Market

Cost-Leadership

Low Urgency

Medium Need

Medium Virality

Low Uniqueness/innovation

Differentiation

Medium Urgency

High Need

High Virality

High Uniqueness/innovation

Narrow Market

Niche

High Urgency (for a group)

High Need Level (for the group)

High Virality (for the group)

High Uniqueness/innovation (for the group) Medium to Low ALL the criteria for the rest of the population!

a. Cost-Leadership. If the team has realized that the offering covers an existing need, without

an urgency as there are already other solutions and differentiation on the market is hard to

be perceived, it should choose to bet on lower costs. This strategy will try to re-segment the

market based on costs, thus any unnecessary features that increases the price should be

removed, and the culture and operations of the team should focus in reducing expenses.

Relatively to the criteria, “low urgency” implies that customers have already other solutions,

and a lower price is something that may convince them to change existing solutions for a

new one. But expecting that a lower pricing will convince customers to shift to another

offering is risky, thus the new offering should be identified at least as a “medium need” for

customers; combined properly with at least a “medium virality” may generate the critical

mass for the idea to spread and establish in the market as a competitive offering, lower

priced solution compared with existing ones. In that case, you may expect a “low

innovation/innovation” for the idea, even if you have to innovate on the operations and the

company structure in order to offer such a competitive offering; the customers will focus on

competitive pricing and so must do the organisation too.

b. Differentiation. If the team has realized that there is a unique, innovative characteristic in

their offering for an existing, highly competitive market, it should choose to go with a

differentiation strategy. In that case, the team may work to re-segment the market that will

allow it to take a larger portion of the market share with a new value proposition, by

competing on unique features and performance issues. The team should recognize its


29

strongest, unique advantages and competencies in order to choose such a strategy, and

work to amplify them in a meaningful way for its customers.

From the customers’ feedback, the team should have identified at least “medium urgency”

for the new offering to go with this strategy, as customers may have other solutions already,

but would welcome the new idea. “High need” throughout the sample implies that the idea

may reach a large market, which needs “high virality” to spread more easily across the

market. Finally, as we talk about a differentiation strategy, “high uniqueness/innovation”

must drive customers’ perception for the idea.

c. Niche. In most cases a new idea from a small team, like a startup, will have to follow a Niche

strategy; a special offering – either low-cost or differentiated – for a narrow market. This

does not mean that the team has no opportunities to grow, the “disruptive innovation8”

framework describes that case, an innovation to change progressively the whole market; Or

alternatively, there are companies that have grown and been sustainable by being Niche

(e.g. Ferrari, Gucci, even Apple with iPhone etc.). In the case the team should focus on

creating a new market or cater a small part of an existing one, low priority should be put on

traditional features and the team should focus on market adoption (e.g. training, customer

support, customization etc.).

To realize that the team has an opportunity to go with a Niche strategy, all of the criteria

should be “Medium to Low” for the majority of the sample, but in the meanwhile a special

group has been identified that shows high correlation, and shows “High” values in urgency,

need level, virality and innovation. This group is expected to uncover a market segment of

“early adopters9”, who have a problem, have realised it and try to find a solution of them,

while they may have tried multiple other solutions that do not cover their needs. This

people is possibly organized in groups where they exchange opinions and try to help each

other (e.g. Facebook groups or forums), and they will possible talk with each other for the

“new solution in the town”. Nevertheless, before assuming that this group is the perfect for

you, you should do a sizing of your market, with the price you expect this group will pay for

your offering, and the expected number of customers you expect to have; your business

must be viable, at least before another strategy is followed when external funding resources

may be required as well.

There are multiple other strategies available out there, but they can be mapped easily on the three generic

strategies of Porter. For example, the “Blue Ocean” strategy where some characteristics are removed from

an offering, some are decreased and innovation is introduced in others, is a subcategory of the

“differentiation” strategy. You should realise clearly where your idea is positioned, and stick to the selected

strategy you are going to follow.

8 https://en.wikipedia.org/wiki/Disruptive_innovation 9 https://en.wikipedia.org/wiki/Early_adopter

https://en.wikipedia.org/wiki/Disruptive_innovation

https://en.wikipedia.org/wiki/Early_adopter


30

7. The strategy where the next decisions will be based has been decided, thus the team should update

both the pitch and the report with the outcomes from the market research and the contact with

customers, and get the final management approval to start developing the offering. The team is

advised to make it clear to the management team that the results are mainly qualitative10 and

cannot be easily generalized in the first iterations, but valid quantitative feedback may be collected

only after an initial launch of the offering to the market. And remember, at this stage you must tell a

story for your customers, rather than emphasize on data11.

If rejected (e.g. brand incompatibility), the team should restart on the same idea but different

approach rebooting step 2, or should look for a new idea rebooting from step 1.

8. If the management team approves the solution, the team is ready to go with the implementation,

where a prioritization should be done in smaller cycles. But, before going into the implementation,

the project management teams should work on the User Acceptance Testing (UAT) plan! In other

words, the team should know what will be measured, how and when each iteration will be available,

to validate that what has been produced is what the customers need. The requirements should

remain in a business level, from the usage and problem-solution perspective of the customer, and no

technical details should be included (i.e. this is technical verification and is scheduled in the

development phase later).

For the User Acceptance Testing Plan, which will be used in Step 9 (Automated Acceptance Testing)

there are various frameworks that the team may use to develop it. The team may decide to adopt

one of the recommended, use an existing one or develop a customized plan based on its needs;

based on the selected framework, the team has to setup quantitative criteria and questionnaires to

extract qualitative feedback and evaluate the developed solution. There are various models that may

be used, like Theory of Planned Behaviour (Ajzen, 1985,1991), Decomposed Theory of Planned

Behaviour (Taylor, S., and Todd, P.A., 1995), Decomposed Theory of Planned Behaviour (Taylor,

S., and Todd, P.A., 1995), Task-Technology fit model (Dishaw, Strong and Bandy, 2002), and Unified

Theory of Acceptance and Use of Technology - UTAUT (Venkatesh, V. et al. , 2003). TAM2 is a good

general model for evaluation of technology, ISO/IEC 25010 is ideal specifically for software, while

HMSAM is appropriate for entertainment products (e.g. video games, music, movies etc.). UEQ12 is a

handful UAT tool, with open and available tools to do the analysis, and it is also supported as a

template in CloudTeams, when you are about to create a new questionnaire.

Table 5-4. Well-known User Acceptance Testing models

UAT Model Name Description

10 http://ethnographymatters.net/blog/2013/05/13/big-data-needs-thick-data/ 11 https://hbr.org/2015/10/the-best-data-scientists-know-how-to-tell-stories 12 http://www.ueq-online.org/

http://ethnographymatters.net/blog/2013/05/13/big-data-needs-thick-data/

https://hbr.org/2015/10/the-best-data-scientists-know-how-to-tell-stories

http://www.ueq-online.org/


31

TAM2

(Venkatesh, V. and

Davis, F.D, 2000)

TAM (Davis, F. D. , 989) is on of the most popular models for evaluating technology

acceptance. The team should extract specific criteria from the following categories, and

end up on a result based on the correlation among the criteria, as TAM describes:

Perceived Ease of Use, Perceived Usefulness, Attitude Toward Using, Behavioural

Intention to Use, Actual System Use.

TAM2 adds the context of the user that evaluates perceived usefulness, based on:

Subjective Norm, Image, Job Relevance, Output Quality, Result Demosntrability

Additionally, evaluate Intention of Use with two more criteria: the experience, and the

voluntariness of the subject

TAM3 is an improvement for e-commerce solutions

ISO/IEC 25010:201113 This is an ISO standard that tries to ensure the Quality of the Software.

The main parameters of the Software Product Quality are: Functional Suitability,

Performance Efficiency, Compatibility, Usability, Reliability, Security, Maintainability,

Portability.

The dimensions of Software Quality in Use14 are: Effectiveness, Efficiency, Satisfaction,

Safety, Usability

Hedonic-motivation

system adoption

model (HMSAM)

HMSAM focuses on more motivational aspects, and is more appropriate for projects

like gaming, e-shops, virtual worlds, e-learning, social networking, dating, music and

generally projects with gamification factors: Perceived Usefulness, Curiosity, Joy,

Control, Behavioural intention to use, Immersion.

UEQ (User Experience

Questionnaire)

The scales of the questionnaire cover a comprehensive impression of user experience,

i.e. measure both classical usability aspects (efficiency, perspicuity, dependability) and

user experience aspects (originality, stimulation).

Step 3: System Backlog Definition & User Stories Validation “The Devil is in the details”, they say, and having recognized a valid need for your customers doesn’t

guarantee your success. The team has to break down the customer need and the idea into requirements,

schedule the roadmap of implementation and also include the innovative characteristics that the team can

offer, in order to differentiate with the competition and current solutions (e.g. lower costs or new features).

This is commonly said the “requirements elicitation and documentation phase”, and this is typically the step

where the customer and business knowledge starts fading to become thinner; development teams many

times tend to forget the context where the solution was perceived and validated.

There are multiple techniques to approach the challenge of breaking customers’ needs into system

requirements. You can hear about multiple other documentation techniques for requirements, like UML

diagrams, class diagrams etc., which are closer to the implementation. Nevertheless, the theories of agile

software development work to identify the minimum amount of documentation that can help the developers

to develop the proper solution, without losing the usage context and without guiding them to the execution

13 http://iso25000.com/index.php/en/iso-25000-standards/iso-25010 14 http://ryreitsma.blogspot.gr/2011/07/software-has-new-quality-model-iso.html


32

level (e.g. we need a button that will do this, and will go there); thus, traditional software engineering tools

have started becoming obsolete, at least during the production phase (e.g. someone may choose to use them

for documentation purposes of the system).Of course, letting the development team improvise means trust,

free information flow internally, processes that allow fast experimentations and improvements, as well as

high skills and experience among team members.

Figure 5-6. The System Backlog Definition and User Stories Validation steps, as visualized in the CloudTeams

methodology

In CloudTeams we suggest following a customer-driven approach, and using mainly four tools with strong

design thinking characteristics in order to address those challenges: (a) user personas, (b) usage scenarios, (c)

user stories and (d) prototypes. For that reason, the CloudTeams methodology has described an approach to

minimize the destroyed knowledge and increase team’s freedom during that phase.

Step 3a – System Backlog definition (see Figure 5-7, left side):

Until this step, the team should have gone through multiple iterations, internal meetings and customer

validation interactions. The outcomes should be a validated idea, a set of personas that represents its market

segments, a business model that it is expected to work (i.e. not validated before launching and initiating

sales), the scenarios where those personas live and which have been presented with some media material

(e.g. videos or storyboards).

Having defined the selected “User Personas” and the context where they are located, the next step is to

develop a full story about how a customer segment represented by each persona will engage with the

offering. The outcome of a first round of collaboration between the business and the technical teams (or

members of the team) should be one “Usage Scenario” per user persona; this scenario should start with the

“point of contact” of the product or service with the persona, or in other word how the interaction of the

customer with the offering starts (e.g. a Google/Facebook Ad or after searching the App Store). It should

cover basic workflows of interaction that “match” with her hypothetical personality, for a teenager is

supposed to focus less on tech and more in movements, or an anti-social media user will not be able to login

with a Facebook account. Exception workflows should spread among the scenarios, as well as descriptions of


33

their need addressed and any virality flow (i.e. letting other customers learn about their good or bad

experience). Usage scenarios should be detailed, not feeling unnatural to the reader, and definitely are not

material of marketing; e.g. the video used to get feedback from the customers in early steps hides

uncomforted events or typical features that a user may perceives as must-haves. Proper documentation,

collaboration, versioning and sharing of those usage scenarios should be enabled, to let any member have

access on them through the implementation phase; CloudTeams offers document management and

collaboration features, and you may use a CloudTeams project to manage your time at this step.

Figure 5-7. System backlog definition and User Stories Validation

Don’t forget the importance of including members of the technical team in that process, to put the business

ideation into a reality frame, identifying the feasibility and the complexity of possible solutions; e.g. someone

implied for an application on iPhone to get access to the messages or the notifications of the system, which is

not allowed by iOS. Nevertheless, the technical feedback should not let the team look for solutions at this

stage of the product development, only to filter out extreme or dummy requests; obstacles identified by the

technical team may be opportunities for further innovation. The main reason to include people of different

background is to get more incentives on design phase, and not let the context of the problem get lost into the

implementation phased, under boring texts overwhelmed with unnecessary technical requirements, missing

the proper usage context.


34

Having defined the Usage Scenarios in the project, the team deals with the greatest challenge of step 3:

breaking the usage scenarios into meaningful “User Stories” that will bring the team into action, without

destroying the context where the previous steps have been developed. In order to overcome such a

challenge, the most appropriate theory is “Jobs to be Done” or the “Jobs Theory”15, which puts the needs of

the customer in the centre of the solution design; the customer will expect to accomplish specific goals and

sub-goals to meet its motivation behind interacting with the offering16.

The user stories are in the format “As a <type of user>, I want <some goal> so that <some reason>”, while

typically they are written into cards to limit the size of each story, where Comments, Priority and Estimated

Delivery Time are indicated in many case on the back of each card17. What many teams miss is the “social”

and “emotional” context behind a need of the customer in many cases18. In that case, the “Value Proposition

Canvas” is ideal to identify the underlying context and include it in the User Stories.

Figure 5-8. Hierarchy in User Stories, relation with the goal of each level, and relative examples

What many teams face in many cases, at this moment, is having an implied hierarchy on user stories. For the

case of user “Uber” for example, a Job-to-Be-Done (JTBD) for a customer would describe, “I want to move

from place A to place B fast and easy”. A high-level User Story would describe “As an asset manager, I want

to move from home to work, fast, easy and inexpensive, so that I go to work every morning”; it gives the

context of the user (asset manager), it gives a specific root that has been identified as the most common

among users, and it also implies requirements for not moving her out of her routine (fast and easy) and not

putting the price tag higher than a taxi (inexpensive). Another case could be “As a college student, I want to

return home safe after drinking nights out, so that I don’t have to drive and lose my license”; in that case,

15 https://hbr.org/ideacast/2016/12/the-jobs-to-be-done-theory-of-innovation 16 Putting the need of the user in the center of the team interest is more appropriate for solutions that are driven by functional needs, rather than emotional ones. For example, gaming and entertainment may applies different rules while defining the user stories. 17 http://www.agilemodeling.com/artifacts/userStory.htm 18 http://innovatorstoolkit.com/content/technique-1-jobs-be-done

https://hbr.org/ideacast/2016/12/the-jobs-to-be-done-theory-of-innovation

http://www.agilemodeling.com/artifacts/userStory.htm

http://innovatorstoolkit.com/content/technique-1-jobs-be-done


35

another usage context has been given (e.g. night, legal and practical limitations), another customer segment

and possibly other price elasticity may be implied. Both User Stories comply with the JTBD described initially;

nevertheless they apply on another persona (i.e. customer segment) and apply on different context. In the

initial case, the JTBD, more space has been given to the manager to innovate with a new service, while a

high-level User Story has focused on specific market segments. But still, it is difficult for the technical team to

start implementing those User Stories without any specific storyboard, a Workflow of User Stories that have

to be implemented, e.g. “As a user, I want to call the vehicle on the place I am, so that I don’t have to walk

around and find a vehicle” or “As a user, I want to split the fair with another passenger, so that we don’t have

to exchange cash after the tip”. Thus, case-specific User Stories must be developed, to cover all the different

levels of high-level User Stories, and completing the implied workflow. On the same time, the JTBD still

should leave freedom to the implementation team to come up with ideas that will make the solution

modular, scalable, and may capture new markets with new features; innovation is a continuous and non-

linear process that cannot be fully controlled by management, theories and frameworks. See Figure 5-8 for a

visual representation of the hierarchy.

Having defined the user stories, it is time to decide what is going to be implemented and delivered to the

customers for initial validation of the market. A voting mechanism is recommended, with the criteria you

have defined. In CloudTeams we have used two criteria “Value” and “Urgency”, with a value from 1 (Very

Low) to 5 (Very High); then we multiply the two scores and based on the median value (i.e. we prefer the

median value because there are many voters who vote for their ideas preferable by giving low grades to

other stories, but you may choose to go with the average value). Then, as seen in Figure 5-9, the stories are

grouped in buckets of score for 25 points (i.e. critical), 15-20 points (i.e. important), 6-12 points (i.e.

moderately important), and 1-5 points (i.e. nice to have). If you use your own criteria, you should define your

own grouping for user stories; many teams want to define a criterion for “effort”, which is meaningful if

resources are limited, nevertheless this step is not a matter of technical analysis and could limit the team to

think of a cheap and nice workaround to mock the described functionality in the initial phase.

Figure 5-9. Putting the user stories in priority

Typically, after the voting process, you will see that all the 25-point user stories make no sense if they are

selected for initial launch; people many times vote for features, not for complete workflows. In that moment,

the team has to call another collaborative meeting (3rd), to put the selected user stories in the narration and

see what is missing. We call the outcome “Usage Workflows”, (see Figure 5-10) and they are nothing more

that selected user stories that drive different stakeholders (i.e. types of users) throughout the version that


36

will be released. The team should start with the 25-point user stories, try to see if there is a meaningful flow

that meets the JTBD and the high-level user stories (i.e. for which persona), and then try to fill in any gaps

with user stories of lower priority, starting from higher to lower scores. To validate that the usage workflows

are complete, another meeting may be called (4th), where the presence of the technical team will give an

estimation of the technical characteristics (abstract) ones that may be delivered based on current capabilities

and resources, but also where the decider of the project (e.g. the project manager) would like to make any

modification to the workflow to meet any business model or strategy goal. The outcomes of those meetings

will be the selected workflow to be implemented, the prioritized backlog of user stories, as well as the

technical estimations of the team (e.g. “we can export the report after 10 minutes at least”, or “we can

generate only low-resolution images on such hardware”).

Figure 5-10. An example of a workflow that completes an interaction (i.e. a user map)

Step 3b – User Stories Validation (see Figure 5-7, right side):

The updated Usage Flows should lead the team to fast prototyping, thus creating mockups of the offering

that may be tested and validated by the users and customers. The technical limitations defined by the

technical team, may lead to mocked limitations in the mockup, or to questionnaires that will try to identify

the intentions of usage and the importance for the users, despite those limitations (i.e. if they are blocking

issues).


37

Figure 5-11. CloudTeams Platform allows the documentation of both Scenarios and User Stories

Then, the interviews with the customers start, and the analysis of the results then. It may seem challenging to

build fast prototypes and test different perceptions of the team. “Sprint framework19” describes how a

prototype can be generated in one day (i.e. if everything else is ready – the storyboard and the problem

definition), and may be tested in another day with 5 interviews with users. You may refer to that book to see

how tasks among team may be allocated to complete the challenge of fast prototyping, how qualitative the

solution should be (hint: the user should perceive the solution as much ready as possible), and how the

interviews may be scheduled and executed. The team should be confident, fast and effective. In each testing,

the prototype may be improved, but the goal is to understand the customer, so every unnecessary effort that

does not help the team on learning should be excluded.

Table 5-5. An example of documenting a Use Case

Use case Id: CODE.#

Name: Building personas from data coming from actual users

Path: Power User Path:

1. A Power User creates an account of CloudTeams Customer Platform

2. The Power User provides personal information asked by the provided form

3. (optional)The Power User synchronizes social media accounts for the completion of persona

Project Manager Path:

1. Project Manager (PM) logins to Team Platform, selects Orbi Project

2. PM access the Persona Builder though the Team Platform

19 http://www.thesprintbook.com/

http://www.thesprintbook.com/


38

3. PM selects to create a new persona that reflects the desired age, tech level and device used

4. Existing users are mapped in an anonymized way to this persona and it can be now used on the project

Related test cases/user

stories: C.01.R1, ST.12.R2

Involved Roles: Power User, Project Manager

Pre-conditions: The project has been created in CloudTeams already

Post-conditions: At least one Persona has been created for the project

Having experimented on the prototype, the team is ready to decide what is going to implement and deliver

to the users. Of course, if the prototypes have identified issues, the team may decide to discontinue (even if

launching will help learning) and go back to step 2 or 1 if there is negative feedback.

The outcomes of step 3b should be a finalized backlog with the relative workflows, the functional and non-

functional requirements identified through the technical meetings and the customer observation, and a

testing plan. The described outcome could be called an MVP (Minimum Viable Product), and should include

the basic pricing characteristics, while being full functional for the goals set by the team; if it is considered a

prototype that improves learning, the team should focus on that period or relaunch step 3. The goal of the

team after entering the implementation phase (step 4 and further) is to test the market.

CloudTeams Platform supports documentation of requirements, and in detail team members can create

Scenarios related to a project and a persona, and assign different User Stories to them (see Figure 5-11). They

are organized per project, and it makes the exchange with team members easier.

The suggested tool for building the testing plan is the production of “Use Cases” (see an example in Table

5-5). In detail, by looking at the usage workflows, smaller workflows should be identified that include a

certain number of user stories. These Use Cases are documented, indexed, named and released as a

document that informs the tester on what scenarios should be implemented. These Use Cases should cover

the expected flows that are covered from the MVP definition (i.e. not user stories or workflows that are not

planned for implementation), and overlapping may be present, as different paths may identify different

issues during testing.

The Testing Plan may be developed asynchronously and is not a prerequisite to move to the next step, as

long as it is ready by the time components start being delivered. Now the team is ready to move to the next

step, the implementation phase of planning and development.

Step 4: Sprint Backlog Definition In that phase, the actual implementation and production of the offering kicks off. If a team has its own

methodology already, it is considered agile and does not want to change, it may consider Steps 4 (Sprint

Backlog Definition) to 9 (Automated Acceptance Testing) as a black box for the CloudTeams methodology, try


39

to apply the User Stories defined in previous steps in their methodology of software development and

testing, and go directly to Step 10 (Regression and Functional Testing).

Figure 5-12. The position of Step 4 (Sprint Backlog Definition) in the implementation phase

For the development of the CloudTeams methodology, we have selected Scrum as the preferred

methodology to manage the production phase. The main reason was the popularity of this method among

the teams we interviewed, but mainly the presence of concepts and tools that match perfectly with the User

Stories as means of driving software engineering, without messing with complex requirements.

Now is the time for the technical team to take the lead. The team takes the updated backlog of user stories,

and all the relative requirements (functional, non-functional). It also goes through the personas, the JTBD and

the high-level user stories, as well as any media material, the prototypes and answers during the validation;

the technical team should present to all its members the relative content, in a half-day meeting, and then the

MVP backlog should be presented.

The team should be aware of the customers’ and business’ priorities, but it may choose to implement the

user stories in a different order, to facilitate development, manage its resources, or put heavy tasks in the

right implementation slot. For that reason, every user story should be evaluated in a matter of “Cost of

development”. Provided that the team uses the Scrum methodology (see the relevant reference20), the team

should consider breaking user stories down into simpler workflows (i.e. creating another, lower lever of

hierarchy), to allow the team development phase fit into the development cycle of the team (max 4 weeks).

The suggested tool to manage and monitor through development phase, under the Scrum methodology, is

the Kanban board21. The Kanban board is a visual tool that allows the team to monitor and perceive instantly

20 https://calvinx.com/2014/05/22/why-scrum-why-agile-development/

21 https://ketiljensen.wordpress.com/2009/10/31/kanban-the-next-step-in-the-agile-evolution/

https://calvinx.com/2014/05/22/why-scrum-why-agile-development/

https://ketiljensen.wordpress.com/2009/10/31/kanban-the-next-step-in-the-agile-evolution/


40

the development phase, includes the Backlog of the User Stories, as well as different steps of testing,

verification and bug fixing. You may feel also free to modify the tool, print it and mount it on a wall where it is

visible to all team members. The board implements the main concept of Scrum that only three (3) User

Stories are allowed to be developed, tested and fixed each time, until a new User Story enters the

production phase (i.e. if a User Story is completed or dropped).

There are multiple other collaborative tools that teams use to manage the production, assign tasks and

improve transparency in the development phase. Trello, Pivotal Tracker, and GitHub (even for managing

releases) are the most popular tools you may try. CloudTeams provides the capability to create and connect

GitHub issues with customers’ feedback and connect to Trello (see Figure 5-13), to monitor your team’s and

product’s progress from one central point.

Figure 5-13. Connecting external services to a CloudTeams project

Step 5: Design and Visual Modelling CloudTeams methodology does not intend to change the way software is modelled throughout the years,

nevertheless it focuses on not losing the context of usage and emphasizing on the customer for decisions. On

this step, the team is called to design both the User Experience (UX) in the front-end of the solution and the

back-end elements of the system.

On the UX-side, things seem more standardized in a matter of tools and techniques, nevertheless the quality

of the outcome depends on the diverse background of the team members and the proper usage of design in

the process; The knowledge of the customer is a key success factor for proper design.


41

On the back-end, an experienced engineering is extremely useful, acting as a system architect to design a

modular, API-driven architecture that will make the system easily convertible to another solution after a

possible business pivot, easily maintainable and scalable; modularity may even enable new business models

for an organization, under a bottom-up innovation process, like Amazon did with its AWS business that

emerged from its technological capacity.

Figure 5-14. The complete diagram of Step 4 “Design and Visual Modelling”

Based on the agreed MVP and the Personas, the team is ready to start designing what will be implemented.

Every team may choose the designs that better match its capabilities, experience and functionality

requirements, nevertheless the suggested categories and subcategories of designs that should be completed

are:

Interaction Design: This set of documents defines how users will interact with the system, not in a

matter of visual components but mainly relatively to the workflow that a user may complete in the

system. The suggested tools are the following:

o Storyboard: The storyboard is a visual representation of a scenario, describing how the user

will interact with the system. It may include some visual components of the system, but it

describes the whole expected experience, even before and after the interaction. The


42

Storyboard concept is vastly used in the Sprint idea (i.e. design thinking and fast

prototyping)22.

o UML: This is a typical software engineering tool, with a predefined visual language that

many systems may even convert into software. It depicts the main user roles and the

allowed interactions with the system. It is standardized, nevertheless for understanding

newer versions, initial training is required. Some information may also be missed, but in

combination with other visual tools (e.g. storyboard or scenarios) it gives a complete

perspective of the solution.

o Sitemap: This tool describes the navigation options of a user into a website. Of course, if a

mobile application or program is developed, it may be a “Screenmap”. The idea is not

focusing on the visual components (e.g. buttons that navigate from one screen to the other),

but see how the user may reach a required step, and if there are any dead-ends on the flow.

o Gamification: The gamification is about putting gaming elements to software, to improve

the user experience in a solution. There is the misperception that gamification is a marketing

tool, or a feature in a platform. Nevertheless, it is an important design decision to align the

business goals of the team with the user goals. E.g., when the user is rewarded with extra

storage in Dropbox for inviting friends, he increases the customer base of the platform but

on the same time it makes his life easier for collaborating with those users over Dropbox.

Thus, Gamification is a strong design decision, that should be created by the team and

should not be driven by requirements. Nevertheless, the proper moment to design such

elements, is a strategic decision as well, as initially the core offering may be tested, before

launching gamification elements.

Information Design: Going technically deeper, the team has to start designing how information will

be modelled into the system. Our recommendation is the following list of documents:

o Models: What main entities will be in the system, what properties will include, and what

interrelation will have (e.g. generalizations) with each other. Typically, an ER diagram, or

some class diagrams will be produced to model the information into the system.

o Database: Based on the type of the database used (i.e. relational or noSQL), different level

of design is needed. Typically, for a relational DB, a relational diagram will be produced.

o Architecture: How the system will break into different components, how they will be

connected, what technologies will be used. There are different levels of architecture, it is a

matter of the technical team in which detail they will be documented.

o Information Flow: How messages will be exchanged among different architectural

components, what specifications are needed, what standards are used and what is expected

to be the outcome of each communication (i.e. including the exception and error messages).

22 https://www.fastcodesign.com/1672917/the-8-steps-to-creating-a-great-storyboard

https://www.fastcodesign.com/1672917/the-8-steps-to-creating-a-great-storyboard


43

Industrial Design: In case that a solution may be accompanied by a device, or there is any

modification on existing device, an industrial design may be needed. Additionally, the brand itself

may affect the user experience, and some modifications may be needed (e.g. different logo layouts,

or different themes to improve UX).

o Hardware: It is not in the goal of this manual to describe how hardware should be

developed, and it may be even a different track of development; running a separate

CloudTeams methodology to develop new hardware, while changing software techniques

with electronics and hardware tools, may be an option that will not be analysed here.

Nevertheless, if a software solution is bound on a dedicated hardware solution (e.g.

smartphone, smartwatch), it should be clear to the team the limitations and the

specifications of the device. Even the device itself should be given to the team, with a set of

solutions that will help them go into the customer experience and design software that is

compliant with the device. For smartphone applications, there are strict design rules and

limitations, which if not applied Apple may even block the application on the testing period.

o Branding: In many cases the team must be aligned with the broader branding rules of the

organization, thus they should be respected in the design of software as well. If any

modifications (e.g. high-contrast logos) or even new branding rules are needed, this is the

moment to work on it; having the business and usage context in mind, it is easier for a

creative director or a designer to produce the proper product branding.

Service Design: In case the solution is delivered as a service, or is accompanied with services, it is an

important step to design properly the interfaces and the means of services to the customers:

o API: Many architectures follow a API-driven approach, where interfaces are built both for

internal and external usage. The team has to describe how the resources will be modelled,

what protocol will be used, and how the services will be exposed.

o AAA (Authorization, Authentication, Accounting): Typically, part of the API design is always

AAA, but as long there are various protocols that may be supported (e.g. oAuth 2.0), it is a

different, important aspect where the team needs to agree on.

o Customer Support: The feedback mechanism needs to be decided here, as well as any

support provided to the customers of the solution. This is an important business decision

that needs to be integrated in the flow. It is very different providing a 24/7 support, direct

the flow through emails, have a chat mechanism to resolve issues on high level, or provide a

FAQ section in the website.

Visual Design: This is a major driving force for the implementation, the visual interface for the user

will interact with. Some product methodologies “allow” even developing only a visual, draft version

of the product before coding, just to test hypotheses on actual users, and collect feedback. There are

different levels of detailed that may be implemented in that step:


44

o Wireframe: This is a draft design of the application UI, on hand or software, navigating the

user through different screens.

o Mockup: This is a front end disconnected from any backend, and simulating a complete user

experience. To achieve it, media content may be used to give a “real-life” experience, why

automated processes may be simulated by humans to test their effectiveness and costs. In

Lean implementations, teams may even skip backend design in the early steps, skip coding

steps, and go with a visual experiment as fast as possible.

CloudTeams does not intend to replicate functionality of well-established tools and engineering practices, on

that level. As a document management and collaboration tool, it allows every project to share files and

folders with the team members, managing access on different levels; in that way, any modelling

documentation may be shared across the team members. Nevertheless, modelling tools can and will be

integrated in the platform; for example, an UML modelling tool called “Snape” is integrated into the

CloudTeams platform, for easier access.

Figure 5-15. The “Snape” application helps software teams manage their UML diagrams

Step 6: Coding At this level of production, the role of the Product Manager should be limited in overviewing teams’ work,

run the daily and weekly meetings as a Scrum master (i.e. if this methodology is selected), and prepare the

product validation steps, as well the plan for launching the product. Nevertheless, it is important to agree

with the Technical Manager and the Software Developers on how the whole team is expected to work. In

small teams, the Product Manager may also be able to have technical skills, and should drive the team at this

level as well.


45

In the CloudTeams methodology we will not introduce ourselves as the masters of Software Engineering,

there are other teams that do it better and for years. There software tools that have emerged through years

of research and development. If someone is looking for best practices in the area, even Google has released a

relevant guide on how they manage software engineering with many custom solutions23.

Nevertheless, we do consider that a Product Manager should be aware of the main steps and concepts in

Software Engineering, at least to understand what the team does, even to recommend to the team any tools

that may help them work faster and more efficient. An overview of those tools, on Coding Authoring, are

available in Table 5-6. These tools can be used throughout the code authoring, testing and verification phase,

from steps 6 to 9, nevertheless there are documented in this step as these steps are many times unclear;

especially the debugging process is continuous and iterative for developers.

For all the nodes defined in a cycle of software development in Table 5-6, a plethora of solutions can be used.

In this section, we will try to discover common development solutions, their collaboration aspects, and also

to identify their role in CloudTeams. As these diagrams require collaboration of the Product Manager with

other technical team members, different diagrams are used to depict the workflows of collaboration.

Table 5-6. 7. The software development cycle in Software Development, Testing and Verification

23 https://arxiv.org/ftp/arxiv/papers/1702/1702.01715.pdf

Software Development Cycle

Integrated

Development

Environment (IDE)

An integrated development environment (IDE) or interactive development environment is a software

application that provides comprehensive facilities to computer programmers for software development.

An IDE normally consists of a source code editor, build automation tools and a debugger. Most modern

IDEs have intelligent code completion. There are also many collaborative IDEs introduced lately, to help

team collaboration on projects, as well as online IDEs; Code9 and SourceLair are some of them.

Code Repositories Code repository is a file archive and web hosting facility where large amounts of source code for

software, but also for web pages are kept, either publicly or privately. Code repositories are allowing

developers to effortlessly work in the same project by providing merging, branching and storing of code

functionalities. Also modern code repositories and version control systems provide more tools that

allow developers to compare and find bugs in a straightforward and easy to use way. GitHub, GitLab

and BitBucket are some of the most popular ones.

Build Automation

Tools

Build automation is the act of scripting or automating a wide variety of tasks that software developers

do in their day-to-day activities including things like compiling and packaging computer source code into

binary code or even running automated tests and deploying to production systems. Modern build

automation tools provide developers with a common build configuration that can work well for a team

that developers used different OS and have different configurations, while in the same time take care of

any dependencies.

Code Quality Review

Tools

Code review is systematic examination of computer source code. It is intended to find and fix mistakes

overlooked in the initial development phase, improving both the overall quality of software and the

developers' skills and help teams to create better quality code, that uses best practices and

documentation.

Deployment

Automation Tools

Continuous Integration is a software development practice where the members of a team frequently

integrate their work – usually each contributor integrates his software code at least daily, leading to

multiple integrations per day. Each integration cycle is verified by an automated build (including test) to


46

CloudTeams has not developed any tool from the list above, however the integration with existing popular

ones makes the life of the Product Manager easier and the team collaboration more intuitive; the team does

not feel the Product Manager isolated but a part of the development process as well. Figure 5-16 shows a

workflow on how Product Manager could create, import and view a code repository, and manage the

members of the repository by using the Software Team dashboard.

Figure 5-16. Suggestion on Product Manager engagement in Code Collaboration through CloudTeams

Step 7: Automated Unit Testing During the code authoring phase, developers are creating the appropriate classes and methods, while testers

have taken the responsibility of creating the unit tests that validate the proper functionality of the smallest

pieces of the code. On each code commit from the developers, automated build is running in order to assure

that each commit is not creating new issues, both on application build and unit testing. Popular libraries are

jUnit, xUnit and nUnit, depending on the used programming language. On the other hand, build automation

is the act of scripting or automating a wide variety of tasks that software developers do in their day-to-day

activities including things like: (i) compiling computer source code into binary code, (ii) packaging binary

detect integration errors as quickly as possible. Continue Integration tools allow the automated

deployment of new versions of a software product each time code is stored in a code repository. This

allows the continuous and direct finding of any issues that can be created from a code bug that is

particularly critical in teamwork environments.

Artifact Repositories An artifact repository is a solution that optimizes the download and storage of binary files used and

produced in software development. Artifact repositories allow the usage of third party libraries from all

team in a straightforward way. Also and most important, they allow collaboration by providing artifacts

developed by one part of a team by the rest of the team.

Bug Tracking Tools A bug tracking system is a software application that keeps track of reported software bugs in software

development projects. Bug Tracking tools allow teams to track and tackle any bugs and issues found in

collaboratively created software.


47

code, (iii) running tests, (iv) deployment to production systems and (v) creating documentation and/or

release notes.

The use of bug and issue tracking tools is very common for every developed software product, in every phase

of software development. As every bug fixing is addressed on the level of code authoring, on this steps it is

expected that the developer will run tests to address any bugs identified in higher verification levels, and

reported by other team members.

By the time this playbook is documented, CloudTeams does not support the integration of any unit testing

library, as it is difficult to integrate on the code-authoring level. Nevertheless, there is integration with some

of the most bug and issue tracking systems, which are also used as code repositories (i.e. GitHub, BitBucket).

Figure 5-17 shows the workflow on how a Product Manager could connect to a bug tracking Platform and

setup a bug tracking mechanism for the project by using the Software Team dashboard, while Software Team

members can use the Software Team dashboard to achieve basic bug tracking tasks.

Figure 5-17. Suggested workflow for Bug Tracking in Software Team Dashboard

Step 8: Continuous Integration Testing Continuous Integration is a software development practice where the members of a team frequently

integrate their work – usually each contributor integrates his software code at least daily, leading to multiple

integrations per day. Each integration cycle is verified by an automated build (including test) to detect

integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced

integration problems and allows a team to develop cohesive software more rapidly. The goal of Continuous

Integration is to constantly verify that all the components of a system under development can be integrated

and tested in combination; This is especially useful when code or models have changed or when new code

does not respect the interfaces provided by other components.


48

The greatest and widest ranging benefit of Continuous Integration is reduced risk and facilitates bug fixing. It

doesn't get rid of bugs, but it does make them dramatically easier to find and remove. In this respect, it's

rather like self-testing code. If a bug is introduced and detected it quickly, it's far easier to get rid of. It’s also

ease to diff debugging - comparing the current version of the system to an earlier one that didn't have the

bug. Bugs are also cumulative. The more bugs you have, the harder it is to remove each one. This is partly

because you get bug interactions, where failures show as the result of multiple faults - making each fault

harder to find. It's also psychological - people have less energy to find and get rid of bugs when there are

many of them - a phenomenon that the Pragmatic Programmers call the Broken Windows syndrome. As a

result projects with Continuous Integration tend to have dramatically less bugs, both in production and in

process.

However, the degree of this benefit is directly tied on how good the test suite is. It's not too difficult to build

a test suite that makes a noticeable difference. Usually, however, it takes a while before a team gets to the

low level of bugs that they have the potential to reach. Frequent deployment is valuable because it allows

your users to get new features more rapidly, to give more rapid feedback on those features, and generally

become more collaborative in the development cycle. This helps break down the barriers between customers

and development - barriers are the biggest ones to successful software development. Continuous integration

should occur frequently enough that no intervening window remains between commit and build, and such

that no errors can arise without developers noticing them and correcting them immediately. Normal practice

is to trigger these builds by every commit to a repository, rather than a periodically scheduled build. Some

well-known tools are CircleCI and CloudBees.

CloudTeams offers continuous integration capabilities through integrating appropriate tools to the Software

Team dashboard. The Product Manager is the lead on this process and is considered responsible for setting

up the Continuous integration process by providing appropriate credentials of the external service to be

integrated and by configuring basic parameters. Software Team members are able to check the continuous

integration build status and history. The team knows that, working on a development server, any change in

code should be deployed and built in the server, in small badges, in order to see if there is any serious

problem on integration. With CloudTeams and the PaaS connector named PaaSport connected in

CloudTeams, it is easy for the team to easily build and deploy automatically any integrated solution

connected with a code repository, and monitor the status of the service. Figure 5-18 shows a workflow on

how a Product Manager could setup the Continuous Integration and Automated PaaS deployment by using

the Software Team dashboard, while Software Team members can view the results of the integrated tools.


49

Figure 5-18. Suggested workflow for Continuous Integration and Automated Deployment

Step 9: Automated Acceptance Testing Task allocation tools, like Trello which is integrated in CloudTeams, allows to track when a developer has

completed a task, and even attach the commit that addresses each issue. The PaaSport component informs

the Product Manager about the status of the server, and the Bug Fixing and Reporting tools (e.g. GitHub) let

the Product Manager see if there are any open issues identified. Thus, what the Product or Technical

Manager can do before starting testing the complete release of the software under the Testing Plan, on Step

10, is to monitor the quality of the code, and there are dedicated tools to complete this task.

The use of code quality review tools is a useful approach for development, especially for collaborating teams.

It is in the scope of the CloudTeams Software Team dashboard to integrate code quality reviewing tools and

offer the information directly to the whole Software Team; SonarQube is already integrated in the

CloudTeams platform. The Product manager is responsible for the setup of appropriate tools to the platform,

while other team member can examine the review results offered. Figure 5-19 shows the workflow on how

Product Manager could connect and setup a Code Quality review platform to the Software Team dashboard,

while Software Team members can use the Software Team dashboard to view the review results during

Sprints.


50

Figure 5-19. Suggested workflow for Code Quality review in Software Team Dashboard

Step 10: Regression & Functional Testing On Step 3 (System Backlog Definition and Update), a Testing Plan should have been released to allow testing

in that phase (Figure 5-20). What it was suggested for the Testing Plan was to group user stories into use

cases, thus workflows of platform usage that will allow testing different cases into the platform, and will

identify problems in the UX.

In this step, the Testing Plan should be executed to evaluate (i.e. verify) the technical implementation. Two

processes are recommended, (a) an unsupervised testing with testing users that actually try to complete the

use case scenarios, and (b) a series of supervised tests from the technical team, together with the business

team. For every bug or issue identified, the backlog of issues is updated, and then based on the management

methodology that the team runs, the selected issues are addressed, and testing is relaunched, until the

product is ready to be evaluated by actual users on its usefulness. For example, the Lean methodology works

on small badges of implementation, which must be complete in order to complete a market test; thus what

has been selected to be released, should be perfect, but scalability and functional requirements may be

overlooked. A linear approach (e.g. waterfall) requires fixing all issues, and meeting all technical

requirements. An Agile approach should focus on the most important metrics in order to move forward and

launch an early version of the platform, and fix the problems before the official launch. In other words, it is a

matter of the management methodology on how strict the team will be with qualifying the tests.


51

Figure 5-20. The relation of step 10 (regression and functional testing) with step 3 (system backlog definition

and update)

For the unsupervised testing, there is a recommended process that we followed in CloudTeams. For each use

case produced in the Testing Plan, there is a form of evaluation, as seen in Table 5-8. Then, on a controlled

sample of users (i.e. in our case the pilot users), the goal was given and the user was observed and evaluated

on how well this goal was executed. For example, “create and account in the platform and setup a new

project, and then make it publicly available”. In the whole process, it is documented if there are any bugs

identified, any obstacles and difficulties, if explanation and help are given by the moderator of the tests, if

the case is completed (e.g. fatal errors that stopped the process), and finally how useful and easy the process

is perceived by the user. As the user is controlled through the whole experience, and certain goals engaged

their interactions, those two latest metrics (i.e. usefulness, easiness) are not perceived as validation metrics,

but an indication of the quality of the use case completed. At the end of those tests, any issues should be

uploaded in the issue tracking tool, and the qualitative feedback should be given to the team. Of course,

Table 5-8 is an indicative verification process, suggested and used in CloudTeams, and every team is free to

modify or use its own process.


52

Figure 5-21. The suggested workflow for regression and functional testing

Table 5-8. The template for unsupervised use case auditing (evaluating use cases)

Release

Number

Use Case ID Use case

Title

Completed

Without Help

Completed

with Help

Completed

with bugs

Not

Completed

because of

bug

Not

Completed

after help

Useful

(1-5)

Easy

(1-5)

Stakeholder Group Name

CODE.#

CODE.#

The supervised testing is controlled by the team, and may include various tests in different levels (see Table

5-9). Initially, the functionality suitability asks to audit the user stories, through running the use cases defined

in the unsupervised testing; in that case, the team itself runs through the use cases, and tries to complete the

defined goals by the Testing Plan. Of course the results are more controlled and biased, but the tester is able

to notice problems that the user may skip (e.g. wrong fonts, text, missing descriptions, broken links etc.). For

the rest of the supervised tests, the team should decide which ones will be concluded and how the test cases

will be completed; based on its experience, the available tools and the functional requirements defined in

collaboration with the technical team. You may see a list of indicative tests and suggested test cases to

complete them in Table 5-9. For each category, subcategories may be created and goals should be have set

by the team, to see if the current release passes the tests.


53

Table 5-9. Different metrics for running supervised testing24

Category of test Test Cases Description

Functional suitability User Stories Auditing Go through the user stories to see if they are functional for all the

stakeholders

Performance efficiency

System analytics Analytics during the operation of the system

Stress tests Run extreme cases to see when the system breaks

Compatibility Installation tests Measure the KPIs during and after installation of the system

Operability

Actual usage tests Using feedback mechanisms to collect opinion from actual users,

e.g. a poll on the landing page

Concept tests The team releases the concept in a form of text or media

System audit Walk through the system to measure the suggested KPIs

Aesthetics tests Use tools to cover different devices and user disabilities

Reliability System analytics Analytics during the operation of the system

Security Security Tests

Choose and run a series of system security, both on systems and

software. A complete list of the tests need to run is available at

OWASP25. Optional services that may automate many tests are

Relic26, Qualys27, SQLmap28.

Maintainability System audit Walk through system components to evaluate their behaviour

Portability Installation tests Measure the KPIs during and after installation of the system

There are various tools that may automate this step, available in the market, like TestComplete29, vTest30,

VersionOne31. In CloudTeams platform, you are not able to connect with Github and Bitbucket, in order to

manage issues in the development phase, as well as PaaSport to monitor the status and history of each build

of the product in the cloud.

24 Based on the ISO/IEC 25010:2011 standard 25 https://www.owasp.org/index.php/Web_Application_Security_Testing_Cheat_Sheet 26 http://newrelic.com/ 27 https://www.qualys.com/ 28 http://sqlmap.org/ 29 https://smartbear.com/product/testcomplete/ 30 http://vtestcorp.com/ 31 https://www.versionone.com/

https://www.owasp.org/index.php/Web_Application_Security_Testing_Cheat_Sheet

http://newrelic.com/

https://www.qualys.com/

http://sqlmap.org/

https://smartbear.com/product/testcomplete/

http://vtestcorp.com/

https://www.versionone.com/


54

Step 11: User Acceptance Testing – UAT At this point, the team has tested and verified the solution under development, and it is time to start

validating if what has been implemented is what the user actually needed. At this step, an evaluation

methodology runs based on the User Acceptance Testing Plan released in Step 2 for User Experience (see

Figure 5-22).

Figure 5-22. The relation of step 11 (user acceptance testing) with step 2 (user experience)

Overall (Figure 5-23), the team should get the Testing Plan as defined in Step 2, update it if needed to match

the context where the product has launched to see if the criteria and questions are still meaningful or not,

setup a questionnaire (e.g. in CloudTeams or Google Forms), and send it to the users of the solution

launched. After collecting the required number of responses, the team should analyse the results and issue a

UAT report. Based on those results, the team should prepare the launch of the solution, taking into

consideration the feedback collected from users, in order to smooth and facilitate the launch. For example, if

the users do not realize how an application works, the team may create manual and supporting material (e.g.

video), or choose to offer some free trial periods to convince them to “pay” for the learning curve period. The

goal of course of this step is not to redefine requirements, but to guide supporting activities that will

facilitate the opening of the solution to the customers.

There are plenty frameworks that may be used to evaluate usage from software users (see Table 5-10).

TAM232 is a well-known model for technology acceptance evaluation, HMSAM33 (Hedonic Motivation System

Adoption System) is another framework to evaluate systems that need to engage users, CloudTeams VnV

defined some business aspects that need to be covered through the evaluation of a solution, while UEQ34 is

an evaluation framework that focuses on running fast and easy-to-fill, light questionnaires that cover a broad

aspect of criteria. The most important criterion for selecting these frameworks is the previous experience of

partners with them, and their conformity to use them in practice. The greatest difference for these

frameworks is the way results are correlated at the end, and if link among different categories are identified

in other words; e.g. Models like TAM and HMSAM require correlation, while UEQ and ISO do not.

32 http://www.vvenkatesh.com/it/organizations/theoretical_models.asp 33 http://is.theorizeit.org/wiki/Hedonic-motivation_system_adoption_model_(HMSAM) 34 http://www.ueq-online.org/

http://www.vvenkatesh.com/it/organizations/theoretical_models.asp

http://is.theorizeit.org/wiki/Hedonic-motivation_system_adoption_model_(HMSAM)

http://www.ueq-online.org/


55

Figure 5-23. Running the User Acceptance Testing.

Table 5-10. Comparing well-known evaluation frameworks where partners are experienced in

Methodology Categories Advantages

ISO 25010:

Quality in use

model

Effectiveness, efficiency, Satisfaction (Usefulness, Trust,

Pleasure, Comfort), Safety (Economic damage risk, Health and

Safety risk, Environmental harm risk), Usability (Learnability,

Flexibility, Accessibility, Content Conformity)

Strict model that covers many technical

aspects from the user perspective.

TAM Perceived usefulness, Perceived ease of use, Intention to use,

Usage behaviour

Popular and trusted model, can find

correlation between user

characteristics, intention and actual

usage, if the correct answers are

selected.

HMSAM33 Perceived ease of use, Perceived usefulness, Curiosity, Joy,

Control, Immersion, Behavioural intention to use

Related to social networks, media

content, and generally projects that

need to trigger users’ interest

CloudTeams

VnV35

Clarity, Value, Need Level, Urgency, Need Coverage,

Innovation/Uniqueness, Virality

User experience related with business

values

UEQ Attractiveness, Perspicuity, Efficiency, Dependability,

Stimulation, Novelty

Fast, ready to evaluate, comprehensive,

user-friendly

35 Additional Characteristics, Sub-characteristics and Relevance to CloudTeams, as described in the CloudTeams methodology in D2.1


56

In CloudTeams, a mixture of metrics was used, covering different aspects of ISO 25010 (for Quality in use

model) and expanded with business metrics defined in CloudTeams V’n’V (i.e. Clarity, Value, Need Level etc.).

Table 5-11 presents the criteria selected to evaluate the actual usage, with some relevant questions; every

team is free to develop multiple questions per category, and then measure the mean value of those multiple

questions to count the score per criterion, and stabilise users’ random answers. The team may also consider

to run any regression analysis among different categories, to find any correlations between different criteria;

e.g. TAM implies correlation among different categories.

Table 5-11. Evaluation Metrics selected for the CloudTeams Pilots Operation

Criteria Related Questions (Scale 1-5)

Effectiveness

Effectiveness Can you accurately complete your goals with the system?

Efficiency

Efficiency Do you think the platform covers the intended purpose?

Satisfaction

Usefulness Do you find the platform useful?

Trust Do you trust the platform and its results?

Pleasure Does the platform please you when you use it?

Comfort Do you feel that the platform provides a comfortable UI and workflow?

Safety

Economic damage risk How sure you are that protects you from exposing you on economic damage?

Privacy harm risk How solid do you feel that the platform is on protecting your privacy?

Usability

Learnability How easy it was for you to learn how to use the platform?

Flexibility How much do you believe the platform can be used the system for other purposes than the

intended use?

Accessibility To which extent do you believe the platform can be used by disabled users?

Content Conformity

How useful do you find the content found in the platform in a matter of quality?

How satisfied are you from the content quantity found in the platform?


57

Criteria Related Questions (Scale 1-5)

Business Value

Clarity How clear was it for you what the platform is about?

Value How much do you feel that the platform increases the value you produce while using it?

Need Level How important is for you the need that the platform covers for you?

Urgency How fast do you expect the platform to be fully functional?

Need Coverage In which degree does the platform covers your need?

Innovation/Uniqueness How innovative do you find the idea of the platform?

Virality How probable is it for you to recommend the platform to someone you know?

CloudTeams is the ideal platform to run this evaluation step. By launching a campaign under a project, the

team cannot only reach existing users but also other users from the CloudTeams pool, invite them into the

campaign based on the persona characteristics and give away CloudCoins to users who reply on

questionnaires. Additionally, for your convenience, UEQ is also offered in the form of template in the

platform (see Figure 5-24), while there are ready-to use spreadsheets from the UEQ team to run the analysis

of users’ answers36.

Figure 5-24. Setting up a UEQ (or other) questionnaire through CloudTeams platform

36 http://www.ueq-online.org/filter/excel-sheet/

http://www.ueq-online.org/filter/excel-sheet/


58

There are multiple other tools that you may want to use in this step, like gTest, eXplorer, VersionOne and T-

Plan.

Step 12: Market Test At this stage, the team has launched the product, either in a controlled environment (e.g. Alpha version) or

publicly, and a production cycle is ready to close (see Figure 5-25). In this step, the team should measure the

actual usage of the solution, with a pricing tag on it if possible. There are many concepts on how you may

evaluate your market performance, and it is not always trivial as many factors may affect the solution, as the

industry maturity, the size of the market and the competition, may require for the team to set the goal lower.

Thus, in that step, well-established frameworks to evaluate the market acceptance are presented.

After experimenting with different ideas on what KPIs the team should track, and various resources may be

found online or in books, we have ended up with “Lean Analytics37” as the most appropriate framework to

align with the CloudTeams methodology. The main idea of the Lean Analytics framework is that every

software company follows a workflow of interactions with its users, which varies based on its business

model; then, depending on the phase where the team is located, it should focus on different metrics,

measure them, set goals that the team needs to reach, and work until this goal is met (i.e. go to Step 2). If

after various iterations the team identifies that the goal cannot be reached (and the goal was feasible for the

market where the team operates), the team should consider to make a Pivot and change some major or

minor parts of its main, business idea (i.e. go to Step 1).

Figure 5-25. Validating the product based on market tests

In more detail, the team has to choose among six (6) basic business models: e-Commerce (i.e. one sells to

many, e.g. Amazon), SaaS (i.e. one platform for everyone, e.g. Dropbox), Free Application/Content (i.e. free or

cheap solutions, revenues from side services, e.g. Candy Crush), Multimedia Content (i.e. selling content of

high quality, e.g. Netflix), User-Generated Content (i.e. users are the producers, e.g. Facebook), Two-sided

market (i.e. facilitating transactions between two market segments, e.g. eBay). Then, based on the selected

business model, the framework drives teams to define the workflow of interactions for users and customers,

37 http://leananalyticsbook.com/

http://leananalyticsbook.com/


59

step by step, including the channels that users use to arrive on the solution (e.g. Google Search, Google Play,

direct URL etc.).

Figure 5-26. Running the “final” step of the Market Test

The next step is the most important, the team should define in which phase of product development it is

among five (5) progressive stages: (a) Empathy (i.e. identifying a real problem), (b) Stickiness (i.e. see if there

is a small audience for your solution), (c) Virality (i.e. see if customers want to share their experience with

others), (d) Revenue (i.e. test if customers will pay), (e) Scale (i.e. improve channels to make development

cost-effective and sustainable). In each phase, the team should track one, different metric that helps it

improve and reach its goal. The team will focus on that metric, with various iterations if needed, until the

goal is reached or the team decides that a pivot is needed as there is no improvement. As this one metric is

important, the team should use relative metrics (e.g. monthly increase in purchases volume) and not vanity

ones (e.g. visitors in the website), while the goal should be reasonable for the phase and the size of the team.

There are detailed examples in the book for each business model on what metrics should be chosen in each

phase of the product development.

Additionally, the team should now when a product is successful or when it is ready to receive additional

funds. For that reason, some general KPIs should be monitored all the time; such metrics may be found in the

Lean Analytics framework, but various online resources suggest such KPIs. For example, a possible set of 12

KPIs38 is: Customer Acquisition Cost (CAC, i.e. average money spent on bringing a customer in the platform),

Customer Retention Rate (i.e. paying customers who remain paying for a given period), Lifetime value (LTV,

38 https://techcrunch.com/2017/02/04/12-kpis-you-must-know-before-pitching-your-startup/

https://techcrunch.com/2017/02/04/12-kpis-you-must-know-before-pitching-your-startup/


60

i.e. net value of average customer), Ratio of CAC to LTV, CAC recovery time (i.e. how much time is needed to

cover the CAC), Overhead (i.e. fixed costs for expenses to acquire new customers), Monthly burn, Runway

(i.e. remaining time for the company), Profit Margin, Conversion Rate (i.e. portion of users becoming paid

customers), Gross Merchandise Volume (GMV, i.e. gross value of sales), and Monthly Active Users (MAU, i.e.

Unique users engaging in one-month period).

Nevertheless, it is expected for a team to have setup the Lean Canvas already in early steps, thus the “Key

Metrics” that the team should track in this phase may be already defined and should only be revised.


61

6 Summary

This book is the first manual that tries to apply Agile methodologies in action, using both theories and tools

that have been available for years, while on the same time a dedicated platform is available to support it.

Typically, someone that introduces a methodology on product development, he leaves the reader to make it

out how this methodology may be realised through certain steps and tools. In many cases we have seen

different terminology or semantics for similar things.

The CloudTeams methodology tried to become something more than a culture, a school of theories or a

dictionary of common terms. CloudTeams wants to be a manual for Product Managers, combining the theory

of a methodology with actual, well tested and newly introduced tools. In CloudTeams we believe that

software development has met many obstacles when collaboration among the team members fails and when

the production phase is isolated from customers’ real needs. The existence of a customer community for

many people in software production may seem a headache, and the “chicken-egg” problem may also be

pointed out. However, we believe that if software teams have the customers in mind in every step, drive the

development phase around them, progressively the outcomes can be impressive. For teams that have a small

customer base, or try to launch on a new market, the CloudTeams SaaS platform is ideal to find new

customers, while for established organizations a private installation may allow the team to control their

community without worrying if they will miss their customers.

Of course, what you have read in this playbook is our initial work. Even if it is based in years of

experimentations and experience, it is expected to have flows and issues that need to be fixed. Thus, do not

hesitate to contact us any time, to share us your ideas and comments; we promise that if there is a relative

volume of comments, we will work all together to establish a separate group where we can collaborate in

order to improve the suggested methodology, and the tools developed and provided throughout this project.

Visit the CloudTeams Collaboration Platform at https://www.cloudteams.eu

https://www.cloudteams.eu/

cloudteams playbook · adapt — remix, transform, and build upon the material for any purpose, ......

Documents