[acm press the extended abstracts of the 32nd annual acm conference - toronto, ontario, canada...

Gamification of Collaborative Idea Generation and Convergence

Abstract

Collaborative brainstorming does not always result in

more ideas or higher quality ideas than working

individually. We designed a system with game elements

to incent participation in a collaborative creative idea

generation processes of brainstorming followed by a

convergence activity. We compared teams using the

system with and without game elements to investigate

the effect of the elements on collaborative work

activities. Preliminary results suggest that game

elements can help teams produce more ideas during

brainstorming and engage in more discussion during a

subsequent convergence activity, without negatively

affecting idea quality.

Author Keywords

Gamification; Brainstorming; Convergence;

Divergence; Collaboration

ACM Classification Keywords

H.5.3 [Information Interfaces and Presentation

(e.g.,HCI)]: Group and Organization Interfaces---

Computer-supported cooperative work;

Introduction

A common strategy for collaborative problem solving is

to brainstorm a number of ideas and then converge on

a subset through discussion, refinement and selection.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for

profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author. Copyright is held by the owner/author(s). CHI 2014, Apr 26 - May 01 2014, Toronto, ON, Canada ACM 978-1-4503-2474-8/14/04. http://dx.doi.org/10.1145/2559206.2581253

Ali Moradian

IBM

36 York Mills Road Suite 200,

Toronto, ON, Canada. M2P 2E9

[email protected]

Maaz Nasir

University of Toronto

40 St. George Street, Toronto,

Ontario, Canada. M5S 2E4

[email protected]

Kelly Lyons

University of Toronto

140 St. George Street, Toronto,

Ontario, Canada. M5S 3G6

[email protected]

Rock Leung

SAP

910 Mainland Street Vancouver,

British Columbia, Canada. V6B 1A9

[email protected]

Susan Elliott Sim

Many Roads Studios,

Toronto, Ontario, Canada.

M4N 1S6

[email protected]

Work-in-Progress CHI 2014, One of a CHInd, Toronto, ON, Canada

1459

http://dx.doi.org/10.1145/2559206.2581253

Collaborative brainstorming involves generating new

ideas, typically without consideration of quality and

using divergent group thinking [10] . In contrast,

judgment, critique, and selection of ideas from a list

generated by brainstorming requires convergent group

thinking and is often accomplished by refining,

combining, and/or selecting ideas [10]. Past research

has found that brainstorming in groups does not always

produce more ideas or higher quality ideas than

working individually [10]. Phenomena such as free

riding have been identified as barriers to group idea

generation. Computer-based group systems have been

designed to help by enabling anonymity in idea

generation and displaying ideas as they are entered in

real time [10]. However, the impact of computer-based

systems is not uniformly positive [2].

We are interested in how game elements can help

support collaborative brainstorming and convergence.

Gamification is an approach to enhancing existing

systems by adding game-related elements such as

goals, feedback, achievements, and rankings, for the

purpose of steering users towards sought-after

behaviors [8]. The use of these elements leverage

research in goal setting and social psychology [7],[9].

Gamified systems need not look like traditional

computer games [4]. In fact, the progress bar in

LinkedIn’s profile, goal setting and progress tracking in

a gym, and creating competition among customers in

Foursquare are examples given of gamification [4].

We created a system to support collaborative

brainstorming and convergence in order to evaluate the

effects of added game elements. Preliminary results

suggest that game elements can have a positive impact

by increasing the quantity of ideas produced in

collaborative brainstorming and may increase the

amount of discussion during the convergence activity.

A Brainstorming and Convergence System

We designed a system to support the collaborative

brainstorming and convergence processes depicted in

Figure 1. The system enables small teams (5 or fewer)

working on a pre-defined problem to quickly produce a

set of ideas [1]. The system was designed to work

either with collocated individuals or individuals working

over distance. We developed our system on the public

development version of SAP’s StreamWork [5], using

OpenSocial API.

During the brainstorming activity, participants generate

ideas anonymously and synchronously in parallel.

Submitted ideas are displayed immediately to peers.

Our implementation of the convergence activity was

modeled after the FastFocus ThinkLet [1]. In this

activity, each team member receives a Segmented List

of Ideas and is asked to select one idea. After all team

members have done this, they must perform

Clarification and Reduction by discussing each of the

selected ideas in turn by posting comments, during

which the team must agree on a final phrasing of the

idea and decide whether to add it to the Final List of

Ideas. After each selected idea has been discussed, the

process iterates over receiving another Segmented List

of Ideas, followed by Clarification and Reduction.

Figures 2 and 3 show the user interface (UI) for the

baseline version of the brainstorming and convergence

tools respectively. In the gamified version, the game

elements shown in Figures 4-8 were added to a panel

on the right side of the UI. These game elements have

been used in past systems to increase motivation and

engagement in performing a task [4], [6], [8]. Other

Figure 1: Brainstorming and Convergence system overview.


1460

than that everything else was the same between two

conditions (e.g. task description and questions).

Goal Setting

Giving specific high-challenge goals, compared to

simply letting people do their best, has been found to

lead to better performance [6], [9]. We selected goals

that encourage collaboration, are relatively difficult,

specific to the task, and can be achieved in a short

amount of time. For the brainstorming activity, each

team using the gamified version was given the ultimate

goal to generate 20 ideas but was presented with a

number of intermediate goals (10 ideas, 5 more, 3

more, 2 more, and bonus ideas; see Fig. 4). For the

convergence activity, gamification condition teams were

given four intermediate goals: generate five ideas in

the final list; all members in a team participate in the

discussion; complete the first round of discussion;

complete three rounds of discussion (see Fig. 6).

Progress Bar

Feedback that indicates progress towards a goal can

increase the usefulness of goals [9]. For the

brainstorming activity, we added a progress bar to

indicate the team’s progress toward each goal. We used

a power function (rather than a linear one) to indicate

faster progress as participants approach the final goal

[3]. For example, when teams generated 6 ideas

towards a goal of 10, the progress shown is 37.45%

(instead of 60%) but for 9/10, the progress 81% (see

Fig. 5). This characteristic was expected to provide

encouragement during brainstorming because idea

generation increases in difficulty over time [2].

Achievement Points and Leader Board

Awarding points is a common approach for providing

feedback and rewarding people for achieving task-

related goals [6], [9]. For the brainstorming activity,

we designed a leader board that displays the number of

ideas that each participant in the team has contributed

so far, thereby enabling participants to evaluate their

performance and compare it with others [7].

For the convergence activity, we created a scoring

system that awards points for both individual

achievements (e.g., selecting an idea for discussion,

participating in the discussion) and team achievements

(e.g., all participants in the team finish a round of

discussion). Participants can evaluate their relative

performance by reviewing the point history (see Fig. 8).

Method

In our study, teams of three collocated individuals

worked on the same collaborative problem-solving task

(brainstorming followed by convergence) using either

the baseline or gamified version of the system. The

teams were assigned randomly to condition-specific

rooms, thereby blinding them to the treatment.

We investigated two research questions: “How does

gamification affect productivity?” and “How does

gamification affect the quality of ideas produced?” To

answer the first question, we looked at the number of

ideas generated by each team during brainstorming,

and the number of ideas discussed, as well as the

number of discussion comments for each idea made by

each team during convergence. To answer the second

one, we examined the quality of ideas generated during

brainstorming and the quality of ideas selected during

convergence. We also conducted a post-activity survey

on participants’ experience.

Participants

The participants in the study were students in a

graduate-level project management course in an

Figure 2: The user interface for

the brainstorming activity

Figure 3: The user interface for

the convergence activity


1461

iSchool. The students had not worked together before

but were assigned to a team for the duration of the

term. Seven teams of three students each (total 21)

participated in the study: four teams used the baseline

system and three used the gamified version.

Procedure

Each student sat at a single computer. The participants

in each team sat near each other but were instructed to

only communicate using the system and not verbally.

All teams were asked to brainstorm on the topic:

“Given the iSchool mission and goals, identify project

ideas that you think will help iSchool achieve their

mission and goals.” Participants were provided a paper

copy of the mission and goals. For brainstorming,

participants were told: “You should think about project

ideas to support the iSchool mission and goals, as well

as focus on the quantity and variety of ideas. You will

have time to refine and reject ideas later. At this stage

all ideas are valuable.” For the convergence activity,

they were told: “You will go through several rounds of

reviewing and refining your brainstormed project ideas

through group discussions. The ultimate goal is to

select and agree on a final list of project ideas.”

All teams were given 75 minutes to complete in the

task. They were told to spend roughly 20 minutes in

the brainstorming activity and the rest of the time in

the convergence activity, but were able to decide as a

group when to switch to the convergence activity.

Data Analysis

We analyzed the data at the team level, which we

present below. In this study, individuals from different

teams did not interact with each other and since the

conditions were fixed, observations between teams are

independent. We used a one-way ANOVA to test the

differences in means and below we report mean (M),

standard deviation (SD) and F ratio.

We analyzed the quality of the brainstormed ideas and

final selected ideas by having one of the co-authors

rate all ideas on a scale from 0 (lowest) to 4 (highest).

The reliability of the rating scheme was validated by

having another co-author rate a random subset of the

data (~25% of ideas), both raters were blind to the

condition; substantial agreement was found (a linearly

weighted Kappa co-efficient = 0.63).

Results

Brainstorming

Although teams in the gamification and baseline

conditions spent similar amounts of time on the

brainstorming activity (M=16 min, SD=6.3 vs. M=15

min, SD=3.5, respectively), teams in the gamification

condition generated significantly more ideas than those

in the baseline condition (F1,5=7.1, p<.05). The

gamification condition teams produced on average 21.3

ideas per team (SD=4.0), while baseline condition

teams produced 13.8 ideas per team (SD=3.5). In fact,

two teams in the gamification condition exceeded the

goal of 20 ideas and none in the baseline condition did.

Teams from both conditions produced a similar number

of good ideas (rated 3 or 4 out of 4) (Gamification:

M=11.7, SD=5.5, 55% of all ideas; Baseline: M=9.0,

SD=3.4; 65% of all ideas).

Convergence

Teams in the gamification condition spent more time on

the convergence activity than those in the baseline

condition (M=50 min, SD=3.6 vs. M=38 min, SD=11.5,

respectively). Gamification condition teams on average

also engaged in more discussion than those in the

baseline condition in terms of average number of

Figure 4: Brainstorming tool

achievement list.

Figure 5: Brainstorming tool

progress bar.

Figure 6: Convergence tool

achievement list.

Figure 7: Convergence tool

leader board.

Figure 8: Convergence tool history of the points.


1462

comments per idea (M=23.11, SD=7.9 vs. M=13.4,

SD=6.9, respectively), but this difference was not

statistically significant (F1,5 = 2.9, ns). Despite these

differences, teams in the gamification and baseline

conditions discussed a similar number of ideas (M=6.7,

SD=1.1vs. M=6, SD=2.9, respectively; F1,5=0.13, ns).

The gamification condition teams, compared to those in

the baseline condition, selected on average more ideas

(M=5.3, SD=1.2 vs. M=4.0, SD=1.8, respectively) and

produced more good ideas in the final list (M=4.0,

SD=2.0 vs. M=2.8, SD=1.7, respectively), but this

difference was not significant (F1,5=0.11, ns).

Participants’ experience

Comments from the post-activity survey suggest that

the added game elements may have positively affected

participants’ perceptions of the brainstorming and

convergence activities. Four people in the gamified

condition chose to answer the open-ended question and

all commented positively: “Loved this program … Really

useful for brainstorming, which I'm usually really bad

at.”; “This is an excellent tool!”; “I really like this online

system. Makes it much easier for me, as a person

traditionally more quiet and observant than outgoing in

group work, to toss ideas out with the buffer of internet

anonymity!”; and “This is a really great system. I feel

that I am usually a wallflower in group discussions. I

tend to be better at adding to ideas and I need more

time, however, the buffer with this system made it

really inviting to be in the group discussion, To be

honest, I was dreading doing this assignment, but it

was great!”.

Three people in the baseline condition answered the

open-ended question, only one making a similar

positive comment, “I thought the tool was extremely

useful for organizations who have project members

distributed over a large geographic area. I did miss the

creativity that can result from verbal/face-to-face

interaction but in situations where this is not possible I

think this tool is an extremely valuable resource…” The

other two comments were less positive, for example:

“The only problem I had with the activity had to do with

the initial brainstorming process. I feel more

comfortable coming up with ideas if there is a dialogue

exchange -- as there was for the other stages of the

activity. I find it more difficult to come up with ideas if I

don't have the ability to discuss my ideas and receive

feedback, suggestions and other from other group

members. I find that good ideas come out of dialogue.”

Discussion and Future Work

In summary, we found that our added game elements

increased idea generation in the brainstorming activity

and may also have increased the amount of discussion

and number of ideas selected during the convergence

activity, though the increases were not statistically

significant.

Comments from the gamification condition teams offer

some evidence that the added game elements helped

them participate in the activity. While participants did

not comment directly on their experience with the

game elements, they also did not state that any

elements made it more difficult to participate (e.g.,

competitive pressure), which was a possibility. We also

noted a common theme in all comments was that

participants did not think they would do well in the

activity. However, participants in the gamification

condition teams found their version of the system to be

“really useful”, “much easier for me”, and “really

inviting.” Perhaps the extra guidance from the game

Figure 9: Quality of ideas produced during brainstorming by each team. Good ideas (scores 3 or better) are

shaded dark.

Figure 10: Quality of ideas produced after the convergence activity by

each team. Good ideas (scores 3 or better) are shaded dark.


1463

elements made it easier to participate. Better

understanding of why game elements helped support

the collaborative activity warrants further investigation.

In general, our work suggests there may be

opportunities for using game elements to support

brainstorming and convergence activities, and further

investigation is needed. First, we plan to run our study

with more teams to increase the statistical power of our

study and confirm if trends observed in our results are

significant. Further studies should also look at the effect

of our added game elements to smaller or larger teams

as the effort involved in collaborative brainstorming and

convergence may change with different team sizes. In

addition, the participants in our study were graduate

students who had not previously worked together. It

would be interesting to repeat the experiment in a

professional work setting with individuals who work

together on a regular basis. Finally, we only looked at

the combined effect of several gamification elements,

and more work is needed to understand the effect of

individual game elements on creative idea generation

processes. Specific elements may have a greater

influence than others over the outcome of the activity.

It may be also possible to fine-tune the design of each

game element for effectiveness in particular situations.

Acknowledgements

We thank anonymous reviewers for feedback, Dr. F.

Camacho (Damos) for help with statistical analysis, and

Dr. S. Szigeti for help with data collection. This work

was supported by an NSERC CRD Grant with SAP

Canada.

References [1] Briggs, R.O., De Vreede, G.J., Nunamaker, J.F., & Tobey, D. ThinkLets: Achieving predictable, repeatable

patterns of group interaction with group support systems (GSS). In Proc. of HICSS 2001, 9 pages.

[2] Ferreira, A., Antunes, P., and Herskovic, V. Improving group attention: An experiment with

synchronous brainstorming. Group Decision and Negotiation 20, 5 (2011), 643–666.

[3] Harrison, C., Amento, B., Kuznetsov, S., and Bell, R. Rethinking the progress bar. In Proc. of UIST 2007, 115–118.

[4] Huotari, K. and Hamari, J. Gamification from the perspective of service marketing. In Proc. of CHI 2011 Workshop Gamification, (2011).

[5] Kanaracus, C. SAP’s ‘Virtual War Room’ Tool Gets a Name: StreamWork | PCWorld. PCWorld, 2010. http://www.pcworld.com/article/192903/article.html.

[6] Li, W., Grossman, T., and Fitzmaurice, G. GamiCAD: a gamified tutorial system for first time autocad users. In Proc. of UIST 2012, 103–112.

[7] Ling, K., Beenen, G., Ludford, P., Wang, X., Chang, K., Li, X., Cosley, D., Frankowski, D., Terveen, L.,

Rashid, A.M., Resnick, P., and Kraut, R. Using social psychology to motivate contributions to online communities. Journal of Computer-Mediated Communication. 10, 4 (2005).

[8] Liu, Y., Alexandrova, T., and Nakajima, T. Gamifying intelligent environments. In Proceedings of the 2011 international ACM workshop on Ubiquitous meta user interfaces, (2011), 7–12.

[9] Locke, E. A. and Latham, G. P. Building a Practically Useful Theory of Goal Setting and Task Motivation: A 35-Year Odyssey. American Psychologist 57, 9, (2002), 705-717.

[10] Rietzschel, E.F.,Nijstad, B.A., and Stroebe, W. Productivity is not enough: A comparison of interactive and nominal brainstorming groups on idea generation and selection. Journal of Experimental Social Psychology 42, 2 (2006), 244–251.


1464

[acm press the extended abstracts of the 32nd annual acm conference - toronto, ontario, canada...

Documents