dynamics of collective performance in collaboration networksvictor/pub/ucsb-mit-sunbelt16.pdf ·...

40
Dynamics of Collective Performance in Collaboration Networks Amelkin V. , Askarisichani O. , Kim Y.J. , Singh, A.K. , Malone T.W. University of California, Santa Barbara, Department of Computer Science {victor,omid55,ambuj}@cs.ucsb.edu Massachusetts Institute of Technology, Sloan School of Management {youngji,malone}@mit.edu INSNA, XXXVI Sunbelt Social Networks Conference, 2016 1 / 18

Upload: others

Post on 14-Mar-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Dynamics of Collective Performancein Collaboration Networks

Amelkin V.†, Askarisichani O.†, Kim Y.J.‡, Singh, A.K.†, Malone T.W.‡

†University of California, Santa Barbara, Department of Computer Science

{victor,omid55,ambuj}@cs.ucsb.edu

‡Massachusetts Institute of Technology, Sloan School of Management

{youngji,malone}@mit.edu

INSNA, XXXVI Sunbelt Social Networks Conference, 2016

1 / 18

Page 2: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Introduction – Motivation

• Small teams of humans working on a sequence of simple tasks.

• Questions:

. Why do some teams perform well while others do not?

. What are the key factors affecting team performance?

. How to accurately predict team performance?

. How does team performance evolve?

. What is the impact of collaboration patterns on team performance?

2 / 18

Page 3: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Introduction – Dataset – Teams

• Dataset

. 68 teams, 4 members each, diverse population found through ads1,2

• Team features (select)

. social sensitivity3

. gender balance

. personality traits (“Big Five”)

. verbal / nonverbal communication

1David Engel et al. “Reading the mind in the eyes or reading between the lines? Theory ofmind predicts collective intelligence equally well online and face-to-face”. In: PloS one 9.12(2014), e115212.

2Anita W Woolley et al. “Evidence for a collective intelligence factor in the performance ofhuman groups”. In: Science 330.6004 (2010), pp. 686–688.

3Simon Baron-Cohen et al. “Reading the Mind in the Eyes test revised version: A study withnormal adults, and adults with Asperger syndrome or high-functioning autism”. In: Journal ofchild psychology and psychiatry 42.2 (2001), pp. 241–251.

3 / 18

Page 4: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Introduction – Dataset – Teams

• Dataset

. 68 teams, 4 members each, diverse population found through ads1,2

• Team features (select)

. social sensitivity3

. gender balance

. personality traits (“Big Five”)

. verbal / nonverbal communication

(reflective)

(disapproving)

Social sensitivity quantified viaReading the Mind in the Eyes test.

1Engel et al., “Reading the mind in the eyes or reading between the lines? Theory of mindpredicts collective intelligence equally well online and face-to-face”.

2Woolley et al., “Evidence for a collective intelligence factor in the performance of humangroups”.

3Baron-Cohen et al., “Reading the Mind in the Eyes test revised version: A study with normaladults, and adults with Asperger syndrome or high-functioning autism”.

3 / 18

Page 5: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Introduction – Dataset – Tasks

• Tasks

Typing(Text)

Typing(Numbers)

Matrixsolving

Detection(Images)

Detection(Words)

Unscramblewords

Memory(Video)

Memory(Images)

SudokuBrainstorm

(Words)

Brainstorm(Brick)

Brainstorm(Equations)

Judgement(Slogans)

Judgement(Pictures)

Judgemet(Pages)

• Task features

. category: generating, choosing, executing, remembering, sensing;

. verbality

4 / 18

Page 6: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Introduction – Dataset – Tasks

• Tasks

0 0.5 10

2

4

Typing Text

0 0.5 10

2

4

Typing Numbers

0 0.5 10

2

4

Matrix Solving

0 0.5 10

2

4

Detection Images

0 0.5 10

2

4

Detection Words

0 0.5 10

2

4

Unscramble Words

0 0.5 10

2

4

Memory Video

0 0.5 10

2

4

Memory Images

0 0.5 10

2

4

Sudoku

0 0.5 10

2

4

Brainstorm Words

0 0.5 10

2

4

Brainstorm Brick

0 0.5 10

2

4

Brainstorm Equations

0 0.5 10

2

4

Judgement Slogans

0 0.5 10

2

4

Judgement Pictures

0 0.5 10

2

4

Judgement Pages

Figure: Task scores (scaled to [0, 1])

• Task features

. category: generating, choosing, executing, remembering, sensing;

. verbality

4 / 18

Page 7: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Introduction – Dataset – Team communication

• Some teams only communicate through text chat.

• For these teams, we have communication logs.

Time Sender Message Sentiment9:33:45 daniel thanks tim !! +0.8

9:33:46 liza type in your own lines +0.3

9:39:50 tim tao, look at the rules +0.3...

......

...9:47:53 tim this is seriously subjective −0.6

9:48:11 liza I think they all suck honestly −0.8

9:49:48 tim when is this going to end? 0.0

9:50:21 tao one is nonsense −0.6

• Logs used to build network representation of teams.

5 / 18

Page 8: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Introduction – Problem statement

Team features Observed team behavior Observed performance

Predicted / explained team performance

6 / 18

Page 9: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Predicting team performance from similar tasks

• Natural Question: How a team will perform on a given task?

• Natural Answer: Similarly to how it performed on similar tasks.

• Verification: Measure the connection between task similarity w.r.t.task features and task similarity w.r.t. how teams score on the tasks.

Task similarity (task features)

Typing TextTyping N

umbers

Matrix Solving

Detection Im

ages

Detection W

ords

Unscram

ble Words

Mem

ory Video

Mem

ory Images

SudokuBrainstorm

Words

Brainstorm Brick

Brainstorm Equations

Judgement Slogans

Judgement Pictures

Judgement Pages

Typing TextTyping Numbers

Matrix SolvingDetection ImagesDetection Words

Unscramble WordsMemory Video

Memory ImagesSudoku

Brainstorm WordsBrainstorm Brick

Brainstorm EquationsJudgement SlogansJudgement Pictures

Judgement Pages0

0.2

0.4

0.6

0.8

1Task similarity (scores)

Typing TextTyping N

umbers

Matrix Solving

Detection Im

ages

Detection W

ords

Unscram

ble Words

Mem

ory Video

Mem

ory Images

SudokuBrainstorm

Words

Brainstorm Brick

Brainstorm Equations

Judgement Slogans

Judgement Pictures

Judgement Pages

Typing TextTyping Numbers

Matrix SolvingDetection ImagesDetection Words

Unscramble WordsMemory Video

Memory ImagesSudoku

Brainstorm WordsBrainstorm Brick

Brainstorm EquationsJudgement SlogansJudgement Pictures

Judgement Pages0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

• Conclusion: On average, teams do not perform similarly on similartasks: actual similarity matrices ∼ random similarity matrices(µ = 0.5, σ = 0.17).

• To predict team performance, we need to go beyond task similarity.

7 / 18

Page 10: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Predicting team performance from similar tasks

• Natural Question: How a team will perform on a given task?

• Natural Answer: Similarly to how it performed on similar tasks.

• Verification: Measure the connection between task similarity w.r.t.task features and task similarity w.r.t. how teams score on the tasks.

Task similarity (task features)

Typing TextTyping N

umbers

Matrix Solving

Detection Im

ages

Detection W

ords

Unscram

ble Words

Mem

ory Video

Mem

ory Images

SudokuBrainstorm

Words

Brainstorm Brick

Brainstorm Equations

Judgement Slogans

Judgement Pictures

Judgement Pages

Typing TextTyping Numbers

Matrix SolvingDetection ImagesDetection Words

Unscramble WordsMemory Video

Memory ImagesSudoku

Brainstorm WordsBrainstorm Brick

Brainstorm EquationsJudgement SlogansJudgement Pictures

Judgement Pages0

0.2

0.4

0.6

0.8

1Task similarity (scores)

Typing TextTyping N

umbers

Matrix Solving

Detection Im

ages

Detection W

ords

Unscram

ble Words

Mem

ory Video

Mem

ory Images

SudokuBrainstorm

Words

Brainstorm Brick

Brainstorm Equations

Judgement Slogans

Judgement Pictures

Judgement Pages

Typing TextTyping Numbers

Matrix SolvingDetection ImagesDetection Words

Unscramble WordsMemory Video

Memory ImagesSudoku

Brainstorm WordsBrainstorm Brick

Brainstorm EquationsJudgement SlogansJudgement Pictures

Judgement Pages0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

• Conclusion: On average, teams do not perform similarly on similartasks: actual similarity matrices ∼ random similarity matrices(µ = 0.5, σ = 0.17).

• To predict team performance, we need to go beyond task similarity.

7 / 18

Page 11: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Predicting team performance from similar tasks

• Natural Question: How a team will perform on a given task?

• Natural Answer: Similarly to how it performed on similar tasks.

• Verification: Measure the connection between task similarity w.r.t.task features and task similarity w.r.t. how teams score on the tasks.

Task similarity (task features)

Typing TextTyping N

umbers

Matrix Solving

Detection Im

ages

Detection W

ords

Unscram

ble Words

Mem

ory Video

Mem

ory Images

SudokuBrainstorm

Words

Brainstorm Brick

Brainstorm Equations

Judgement Slogans

Judgement Pictures

Judgement Pages

Typing TextTyping Numbers

Matrix SolvingDetection ImagesDetection Words

Unscramble WordsMemory Video

Memory ImagesSudoku

Brainstorm WordsBrainstorm Brick

Brainstorm EquationsJudgement SlogansJudgement Pictures

Judgement Pages0

0.2

0.4

0.6

0.8

1Task similarity (scores)

Typing TextTyping N

umbers

Matrix Solving

Detection Im

ages

Detection W

ords

Unscram

ble Words

Mem

ory Video

Mem

ory Images

SudokuBrainstorm

Words

Brainstorm Brick

Brainstorm Equations

Judgement Slogans

Judgement Pictures

Judgement Pages

Typing TextTyping Numbers

Matrix SolvingDetection ImagesDetection Words

Unscramble WordsMemory Video

Memory ImagesSudoku

Brainstorm WordsBrainstorm Brick

Brainstorm EquationsJudgement SlogansJudgement Pictures

Judgement Pages0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

• Conclusion: On average, teams do not perform similarly on similartasks: actual similarity matrices ∼ random similarity matrices(µ = 0.5, σ = 0.17).

• To predict team performance, we need to go beyond task similarity.

7 / 18

Page 12: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Predicting team performance from similar tasks

• Natural Question: How a team will perform on a given task?

• Natural Answer: Similarly to how it performed on similar tasks.

• Verification: Measure the connection between task similarity w.r.t.task features and task similarity w.r.t. how teams score on the tasks.

Task similarity (task features)

Typing TextTyping N

umbers

Matrix Solving

Detection Im

ages

Detection W

ords

Unscram

ble Words

Mem

ory Video

Mem

ory Images

SudokuBrainstorm

Words

Brainstorm Brick

Brainstorm Equations

Judgement Slogans

Judgement Pictures

Judgement Pages

Typing TextTyping Numbers

Matrix SolvingDetection ImagesDetection Words

Unscramble WordsMemory Video

Memory ImagesSudoku

Brainstorm WordsBrainstorm Brick

Brainstorm EquationsJudgement SlogansJudgement Pictures

Judgement Pages0

0.2

0.4

0.6

0.8

1Task similarity (scores)

Typing TextTyping N

umbers

Matrix Solving

Detection Im

ages

Detection W

ords

Unscram

ble Words

Mem

ory Video

Mem

ory Images

SudokuBrainstorm

Words

Brainstorm Brick

Brainstorm Equations

Judgement Slogans

Judgement Pictures

Judgement Pages

Typing TextTyping Numbers

Matrix SolvingDetection ImagesDetection Words

Unscramble WordsMemory Video

Memory ImagesSudoku

Brainstorm WordsBrainstorm Brick

Brainstorm EquationsJudgement SlogansJudgement Pictures

Judgement Pages0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

• Conclusion: On average, teams do not perform similarly on similartasks: actual similarity matrices ∼ random similarity matrices(µ = 0.5, σ = 0.17).

• To predict team performance, we need to go beyond task similarity.

7 / 18

Page 13: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Team features vs. Team performance

• Goal: use a team’s static features (e.g., gender balance, mean socialsensitivity, mean age) to predict its performance on each task.

• Methods:

. Correlation analysis

. Team performance modeling

- Linear / non-linear regularized regression

- Regression in latent space

- Support Vector Regression

- Gaussian Process

- REP Trees

results not reported

8 / 18

Page 14: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Team features vs. Team performance – Correlation

Correlation: Team features vs. Scores (p-val < 0.05)

Mean Age

Mind in the Eyes

# Females

Big-5 Sympathetic

Big-5 ExtravertBig-5 D

ependableTeam

TalksTeam

Cohesion

Big-5 Calm

Big-5 Open to N

ew

Typing Text

Typing Numbers

Matrix Solving

Detection Images

Detection Words

Unscramble Words

Memory Video

Memory Images

Sudoku

Brainstorm Words

Brainstorm Brick

Brainstorm Equations

Judgement Slogans

Judgement Pictures

Judgement Pages -0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

Team Feature ρ, [min,max] max p-valueMean Age [−0.53,−0.25] 0.04Mind in the Eyes [+0.24,+0.48] 0.05# Females [+0.26,+0.38] 0.03

Table: Top team features significantly correlated with task scores.

9 / 18

Page 15: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Team features vs. Team performance – Regression

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

Typing TextRMSE = 0.20

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

Typing NumbersRMSE = 0.20

0.2 0.4 0.6 0.8 1

0.2

0.4

0.6

0.8

1

Matrix SolvingRMSE = 0.20

0 0.5 10

0.2

0.4

0.6

0.8

1

Detection ImagesRMSE = 0.24

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

Detection WordsRMSE = 0.27

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

Unscramble WordsRMSE = 0.23

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

Memory VideoRMSE = 0.27

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

1

Memory ImagesRMSE = 0.24

0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

SudokuRMSE = 0.22

0 0.2 0.4 0.60

0.2

0.4

0.6

Brainstorm WordsRMSE = 0.13

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

Brainstorm BrickRMSE = 0.18

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

Brainstorm EquationsRMSE = 0.20

0.4 0.6 0.8 1

0.4

0.6

0.8

1

Judgement SlogansRMSE = 0.20

0.2 0.4 0.6 0.8 1

0.2

0.4

0.6

0.8

1

Judgement PicturesRMSE = 0.22

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

Judgement PagesRMSE = 0.21

Figure: True vs. predicted task scores (linear regression, 80% LASSO + 20% ridge)

10 / 18

Page 16: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Team features vs. Team performance – Regression

Team feature significance

# Females

Big-5 ExtravertM

ean AgeM

ind in the EyesBig-5 O

pen to New

Big-5 Sympathetic

Big-5 Dependable

Team Talks

Big-5 Calm

Team C

ohesion

Typing Text

Typing Numbers

Matrix Solving

Detection Images

Detection Words

Unscramble Words

Memory Video

Memory Images

Sudoku

Brainstorm Words

Brainstorm Brick

Brainstorm Equations

Judgement Slogans

Judgement Pictures

Judgement Pages-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Team Feature min |βi| max |βi| mean |βi|# Females 0.19 0.89 0.47Big-5 Extravert 0.07 0.46 0.24Mean Age 0.04 0.36 0.18Mind in the Eyes 0.01 1.00 0.14

Table: Top team features emphasized in the best linear regression models. Sorted by mean |βi|,where βi is the model’s coefficient corresponding to the team feature.

10 / 18

Page 17: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Team features vs. Team performance dynamics

• Can we explain the dynamics of team performance in terms of teamfeatures?

0

.5

1Team 3

0

.5

1Team 10

0

.5

1Team 64

0

.5

1Team 67

0

.5

1Team 48

0

.5

1Team 5

0

.5

1Team 59

0

.5

1Team 4

0

.5

1Team 46

0

.5

1Team 9

0

.5

1Team 40

0

.5

1Team 54

0

.5

1Team 20

0

.5

1Team 55

0

.5

1Team 15

0

.5

1Team 18

0

.5

1Team 58

0

.5

1Team 33

0

.5

1Team 28

0

.5

1Team 21

0

.5

1Team 1

0

.5

1Team 26

0

.5

1Team 41

0

.5

1Team 68

0

.5

1Team 23

0

.5

1Team 14

0

.5

1Team 7

0

.5

1Team 65

0

.5

1Team 36

0

.5

1Team 32

0

.5

1Team 62

0

.5

1Team 66

0

.5

1Team 16

0

.5

1Team 19

0

.5

1Team 63

0

.5

1Team 11

0

.5

1Team 6

0

.5

1Team 17

0

.5

1Team 47

0

.5

1Team 34

0

.5

1Team 52

0

.5

1Team 56

0

.5

1Team 57

0

.5

1Team 43

0

.5

1Team 51

0

.5

1Team 2

0

.5

1Team 60

0

.5

1Team 39

0

.5

1Team 8

0

.5

1Team 13

0

.5

1Team 61

0

.5

1Team 30

0

.5

1Team 53

0

.5

1Team 25

0

.5

1Team 45

0

.5

1Team 31

0

.5

1Team 42

0

.5

1Team 22

0

.5

1Team 49

0

.5

1Team 50

0

.5

1Team 35

0

.5

1Team 38

0

.5

1Team 24

0

.5

1Team 44

0

.5

1Team 27

0

.5

1Team 29

0

.5

1Team 37

0

.5

1Team 12

Figure: Series of 15 task scores for each of 68 teams. Teams are ordered by their overallperformance, from low (top left) to high (bottom right) performers.

11 / 18

Page 18: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Team features vs. Team performance dynamics

Correlation: Team features vs. Score dynamics features

CI

# Females

Mind in the Eyes

Team C

ohesionBig-5 ExtravertBig-5 Sym

patheticBig-5 D

ependableBig-5 C

almBig-5 O

pen to New

Mean Age

first scorelast score

median scoremean score

mean wavelet scorescore std

score variancescore skewness

score kurtosisbest-fit line, Y(first)best-fit line, Y(last)

best-fit line, Y(middle)best-fit line slope (tan)

best-fit line slope (angle)last score - first score

# below-median score changes# above-median score changes

# score increases# score decreases

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

• Teams with a higher proportion of females (ρ = 0.30) and socialperceptiveness (ρ = 0.53) perform better overall.

• Better-performing teams start doing so early (first task ρ = 0.88) andperform well consistently (slope ρ = −0.42, skewness ρ = −0.50).

• Older teams perform worse on the first few tasks (ρ = −0.55).

11 / 18

Page 19: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Observed performance dynamics vs. Team performanceTypingText(training)

TypingNumbers(training)

MatrixSolving(training)

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

DetectionImagesRMSE = 0.24

0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

DetectionWordsRMSE = 0.25

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

UnscrambleWordsRMSE = 0.22

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

MemoryVideoRMSE = 0.27

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

1

MemoryImagesRMSE = 0.23

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

SudokuRMSE = 0.22

0.1 0.2 0.3 0.4 0.50

0.2

0.4

BrainstormWordsRMSE = 0.14

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

BrainstormBrickRMSE = 0.17

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

BrainstormEquationsRMSE = 0.18

0.4 0.6 0.8 1

0.4

0.6

0.8

1

JudgementSlogansRMSE = 0.19

0.4 0.6 0.8

0.4

0.6

0.8

JudgementPicturesRMSE = 0.21

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

JudgementPagesRMSE = 0.20

Figure: True vs. predicted task scores; linear regression using features derived from the past teamperformance dynamics.

Features RMSEs for task score predictions

Static .20 .20 .20 .24 .27 .23 .27 .24 .22 .13 .18 .20 .20 .22 .21

Dynamic × × × .24 .25 .22 .27 .23 .22 .14 .17 .18 .19 .21 .20

• Performance prediction based on (static) team features is as goodas prediction based on the observed performance dynamics.

12 / 18

Page 20: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Observed performance dynamics vs. Team performanceTypingText(training)

TypingNumbers(training)

MatrixSolving(training)

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

DetectionImagesRMSE = 0.24

0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

DetectionWordsRMSE = 0.25

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

UnscrambleWordsRMSE = 0.22

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

MemoryVideoRMSE = 0.27

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

1

MemoryImagesRMSE = 0.23

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

SudokuRMSE = 0.22

0.1 0.2 0.3 0.4 0.50

0.2

0.4

BrainstormWordsRMSE = 0.14

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

BrainstormBrickRMSE = 0.17

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

BrainstormEquationsRMSE = 0.18

0.4 0.6 0.8 1

0.4

0.6

0.8

1

JudgementSlogansRMSE = 0.19

0.4 0.6 0.8

0.4

0.6

0.8

JudgementPicturesRMSE = 0.21

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

JudgementPagesRMSE = 0.20

Figure: True vs. predicted task scores; linear regression using features derived from the past teamperformance dynamics.

Features RMSEs for task score predictions

Static .20 .20 .20 .24 .27 .23 .27 .24 .22 .13 .18 .20 .20 .22 .21

Dynamic × × × .24 .25 .22 .27 .23 .22 .14 .17 .18 .19 .21 .20

• Performance prediction based on (static) team features is as goodas prediction based on the observed performance dynamics.

12 / 18

Page 21: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs for Team Performance Prediction

• For chatting teams, we have team member communication logs:

Time Sender Message Sentiment9:33:45 daniel thanks tim !! +0.89:33:46 liza type in your own lines +0.3

.

.

....

.

.

....

• Goal: connect team performance with features derived from logs.

• General log-based features: (select)

. # of turns in conversation

. time delays between chat messages

. mean # words/characters produced per team member

. message sentiment

13 / 18

Page 22: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs for Team Performance Prediction

• General log-based features: (select)

. # of turns in conversation

. time delays between chat messages

. mean # words/characters produced per team member

. message sentiment

• Network-level features:

. A team is viewed as a network; team members – nodes.

. Edge weight wij = how often member i responds4 to member j.

. Features extracted from the network:- node degree

- edge density

- clustering coefficients

- betweenness centrality

- max/mean shortest path lengths

- · · ·4The likelihood that one message is a response to another message is computed based on their

temporal order and proximity.13 / 18

Page 23: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs to Networks

• Networks derived from logs:

Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.

.

.

....

.

.

.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah

14 / 18

Page 24: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs to Networks

• Networks derived from logs:

Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.

.

.

....

.

.

.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah

nick kate

jeral greg

14 / 18

Page 25: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs to Networks

• Networks derived from logs:

Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.

.

.

....

.

.

.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah

nick kate

jeral greg

14 / 18

Page 26: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs to Networks

• Networks derived from logs:

Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.

.

.

....

.

.

.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah

nick kate

jeral greg

14 / 18

Page 27: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs to Networks

• Networks derived from logs:

Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.

.

.

....

.

.

.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah

nick kate

jeral greg

14 / 18

Page 28: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs to Networks

• Networks derived from logs:

Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.

.

.

....

.

.

.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah

nick kate

jeral greg

14 / 18

Page 29: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs to Networks

• Networks derived from logs:

Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.

.

.

....

.

.

.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah

nick kate

jeral greg

14 / 18

Page 30: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs to Networks

• Networks derived from logs:

Time Sender Message36:38 nick So, I think the salt is really low importance.37:00 kate i don’t think the aircraft compass or motor oil are very useful37:05 nick kate, sun umbrella is high (we need shade!)37:11 kate i would put umbrella and food next after water37:16 nick Well, I think the compass works regardless, but yeah.

.

.

....

.

.

.38:13 jeral Mirror is high because it’s a signal38:17 jeral You can reflect the sun38:25 greg depends how big it is38:29 kate well yeah

nick kate

jeral greg

14 / 18

Page 31: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs to Networks

• Networks derived from logs:

nick kate

jeral greg

Dense, Weighted

nick kate

jeral greg

Sparse, Unweighted

14 / 18

Page 32: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs to Networks

• Networks derived from logs:

nick kate

jeral greg

Dense, Weighted

nick kate

jeral greg

Sparse, Unweighted

• Typical weighted adjacency matrices:

0

0.1

0.2

0.3

0.4

Low-performing team

0

0.1

0.2

0.3

High-performing team

14 / 18

Page 33: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs vs. Team performance – CorrelationCorrelation: Log-based features vs. Scores (p-val < 0.05)

pwn-nturns

pwn-delays-m

ean

pwn-delays-m

edian

sn25-cluscoeff-mean

pwn-outdeg-m

ean-m-std

pwn-delays-std

sn25-density

sn25-outdegree-mean

sn25-indegree-mean

pwn-sent-total-std

sn25-nsccsn50-nccpw

n-indegree-std

pwn-indeg-m

ean-m-std

sn50-outdegree-mean

sn50-indegree-mean

pwn-nw

ords-total

sn50-density

pwn-nw

ords-per-mem

ber-mean

pwn-delays-m

ax90

pwn-delays-m

ax80

pwn-nchars-total

pwn-nchars-per-m

ember-m

ean

pwn-delays-m

ax70

sn50-avg-path-len

sn50-centrality-mean

sn50-diameter

pwn-outdegree-m

ean

pwn-indegree-m

ean

pwn-outdegree-std

sn50-cluscoeff-mean

sn50-nsccsn50-indegree-std

sn25-nccsn75-nccpw

n-sent-total-mean

pwn-sent-nn-std

sn25-avg-path-len

pwn-sent-nn-m

ean

pwn-sent-neg-std

sn75-outdegree-mean

sn75-indegree-mean

sn50-outdegree-std

pwn-sent-neg-m

ean

sn75-avg-path-len

pwn-reciprocity-m

ean

pwn-reciprocity-std

sn25-outdegree-std

sn25-indegree-std

sn25-diameter

sn25-centrality-mean

sn75-density

sn75-outdegree-std

sn75-indegree-std

sn75-diameter

sn75-nsccsn75-centrality-m

ean

sn75-cluscoeff-mean

pwn-nchars-per-m

ember-std

pwn-nw

ords-per-mem

ber-std

pwn-sent-pos-m

ean

pwn-sent-pos-std

Typing TextTyping Numbers

Matrix SolvingDetection ImagesDetection Words

Unscramble WordsMemory Video

Memory ImagesSudoku

Brainstorm WordsBrainstorm Brick

Brainstorm EquationsJudgement SlogansJudgement Pictures

Judgement Pages -0.5

0

0.5

Team Feature ρ, [min,max] max p-value# of turns in conversation [ 0.37, 0.37] 0.03Median delay between chat messages [−0.44,−0.35] 0.05Mean clustering coefficient [ 0.36, 0.54] 0.04Node degree, mean− std [ 0.35, 0.49] 0.04Edge density [ 0.42, 0.52] 0.01Message sentiment variance [−0.52,−0.39] 0.02# connected components [−0.48,−0.35] 0.04# words typed [ 0.35, 0.41] 0.05Mean path length [ 0.39, 0.50] 0.02Mean betweenness [ 0.41, 0.45] 0.05

Table: Top 10 (out of 63) log-based team features significantly correlated with task scores.

15 / 18

Page 34: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs vs. Team performance – Regression

0.2 0.4 0.60

0.2

0.4

0.6

Typing TextRMSE = 0.21

0 0.2 0.4 0.60

0.2

0.4

0.6

Typing NumbersRMSE = 0.17

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

Matrix SolvingRMSE = 0.27

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

Detection ImagesRMSE = 0.24

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

Detection WordsRMSE = 0.27

0 0.5 10

0.2

0.4

0.6

0.8

1

Unscramble WordsRMSE = 0.24

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

Memory VideoRMSE = 0.27

0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

Memory ImagesRMSE = 0.24

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

1

SudokuRMSE = 0.25

0 0.2 0.4 0.6

0.1

0.2

0.3

0.4

0.5

0.6

Brainstorm WordsRMSE = 0.18

0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

Brainstorm BrickRMSE = 0.20

0.2 0.4 0.6 0.8

0.2

0.4

0.6

0.8

Brainstorm EquationsRMSE = 0.19

0.4 0.6 0.8 1

0.4

0.6

0.8

Judgement SlogansRMSE = 0.23

0.2 0.4 0.6 0.8 1

0.2

0.4

0.6

0.8

Judgement PicturesRMSE = 0.26

0 0.5 10

0.2

0.4

0.6

0.8

1

Judgement PagesRMSE = 0.23

Figure: True vs. predicted task scores

16 / 18

Page 35: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs vs. Team performance – Regression

Log-based feature significance (model coefficients)

pwn-outdeg-m

ean-m-std

sn25-cluscoeff-mean

sn50-centrality-mean

sn25-densitysn50-densitypw

n-nturnssn50-nccpw

n-nwords-total

sn25-nsccpw

n-delays-mean

pwn-delays-std

sn50-diameter

sn50-avg-path-len

Typing TextTyping Numbers

Matrix SolvingDetection ImagesDetection Words

Unscramble WordsMemory Video

Memory ImagesSudoku

Brainstorm WordsBrainstorm Brick

Brainstorm EquationsJudgement SlogansJudgement Pictures

Judgement Pages -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Team Feature min |βi| max |βi| mean |βi|Node degree, mean− std 0.15 1.00 0.62Mean clustering coefficient 0.47 0.70 0.58Mean betweenness 0.14 0.66 0.31# connected components −0.34 −0.26 −0.30

Table: Top 4 team features emphasized in the best linear regression models. Sorted bymean |βi|, where βi is the model’s coefficient corresponding to the team feature.

16 / 18

Page 36: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Communication Logs vs. Team performance – Regression

Log-based feature significance (model coefficients)

pwn-outdeg-m

ean-m-std

sn25-cluscoeff-mean

sn50-centrality-mean

sn25-densitysn50-densitypw

n-nturnssn50-nccpw

n-nwords-total

sn25-nsccpw

n-delays-mean

pwn-delays-std

sn50-diameter

sn50-avg-path-len

Typing TextTyping Numbers

Matrix SolvingDetection ImagesDetection Words

Unscramble WordsMemory Video

Memory ImagesSudoku

Brainstorm WordsBrainstorm Brick

Brainstorm EquationsJudgement SlogansJudgement Pictures

Judgement Pages -1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Features RMSEs for task score predictions

Static .20 .20 .20 .24 .27 .23 .27 .24 .22 .13 .18 .20 .20 .22 .21

Dynamic × × × .24 .25 .22 .27 .23 .22 .14 .17 .18 .19 .21 .20

Log-based .21 .17 .27 .24 .25 .22 .27 .23 .22 .14 .17 .18 .19 .21 .20

16 / 18

Page 37: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Conclusion

• Team performance prediction – Summary

. Team performance prediction via task similarity is intuitive,but does not work.

. Performance prediction via regression on static team features worksreasonably well. Key positive factors – number of females and socialsensitivity.

. Performance prediction based on a team’s performance history worksslightly better (although the history’s often being unavailable).

. Predictions based on log-data are of the same quality, yet,have different model interpretation.

- positive factors: # of turns in conversion, node degrees and clusteringcoefficients, edge density

- negative factors: communication delays, network disconnectedness, messagesentiment variance.

17 / 18

Page 38: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Conclusion

• Recipe for a better-performing team

. Team members are socially apt, as reflected in their social sensitivity.

. The team is gender-balanced∗.∗Number of females is likely emphasized by the predictive models

because of its positive connection with social sensitivity5

. Team members enthusiastically interact and

equally contribute to the discussion.

5Baron-Cohen et al., “Reading the Mind in the Eyes test revised version: A study with normaladults, and adults with Asperger syndrome or high-functioning autism”.

17 / 18

Page 39: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Conclusion

• Recipe for a better-performing team

. Team members are socially apt, as reflected in their social sensitivity.

. The team is gender-balanced∗.∗Number of females is likely emphasized by the predictive models

because of its positive connection with social sensitivity5

. Team members enthusiastically interact and

equally contribute to the discussion.

• Future Work

. Capturing actual communication/collaboration between teammembers.

. Varying task sequence composition and, particularly, task order.

. Tracking team members’ emotions.

. Studying effect of particular collaboration patterns on teamperformance dynamics.

5Baron-Cohen et al., “Reading the Mind in the Eyes test revised version: A study with normaladults, and adults with Asperger syndrome or high-functioning autism”.

17 / 18

Page 40: Dynamics of Collective Performance in Collaboration Networksvictor/pub/ucsb-mit-sunbelt16.pdf · 2016-04-11 · Dynamics of Collective Performance in Collaboration Networks Amelkin

Questions

∼ Thanks ∼

18 / 18