cobi: a community-informed conference scheduling tool. uist 2013 slides

Cobi: A Community-Informed Conference Scheduling Tool

Juho Kim Haoqi Zhang

Paul André Lydia B. Chilton

Wendy Mackay Michel Beaudouin-Lafon

Robert C. Miller Steven P. Dow

1

2

420 papers

16 parallel rooms

200 sessions in 15 venues

4 days and 3442 attendees

Room sizeAwarded papersSession length

Themes of the daySimilar sessions in nearby rooms

…

1. Lack data on paper affinities

2. Lack data on what people want

3. Lack software for resolving conflicts

4. Multi-dimensional requirements

Challenges for conference organizers

3

?

?

??

1pm

3pm

good or bad?

101 102 103 104

Cobicommunity-informed conference scheduling

4

Community-sourcing

5

Encoding constraints

Mixed-initiative

scheduling

1 2 3

The Cobi Scheduling Tool

6

Conflictshow are they detected?

how are they resolved?

7

8

Wikipedia Classroom Experiment

A Pilot Study of Using Crowds in the Classroom

✓✓

✓ ✓

X

“papers of mutual interest shouldnot be in opposing sessions.”constraint

input mediationEncode if two or more people express interest in seeing both.

Authors, do you want to see the other paper?

9

constraint“Wikipedia Classroom Experiment” and “A Pilot Study of Using Crowds in the Classroom” should not be in opposing sessions.

conflictWikipedia Classroom

ExperimentA Pilot Study of Using

Crowds in the Classroom

Room 2Room 1

1pm

Conflictshow are they detected?

how are they resolved?

10

✓

The Cobi Scheduling Tool

11

Detected conflicts

12

Schedule table

13

14

Step 1. Locate a conflict to resolve

Step 2. Study the conflict & propose move

15

Conflict information

16

Step 3. Review recommendations and pick

17

Step 4. Make the final scheduling decision

Showing how our conflict

will be resolved by this move

18

Sessions successfully swapped

Cobi Deployment for CHI 2013

19

Community-sourcing

20

Encoding constraints

Mixed-initiative

scheduling

1 2 3

Community-sourcing

21

Committee members create initial affinity

scores between papers.

1

22

Committee Clustering

1722 paper affinities from

64 committee members• Paul André, Haoqi Zhang, Juho Kim, Lydia B. Chilton, Steven P. Dow, and Robert

C. Miller. Community clustering: Leveraging an academic crowd to form coherent conference sessions. HCOMP 2013, to appear.

23

?

?

?

?

paper D

paper B

paper Cpaper E

paper A

Community-sourcing

24

Authors judge relevance and interest.

1

25

Authorsourcing

8651 preferences and constraints from

645 authors, covering 87% of accepted

submissions

26

my paper

fits with my paper does not fit

paper E

paper D

paper B

paper C

paper A

paper Z

The value of community input

“Authors were asked for input, most gave it, we tried hard to accommodate them, and almost nobody complained.”

27

28

Encoding constraints

Data from committee & authors

Cobidetected

238conflictsPreliminary

schedule from TP meeting

2

29

Mixed-initiative

scheduling

Organizers make scheduling changes.

3

Three organizers (general chair + TP chairs)

815 scheduling operations over 42 days

30

0

50

100

150

200

250

2/10/13 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13

Conflict Count

Date

conflict count

Conflicts: 238 70 (168 resolved)

31

0

50

100

150

200

250

0

30

60

90

120

150

180

2/10/13 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13

Conflict CountEdit Count

Date

session moves

paper moves

chair moves

meta edits

conflict count

32

0

50

100

150

200

250

0

30

60

90

120

150

180

2/10/13 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13

Conflict CountEdit Count

Date

session moves

paper moves

chair moves

meta edits

conflict count

33

Scheduling phasesResolving conflicts

Themes & fine-tuning

Session chair assignment

0

50

100

150

200

250

0

30

60

90

120

150

180

2/10/13 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13

Conflict CountEdit Count

Date

session moves

paper moves

chair moves

meta edits

conflict count

Adjust session length

Resolve author conflicts

Make coherent sessions

Phase 1. Resolving conflicts

34

Author conflicts

All co-authors (not just presenters) can attend all of their paper presentations.

30 0 in 29 minutes

35

“Cobi trivialized conflict resolution.”

36

0

50

100

150

200

250

0

30

60

90

120

150

180

2/10/13 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13

Conflict CountEdit Count

Date

session moves

paper moves

chair moves

meta edits

conflict count

Edit session titles

Reorder papers

Switch rooms

Balance awards

Phase 2. Themes and fine-tuning

37

Mixed-initiative problem solving

“I was by and large driven by what Cobi was suggesting. As you make progress you can progressively integrate other criteria that are not explicit in the system.”

38

0

50

100

150

200

250

0

30

60

90

120

150

180

2/10/13 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13

Conflict CountEdit Count

Date

session moves

paper moves

chair moves

meta edits

conflict count

Assign session chairs

Phase 3. Session chair assignment

39

40

http://projectcobi.com

committeesourcing authorsourcingFrenzysession making

Cobi scheduling tool myCHI for attendees

41

Paul André

CMU

Anant Bhardwaj

MIT

Lydia Chilton

UW

Steven Dow

CMU

Rob Miller

MIT

David Karger

MIT

Haoqi Zhang

Northwestern

Juho Kim

MIT

42

Upcoming

Michel Beaudouin-Lafon Wendy Mackay Patrick Baudisch

Cobi is a community-informed mixed-initiative system for a

large-scale planning problem.

Communitysourcing applications

Encoding constraints from community data

Mixed-initiative interface for conflict resolution

43

Cobi: Community-Informed Conference Scheduling

Juho Kim | juhokim@mit.eduhttp://projectcobi.com/

44

45

Backup Slides

46

Different crowd, Different expertise

• Organizers: 3

– Session balance, theme of the day

• Committee members: 200+

– Affinities between papers in their subfield

• Authors: 1000+

– Fit with their own paper

• Attendees: 3000+

– Individual preferences for attending talks

50

Techniques for Affinity Creation

• Manual grouping from TP meeting

• Automatically generated affinity scores using TF-IDF

• Committee-generated affinity scores

using community clustering

51

Fine-grained affinity data

• How relevant is this paper to yours?

– “should it be in the same session as your paper?”

• Is this paper interesting to you?

– “Would you like to see this paper’s presentation?”

52

Constraint Type Data Source Severity Encodedauthor with papers in opposing sessions

system-generated high -

topics of interest to a persona in opposing sessions

system-generated medium -

Session-related

53

Constraint Type Data Source Severity Encodedpapers of mutual interests in opposing sessions authorsourcing high 923 papers that do not fit well in the same session authorsourcing medium 651

Preference Type Data Source Severity Totalpapers good in the same session authorsourcing N/A 805

Paper-related

54

Constraint Type Data Source Severity Encoded

chair's paper in own sessionsystem-

generated high -

chair's paper in opposing sessionssystem-

generated high -chair interested in opposing sessions authorsourcing medium 243 chair in a session with a bad fit chairs medium -

Preference Type Data Source Severity Totalchair fits well in the session chairs N/A -

Chair-related

55

Constraint Type Related to Data Source Severity Total Initial Final Change

author with papers in opposing sessions Session

system-generated high - 31 0 -31

topics of interest to a persona in opposing sessions Session

system-generated medium - 6 4 -2

papers of mutual interests in opposing sessions Paper authorsourcing high 923 40 19 -21

papers that do not fit well in the same session Paper authorsourcing medium 651 129 42 -87

chair's paper in own session Chairsystem-

generated high - 21 0 -21

chair's paper in opposing sessions Chair system-

generated high - 6 0 -6

chair interested in opposing sessions Chair authorsourcing medium 243 5 4 -1

chair in a session with a bad fit Chair chairs medium - 0 1 1

Total violated 238 70 -168

Preference Type Data Source Severity Total Initial Final Change

papers good in the same session Paper authorsourcing N/A 805 268 272 4

chair fits well in the session Chair chairs N/A - 90 78 -12

Total satisfied 358 350 -8

56

0

10

20

30

0 200 400 600 800

Conflict Count

Edit Number

author in opposing sessions

persona in opposing sessions

Resolve author conflicts

Switch rooms

Balance awards

Adjust session length

System-defined conflicts over time

57

0

30

60

90

120

0 200 400 600 800

Conflict Count

Edit Number

papers that do not fit well in the same session

papers of mutual interests in opposing sessions

Make coherent sessions

Balance awards

Adjust session length

Switch rooms

Community-defined conflicts over time

58

0

10

20

30

270 420 570 720

Conflict Count

Edit Number

chair's paper in own session

chair's paper in opposing sessions

chair interested in opposing sessions

chair in a session with a bad fit

Assign session chairs

59

60

61

Encoded constraints/preferences

62

Encoded constraints/preferences

63

Schedule table

64

Schedule table

65

Scratch space for unscheduled items

66

Scratch space for unscheduled items

67

Search and notification

68

Search and notification

69

70

Step 1. Locate a conflict to resolve

71

conflict preference

award duration

72

73

74

75

76

77

78

79

Consequence of a swap

80

81