cobi: a community-informed conference scheduling tool. uist 2013 slides
DESCRIPTION
Juho Kim, Haoqi Zhang, Paul André, Lydia B. Chilton, Wendy Mackay, Michel Beaudouin-Lafon, Robert C. Miller, and Steven P. Dow. Cobi: A Community-Informed Conference Scheduling Tool. UIST 2013. http://dl.acm.org/citation.cfm?id=2502034TRANSCRIPT
Cobi: A Community-Informed Conference Scheduling Tool
Juho Kim Haoqi Zhang
Paul André Lydia B. Chilton
Wendy Mackay Michel Beaudouin-Lafon
Robert C. Miller Steven P. Dow
1
2
420 papers
16 parallel rooms
200 sessions in 15 venues
4 days and 3442 attendees
Room sizeAwarded papersSession length
Themes of the daySimilar sessions in nearby rooms
…
1. Lack data on paper affinities
2. Lack data on what people want
3. Lack software for resolving conflicts
4. Multi-dimensional requirements
Challenges for conference organizers
3
?
?
??
1pm
3pm
good or bad?
101 102 103 104
Cobicommunity-informed conference scheduling
4
Community-sourcing
5
Encoding constraints
Mixed-initiative
scheduling
1 2 3
The Cobi Scheduling Tool
6
Conflictshow are they detected?
how are they resolved?
7
8
Wikipedia Classroom Experiment
A Pilot Study of Using Crowds in the Classroom
✓✓
✓ ✓
X
“papers of mutual interest shouldnot be in opposing sessions.”constraint
input mediationEncode if two or more people express interest in seeing both.
Authors, do you want to see the other paper?
9
constraint“Wikipedia Classroom Experiment” and “A Pilot Study of Using Crowds in the Classroom” should not be in opposing sessions.
conflictWikipedia Classroom
ExperimentA Pilot Study of Using
Crowds in the Classroom
Room 2Room 1
1pm
Conflictshow are they detected?
how are they resolved?
10
✓
The Cobi Scheduling Tool
11
Detected conflicts
12
Schedule table
13
14
Step 1. Locate a conflict to resolve
Step 2. Study the conflict & propose move
15
Conflict information
16
Step 3. Review recommendations and pick
17
Step 4. Make the final scheduling decision
Showing how our conflict
will be resolved by this move
18
Sessions successfully swapped
Cobi Deployment for CHI 2013
19
Community-sourcing
20
Encoding constraints
Mixed-initiative
scheduling
1 2 3
Community-sourcing
21
Committee members create initial affinity
scores between papers.
1
22
Committee Clustering
1722 paper affinities from
64 committee members• Paul André, Haoqi Zhang, Juho Kim, Lydia B. Chilton, Steven P. Dow, and Robert
C. Miller. Community clustering: Leveraging an academic crowd to form coherent conference sessions. HCOMP 2013, to appear.
23
?
?
?
?
paper D
paper B
paper Cpaper E
paper A
Community-sourcing
24
Authors judge relevance and interest.
1
25
Authorsourcing
8651 preferences and constraints from
645 authors, covering 87% of accepted
submissions
26
my paper
fits with my paper does not fit
paper E
paper D
paper B
paper C
paper A
paper Z
The value of community input
“Authors were asked for input, most gave it, we tried hard to accommodate them, and almost nobody complained.”
27
28
Encoding constraints
Data from committee & authors
Cobidetected
238conflictsPreliminary
schedule from TP meeting
2
29
Mixed-initiative
scheduling
Organizers make scheduling changes.
3
Three organizers (general chair + TP chairs)
815 scheduling operations over 42 days
30
0
50
100
150
200
250
2/10/13 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13
Conflict Count
Date
conflict count
Conflicts: 238 70 (168 resolved)
31
0
50
100
150
200
250
0
30
60
90
120
150
180
2/10/13 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13
Conflict CountEdit Count
Date
session moves
paper moves
chair moves
meta edits
conflict count
32
0
50
100
150
200
250
0
30
60
90
120
150
180
2/10/13 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13
Conflict CountEdit Count
Date
session moves
paper moves
chair moves
meta edits
conflict count
33
Scheduling phasesResolving conflicts
Themes & fine-tuning
Session chair assignment
0
50
100
150
200
250
0
30
60
90
120
150
180
2/10/13 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13
Conflict CountEdit Count
Date
session moves
paper moves
chair moves
meta edits
conflict count
Adjust session length
Resolve author conflicts
Make coherent sessions
Phase 1. Resolving conflicts
34
Author conflicts
All co-authors (not just presenters) can attend all of their paper presentations.
30 0 in 29 minutes
35
“Cobi trivialized conflict resolution.”
36
0
50
100
150
200
250
0
30
60
90
120
150
180
2/10/13 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13
Conflict CountEdit Count
Date
session moves
paper moves
chair moves
meta edits
conflict count
Edit session titles
Reorder papers
Switch rooms
Balance awards
Phase 2. Themes and fine-tuning
37
Mixed-initiative problem solving
“I was by and large driven by what Cobi was suggesting. As you make progress you can progressively integrate other criteria that are not explicit in the system.”
38
0
50
100
150
200
250
0
30
60
90
120
150
180
2/10/13 2/17/13 2/24/13 3/3/13 3/10/13 3/17/13
Conflict CountEdit Count
Date
session moves
paper moves
chair moves
meta edits
conflict count
Assign session chairs
Phase 3. Session chair assignment
39
40
http://projectcobi.com
committeesourcing authorsourcingFrenzysession making
Cobi scheduling tool myCHI for attendees
41
Paul André
CMU
Anant Bhardwaj
MIT
Lydia Chilton
UW
Steven Dow
CMU
Rob Miller
MIT
David Karger
MIT
Haoqi Zhang
Northwestern
Juho Kim
MIT
42
Upcoming
Michel Beaudouin-Lafon Wendy Mackay Patrick Baudisch
Cobi is a community-informed mixed-initiative system for a
large-scale planning problem.
Communitysourcing applications
Encoding constraints from community data
Mixed-initiative interface for conflict resolution
43
45
Backup Slides
46
Different crowd, Different expertise
• Organizers: 3
– Session balance, theme of the day
• Committee members: 200+
– Affinities between papers in their subfield
• Authors: 1000+
– Fit with their own paper
• Attendees: 3000+
– Individual preferences for attending talks
50
Techniques for Affinity Creation
• Manual grouping from TP meeting
• Automatically generated affinity scores using TF-IDF
• Committee-generated affinity scores
using community clustering
51
Fine-grained affinity data
• How relevant is this paper to yours?
– “should it be in the same session as your paper?”
• Is this paper interesting to you?
– “Would you like to see this paper’s presentation?”
52
Constraint Type Data Source Severity Encodedauthor with papers in opposing sessions
system-generated high -
topics of interest to a persona in opposing sessions
system-generated medium -
Session-related
53
Constraint Type Data Source Severity Encodedpapers of mutual interests in opposing sessions authorsourcing high 923 papers that do not fit well in the same session authorsourcing medium 651
Preference Type Data Source Severity Totalpapers good in the same session authorsourcing N/A 805
Paper-related
54
Constraint Type Data Source Severity Encoded
chair's paper in own sessionsystem-
generated high -
chair's paper in opposing sessionssystem-
generated high -chair interested in opposing sessions authorsourcing medium 243 chair in a session with a bad fit chairs medium -
Preference Type Data Source Severity Totalchair fits well in the session chairs N/A -
Chair-related
55
Constraint Type Related to Data Source Severity Total Initial Final Change
author with papers in opposing sessions Session
system-generated high - 31 0 -31
topics of interest to a persona in opposing sessions Session
system-generated medium - 6 4 -2
papers of mutual interests in opposing sessions Paper authorsourcing high 923 40 19 -21
papers that do not fit well in the same session Paper authorsourcing medium 651 129 42 -87
chair's paper in own session Chairsystem-
generated high - 21 0 -21
chair's paper in opposing sessions Chair system-
generated high - 6 0 -6
chair interested in opposing sessions Chair authorsourcing medium 243 5 4 -1
chair in a session with a bad fit Chair chairs medium - 0 1 1
Total violated 238 70 -168
Preference Type Data Source Severity Total Initial Final Change
papers good in the same session Paper authorsourcing N/A 805 268 272 4
chair fits well in the session Chair chairs N/A - 90 78 -12
Total satisfied 358 350 -8
56
0
10
20
30
0 200 400 600 800
Conflict Count
Edit Number
author in opposing sessions
persona in opposing sessions
Resolve author conflicts
Switch rooms
Balance awards
Adjust session length
System-defined conflicts over time
57
0
30
60
90
120
0 200 400 600 800
Conflict Count
Edit Number
papers that do not fit well in the same session
papers of mutual interests in opposing sessions
Make coherent sessions
Balance awards
Adjust session length
Switch rooms
Community-defined conflicts over time
58
0
10
20
30
270 420 570 720
Conflict Count
Edit Number
chair's paper in own session
chair's paper in opposing sessions
chair interested in opposing sessions
chair in a session with a bad fit
Assign session chairs
59
60
61
Encoded constraints/preferences
62
Encoded constraints/preferences
63
Schedule table
64
Schedule table
65
Scratch space for unscheduled items
66
Scratch space for unscheduled items
67
Search and notification
68
Search and notification
69
70
Step 1. Locate a conflict to resolve
71
conflict preference
award duration
72
73
74
75
76
77
78
79
Consequence of a swap
80
81