[width=4cm]/home/kfort/documents/cours/coursparis4/images/zlbanniere.png defying ... · 2020. 12....
TRANSCRIPT
![Page 1: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/1.jpg)
defying complexity (lessons learned)
Karen Fort & Bruno Guillaume
[email protected] / [email protected]
October, 2016
1 / 31
![Page 2: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/2.jpg)
1 Overview of the game
2 Motivating players
3 Behind the curtain
4 Obtained results [Guillaume et al., 2016]
5 Conclusion and future plans
2 / 31
![Page 3: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/3.jpg)
Overview of the game
1 Overview of the gameDependency syntax annotationZombiLingo
2 Motivating players
3 Behind the curtain
4 Obtained results [Guillaume et al., 2016]
5 Conclusion and future plans
3 / 31
![Page 4: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/4.jpg)
Overview of the game Dependency syntax annotation
A complex annotation type
annotation guidelines:I 29 relation typesI approx. 50 pages
counter-intuitive decisions
→ decompose the complexity of the task [Fort et al., 2012],not simplify it!
4 / 31
![Page 6: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/6.jpg)
Overview of the game ZombiLingo
6 / 31
![Page 7: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/7.jpg)
Overview of the game ZombiLingo
7 / 31
![Page 8: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/8.jpg)
Overview of the game ZombiLingo
8 / 31
![Page 9: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/9.jpg)
Motivating players
1 Overview of the game
2 Motivating playersAttracting playersKeeping players playing
3 Behind the curtain
4 Obtained results [Guillaume et al., 2016]
5 Conclusion and future plans
9 / 31
![Page 10: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/10.jpg)
Motivating players Attracting players
General features
Bring the fun through:
zombie design
use of (crazy) objects
regular challenges (specific corpus and design) on a trendy topic:I Star Wars (when the movie was playing)I soccer (during the Euro)I Pokemon (well...)
10 / 31
![Page 11: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/11.jpg)
Motivating players Keeping players playing
LeaderboardS (for achievers)
Criteria:
number of annotations or points
in total, during the month, during the challenge11 / 31
![Page 12: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/12.jpg)
Motivating players Keeping players playing
Hidden features (for explorers)
appearing randomly
with different effects: objects, other game, etc.
12 / 31
![Page 13: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/13.jpg)
Motivating players Keeping players playing
Duels (for socializers (and killers?))
select an enemy
challenge them on a specific type of relation
13 / 31
![Page 14: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/14.jpg)
Motivating players Keeping players playing
Badges (?) (for collectors)
play all the sentences for a relation type, for a corpus
play all the sentences from a corpus
14 / 31
![Page 15: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/15.jpg)
Behind the curtain
1 Overview of the game
2 Motivating players
3 Behind the curtainPreprocessingEnsuring quality
4 Obtained results [Guillaume et al., 2016]
5 Conclusion and future plans
15 / 31
![Page 16: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/16.jpg)
Behind the curtain Preprocessing
Preprocessing data (freely available corpora)
Pre-annotation with two parsers:
1 a statistical parser : Talismane [Urieli, 2013]
2 a symbolic parser, based on graph rewriting :FrDep-Parse [Guillaume and Perrier, 2015]
→ play the items for which the two parsers give different annotations
16 / 31
![Page 17: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/17.jpg)
Behind the curtain Ensuring quality
Training, control and evaluationReference: 3,099 sentences of the Sequoia corpus [Candito and Seddah, 2012]
REFTrain&Control REFEval Unused
50% 25% 25%1,549 sentences 776 sentences 774 sentences
REFTrain&Control is used to train the players
REFEval is used like a raw corpus, to evaluate the producedannotations
17 / 31
![Page 18: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/18.jpg)
Behind the curtain Ensuring quality
Training the playersCompulsory for each dependency relation
sentences are taken from the REFTrain&Controlcorpus
a feedback is given in case of error
18 / 31
![Page 19: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/19.jpg)
Behind the curtain Ensuring quality
Dealing with cognitive fatigue and long-term playersControl mechanism
Sentences from the REFTrain&Controlcorpus are proposed regularly:
if the player fails to find the right answer, a feedback with thesolution is given
after a given number of failures on the same relation, the playercannot play anymore and has to redo the corresponding training
→ we deduce a level of confidence for the player on this relation
19 / 31
![Page 20: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/20.jpg)
Obtained results [Guillaume et al., 2016]
1 Overview of the game
2 Motivating players
3 Behind the curtain
4 Obtained results [Guillaume et al., 2016]QuantityQuality
5 Conclusion and future plans
20 / 31
![Page 21: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/21.jpg)
Obtained results [Guillaume et al., 2016] Quantity
Production: game corpus sizecompared to other existing French dependency syntax corpora
As of July 10, 2016:
647 players
who produced 107,719 annotations
Sequoia 7.0 UD-French 1.3 FTB-UC FTB-SPMRL Game
Sentences 3,099 16,448 12,351 18,535 5,221
Tokens 67,038 401,960 350,947 557,149 128,046Tokens/sent. 21.6 24.4 28.4 30.1 24.5
21 / 31
![Page 22: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/22.jpg)
Obtained results [Guillaume et al., 2016] Quantity
Production: game corpus sizecompared to other existing French dependency syntax corpora
As of July 10, 2016:
647 players
who produced 107,719 annotations
Sequoia 7.0 UD-French 1.3 FTB-UC FTB-SPMRL Gamefree free not free not free free
Sentences 3,099 16,448 12,351 18,535 5,221
Tokens 67,038 401,960 350,947 557,149 128,046Tokens/sent. 21.6 24.4 28.4 30.1 24.5
22 / 31
![Page 23: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/23.jpg)
Obtained results [Guillaume et al., 2016] Quantity
Production: game corpus sizecompared to other existing French dependency syntax corpora
As of July 10, 2016:
647 players
who produced 107,719 annotations
Sequoia 7.0 UD-French 1.3 FTB-UC FTB-SPMRL Gamefree free not free not free free
validated errors validated validated validated
Sentences 3,099 16,448 12,351 18,535 5,221
Tokens 67,038 401,960 350,947 557,149 128,046Tokens/sent. 21.6 24.4 28.4 30.1 24.5
+ (ever)growing resource!
23 / 31
![Page 24: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/24.jpg)
Obtained results [Guillaume et al., 2016] Quality
Evaluating qualityon the REFEval corpus
aux.tps
sujaux.pass
aff detobj.cpl
aobj
mod.rel
dep.coord
obj.pats
pobj.o
deobj
coord
objm
od
0
0.5
1
F-m
easu
re
Talismane FrDep-Parse Game
NB: left part of the figure = density of annotation > 124 / 31
![Page 25: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/25.jpg)
Obtained results [Guillaume et al., 2016] Quality
Annotation densityon the REFEval corpus
aux.tps
sujaux.pass
aff detobj.cpl
aobj
mod.rel
dep.coord
obj.pats
pobj.o
deobj
coord
objm
od
0
2
4
nu
mb
erof
answ
ers
per
ann
otat
ion
→ need more annotations on some relations
25 / 31
![Page 26: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/26.jpg)
Conclusion and future plans
1 Overview of the game
2 Motivating players
3 Behind the curtain
4 Obtained results [Guillaume et al., 2016]
5 Conclusion and future plans
26 / 31
![Page 27: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/27.jpg)
Conclusion and future plans
Improving gamification
Give more to:
explore and collect
build a real story
build a sense of community
27 / 31
![Page 28: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/28.jpg)
Conclusion and future plans
Improving the exported resource
Test the influence of:
the pre-annotation score
the level of the player in the game
the confidence we have in the player for the relation type at hand
28 / 31
![Page 29: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/29.jpg)
Conclusion and future plans
Expand to new languagesand new annotation types
New languages:
English
less-resourced languages
New annotation types:
POS,
corpus gathering, etc.
Alice Millour (PhD student)
29 / 31
![Page 30: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/30.jpg)
Conclusion and future plans
Building a Community
GWAPs for research should form a network, to:
attract more players,
share them,
share the burden of communication
30 / 31
![Page 31: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/31.jpg)
Conclusion and future plans
Thanks!
Nicolas Lefebvre (engineer)
31 / 31
![Page 32: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/32.jpg)
Bibliographie
Candito, M. and Seddah, D. (2012).Le corpus Sequoia : annotation syntaxique et exploitation pourl’adaptation d’analyseur par pont lexical.In Proceedings of the Traitement Automatique des Langues Naturelles(TALN), Grenoble, France.
Fort, K., Nazarenko, A., and Rosset, S. (2012).Modeling the complexity of manual annotation tasks: a grid ofanalysis.In International Conference on Computational Linguistics (COLING),pages 895–910, Mumbai, India.
Guillaume, B., Fort, K., and Lefebvre, N. (2016).Crowdsourcing complex language resources: Playing to annotatedependency syntax.In Proceedings of the International Conference on ComputationalLinguistics (COLING), Osaka, Japan.
Guillaume, B. and Perrier, G. (2015).
![Page 33: [width=4cm]/home/kfort/Documents/Cours/CoursParis4/Images/ZLBanniere.png defying ... · 2020. 12. 2. · Motivating players Attracting players. General features. Bring the fun through:](https://reader035.vdocument.in/reader035/viewer/2022071507/6127ab58577998271d0a2404/html5/thumbnails/33.jpg)
Bibliographie
Dependency Parsing with Graph Rewriting.InProceedings of IWPT 2015, 14th International Conference on Parsing Technologies,pages 30–39, Bilbao, Spain.
Urieli, A. (2013).Robust French syntax analysis: reconciling statistical methods andlinguistic knowledge in the Talismane toolkit.PhD thesis, Universite de Toulouse II le Mirail, France.