![Page 1: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/1.jpg)
EXYSTENCE Topical WorkshopTrust-Based Networks and Robustness in Organizations
ETH Zurich, March 15, 2006
Collective Evaluation: Practice and Theory
Yi-Cheng Zhang (Fribourg University, Switzerland)
![Page 2: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/2.jpg)
Collective Evaluation: Practice and Theory
Main Themes●New generation of Web Sites● From— People—For—People
Wikipedia(user generated encyclopedia)de l.icio.us (social bookmarks)
Ohmynews.com society digg.com reddit.com technocrati.com
flickr.comblogdex.com
SWITZERLAND
![Page 3: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/3.jpg)
Collective Evaluation: Practice and Theory
GoogleAmazon.comEpinions.comRatemyteachers.com
...professor.comEconomy ...doctor.com Ebay.com
ZAGAT Guides(restaurant) Dianping.com
Douban.com
.
SWITZERLAND
![Page 4: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/4.jpg)
Collective Evaluation: Practice and Theory
Successful Case impacting all of us:Google, Amazon.com, Ebay.com,
Wikipedia.org, Slashdot.org, World66.com....
(new “animals ” are added fast by the day,explicitly&implicitly using “New” info-Theory)
Google: (Page Rank) by S. Brin & L. Page and by Kleinberg( 96~98)
Hubs vs Authorities
SWITZERLAND
![Page 5: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/5.jpg)
Collective Evaluation: Practice and Theory
Old Model:newspapers, government, agency.
New Model:“Matchmaker site”
SWITZERLAND
![Page 6: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/6.jpg)
Collective Evaluation: Practice and Theory
Objects can be:
books, products, news, stories, tourist places, ideas, favorite actress/actors, music, scientists, teachers(!)
SWITZERLAND
![Page 7: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/7.jpg)
Collective Evaluation: Practice and Theory Advantages Disadvntages Old expertise corruption
easy control manipulation
New people are not noisy corrupted!!! contradictory distortion everyone is good at pseudo-experts something, sometimes laziness even better than experts Not reliable!!!
SWITZERLAND
![Page 8: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/8.jpg)
Collective Evaluation: Practice and Theory
Science may help?Objective: to get the best approximation possible to assure quality information.
1) pure democracy one-man-one-vote2) authority; people NO influence
our challenge: find the best balance of the two- distribution of weights
SWITZERLAND
![Page 9: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/9.jpg)
Collective Evaluation: Practice and Theory
Simple Model (XY):
SWITZERLAND
Q l l=1 ,2 , .... , M Intrinsic quality ( “God-Given)”
ql Approximate evaluation
X il i=1 ,2 , ... , Nl=1 ,2 , ... , M
“i”th expert's evaluationon object α
Y i “God -Given”judging capacity
![Page 10: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/10.jpg)
Naive democratic method:
“Tyranny of Majority”!
More refined approach:
Collective Evaluation: Practice and Theory
SWITZERLAND
ql=1N∑i=1
N
xil
ql=∑i=1
N
w i x il w i : Weight of “i”th expert
What weight to assign?
![Page 11: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/11.jpg)
Collective Evaluation: Practice and Theory
ql=∑i=1
N
w i x il
i2≈V i≡
1M∑
l=1
M
xil−ql2
SWITZERLAND
![Page 12: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/12.jpg)
Collective Evaluation: Practice and Theory
∑i=1
N 1V i
x il−ql =0 l=1 ,2 , ...M ;
∑l=1
M x il−q l2
V i=M i=1 ,2 , ...N
SWITZERLAND
Rewriting as:
![Page 13: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/13.jpg)
Collective Evaluation: Practice and Theory
SWITZERLAND
= 1M∑
l=1
M
ql−Q l 2
I p≡ 1[ pM ] ∑R=1
[ pM ]
nR/R
[ I p]0=[ pM ]1
2M p /2 M≫1
![Page 14: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/14.jpg)
Collective Evaluation: Practice and Theory
“New” Information Theory for the Internet Age
Classical Information Theory (Claude Shannon 1948)
Original message
SWITZERLAND
0 0 0 1 0 1 0 0 0 1 1 0 0 1
noise transmission
1 0 0 1 1 1 0 0 0 1 1 0 0
Error
Need of redundancy(large number repeats)
![Page 15: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/15.jpg)
Collective Evaluation: Practice and Theory
The goal:
SWITZERLAND
Use the least number of repeats to recover original message
Give a finite number of repeats, get the best approximation
![Page 16: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/16.jpg)
Collective Evaluation: Practice and Theory
Challenge of Noise
Classical Information Theory: natural noise
“New” Information Theory:natural noise
intentional noise herding noise
SWITZERLAND
![Page 17: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/17.jpg)
Collective Evaluation: Practice and Theory
When objects are too complex, need intermediate layers XY→XYZ
Amazon.com example
book 1 book 2.... ...... ....... ......book M
reviewson books
Readers
Reviews are easier than books to evaluateSWITZERLAND
![Page 18: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/18.jpg)
Collective Evaluation: Practice and Theory
Collaborative Filtering & Predictions
Challenge: “Movielens Project”
(Michigan University 1997-2000)
3 Millions data 2000 movies rated by 17'000 viewers
SWITZERLAND
![Page 19: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/19.jpg)
Collective Evaluation: Practice and Theory
example
Movies
123456
8
109
1 2 3 4 5 6 .. .. .. .. .. ... .. 20
√
√
√
√
√√
√
√
√
√√
√7
Viewers
√√
√
√
√
√
√
√
SWITZERLAND
![Page 20: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/20.jpg)
Collective Evaluation: Practice and Theory
SWITZERLAND
v i Viewer i votes on movie α
I, j = 1,2,...Nα, β =1,2,...M
v i=1 ,2 ,3 , 4 ,5 Stars
![Page 21: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/21.jpg)
Collective Evaluation: Practice and Theory
SWITZERLAND
ij=∑=1
M
vi v j
∑=1
M
∣vi∣∣v j ∣, ij∈−1 ,1
Person-to-person relationship:
d ij=1−ij/2Distance
![Page 22: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/22.jpg)
Collective Evaluation: Practice and Theory
=∑i=1
N
vi vi
∑i=1
N
∣vi∣∣vi ∣, ∈−1 ,1
d =1−/2
Movie-to-movie relationship:
Distance
SWITZERLAND
![Page 23: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/23.jpg)
Collective Evaluation: Practice and Theory
SWITZERLAND
2 23...
'=
1−
More Refined Approach
![Page 24: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/24.jpg)
Collective Evaluation: Practice and Theory
SWITZERLAND
=∑ij∑S i
∣V test∣
≃0.83 0.9
Predictions
![Page 25: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/25.jpg)
Collective Evaluation: Practice and Theory
Theory Group for“Interdisciplinary Physics”
Post.doc. Paolo Laureti (Roma) Ph.D. Marcel Blattner (Zürich)
Ph.D. Lionel Moret (EPFL)Post.doc. Hasan Masum (Ottava, Canada)
SWITZERLAND
Fribourg Members:
![Page 26: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/26.jpg)
Collective Evaluation: Practice and Theory
SWITZERLAND
External Members:
Yi-kuo Yu (NIH, Washington)Damien Challet (Oxford)Matteo Marsili (ICTP; Italy)Sergei Maslov (BNL, New York)
![Page 27: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/27.jpg)
Collective Evaluation: Practice and Theory
Thank You very much for your attention!
SWITZERLAND
![Page 28: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/28.jpg)
Workshop: Internet @nd Information Filtering Mechanism
SWITZERLAND
![Page 29: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/29.jpg)
Workshop: Internet @nd Information Filtering Mecanisme
SWITZERLAND
![Page 30: Collective Evaluation: Practice and Theoryweb.sg.ethz.ch/workshops/TW_Trust/Download/Zhang.pdf · Collective Evaluation: Practice and Theory Main Themes New generation of Web Sites](https://reader035.vdocument.in/reader035/viewer/2022081400/5f0dadfa7e708231d43b8c8c/html5/thumbnails/30.jpg)
Workshop: Internet @nd Information Filtering Mechanism
much harder... but our task is modest:
best approximation,can not dream about decoding “God-Given”
SWITZERLAND