2011 06-14 cristhian-parra_u_count
DESCRIPTION
Presentation about our community-driven approach for reputation eliciting and estimation, given at the Altmetrics Workshop, during WebSci Conference 2011 held in Koblenz, Germany.TRANSCRIPT
![Page 1: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/1.jpg)
UCount: A community-driven approach for measuring Scientific Reputation
Cristhian ParraUniversity of Trento, Italy
Altmetrics Workshop / websci2011
![Page 2: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/2.jpg)
Context
http://beta.kspaces.net/ic/ http://reseseval.org/http://liquidjournal.org/
![Page 3: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/3.jpg)
What is Scientific Reputation?
Scientific Reputation is the social evaluation (opinion) by the scientific community of a
researcher or its contributions given a certain criterion (scientific impact)
![Page 4: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/4.jpg)
Main Goal
How, Why ?
To understand the way reputation is formed within and across scientific communitiesunderstand
![Page 5: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/5.jpg)
[Insert footer]http://reseval.org/survey
DatasetTop H-Index (>200)
79 total Replies8 Online Surveys
ICWE (18)BPM (20)
VLDB (15)...
http://www.cs.ucla.edu/~palsberg/h-number.html
Experiment #1: LiquidReputation Surveys
![Page 6: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/6.jpg)
Results published in ISSI2011 and SEBD2011
1
23
# Publications (DBLP)
H-Index (Palsberg)
H-Index (Script)
Correlation Results
![Page 7: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/7.jpg)
MIUR* CNRS**
Researchers 664 >1000
Areas 12 45
Output Winner/Loser Pairs Selected ResearcherRanked Waiting Lists
# Rankings 333 pairs (with H-Index >= 0)208 pairs (with H-Index > 0)
196 Rankings of 5 researchers in average
(*) http://reclutamento.murst.it/ (**) http://intersection.dsi.cnrs.fr/intersection/resultats- cc- en.do
Experiment #2: Position Contests Analysis
![Page 8: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/8.jpg)
Results
• Surveys:– Correlation between bibliometric indicators and
reputation is always in the rank of (-0.5:0.5)• Research Position Contests– CNRS dataset: same result as in surveys– Italian dataset: around 50% of effectiveness in
predictions for all metrics
Bibliometrics are not a good describer of real reputation
![Page 9: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/9.jpg)
UCount Methodology
UCount Sci. ExcellenceUCount Reviewer Score
![Page 10: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/10.jpg)
UCount
Eliciting Reputation
Community oriented Surveys
Peer Review based assessment(Research Position Contests)
SurveysBeen there
Peer Review Feedback
![Page 11: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/11.jpg)
UCount Surveys
DBLP Coauthorship Graph
ICST
Palsberg
http://www.cs.ucla.edu/~palsberg/h-number.htmlhttp://icst.org/icst-transactions/
Editorial Boards
Top H Researchers
AffinityShortest Path +
Jaccard
List of Candidates
http://icst.org/UCount-Survey/
![Page 12: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/12.jpg)
UCount
Surveys Results
Derive Reputation Functions
Peer Review Feedback
UCount Scientific Impact
UCount Reviewer Score
![Page 13: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/13.jpg)
Reverse Engineering of Reputation
# Real reputation
5.5 Alon Halevy
4.7 Stefano Ceri
3.9 Tim Berners Lee
3 Jim Gray
# Estimated Rep.
5.1 Alon Halevy
4.2 Tim Berners Lee
3.7 Stefano Ceri
2.9 Jim Gray
Features
H-Index
Affiliation
Citations
Readership
Combine
Minimum Distance
Other Features?
![Page 14: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/14.jpg)
UCount
Surveys Results
Derive Reputation Functions
Peer Review Feedback
UCount Scientific Impact
UCount Reviewer Score
Community Reputation
Functions Library
![Page 15: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/15.jpg)
Reverse Engineering Approaches
• Decision Trees– No tree with more than 60% of accuracy
• Unsupervised Methods– Genetic algorithms applied on CNRS Dataset improved
correlation in an average of 15% (running only for 5 minutes)– Highly improved correlation for fields Research Management
and Politics. • Next
– Applying Machine Learning techniques– Explore other techniques (e.g. neural networks)– Obtain other types of features (e.g. keynotes, advisory
networks)– http://code.google.com/p/revengrep/ – https://github.com/cdparra/melquiades/
![Page 16: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/16.jpg)
Where are we now?
![Page 17: 2011 06-14 cristhian-parra_u_count](https://reader030.vdocument.in/reader030/viewer/2022020723/547dec8fb4af9fc3158b552d/html5/thumbnails/17.jpg)
Thanks!
Questions?
Ideas?