liquidpub: services at service of science
Post on 10-May-2015
537 Views
Preview:
DESCRIPTION
TRANSCRIPT
LiquidPub: Services at the service of
scienceMarcos Baez, Fabio Casati , and the LiquidPub
teamDepartment of Information Engineering and Computer Science
University of Trento
ECOWS 2009 – Eindhoven, The Netherlands – 10 Nov 2009
In a nutshell: the goal
Capture the lessons learned and opportunities provided by the Web and open source, agile development to develop concepts, models, metrics, and tools for an efficient (for people), effective (for science), and sustainable (for publishers and the community) way of creating, disseminating, evaluating, and consuming scientific knowledge. Understand what’s good for science, and make it happen
project.liquidpub.org
•Publish and perish: why the current publication and review model is killing
research and wasting your money (ACM Ubiquity 8(3), Feb 2007), and
•Liquid Publications: Scientific Publications meet the Web
With Fausto Giunchiglia and Maurizio Marchese.
Objective
Understand
Dissemination models and
overhead
Evaluation processes, and
quality/impact of research
(people, papers, projects)
Peer review and
innovation
Scientific communities
Funding models and
processes
Improve
Better ways to do the
same things
Different things
Principles
Models
IT services
project.liquidpub.org 3
Understanding
project.liquidpub.org 4
PR: Initial Goals
Understand how well peer review works
Understand how to improve the process
Metrics + Analysis
(refer to liquid doc)
Focus only on gatekeeping aspect
“Not everything that can be counted counts,and not everything that counts can
be counted.” -- Albert Einstein
Comparing rankings
28
17
2
45
67
..
..
89
33
..
33
89
2
17
67
..
28
..
..
45
Comparing rankings
T=3
N=10
Divergence: Div (t,N)
Kendall τ
8
Results: peer review ranking vs. citation
count
9
Divergence
Div
Normalized t
Quality-related Metrics: Statistics
11
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0 1 2 3 4 5 6 7 8 9 10
Pro
ba
bili
ty
Marks
Distribution of marks (integer marks)
Also: (-3,3) very different from (1,7)
Disagreement vs number of reviews
1
1.5
2
2.5
R1 R2 R3+
Fairness
Definition: A review process is fair if and only of
the acceptance of a contribution does not
depend on the particular set of PC members
that reviews it
The key is in the assignment of a paper to
reviewers: A paper assignment is unfair if the
specific assignment influences (makes more
predictable) the fate of the paper.
The road ahead
About citations…
project.liquidpub.org
16
Mikalai Krapivin, Maurizio Marchese, Fabio Casati: Exploring and
Understanding Scientific Metrics in Citation Networks. Complex 2009
17
Positio
nal eff
ect on c
itation a
nd r
eaders
hip
in a
rXiv
, by H
aque
and G
insparg
project.liquidpub.org Fabio Casati - ECOWS 2009 18
Positio
nal eff
ect on c
itation a
nd r
eaders
hip
in a
rXiv
, by H
aque
and G
insparg
Improving
Principles
Models
IT Services
project.liquidpub.org Fabio Casati - ECOWS 2009 19
Principles and objectives
Everything counts!! (not just papers, not just ―innovations‖)
Minimal dissemination overhead
Early sharing
Early feedback/interaction
Find diversity
Interestingness and sharing as measures of reputation
No gatekeeping. Use the filtering power of the community
We are not necessarily right!! Let the community select the principles and models
Model:
Liquid Journals
Liquid Journals
project.liquidpub.org Fabio Casati - ECOWS 2009 22
project.liquidpub.org Fabio Casati - ECOWS 2009 23
project.liquidpub.org Fabio Casati - ECOWS 2009 24
Data
Internet
project.liquidpub.org Fabio Casati - ECOWS 2009 25
Data
project.liquidpub.org Fabio Casati - ECOWS 2009 26
3. What you can do with it
project.liquidpub.org Fabio Casati - ECOWS 2009 27
2. How it gets filled…
Can be edited, but can be (semi-)automated
Topics and tags (+ tags by specific people)
Subscriptions/preferences
Journals—editing power of the community!!
Authors
Wisdom of people you trust (as editor)
Wisdom of the crowd (if any)
Diversity!!!
Results
Everybody becomes a journal editor (selfishly)
No gatekeeping, quality enforced by the community (gatekeeping is the noise)
You get things you want to read (or you want your group to read)
You get diversity
You leverage the selection work of your friends
Measure interestingness and quality by sharing
Reduce (optimize) dissemination overhead and encourage early sharing
project.liquidpub.org Fabio Casati - ECOWS 2009 29
IT Services: LiquidPubinfrastructure
project.liquidpub.org Fabio Casati - ECOWS 2009 30
Engineer a system that
Accesses heterogeneous src w/ no API
Is modular so that functions can be reused
In a world where modularity had little success
Facilitates the creation of arbitrary dissemination and evaluation models (by non-programmers)
Provides commonly needed research services
Implements liquid journals (agile – our main macro-story)
Simple and lightweight
project.liquidpub.org Fabio Casati - ECOWS 2009 31
Access
Cached metadata
Liquid Journals dashboard
Diversity Caching
Search computing
Crawling
Reputation
Sharing
Search by topic
Personalized tagging
Disambiguation
Liquid journals API
project.liquidpub.org Fabio Casati - ECOWS 2009 33
Research servicesA
P
I
Karaku client-code
KarakuREST API
REST
AP
IR
EST
AP
I
ResMan
REST API
Adapters
UI
Cached SKOs
Crawling
Cache mgmt
Subscriptions/stream
Disambiguation
Personalized taggingLiquidity
Rendering
Karaku: Uniform access
Handling semantically-equivalent resources with the same set of actions and interface
Interoperability at the type level
The semantics are defined by the type
Resource ActionsResource Manager
Resource type Action typeResource
Manager type
1..N1..11..1 1..N
Google Docs Documents in Google Docs GDocs-specific Operations
Repositories Documents (CRUD)
1..N 1..N 1..N 1..N
Abstraction
Vertical services
project.liquidpub.org Fabio Casati - ECOWS 2009 35
Community discovery
Search computing
ResevalTopicSearch
Gelee
Mashart
Liquid journals Liquid books …
Co
llab
ora
tive
p
rogr
amm
ing
[Insert header here] [Insert footer]
Gelee
[Insert header here] [Insert footer]
Paper at ICDE 09. Video avail online and demo at icsoc
Mashart
project.liquidpub.org 38
Paper at ER‘09, demo at icsoc ‗09
project.liquidpub.org 39
RESEVAL http://demo.liquidpub.org/reseval/ complex’09
Group Management
Group Management
Compare the metrics of people in
group.
Total citations
Total publications
H-index
G-index
Noise ratio
comparison
―Search comparison‖ shows all the comparison made.
We can see and recompute them (for updated data).
Communities
project.liquidpub.org 43
TELETEACHING/HUM_INT (chi,hicss)
AI/DB (icai,aaai)
ROBOTIC/M.MEDIA (icra,icpr)
TELECOM (icc,globecom)
APPLIED COMPUTING/CRYPTO(sac,compsac)
SOFTWARE ENGINEERING(kbse,icse)
DIST. SYSTEM/COMPILER(ipps,iccS)
GENETIC AND EVO ALG(cec,gecco)HUMMAN –COMP INTER(icchp,hci)
Overview of the Community Network
Community Comparison and normalization
0
5
10
15
20
25
30
35
40
45
50
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61
h-i
nd
ex
[icc- globecom]
[ijcai- aaai]
[icra- icpr]
[iscas- date]
[kbse- icse]
Author
Create Journal
You haven’t created any LJ yet.
Start sharing scientific content!U
Hi Alex!
Journal Name
Description
Editorial board
Untitled1 General Info
2Sources to overlay
3 Process
NextCancel
Peer-review journal
Journal that groups the relevant
Information about peer-review
Joe
Alex Joe
Hi Alex!
Peer-review journal1 General Info
2Sources to overlay
3 Process
NextCancel
To be included in your journal
peer review
papers
blogs
Peer review, an overviewPeter, Pablo, Pedro
The world of scientif ic publications has been largely…
Rethinking peer reviewJoe
Exploring new ideas for peer review in physics
Anonymized review dataJuan, Alejandro
Review data f rom 10 conferences in biology
(0) LJ references(4) citations
Peer review, scientific publications
Peer review, open access
Springer
Arxiv
I WANT
• papers, • blogs,
• datasets
TALKING ABOUT
peer-review
FROMSpringer
Arxiv
UPDATE RATE
weekly
(1) LJ references(3) citations
Contributions
dataset
wikis
Edit relations | Advanced filters
Peer review,
Arxiv(1) LJ references(1) citations
Rank by citations
[none]
[Carl’s LJ]
[Liquidpub LJ]
Sources
Springer
ArxivScopus
Scholar
Test
Hi Alex!
Peer-review journal1 General Info
2Sources to overlay
3 Process
DoneCancel
Traditional journal processModel for traditional journals
Publication via voting “Democratic model” for agreement
Total agreement Full agreement model
Hi Alex!
General InfoCreated on 11 Apr 9.00amOwner AlexSubscribers 0
Editors
Issues[no issues yet]
Latests activities
Journal Created11 Apr 9.00am
Latests subscribers
Create Issue
Home
ContentIssues
Stats
Settings
We are still learning from you.
Soon you’ll get our suggestions :)U
No suggestions from your friendsU
Settings
Settings
Disable
Disable
Peer-review journal
Peer review, an overviewPeter, Pablo, Pedro
The world of scientif ic publications has …
Rethinking peer reviewJoe
Exploring new ideas for peer review in …
Anonymized review dataJuan, Alejandro
Review data f rom 10 conferences in …
(0) LJ references(4) citations
Peer review, scientific publications,
Peer review, open access
Springer
Arxiv(1) LJ references(3) citations
Peer review,
Arxiv(1) LJ references(1) citations
[none]
[Carl’s LJ]
[Liquidpub LJ]
Alex
Joe
[no subscribers yet]
Update Now
Hi Alex!
Editors
Issues[no issues yet]
Latests subscribers
Create Issue
Home
ContentIssues
Stats
Settings
Peer-review journal
Peer review, an overviewPeter, Pablo, Pedro
The world of scientif ic publications has …
Rethinking peer reviewJoe
Exploring new ideas for peer review in …
Anonymized review dataJuan, Alejandro
Review data f rom 10 conferences in …
(0) LJ references(4) citations
Peer review, scientific publications,
Peer review, open access
Springer
Arxiv(1) LJ references(3) citations
Peer review,
Arxiv(1) LJ references(1) citations
[none]
[Carl’s LJ]
[Liquidpub LJ]
Alex
Joe
[no subscribers yet]
Add Contribution Citations
Popular Tags
Peer reviewScientific publicationsOpen access
Hi Alex!
Latests activities
Joe joined the LJ11 Apr 9.02am
Journal Created by Alex11 Apr 9.00am
Editors
Issues
Create Issue
Home
ContentIssues
Stats
Settings
Alex
Joe
Peer-review journal » Issue on new review models
Peer review, an overviewPeter, Pablo, Pedro
The world of scientif ic publications has …
Rethinking peer reviewJoe
Exploring new ideas for peer review in …
Benefits of open peer reviewCharles
This paper illustrates the benef its of …
Peer review, past and futureRonald, James
This paper illustrates the benef its of …
(0) LJ references(4) citations
Peer review, scientific publications,
Peer review, open access
Springer
Arxiv(1) LJ references(3) citations
[none]
[Carl’s LJ]
Published
by Alex
by Alex
Hi Alex!
Issue on new
review models(4)
Peer review, open access
(0) LJ references(2) citations
by Joe
Arxiv[none]
Peer review
Scopus (0) LJ references(1) citations
by Alex
[none]
Latests subscribers
[no subscribers yet]
Popular Tags
Peer reviewScientific publicationsOpen access
Latests activities
“Issue on new review models” moved to the Publication phase, by Joe
11 Apr 9.09am(1) Contributions added to the Issue on new review models” by Alex
11 Apr 9.08am“Issue on new review models” moved to the Selection phase, by Alex
11 Apr 9.07am
Take-Home Message
Flaws of current practices (or, lack of evidence
that they work as expected)
Research services for novel dissemination model
Principles, models, composable IT services in a
restricted domain
Liquid journals: Interestingness, reward
innovation, sharing, everything contributes, use
the filtering power of the community
project.liquidpub.org Fabio Casati - ECOWS 2009 53
Google liquidpub-announce and subscribe if
you are interested (VERY low traffic)
First release at the end of the month
project.liquidpub.org Fabio Casati - ECOWS 2009 54
How to collaborate
Liquid journals server
http://demo.liquidpub.org/liquidjournals/
Liquid journals dashboard
http://demo.liquidpub.org/liquidjournals/web/workspace/
Liquid journals SVN
https://dev.liquidpub.org/svn/liquidpub/prototype/liquidjournals/
How to collaborate
http://project.liquidpub.org/karaku/
http://project.liquidpub.org/resman/
project.liquidpub.org Fabio Casati - ECOWS 2009 56
57
top related