liquidpub: services at service of science

LiquidPub: Services at the service of

scienceMarcos Baez, Fabio Casati , and the LiquidPub

teamDepartment of Information Engineering and Computer Science

University of Trento

ECOWS 2009 – Eindhoven, The Netherlands – 10 Nov 2009

In a nutshell: the goal

Capture the lessons learned and opportunities provided by the Web and open source, agile development to develop concepts, models, metrics, and tools for an efficient (for people), effective (for science), and sustainable (for publishers and the community) way of creating, disseminating, evaluating, and consuming scientific knowledge. Understand what’s good for science, and make it happen

project.liquidpub.org

•Publish and perish: why the current publication and review model is killing

research and wasting your money (ACM Ubiquity 8(3), Feb 2007), and

•Liquid Publications: Scientific Publications meet the Web

With Fausto Giunchiglia and Maurizio Marchese.

Objective

Understand

Dissemination models and

overhead

Evaluation processes, and

quality/impact of research

(people, papers, projects)

Peer review and

innovation

Scientific communities

Funding models and

processes

Improve

Better ways to do the

same things

Different things

Principles

Models

IT services

project.liquidpub.org 3

Understanding


PR: Initial Goals

Understand how well peer review works

Understand how to improve the process

Metrics + Analysis

(refer to liquid doc)

Focus only on gatekeeping aspect

“Not everything that can be counted counts,and not everything that counts can

be counted.” -- Albert Einstein

Comparing rankings

28

17

2

45

67

..

..

89

33

..

33

89

2

17

67

..

28

..

..

45

Comparing rankings

T=3

N=10

Divergence: Div (t,N)

Kendall τ

Results: peer review ranking vs. citation

count

9

Divergence

Div

Normalized t

Quality-related Metrics: Statistics

11

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18

0 1 2 3 4 5 6 7 8 9 10

Pro

ba

bili

ty

Marks

Distribution of marks (integer marks)

Also: (-3,3) very different from (1,7)

Disagreement vs number of reviews

1

1.5

2

2.5

R1 R2 R3+

Fairness

Definition: A review process is fair if and only of

the acceptance of a contribution does not

depend on the particular set of PC members

that reviews it

The key is in the assignment of a paper to

reviewers: A paper assignment is unfair if the

specific assignment influences (makes more

predictable) the fate of the paper.

The road ahead

About citations…

project.liquidpub.org

16

Mikalai Krapivin, Maurizio Marchese, Fabio Casati: Exploring and

Understanding Scientific Metrics in Citation Networks. Complex 2009

17

Positio

nal eff

ect on c

itation a

nd r

eaders

hip

in a

rXiv

, by H

aque

and G

insparg

project.liquidpub.org Fabio Casati - ECOWS 2009 18

Positio

nal eff

ect on c

itation a

nd r

eaders

hip

in a

rXiv

, by H

aque

and G

insparg

Improving

Principles

Models

IT Services


Principles and objectives

Everything counts!! (not just papers, not just ―innovations‖)

Minimal dissemination overhead

Early sharing

Early feedback/interaction

Find diversity

Interestingness and sharing as measures of reputation

No gatekeeping. Use the filtering power of the community

We are not necessarily right!! Let the community select the principles and models

Model:

Liquid Journals

Liquid Journals



Data

Internet


Data

3. What you can do with it


2. How it gets filled…

Can be edited, but can be (semi-)automated

Topics and tags (+ tags by specific people)

Subscriptions/preferences

Journals—editing power of the community!!

Authors

Wisdom of people you trust (as editor)

Wisdom of the crowd (if any)

Diversity!!!

Results

Everybody becomes a journal editor (selfishly)

No gatekeeping, quality enforced by the community (gatekeeping is the noise)

You get things you want to read (or you want your group to read)

You get diversity

You leverage the selection work of your friends

Measure interestingness and quality by sharing

Reduce (optimize) dissemination overhead and encourage early sharing


IT Services: LiquidPubinfrastructure


Engineer a system that

Accesses heterogeneous src w/ no API

Is modular so that functions can be reused

In a world where modularity had little success

Facilitates the creation of arbitrary dissemination and evaluation models (by non-programmers)

Provides commonly needed research services

Implements liquid journals (agile – our main macro-story)

Simple and lightweight


Access

Cached metadata

Liquid Journals dashboard

Diversity Caching

Search computing

Crawling

Reputation

Sharing

Search by topic

Personalized tagging

Disambiguation

Liquid journals API


Research servicesA

P

I

Karaku client-code

KarakuREST API

REST

AP

IR

EST

AP

I

ResMan

REST API

Adapters

UI

Cached SKOs

Crawling

Cache mgmt

Subscriptions/stream

Disambiguation

Personalized taggingLiquidity

Rendering

Karaku: Uniform access

Handling semantically-equivalent resources with the same set of actions and interface

Interoperability at the type level

The semantics are defined by the type

Resource ActionsResource Manager

Resource type Action typeResource

Manager type

1..N1..11..1 1..N

Google Docs Documents in Google Docs GDocs-specific Operations

Repositories Documents (CRUD)

1..N 1..N 1..N 1..N

Abstraction

Vertical services


Community discovery

Search computing

ResevalTopicSearch

Gelee

Mashart

Liquid journals Liquid books …

Co

llab

ora

tive

p

rogr

amm

ing

[Insert header here] [Insert footer]

Gelee

[Insert header here] [Insert footer]

Paper at ICDE 09. Video avail online and demo at icsoc

Mashart


Paper at ER‘09, demo at icsoc ‗09


RESEVAL http://demo.liquidpub.org/reseval/ complex’09

http://demo.liquidpub.org/reseval/

Group Management

Group Management

Compare the metrics of people in

group.

Total citations

Total publications

H-index

G-index

Noise ratio

comparison

―Search comparison‖ shows all the comparison made.

We can see and recompute them (for updated data).

Communities


TELETEACHING/HUM_INT (chi,hicss)

AI/DB (icai,aaai)

ROBOTIC/M.MEDIA (icra,icpr)

TELECOM (icc,globecom)

APPLIED COMPUTING/CRYPTO(sac,compsac)

SOFTWARE ENGINEERING(kbse,icse)

DIST. SYSTEM/COMPILER(ipps,iccS)

GENETIC AND EVO ALG(cec,gecco)HUMMAN –COMP INTER(icchp,hci)

Overview of the Community Network

Community Comparison and normalization

0

5

10

15

20

25

30

35

40

45

50

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

h-i

nd

ex

[icc- globecom]

[ijcai- aaai]

[icra- icpr]

[iscas- date]

[kbse- icse]

Author

Create Journal

You haven’t created any LJ yet.

Start sharing scientific content!U

Hi Alex!

Journal Name

Description

Editorial board

Untitled1 General Info

2Sources to overlay

3 Process

NextCancel

Peer-review journal

Journal that groups the relevant

Information about peer-review

Joe

Alex Joe

Hi Alex!

Peer-review journal1 General Info

2Sources to overlay

3 Process

NextCancel

To be included in your journal

peer review

papers

blogs

Peer review, an overviewPeter, Pablo, Pedro

The world of scientif ic publications has been largely…

Rethinking peer reviewJoe

Exploring new ideas for peer review in physics

Anonymized review dataJuan, Alejandro

Review data f rom 10 conferences in biology

(0) LJ references(4) citations

Peer review, scientific publications

Peer review, open access

Springer

Arxiv

I WANT

• papers, • blogs,

• datasets

TALKING ABOUT

peer-review

FROMSpringer

Arxiv

UPDATE RATE

weekly


Contributions

dataset

wikis

Edit relations | Advanced filters

Peer review,

Arxiv(1) LJ references(1) citations

Rank by citations

[none]

[Carl’s LJ]

[Liquidpub LJ]

Sources

Springer

ArxivScopus

Scholar

Test

Hi Alex!

Peer-review journal1 General Info

2Sources to overlay

3 Process

DoneCancel

Traditional journal processModel for traditional journals

Publication via voting “Democratic model” for agreement

Total agreement Full agreement model

Hi Alex!

General InfoCreated on 11 Apr 9.00amOwner AlexSubscribers 0

Editors

Issues[no issues yet]

Latests activities

Journal Created11 Apr 9.00am

Latests subscribers

Create Issue

Home

ContentIssues

Stats

Settings

We are still learning from you.

Soon you’ll get our suggestions :)U

No suggestions from your friendsU

Settings

Settings

Disable

Disable

Peer-review journal


The world of scientif ic publications has …


Exploring new ideas for peer review in …


Review data f rom 10 conferences in …


Peer review, scientific publications,


Springer


Peer review,


[none]

[Carl’s LJ]

[Liquidpub LJ]

Alex

Joe

[no subscribers yet]

Update Now

Hi Alex!

Editors

Issues[no issues yet]

Latests subscribers

Create Issue

Home

ContentIssues

Stats

Settings

Peer-review journal






Review data f rom 10 conferences in …




Springer


Peer review,


[none]

[Carl’s LJ]

[Liquidpub LJ]

Alex

Joe


Add Contribution Citations

Popular Tags

Peer reviewScientific publicationsOpen access

Hi Alex!

Latests activities

Joe joined the LJ11 Apr 9.02am

Journal Created by Alex11 Apr 9.00am

Editors

Issues

Create Issue

Home

ContentIssues

Stats

Settings

Alex

Joe

Peer-review journal » Issue on new review models





Benefits of open peer reviewCharles

This paper illustrates the benef its of …

Peer review, past and futureRonald, James

This paper illustrates the benef its of …




Springer


[none]

[Carl’s LJ]

Published

by Alex

by Alex

Hi Alex!

Issue on new

review models(4)



by Joe

Arxiv[none]

Peer review

Scopus (0) LJ references(1) citations

by Alex

[none]

Latests subscribers


Popular Tags

Peer reviewScientific publicationsOpen access

Latests activities

“Issue on new review models” moved to the Publication phase, by Joe

11 Apr 9.09am(1) Contributions added to the Issue on new review models” by Alex

11 Apr 9.08am“Issue on new review models” moved to the Selection phase, by Alex

11 Apr 9.07am

Take-Home Message

Flaws of current practices (or, lack of evidence

that they work as expected)

Research services for novel dissemination model

Principles, models, composable IT services in a

restricted domain

Liquid journals: Interestingness, reward

innovation, sharing, everything contributes, use

the filtering power of the community


Google liquidpub-announce and subscribe if

you are interested (VERY low traffic)

First release at the end of the month


How to collaborate

Liquid journals server

http://demo.liquidpub.org/liquidjournals/

Liquid journals dashboard

http://demo.liquidpub.org/liquidjournals/web/workspace/

Liquid journals SVN

https://dev.liquidpub.org/svn/liquidpub/prototype/liquidjournals/

http://demo.liquidpub.org/liquidjournals/





How to collaborate

http://project.liquidpub.org/karaku/

http://project.liquidpub.org/resman/


http://project.liquidpub.org/karaku/

http://project.liquidpub.org/resman/

liquidpub: services at service of science

Education

orgfabio casati ecows

community gatekeeping

evaluation models

liquid journals agile

research people

scientific metrics

models innovation

review process