An Open Mashup Platform ForResearch Impact Evaluation

By: Muhammad Imran

EXISTING SOLUTIONS:

Web of Science

Publish or Perish ToolDBLP

• Funding of research projects• Govt. policy for future investment

External EntitiesGovernment agencies/funding

organizations

• Faculty recruitment and promotionUniversity Management

Management, including committees,

• Institutes, department and individual evaluation

University DepartmentsInstitutional research,

academic affairs, tech transfer, etc.

• Finding experts in some specific areaIndividuals Faculty, Staff, Students

PRIMARY USERS OF RESEARCH EVALUATION

Different Things

Different Purposes

Different Data

Sources

Different Metrics

Different Algorithms

•Individual•Groups•Artifacts•Universities•Countries…

•Funding•Future Policy•Finding Expert•Hiring•Promotions…

•Google Scholar•DBLP•Scopus•ACM•User’s DS…

•H-Index•G-Index•C-Index•N-Index…

•Diff aggregation•Diff normalization•Self-Citation•Filters…

DIMENSIONS IN RESEARCH EVALUATION

Research Evaluation

Different Things

Different Purposes

Different Data

Sources

Different Metrics

Different Algorithms

DIMENSIONS INRESEARCH EVALUATION

HOW IS RESEARCH EVALUATION DONE?

• Available Data– Number of and value of Grants awarded – Number of awards (e.g. Nobel Prizes)– Number of patents– Number of post-graduate researchers– Publication counts– Citation counts – Bibliometric metrics– Voting – Reputation, Reviews, Rating (Liquid Journal)– And many more …

• Peer Evaluation– Expensive, time consuming

• Individuals• Groups of researchers• Artifacts (Books, Papers, Articles etc)• Institutions• Geographical units (City, Region, Country)• …

UNIT OF EVALUATION

Complex Metric Computation LogicDiversity Autonomy Variability

Show h-index of Bob, excluding self-citations and the 3 top co-authors contributing to his h-index.

How many different authors contributing to Bob’s h-index

Given a group of people that is considering to hire a person, how diverse is this person from the group

PROPOSED SOLUTIONPutting everything together

User

Mashup Magic

User Input Data Presentation

DataData

Common platform for various data sources

Personalized metrics support

Capability to define complex logic

Evaluate individual, groups …

Resource Space Management System

MASHUP UI

Mashup Engine

ResEval Components

ResEval Domain Model

ResEval Mashup

Language

Mashup Exec Logic

ARCHITECTURE

ResEval Mashup 1 (the vision)

ResEval Mashup 2

ResEval Mashup 3

CONCLUSION & FUTURE WORK

• We proposed architecture for– Personalized metrics definition– Resource-Oriented – Provide Impact evaluation for individual & groups

• Future work– Mashup platform for complex metric logic– Components as web service

We know we don’t have a corner on creativity. There are creative people all around the world, hundreds of millions of them. And they are going to think of things to do with our basic platform that we didn’t think of. So the mashups stuff is a wonderful way of allowing people to find new ways of applying the basic infrastructures we propagating. -Vint Cerf

Thank you.