Analysis of scientific research

Mario SangiorgioGiordano Tamburrelli

The origin of this work

Carlo Ghezzi’s keynote:Reflections on 40+ years of

software engineering research and beyond: an

insider’s view

Analysis based on papers

Lack of tools to perform the

analysis

WHATresearch topics

WHOcontributors

HOW/WHENtrends

The origin of this work

Time consuming Boring

Requires an expert

Lack of tools to perform the

analysis

Automatic analysis

Faster

ScalableGeneral method

One-click(After

training)

Feasible with data mining techniques

BUTstill not perfect

(it is not semantic-based)

Steps of the analysis

Identificationof subtopics

Interpretation ofpaper content

Trend analysis(So far)

CLUSTERING

CLASSIFICATION

CLUSTERING

STATISTICS

Clustering

ClusteringHierarchical Expectation

Maximization Algorithm

The tool used is Crossbow

Thanks to Gianluca Staffiero and Gabriele Valentini

Abstracts of papers from both general and specificconferences and journals

The clustering process

Classification

Classification

Bayesian classifier

Ad hoc tool using Mallet

Analysis based on the abstract of the papers

Result evaluation

Clustering was iterated until the results were good

Classification performs well:high precision and recall values

human expert agrees with the classifier

Outcomes

Research analysistrends on main

conferences and journals

Tools to support research

automatic bidding

Some trends found

Data from IEEE Transactions on Software Engineering

Some trends found

Data from IEEE Transactions on Software Engineering

Automatic bidding

Build upon analysis methodologies and results

Bidding processGrouping the

submissions by topicCreation of a profilefor the reviewers

Matching papers’ topicwith reviewers’ interests

CLASSIFICATION

CLASSIFICATION

SELECTION

Grouping the submissions

Creation of the reviewer profile

Matching profiles and submissions

Result evaluation

ICSM 2010

Reviewers’ profiles

Carlo GhezziProfile:

web-servicesformal methods

middleware for distributed systemsmodels

software componentseducationCONFI

RMED

Harald GallProfile:

software miningmiddleware for distributed

systemsmodels

empirical studies

Do you agree?

Comparison with actual bids

Results apparently not so good: recall it is about 53%

BUT

The actual bid is not an oracle

We are suggesting papers for the most

relevant topics

Live Testing: ICSE 2011

Propose our bids to the reviewers

Get a feedback on our suggestions, based on reviewer impressions

Future worksImprovement of the system

Ranking of the suggested papers

Deeper statistical analysis

Paper assignment based onGenetic Algorithms assignment