systematic mapping studies - uni koblenz-landaumarcel heinz systematic mapping studies 23. juli 2014...

Systematic Mapping Studies

Marcel Heinz

23. Juli 2014

Marcel Heinz Systematic Mapping Studies 23. Juli 2014 1 / 44

Presentation Overview

1 Motivation

2 Systematic Mapping Studies

3 Comparison to Systematic Reviews

4 Guidelines


Studies in Software Engineering

Task: Get an overview of a certain research area and how far it’scovered in research.Approach: Study the research field by using methods frominformation retrieval and statistical analysis.


Systematic Review : History

Creating systematic reviews is an established method in evidencebased medicine.Common in the field of software engineering.1

1Kitchenham & Charters 2007Marcel Heinz Systematic Mapping Studies 23. Juli 2014 4 / 44

Systematic Review : Goal

Go through existing primary reportsReview the primary reportsDescribe their methodology and results


Systematic Mapping Studies : History2

Systematic Mapping Studies are an established method inevidence based medicine.Even though it seems to be faithful there, it gets neglected a lotin the field of software engineering.Less common in the field of software engineering thansystematic reviews.

2Petersen, Kai, et al. ”Systematic mapping studies in software engineering.” 12thInternational Conference on Evaluation and Assessment in Software Engineering. Vol.17. 2008.


Systematic Mapping Studies : Goals

Build a classification scheme and structure a field of interest.Structure of the type of research and results by categorizing afield.Show frequencies of publications for categories in the scheme.Determine coverage in a certain field.Combine the results to answer more specific research questions.Provide a visual summary by mapping the results.In general it tries to provide a more coarsed grained overview


Systematic Mapping Studies: Example

Ex V. Elberzhager, Frank, Jurgen Munch, and Vi Tran Ngoc Nha. ”Asystematic mapping study on the combination of static anddynamic quality assurance techniques.” Information and SoftwareTechnology 54.1 (2012): 1-15.


Systematic Mapping Studies: Example Context

There exist various quality assurance techniques.Most of these are often applied in isolation.But a combination of various quality assurance techniquespromises to exploit synergy effects.


Systematic Mapping Studies: Example Objective

Classification and thematic analysis of existing approaches,which try to exploit a combination.Include reported effects, characteristics and constraints.Result is an overview of existing approaches and a suitable basisfor identifying future research directions.


Systematic Mapping Studies: Process



1 Define the research question⇒ Review Scope

2 Conduct a search⇒ All Papers

3 Screen the papers⇒ Relevant Papers

4 Assign keywords to each papers by using the abstract⇒ Classification scheme

5 Data extracting and mapping process⇒ Systematic map


Task 1: Define the research questions

Find out, what you want to accomplishFind out, where you want to search for your informations.⇒ Identify forums for research areas.


Task 1: Example part 1

Nr. Question RationaleRQ1 What are existing approaches that

combine static and dynamic qua-lity assurance techniques and howcan they be classified?

The first research question definesthe basis of this systematic map-ping study and provides an over-view of the existing approachesthat combine static and dynamicquality assurance techniques.

RQ2 In which sources and in whichyears were approaches regardingthe combination of static and dy-namic quality assurance techni-ques published?

The second research question in-dicates whether there are specificpublication channels and when ef-fort regarding this research areawas made.



Nr. Question RationaleRQ3 Is any kind of evidence presented

with respect to the combination ofquality assurance techniques and ifso, which kind of evidence is gi-ven?

The third research question showswhether the approaches were em-pirically evaluated or whether justinitial ideas are presented. This in-formation was used to evaluate thematurity of the approaches.

RQ4 What are the objectives of combi-ned quality assurance approaches?

The fourth research question pro-vides detailed information whatthe purpose of each approach isand what is addressed and impro-ved when applying a combined ap-proach.



Nr. Question RationaleRQ5

Which static and dynamic quali-ty assurance techniques are usedin combined quality assurance ap-proaches?

The fifth research question pres-ents the concrete static and dyna-mic QA techniques that are com-bined.

RQ6 Which input is used for static anddynamic quality assurance techni-ques in combined quality assuran-ce approaches?

The sixth research question gi-ves information about the data orinformation needed to apply thecombined approach, with respectto both static and dynamic QAtechniques.

⇒ Four reference databases: Inspec, Compendex, IEEE Xplore, andACM Digital Library


Task 2: Search for primary studies

Use information retrieval methodsbrowse manually in journals etc.Use search strings in scientific databases

search-string creation by structuring in terms of population,intervention, comparison and outcomesearch-string driven by research questionsearch-string from each aspect of the structure

If one only considers certain types of studies, the overview willbecome biased and the result map is incomplete


Task 2: Example

Search-String: (inspection or review or “static analysis” or“static quality assurance”) AND (test* or “dynamic qualityassurance” or “dynamic analysis”) AND software AND (combin*or integrat* or synergy or “trade off”)Was applied to check keyword, title, and abstract fields withinthe corresponding databases.


Task 3: Screening of papers

Define inclusion and exclusion criteriaThe criteria should be derived from the research questions.


Task 3: Example


Task 4: Keywording of abstracts

Reviewers read abstracts.Look for keywords and conceptsReflect the contribution of a paperIdentify contexts


Task 4: Keywording of abstracts

Put all keywords together from all found papersDevelop higher level view on the researchHelps with definining categories representing the underlying setsof papersWhen abstracts don’t contain enough information, theintroduction and conclusion are reviewed as well.The final set of keywords is used to cluster and form categories


Task 4: Building the classification scheme


Task 4: Example


Task 5: Data Extraction and Mapping of studies

Frequencies can be derived from a final classification tableThis shows the latest research focus and possible research forthe future.Different facet combinations are possible→ e.g. Context facet + research facet or contribution facet


Comparison to Systematic Reviews

Discussion !What are your thoughts on a comparison?


Comparison - Overview

The methods are different in terms of goals, breadth, validityissues and implications.They should be used complementarily.


Comparing the goals

SR SMS- Focus on establishing thestate of evidence

- Focus on classification, the-matic analysis and identifyingpublication fora

- Mostly used to identify bestpractises based on empiricalevidence- Shows where evidence ismissing or where it’s insuffi-cient

- Can’t show, that evidence ismissing or insufficient

- Identify research gaps - Identify research gaps


Comparing the process

SR SMS- Quality is evaluated - Quality is not evaluated- Meta analysis - Thematic analysis

⇒ Both require a different level of data extraction.


Comparing breadth and depth

SR SMS- States outcome and qualityas its major focus

- Reflects based on searchstrings and inclusion criteria

- Increased depth and effort - Covers a higher breadth.More articles can be covered

- Fewer studies can be inclu-ded

- Can structure a larger field


Comparing classification

SR SMS- Mentions lack of metho-dological precision in primarystudies3

- Doesn’t restrict itself tosuch small portions

- A bias might be introducedby a SR

- Overview is more complete

-More fine grained categoriesare possible

-High level categories

-Especially concerning e.g.research methods and rese-arch approaches

3Mendes2005Marcel Heinz Systematic Mapping Studies 23. Juli 2014 40 / 44

Comparing validity

A major problemSome terms might be used in different meanings. (e.g.: ”experiment”)

SR SMS-Takes details into account.The thread of false classifica-tion is minimized.

- Doesn’t go into details,which might lead to wrongclassification.


Comparing industrial accessibility and relevance

BackgroundWe want to give a good introduction to a field.

SR SMS- It’s more difficult to accessresults

- Easier to spark interest

- Results might be too detai-led, though details might beimportant to practicioners.- The visual appeal should bechanged

- It is probably visually moreappealing


User’s guide

Complementary use is helpfulTake an adaptive reading depth for classificationAlso Classify papers based on evidence and noveltyVisualize your data


Summary

1 Motivation

2 Systematic Mapping Studies

3 Comparison to Systematic Reviews

4 Guidelines


systematic mapping studies - uni koblenz-landaumarcel heinz systematic mapping studies 23. juli 2014...

Documents