Methods for SE Research

Design ScienceTomi Männistö

This material is licensed under the Creative Commons BY-NC-SA License

Background • Sciences of the artificial • Constructive research • Design Science (DS)

Framework for DS • Framework, process, guidelines

Evaluation • Shaw on SE research • Design Evaluation methods • Evaluation of artefact vs. utility

Towards Theory • About theory, theory types • Technological rule

Conclusions

OUTLINE FOR TODAY

Tomi Männistö

BACKGROUND

Tomi Männistö

Herbert A. Simon (1996): Sciences of the Artificial

■ ”Historically and traditionally, it has been the task of the science disciplines to teach about natural things: how they are and how they work.

■ It has been the task of engineering schools to teach about artificial things: how to make artifacts that have desired properties and how to design.

■ Engineering, medicine, business, architecture, and painting are concerned not with the necessary but with the contingent – not with how things are but how they might be...”

Tomi Männistö

(Järvinen 2004)

Research stressing what is reality

Mathematical

Research approaches

Conceptual analytical

Theorytesting

Theory creating

Research stressing utility of artifacts

Approaches studying reality

Approaches for empirical studies

Artifacts-building

Artifacts-evaluating

Tomi Männistö

Categories of scientific disciplines (van Aken 2004)

■ Formal sciences ■ E.g., philosophy and mathematics ■ Empirically ”void”, main test internal logical consistency

■ Explanatory sciences ■ E.g., natural sciences and major section of social sciences ■ Describe, explain and possibly predict observable phenomena within its field

■ Design sciences ■ Engineering sciences, medical sciences and modern psychotherapy ■ Solve construction problems or improvement problems

Tomi Männistö

Design science, definition

The design-science paradigm seeks to extend the boundaries of

human and organizational capabilities by creating new and innovative artifacts

(Hevner et al. 2004)

Tomi Männistö

Sample artefacts (DS artefacts?)

■ Ontology, conceptualisation, modelling language ■ Feature modelling language

■ Algorithm ■ Constraint solving algorithm

■ Method ■ Variability management method, code inspection method

■ Prototype tool ■ Modeller, sw component database, OS license checker

■ Information system (prototype) ■ Product data management system, sales configurator

Tomi Männistö

Tomi Männistö

Problem Study Evaluation

Scientific knowledge (previous work)

Context(industry, …)

Research Questions

Results

Publication

Understanding

Conceptualisation

Scientific knowledge (related work)

Scientific knowledge (methodological

literature)

New knowledge

Novelty

Relevance

Research approach Existing

knowledge

Context(industry, …)

Applicability,Utility

New knowledge

Study Design

FRAMEWORK

Tomi Männistö

Phases (Kasanen et al. 1993)

1. Find a practically relevant problem which also has research potential. 2. Obtain a general and comprehensive understanding of the topic. 3. Innovate, i.e., construct a solution idea. 4. Demonstrate that the solution works. 5. Show the theoretical connections and the research contribution of the

solution concept. 6. Examine the scope of applicability of the solution.

Tomi Männistö

DesignScienceResearchMethodology Process model (Peffers et al. 2008)

Tomi Männistö

(Hevner et al. 2004) Tomi Männistö

(Hevner et al. 2004) Tomi Männistö

Design Science Research Cycles (Hevner, 2007)

Tomi Männistö

(Vesiluoma 2009) KS = Knowledge sharing (in SE)KSF = KS Framework Tomi Männistö

(Tiihonen 2014) Tomi Männistö

EVALUATION

Tomi Männistö

(Hevner et al. 2004) Tomi Männistö

Building blocks for research (Shaw@ICSE01)

Tomi Männistö

Possible plans (Shaw@ICSE01)

§A common plan §Method - System - Evaluation

§Can X be done better? - Build Y - Measure Y, Compare to X

§Common, but bad §Method - Technique - Persuasion

§Can X be done better? - Devise a technique - "Look it works!!"

§Other good plan §Feasibility - System - Persuasion

§Can X be done at all? - Build Y that does X - "Look it works!!"

§Sometimes breakthrough sometimes nonsense §Change basic assumptions -

Formulate new approach - Argue carefully on merits

Validation (Shaw@ICSE01)

■ Persuasion ■ I thought hard about

this, and I believe... ■ Early architectural

models ■ Implementation

■ Here is a prototype of a system that...

■ Early architecture description languages

■ Evaluation ■ Given these criteria,

the object rates as... ■ Taxonomies;

performance improvement

■ Analysis ■ Given the facts, here

are consequences... ■ Many predictive

models ■ Formal model

■ Rigorous verification and proof

■ Empirical model ■ Data on use in

controlled situation ■ User interface

structures

■ Experience ■ Report on use in

practice ■ Case studies ■ Qualitative model

■ Narrative ■ Architectural patterns

■ Decision criteria ■ Comparison of

systems in actual use ■ Domain-specific

architectures ■ Empirical model

■ Data, usually statistical, on practice

■ Communication and project complexity

Tomi Männistö

On evaluation in Design Science

IT artifacts can be evaluated in terms of functionality, completeness, consistency, accuracy, performance, reliability, usability, fit with the organization, and other relevant quality attributes.

(Hevner et al. 2004)

Evaluating utility Utility refers to stakeholders having a value or satisfaction in

the achievement of a desired goal.

Tomi Männistö

TOWARDS THEORY

Tomi Männistö

About theory (Gregor 2006)

aim to describe, explain, and enhance understanding of the world and,

At a general level theories are abstract entities that

in some cases, to provide predictions of what will happen in the future and to give a basis for intervention and action.

Tomi Männistö

Theory Types in Information Systems (Gregor 2006)

Tomi Männistö

About theory (Gregor 2006)

Prescription A special case of prediction exists where the theory provides a description of the method or structure or both for the construction of an artifact (akin to a recipe). The provision of the recipe implies that the recipe, if acted upon, will cause an artifact of a certain type to come into being.

Tomi Männistö

Components of Theory in Information Systems (Gregor and Jones 2007)

Tomi Männistö

(Myllärniemi 2015)Tomi Männistö

(Myllärniemi 2015)Tomi Männistö

G. H. von Wright's concept of technical norm, which is a factual statement about the relation between means and ends:

If you want to make a hut habitable, you ought to heat it.

(Niiniluoto 1993)

If you want B, and you believe that you are in a situation A, then you ought to do X.

Weaker forms:It is rational for you to do X.It is profitable for you to do X.

Tomi Männistö

Technological rule (van Aken 2004)

■ A chunk of general knowledge, linking an intervention or artefact with a desired outcome or performance in a certain field of application

■ Real breakthrough when tested technological rules could be grounded on scientific knowledge

■ Typically studies within its intended context of application, in order to, be as sure as possible of its effectiveness ■ Testing by originator (α), testing by others (β)

Tomi Männistö

75

In company A, the number of product line requirements has been a problem. Complicated decisions must be made regarding the extent of the variability provided in the product line when studying hundreds, if not thousands, of requirements. In real life, this is impossible. During Study I, the number of requirements was reduced by selecting the important high-level requirements of company A. In Study VI the technique T4 was created to make this happen systematically. The author (A) tested the technique with real product line requirements from the mobile phone domain in company C. Although the technique works in practice using real requirements, it has not yet been deployed for daily use. This is probably because the technique is most useful when creating a new product line, not during the evolution of an existing product line, which is the current case in company C.

Table 14 Summary of the validation case studies and application domains

Id Technique Study Application domain

Active stakeholdersa

Market test

T1 Defining independent product

prioritisations and deductive properties

for product line requirements set

I,II Weather station A, GP Weak market test

Automotive systems

A, GP Works in practice

T2 Assigning declarative assumptions to the

product line requirements set

II Weather station A, GP Weak

market test

T3 Checking consistency of the product line

requirements model

II Weather station A, GP Weak market test

Mobile phones A, GP Works in practice

T4 Identifying key requirements and

checking requirements

differentiation against deductive properties

VI Mobile phones A

Works in practice

T5 Mapping variability in requirements to

variability in features

IV Mobile phones A Works in practice

T6 Modelling feature dependencies

III Automotive systems

A, GP, UP, IP Weak market test

T7 Checking feature dependencies

V Mobile Phones A Weak market test

a Active stakeholders are categorised as: the author (A), a guided party (GP), an unguided party (UP), and an independent party (IP).

(Savolainen 2011)Tomi Männistö

CONCLUSIONSIf you’re constructing something as a part of your research work,

Design Science provides a good guidelines and framework – So, take a look!

Solution-oriented, engineering by nature, means searched for getting out some theory

Rationale – Artefact – Evaluation (Utility) – Theory (Prescription)

Tomi Männistö

ReferencesGregor. 2006. The nature of theory in information systems. Management Information Systems Quarterly.30(3):611–642. Gregor S & Jones D. 2007. The anatomy of a design theory. Journal of the Association for Information Systems, 8(5):312–335. Hevner AR, March ST, Park J, Ram S. 2004. Design Science in Information Systems Research. MIS Quarterly, 28(1). Hevner, AR. 2007. A Three Cycle View of Design Science Research. Scandinavian Journal of Information Systems, 19(2):87–92. Järvinen P. 2004. On research methods. Tampere: Tampereen yliopistopaino. Järvinen P. 2004. Research Questions Guiding Selection of an Appropriate Research Method, University of Tampere. Kasanen E, Lukka K, Siitonen A. The Constructive Approach in Management Accounting Research. Journal of Management Accounting Research. 1993 Fall;5(Fall):243-64. Myllärniemi V. 2015. Quality Attribute Variability in Software Product Lines – Varying Performance and Security Purposefully, PhD Thesis, Aalto University, Finland. Peffers, K et al. 2008. A Design Science Research Methodology for Information Systems Research. Journal of Management Information Systems, 24(3):45–77. Savolainen J. 2011. Product Line Management Techniques with Requirement and Feature Models, PhD Thesis, Aalto University, Finland. Shaw M. 2001- The Coming-of-Age of Software Architecture Research. Proceedings of the International Conference on Software Engineering, ICSE2001. Tiihonen, J. 2014. Support for configuration of physical products and services. PhD Thesis, Aalto University, Helsinki, Finland. van Aken JE. Management Research Based on the Paradigm of the Design Sciences: The Quest for Field-Tested and Grounded Technological Rules. Journal of Management Studies. 2004 March;41(2):219-46. Vesiluoma S. 2009. Understanding and Supporting Knowledge Sharing in Software Engineering, PhD Thesis, Tampere University of Technology, Finland.

Tomi Männistö