CC alijavam at Flickr

Theory Building Attempts in Software Engineering PER RUNESON, LUND UNIVERSITY

Definition “A theory provides explanations and understanding in terms of basic concepts and underlying mechanisms, which constitute an important counterpart to knowledge of passing trends and their manifestation”.

Hannay et al. [6]

Systematic Literature Review 113 experiment 1993–2002 40 theories in 23 articles Only two of the theories were used in more than one article!

A Systematic Review of Theory Usein Software Engineering ExperimentsJo E. Hannay, Dag I.K. Sjøberg, Member, IEEE, and Tore Dyba, Member, IEEE

Abstract—Empirically based theories are generally perceived as foundational to science. However, in many disciplines, the nature,

role and even the necessity of theories remain matters for debate, particularly in young or practical disciplines such as software

engineering. This article reports a systematic review of the explicit use of theory in a comprehensive set of 103 articles reportingexperiments, from of a total of 5,453 articles published in major software engineering journals and conferences in the decade 1993-

2002. Of the 103 articles, 24 use a total of 40 theories in various ways to explain the cause-effect relationship(s) under investigation.The majority of these use theory in the experimental design to justify research questions and hypotheses, some use theory to provide

post hoc explanations of their results, and a few test or modify theory. A third of the theories are proposed by authors of the reviewedarticles. The interdisciplinary nature of the theories used is greater than that of research in software engineering in general. We found

that theory use and awareness of theoretical issues are present, but that theory-driven research is, as yet, not a major issue inempirical software engineering. Several articles comment explicitly on the lack of relevant theory. We call for an increased awareness

of the potential benefits of involving theory, when feasible. To support software engineering researchers who wish to use theory, weshow which of the reviewed articles on which topics use which theories for what purposes, as well as details of the theories’

characteristics.

Index Terms—Theory, experiments, research methodology, empirical software engineering.

Ç

1 INTRODUCTION

THERE are many arguments in favor of theory use.Theories offer common conceptual frameworks that

allow the structuring of knowledge in a concise and precisemanner, thus facilitating the communication of ideas andknowledge. Their level of abstraction enables the general-ization of knowledge independently of a specific time andplace [5], [66], [91], [95], [98]. Theory is the means throughwhich one may generalize analytically [85], [103], thusenabling generalization from situations in which statisticalgeneralization is not desirable or possible, such as from casestudies [103], across populations [64], and indeed, oftenfrom experiments, especially those in social and behavioralsciences [85], with which experiments in empirical softwareengineering share essential features.

Such arguments have been voiced in the softwareengineering community as well, e.g., [6], [27], [43], [57],[89]. A theory provides explanations and understanding interms of basic concepts and underlying mechanisms, whichconstitute an important counterpart to knowledge ofpassing trends and their manifestations. When developingbetter software engineering technology for long-livedindustrial needs, building theories is a means to go beyond

the mere observation of phenomena and to try to under-stand why and how these phenomena occur.

Thus, theories are of potential use to both researchersand practitioners. However, the usefulness of theories forsoftware engineering is a subject of discussion, and theactual use of theory in empirical studies of softwareengineering is not well known. Decisions and discussionsregarding issues of theory in empirical software engineer-ing must be founded not only on an understanding of whata theory is and how it can be useful, but also on knowledgeof the actual use of theories. The main motivation of thisarticle is to contribute to the latter by reviewing the use ofexplanatory theories for software engineering.

An importantundertaking in empirical software engineer-ing is to determine what development technology to deployandwhatdevelopers touse inwhat situations; inotherwords,interest lies in comparing various technologies and skills andin determining their effects on software development. Forthis, the experiment is one choice of researchmethod in that itmay be applied more or less directly to make suchcomparisons and to measure such effects.

According to Shadish et al., experiments are suitable forcausal description, that is, “describing the consequencesattributable to deliberately varying a treatment” [85, p. 9],but do not in themselves provide causal explanation, that is,“clarifying the mechanisms through which and the condi-tions under which [the cause-effect relationship] holds” [85,p. 9]. The desire for causal explanation motivates thedevelopment of theory. Thus, the focus of this review is ontheories that are used to explain, in one way or another, thecause-effect relationships investigated by experiments.

A challenge facing software engineering researchers whoare considering using theories to support their research, is

IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, VOL. 33, NO. 2, FEBRUARY 2007 87

. J.E. Hannay and D.I.K. Sjøberg are with Simula Research Laboratory,Department of Software Engineering, Pb. 134, NO-1325 Lysaker, Norway.E-mail: {johannay, dagsj}@simula.no.

. T. Dyba is with Simula Research Laboratory and with SINTEF ICT,Department of Software Engineering, Safety, and Security, NO-7465Trondheim, Norway. E-mail: tore.dyba@sintef.no.

Manuscript received 2 June 2006; revised 6 Sept. 2006; accepted 3 Nov. 2006;published online 28 Dec. 2006.Recommended for acceptance by D. Rombach.For information on obtaining reprints of this article, please send e-mail to:tse@computer.org, and reference IEEECS Log Number TSE-0124-0606.

0098-5589/07/$25.00 ! 2007 IEEE Published by the IEEE Computer Society

Endres and Rombach

•  50 ‘laws’ – a repeatable phenomenon in a natural sciences context.

•  Many are general management •  In their notion, theories explain the

‘laws’, hypotheses propose a tentative explanation for the phenomenon, while a conjecture is a guess about the phenomenon.

•  25 hypotheses and 12 conjectures appeared in the software engineering literature.

‘giving the reader aproven basis forengineering complexsoftware systems’

Empirical Observations, Laws and Theories

Empirical Observations, Laws and Theories

Empirical O

bservations, Laws and Theories

Albert Endres Dieter Rombach

EndresRombach

Albert Endres & Dieter Rombach

A Handbook of Softwareand Systems Engineering

The Fraunhofer IESE Series on Software EngineeringThe Fraunhofer IESE Series on Software Engineering

Computers are the most pervasive tools of modern society. Their deployment relies onadvanced methods of software and systems engineering. Based on repeated and consistentobservations, key lessons of these fields can now be formulated into rules or even laws,providing initial building blocks towards a theoretical foundation that is essential for furtherresearch, for teaching and for the practice of software development.

Intended as a handbook for students and professionals alike, this book is the first to identifyand discuss such rules and laws. They are largely independent of technologies, and thus forma basis for the principles underlying software and systems engineering. Software and systemengineers should be aware of this proven body of knowledge, to ensure professionalism anddue diligence in their work.

The book is structured around the software development lifecycle. It begins with requirementsdefinition and goes on to maintenance and withdrawal. In different process models, thesetasks have different importance or are applied in a different sequence, or even cyclically. Thebook provides the reader with:

• clear statements of software and systems engineering laws and their applicability,

• empirical evidence that proves the usefulness of the material covered,

• unique knowledge to apply in an industrial setting.

Albert Endres had a long and distinguished career in industrial software development withIBM Germany, with responsibilities for compiler, operating system, and software toolsdevelopment. He is an honorary professor in computer science at the University of Stuttgart,Germany; and was full professor of computer science at the Technical University of Munich,Germany from 1993 to 1997. He was the founding chairman of the German chapter of theACM, is a senior member of IEEE and a member of the German Informatics Society (GI).

Dieter Rombach holds the Chair of Software Engineering in the department of computerscience at the University of Kaiserslautern, Germany, and he is also the Executive Director ofthe Fraunhofer Institute for Experimental Software Engineering (IESE). Together, these twoorganizations form one of the largest centers for Software Engineering research in Europe. Hewas formerly a faculty member of the department of computer science at the University ofMaryland at College Park, MD, USA. There, he was a Project Leader in the SoftwareEngineering Laboratory, a joint venture between NASA Goddard Space Flight Center, theComputer Sciences Corporation, and the University of Maryland. He is an IEEE Fellow and amember of ACM as well as of the German Informatics Society (GI).

A Handbook of Softwareand Systems Engineering

A H

andbook of Softw

areand

System

s Engineering

www.pearsoneduc.com

Zendler

•  A “preliminary software engineering theory”,

–  3 fundamental hypotheses,

–  6 central hypotheses, and –  4 elementary hypotheses.

An example theory is “object-oriented programming techniques have advantages against structured programming techniques”.

! "#$%&'&()#* +,-./)#$ 0(1&($$#&(1 23$,#*)4 5(6$4.&1).$7 8* "98%&43$7 0:;$#&'$(.4

!"#$%!& '%"#(%$ )*+,-.-/012310!"#$%&'#() *+ ,*('-./0 1"'(#(2(% *+ 3*/42(%& 56#%"6%'0 787879 ,*('-./0 :/ ;%2%" ,.<.#' 7=0 >%&/.")

!84.#)<.= 450 6*20786.+869* 9: 79:8;+<0 0*.6*00<6*. 80=5*6>?07 @A ?76*. 850 0BC0<6-0*8 7087 ?C 850167=6CD6*0 9: 0BC0<6-0*8+D 79:8;+<0 0*.6*00<6*.3 450 0E0=8620*077 +*1 0F=60*=A 9: 79:8;+<0 0*.6*00<6*.80=5*6>?07 5+20 @00* 78?1601 6* 0BC0<6-0*87 +*1 =9*:<9*801 ;685 0-C6<6=+D 1+8+3 450 <07?D87 9: 850700BC0<6-0*87 +<0 C?@D67501 6* + 1620<768A 9: G9?<*+D7 +*1 C<9=0016*. C+C0<73 H9;020<I 850<0 67 *98 +*920<260; 79 :+< ;56=5 <0C<070*87 850 +2+6D+@D0 <07?D87 7A780-+86=+DDA3 JA 8+K6*. 850 <07?D87 :<9- C?@D675010BC0<6-0*87 85+8 10+D ;685 +*+DA767I 1076.*I 6-CD0-0*8+869*I 8078I -+6*80*+*=0I >?+D68A +77?<+*=0I +*1<0?70 80=5*6>?07 + C<0D6-6*+<A 79:8;+<0 0*.6*00<6*. 8509<A 67 1020D9C013 L<9- 8567 8509<AI :<?68:?DC<9@D0-7I 7?..07869*7 :9< .+850<6*. *0; 1+8+I +*1 0*86<0DA *0; D6*07 9: 6*20786.+869* +<0 101?=013

>$*/,#74? %BC0<6-0*8+D 79:8;+<0 0*.6*00<6*.I 8509<A 9: 79:8;+<0 0*.6*00<6*.3

@= 5(.#,79<.&,(

&9:8;+<0 0*.6*00<6*. 67 =9*=0<*01 ;685 80=5*6>?07 89 1020D9CI <?*I +*1 -+6*8+6* +C/CD6=+869* 7A780-73 %-C6<6=+D 79:8;+<0 0*.6*00<6*. 67 850 7?@167=6CD6*0 9: 79:8;+<0 0*/.6*00<6*. 85+8 6*20786.+80 79:8;+<0 0*.6*00<6*. 80=5*6>?07 @A ?76*. 0-C6<6=+D -085917M=+70 78?1AI 6*>?6<AI +*1 0BC0<6-0*83 %-C6<6=+D 79:8;+<0 0*.6*00<6*. 5+7 @00* C?7501@A 6*80<*+869*+DDA <0=9.*6)01 <070+<=5 6*7868?807 7?=5 +7 &9:8;+<0 %*.6*00<6*. N*786/8?80 O&%NPI %-C6<6=+D L9?*1+869*7 9: Q9-C?80< &=60*=0 O%L9Q&PI %-C6<6=+D &9:8;+<0%*.6*00<6*. (+@9<+89<A O%&%(PI N*7868?80 9: %BC0<6-0*8+D &9:8;+<0 %*.6*00<6*.ON%&%PI Q0*8<0 :9< !12+*=01 %-C6<6=+D &9:8;+<0 $070+<=5 OQ!%&!$PI %-C6<6=+DN*:9<-+86=7 $070+<=5 O%N$PI +*1 "!&!R7 &9:8;+<0 %*.6*00<6*. (+@9<+89<A O&%(P3!* 6-C9<8+*8 <9D0 8+K07 N*80<*+869*+D &9:8;+<0 %*.6*00<6*. $070+<=5 "08;9<K ON&/%$"PI ;56=5 67 + ?*69* 9: -9<0 85+* ST <070+<=5 6*7868?8073%BC0<6-0*8+D 79:8;+<0 0*.6*00<6*. 67 850 7?@167=6CD6*0 9: 0-C6<6=+D 79:8;+<0 0*.6/

*00<6*. 85+8 ?707 850 0BC0<6-0*8 89 2+D61+80I 6-C<920I +*1 70D0=8 79:8;+<0 0*.6*00<6*.80=5*6>?07 ;56=5 C0<-68 +* 0F=60*8 +CCD6=+869* 1020D9C-0*83 %BC0<6-0*87 5+20 @00*=9*1?=801 89 6*20786.+80 +*+DA767I 1076.*I 6-CD0-0*8+869*I 80786*.I -+6*80*+*=0I>?+D68A +77?<+*=0I +*1 <0?70 80=5*6>?073 45070 0BC0<6-0*87 5+20 @00* C?@D67501 6*16E0<0*8 G9?<*+D7 +*1 C<9=0016*. C+C0<73 ! =9-C<050*7620 920<260; 85+8 C<070*87 850<07?D87 9: 850 79:8;+<0 0*.6*00<6*. 0BC0<6-0*87 +*1 <07?D87 67 -6776*.3 U9<0920<I +7A780-+86= =9*80B8 67 *98 +2+6D+@D0 85+8 .?6107 850 70D0=869* 9: ;5+8 89 0BC0<6-0*8 *0B83450 C?<C970 9: 8567 C+C0< 67 89 1020D9C + 7A780-+86= =9*80B8V+ C<0D6-6*+<A

79:8;+<0 0*.6*00<6*. 8509<AV:9< 850 <07?D87 9: 79:8;+<0 0*.6*00<6*. 0BC0<6-0*8785+8 +DD9; 89 +7K :9< *0; 79:8;+<0 0*.6*00<6*. C<9@D0-7I 89 =9DD0=8 :?<850< 1+8+ 89

%-C6<6=+D &9:8;+<0 %*.6*00<6*.I WI XWXYXZTI STTX3! STTX [D?;0< !=+10-6= \?@D6750<73 U+*?:+=8?<01 6* 450 "0850<D+*173

%-C6<6=+D &9:8;+<0 %*.6*00<6*.I WI XWXYXZTI STTX3! STTX [D?;0< !=+10-6= \?@D6750<73 U+*?:+=8?<01 6* 450 "0850<D+*173

Types of theory

By Gregor [5] adapted by Hannay et al. [6]: 1.  Analysis: taxonomies, classifica- tions and ontologies. 2.  Explanation: explains, for example, why something

happens. 3.  Prediction: predict what will happen, for example, in

terms of mathematical or probabilistic models. 4.  Explanation and prediction: combine types 2 and 3;

denoted an “empirically-based theory”.

5.  (Design and action: Theories that describe how to do things, typically prescriptive in the form of design science).

Framework for SE theories

i.  Constructs - entities in which the theory are expressed, and to which the theory offers a description, explanation or prediction

ii.  Propositions - proposed relationships between the constructs.

iii.  Explanations - originate from logical reasoning or empirical observations of the propositions

iv.  Scope - the circumstances, under which the theory is assumed to be applicable

Proposed Framework [Sjøberg et al]

Archetype class

Subclasses

Actor Individual, team, project, organisation or industry Technology Process model, method, technique, tool or

language Activity Plan, create, modify or analyze (a software

system) Software system

Software systems may be classified along many dimensions, such as size, complexity, application domain, business/scientific/student project or administrative/embedded/real time, etc.

What defines a theory?

•  Empirically founded

•  Defined scope •  Relevant