seminar on enterprise software - ut
TRANSCRIPT
Seminar on Enterprise Software
Fabrizio Maria Maggi
(based on lecture material by Raimundas Matulevičius and Dietmar Pfahl)
Writing Good Software Engineering
Research Papers
by Mary Shaw
Proceedings of the 25th International Conference on Software
Engineering, IEEE Computer Society, 2003, pp. 726-736.
presented by Fabrizio Maria Maggi
University of Tartu, Estonia
A good research paper or thesis should
answer a number of questions• What, precisely, was your contribution?
– What question did you answer?
– Why should the reader care?
– What larger question does this address?
• What is your new result?
– What new knowledge have you contributed that the reader can use elsewhere?
– What previous work (yours or someone else’s) do you build on? What do you provide a superior alternative to?
– How is your result different from and better than this prior work?
– What, precisely and in detail, is your new result?
• Why should the reader believe your result?
– What standard should be used to evaluate your claim?
– What concrete evidence shows that your result satisfies your claim?
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
3
A good research paper or thesis should
answer a number of questions• What, precisely, was your contribution?
– What question did you answer?
– Why should the reader care?
– What larger question does this address?
• What is your new result?
– What new knowledge have you contributed that the reader can use elsewhere?
– What previous work (yours or someone else’s) do you build on? What do you provide a superior alternative to?
– How is your result different from and better than this prior work?
– What, precisely and in detail, is your new result?
• Why should the reader believe your result?
– What standard should be used to evaluate your claim?
– What concrete evidence shows that your result satisfies your claim?
� If you answer these questions clearly
� You will probably communicate your result well
� Your result represents an interesting, sound, and
significant contribution
� You will have a good chance of getting thesis accepted
for publication in a conference or journal
� You will have a good chance of publishing the paper (or
few) in a prestigious conference or journal
� If you answer these questions clearly
� You will probably communicate your result well
� Your result represents an interesting, sound, and
significant contribution
� You will have a good chance of getting thesis accepted
for publication in a conference or journal
� You will have a good chance of publishing the paper (or
few) in a prestigious conference or journal
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
4
What, precisely, was your
contribution?
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
5
What kinds of questions do software
engineers investigate?
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
6
What do thesis committees look for?
• A clear statement of the specific problem you solved
• The question about software development you answered
• An explanation of how the answer will help solve an important software engineering problem
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
7
What is your new result?
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
8
What kinds of results do you produce?
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
9
What do thesis committees look for?
• Interesting, novel, exciting results that
Significantly enhance the ability
• to develop and maintain software
• to know the quality of the software
• to recognize general principles about software
• to analyze properties of software
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
10
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
11
What’s new here?
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
12
What has been done
before?
How is your work
different or better?
Questions that committee asks about
your thesis
What, precisely, is the result?
• If you introduce a new model
• be clear about its power
• empirical data, formal semantics, mathematical principles
• If you introduce a new metric
• define it precisely
• If you introduce a new architectural style, design pattern, or
similar design element,
• treat it as if it were a new generalization or model
• If your contribution is principally the synthesis or integration of
other results or components
• be clear about why the synthesis is itself a contribution
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
13
Questions that committee asks about
your thesis
What, precisely, is the result?
• If your paper is chiefly a report on experience applying
research results to a practical problem
• say what the reader can learn from the experience
• If a tool plays a featured role in your paper
• what is the role of the tool?
• If a system implementation plays a featured role in your paper
• what is the role of the implementation?
• archiecture, implementation technique, capabiliy, performance,
system is a result itself
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
14
Why should the reader believe your
result?
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
15
What do committees look for?Why should the reader believe your result?
• If you claim to improve on prior art
– compare your result objectively to the prior art
• If you used an analysis technique
– follow the rules of that analysis technique
• If the technique is not a common one
– explain the technique and standards of proof, and be clear about
your adherence to the technique
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
16
What do committees look for?Why should the reader believe your result?
• If you offer practical experience as evidence for your result
– establish the effect your research has
• If you performed a controlled experiment
– explain the experimental design
• If you performed an empirical study
– explain what you measured, how you analyzed it, and what you concluded
• If you use a small example for explaining the result
– provide additional evidence of its practical use and scalability
Shaw M.: Writing Good Software Engineering Research Papers,
Proc. of the 25th International Conference on Software Engineering,
IEEE Computer Society, 2003, pp. 726-736.
17
Systematic Literature Survey – Protocol
• An example: https://arxiv.org/pdf/1705.02288.pdf
• Background: The rationale for the survey.
• Research questions that the review is intended to answer.
• Search strategy that will be used to search for primary studies
– including search terms and
– resources to be searched.
Resources include
– digital libraries,
– specific journals, and
– conference proceedings.
Systematic Literature Survey – Protocol
• Background: The rationale for the survey.
• Research questions that the review is intended to answer.
• Search strategy that will be used to search for primary studies
– including search terms and
– resources to be searched.
Resources include
– digital libraries,
– specific journals, and
– conference proceedings.
Software Engineering Digital Libraries
•Scopus
•Web of Science
•Google scholar
•IEEExplore
•ACM Digital library
•SpringerLink
Systematic Literature Survey – Protocol
• Study selection criteria
– Study selection criteria are used to determine which
studies are included in, or excluded from, a systematic
review
Systematic Literature Survey – Protocol
• Study quality assessment checklists and procedures
– The researchers should develop quality checklists to
assess the individual studies
• Data extraction strategy
– Design data extraction forms to accurately record the
information researchers obtain from the primary studies
– Linked to the research questions
Systematic Literature Survey – Protocol
• Synthesis of the extracted data
• Collating and summarising the results of the included primary studies
– Descriptive (non-quantitative)
– Quantitative
How bibliography needs to be organized?
References should appear as
• Book with one author:
– Author, A. A. (2005). Title of work.
Location/City, State: Publisher.
• Book with two authors:
– Author, A. A., & Author, B. B.
(2005). Title of work.
Location/City, State: Publisher.
• Book with more than two
authors:
– Author, A. A., Author, B. B., &
Author, C. C. (2005). Title of
work. Location/City, State:
Publisher.
• Journal article:
– Sawyer, S., & Tapia, A. (2005).
The sociotechnical nature of
mobile computing work: Evidence
from a study of policing in the
United States. International
Journal of Technology and
Human Interaction, 1(3), 1-14.
• A publication in press:
– Junho, S. (in press). Roadmap for
e-commerce standardization in
Korea. International Journal of IT
Standards and Standardization
Research.
How bibliography needs to be organized?
References should appear as
• Edited book:
– Zhao, F. (Ed.). (2006). Maximize
business profits through e-
partnerships. Hershey, PA: IRM
Press.
• Chapter in an edited book:
– Jaques, P. A., & Viccari, R. M.
(2006). Considering students’
emotions in computer-mediated
learning environments. In Z. Ma
(Ed.), Web-based intelligent e-
learning systems: Technologies
and applications (pp. 122-138).
Hershey, PA: Information Science
Publishing.
• Report from a university:
– Broadhurst, R. G., & Maller, R. A.
(1991). Sex offending and
recidivism (Tech. Rep. No. 3).
Nedlands, Western Australia:
University of Western Australia,
Crime Research Centre.
• Published proceedings:
– Deci, E. L., & Ryan, R. M. (1991).
A motivational approach to self:
Integration in personality. In R.
Dienstbier (Ed.), Nebraska
Symposium on Motivation: Vol.
38. Perspectives on motivation
(pp. 237-288). Lincoln: University
of Nebraska Press.
How bibliography needs to be organized?
References should appear as• Unpublished doctoral
dissertation or master’s
thesis:
– Wilfley, D. (1989). Interpersonal
analyses of bulimia: Normal-
weight and obese. Unpublished
doctoral dissertation, University of
Missouri, Columbia.
• A presented paper:
– Lanktree, C., & Briere, J. (1991,
January). Early data on the
Trauma Symptom Checklist for
Children (TSC-C). Paper
presented at the meeting of the
American Professional Society on
the Abuse of Children, San
Diego, CA.
• Web site:
– VandenBos, G., Knapp, S., &
Doe, J. (2001). Role of reference
elements in the selection of
resources by psychology
undergraduates. Journal of
Bibliographic Research, 5, 117-
123. Retrieved October 13, 2001,
from http://jbr.org/articles.html
How bibliography needs to be organized?
In-Text Citations• In-text citations should appear with author surname followed by
publication year in parentheses
(Brown, 2002)
• Citing several references in-text:
– In most organizations, data resources are considered to be a major
resource (Brown, 2002; Krall & Johnson, 2005; Smith, 2001).
– Brown (2002) states that the value of data is recognized by most
organizations
– “In most organizations, data resources are considered to be a major
organization asset” (Smith, 2001, pp. 35-36) and must be carefully
monitored by the senior management.
– Brown (2002) states that “the value of data is realized by most
organizations” (p. 45).
How bibliography needs to be organized?
In-Text Citations
• If you have organized the citations with number in
brackets
– In most organizations, data resources are
considered to be a major resource [15; 30; 84].
Research Methods
• Experiment
– Controlled Experiment
– Quasi-Experiment
• Case Study
– Descriptive Case Study
– Exploratory Case Study
– Confirmatory Case Study
• Survey
– Questionnaire-based
– Literature-based
Case Study – Variants
• Descriptive Case Study
– Purely observational / Focus on “What happens?”
• Explorative Case Study
– Initial investigation of some phenomena to derive new
hypotheses and build theories / Focus on “What and Why?”
• Confirmatory Case Study
– Start out with a given theory and try to refute it, ideally with a
series of case studies covering various contexts
More on Case Study design (SE Group at Lund University):
http://serg.cs.lth.se/education/case_study_research/
Case Study – Characterisation
� Characteristics:
– When to use? --> When 'rich'
information is requested
– 'Rich' (in-vivo=in the living),
– More focus on qualitative data -->
allows for better understanding of
conditions under which a
technique/tool works
� Issues:
– Important: Proper case selection /
clearly stated research question /
clearly defined framework for
interpreting the observations
– 'Generalisability' (1 case --> only 1
context)
� Definition:
– An empirical enquiry that
investigates a contemporary
phenomenon within its real-life
context (in-vivo=in the living),
especially when the boundaries
between phenomenon and context
are not clearly evident.
� Example:
– Experienced practitioners
communicate about software
design in many different ways (and
other than, say, undergraduate
students). Thus, one may conduct
a case study in a local company.
Research Methods
• Experiment
– Controlled Experiment
– Quasi-Experiment
• Case Study
– Descriptive Case Study
– Exploratory Case Study
– Confirmatory Case Study
• Survey
– Questionnaire-based
– Literature-based
Survey – Characterisation
• A survey is a data collection method or tool used to gather
information about individuals in order to identify the
characteristics of a broad population
• The defining characteristic is the selection of a representative
sample from a well-defined population with the aim to
generalise from the sample to the population.
• Usually conducted with questionnaires, but can also involve
structured interviews or data logging techniques
• Example:
– Investigate to what extent, for which purpose, by which
companies, and by whom within the companies, UML
diagrams are used.
Survey – Characterisation (cont'd)
When to use it?
– Either at start of research to get an understanding of the
current situation (
– or at the end of a research phase to see the
impact/acceptance/etc. of a new method/technique/tool
Issues:
– 'Superficial' --> no explanation / no causality --> not
suitable for hypothesis testing
– 'Generalisability' of results depends on the choice of
population and 'response rate', as well as validity and
reliability of the data collection instrument
Survey – Example
What?
Research Questions:
- How is Agile practiced at Microsoft?
– i.e. What do engineers do?
- How do engineers feel about it?
– i.e. Do they like it?
Who, Where, and When?
Microsoft (worldwide, 2006)
Anonymous survey sent to 2821 engineers
• 10% random sampling of all developers, testers, program managers at Microsoft in October 2006
487 valid responses
• 44% developers, 28% testers, 17% program managers
Source: Andrew Begel and Nachiappan Nagappan,
Usage and Perceptions of Agile Software Development
in an Industrial Context: An Exploratory Study, in First
International Symposium on Empirical Software
Engineering and Metrics, IEEE Computer Society,
September 2007
Why?
• Many agile
approaches
exist – what's
in it for
Microsoft?
Survey – Example (cont'd)Qantitative Results (Highlights)
• 33% of respondents (spread across
divisions) report their team uses Agile
methodologies.
• They mainly use Scrum (68%).
• Used for many legacy products.
• Agile usage does not appear to depend
on team co-location.
• Test-driven development and pair
programming are not very common.
Qualitative Results (Highlights)
• MS engineers who have used Agile like it
for their local team, but not necessarily
for their organization.
• They worry about scale, overhead, and
management buy-in.
Perceived benefits (687 comments, 44 themes)
Perceived problems (565 comments, 58 themes)
Research Methods
• Experiment
– Controlled Experiment
– Quasi-Experiment
• Case Study
– Descriptive Case Study
– Exploratory Case Study
– Confirmatory Case Study
• Survey
– Questionnaire-based
– Literature-based
Karli Oruste, Process Mining in Industry, 2017
Allar Soo, Automated Process Discovery: A
Literature Review and a Comparative Evaluation
with Domain Experts, 2017
Research Methods
• Survey
– Questionnaire-based
– Literature-based
• Experiment
– Controlled Experiment
– Quasi-Experiment
• Case Study
– Descriptive Case Study
– Exploratory Case Study
– Confirmatory Case Study
Controlled Experiment – Characterisation
• An investigation of a testable hypothesis where one or more
independent variables are manipulated to measure their effect on one
or more dependent variables.
• In Software Engineering, typically, experiments require human
subjects to perform some task.
… …
Treatments
(Interventions)
Independent
Variables
Dependent
Variables
E
C
Controlled Experiment – Simple Example
• Independent Variable: UML diagram usage (yes or no)
• Dependent Variable: Design Quality
• Treatments: E = use UML / C = don’t use UML
E
Treatments
(1 Factor / 2 Levels)
Independent
Variable
Dependent
VariableC
NB: Design can be
within-subject or
between-subject
Controlled Experiment vs. Quasi-Experiment
Randomization is a
prerequisite for a
controlled experiment!
Experiment – Characterisation (cont'd)
�Characteristics:
–When to use? --> When in-depth understanding about causal relationships is requested
–Focus on quantitative data --> allows for use of statistics for hypothesis testing
–Due to control over independent variables (in vitro = 'in glass', i.e. laboratory), helps detect causal relationships (--> tests hypotheses --> theory building)
� Issues:
–Expensive, 'narrow' (in-depth)
–'Generalisability'? ('artificial' lab setting --> test subjects, tasks, artifacts)
Experiment – Example
What?
Research Question:
• What is best – Pair Programming or Solo Programming?
Who, Where, and When?
Norway, 2007
295 junior, intermediate and senior
professional Java consultants from 29
companies were paid to participate (one work
day)
99 individuals; 98 pairs
The pairs and individuals performed the same
Java maintenance tasks on either:
• a ”simple” system (centralized control style), or
• a ”complex” system (delegated control style)
They measured:
• duration (elapsed time)
• effort (cost)
• quality (correctness) of their solutions
Source: E. Arisholm, H. Gallis, T. Dybå, and D. Sjøberg, “Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise,” IEEE Transactions on Software Engineering, 2007, 33(2): 65-86.
Why?
Many studies with contradicting
results – mostly conducted with
students (not with professional
developers)
Total Effect of PP
84 %
7 %
-8 %
-40 %
-20 %
0 %
20 %
40 %
60 %
80 %
100 %
120 %
140 %
160 %
Duration Effort Correctness
Dif
fere
nc
e f
ro
m in
div
idu
als
Experiment – Example (cont'd)
Effect of PP for Juniors
5 %
111 %
73 %
-40 %
-20 %
0 %
20 %
40 %
60 %
80 %
100 %
120 %
140 %
160 %
Duration Effort Correctness
Dif
fere
nc
e f
rom
in
div
idu
als
Moderating Effect of System Complexity for Juniors
4 %
109 %
32 %
6 %
112 %
149 %
-40 %
-20 %
0 %
20 %
40 %
60 %
80 %
100 %
120 %
140 %
160 %
Duration Effort Correctness
Dif
fere
nc
e f
rom
in
div
idu
als
CC (easy)
DC (complex)
Moderating Effect of System Complexity for Seniors
55 %
-13 %
8 %
115 %
-23 %
-2 %
-40 %
-20 %
0 %
20 %
40 %
60 %
80 %
100 %
120 %
140 %
160 %
Duration Effort Correctness
Dif
fere
nc
e f
rom
in
div
idu
als
CC (easy)
DC (complex)
Effect of PP for Seniors
-9 %
83 %
-8 %
-40 %
-20 %
0 %
20 %
40 %
60 %
80 %
100 %
120 %
140 %
160 %
Duration Effort Correctness
Dif
fere
nc
e f
rom
in
div
idu
als
Experiment – Example (cont'd)
So, when should we use PP?
The question of whether PP is best, or not, is meaningless!
One should ask: In which situation is PP best to achieve a defined goal?
Importance of Context:Helps construct/refine theory about when and how to do 'Pair Programming'
Selecting the Research Method (to be
submitted)
• The choice of method depends among other things on:
– Suitable study subject (e.g., do participants have enough experience?)
– Possibility to control the environment
– The size/scale/cost of the study
– The need for generality in the results
– Availability of information/data and other resources
– What is the purpose of the study? (exploration, prediction,
understanding of cause-effect relations, applicability of results in
industry, ....)
• Difficult to provide general recommendation with respect to choice of
method
Experiment or Case Study or Survey?
• Experiments give the researchers freedom to isolate a defined effect and
to hold other things constant (-> research in-depth)
– Often difficult to avoid that people respond differently than they would have
done in a natural environment
– Also, you might have a too restricted setting and omit important influencing
factors that play a role in a natural environment
• Case-studies have the advantage that you observe people doing what
they are actually doing in their natural environment (-> research in-the-
typical)
– It is limited what you can control without interfering with natural activity
• Surveys provide researchers with information about what many people
(think they) are doing (-> research in-breadth)
– No control whatsoever (issues relate to data validity and representativeness)
Validity & Reliability of Empirical Studies
• Construct Validity
– Concepts being studied are operationalized and measured correctly (do the measures used actually represent the concepts you want to measure?)
• Internal Validity
– Establish a causal relationship and sort out spurious relationships (exclude confounding variables / by: random sampling)
• Conclusion Validity
– Do proper statistical inference
• External Validity
– Establish the domain to which a
study’s findings can be
generalized (precisely describe
the population and
experimental conditions)
• Reliability
– The study can be repeated (i.e.,
by other researchers) and
yields the same results
– The measurement instrument is
reliable (interrater agreement)
Example (page 12):
http://ieeexplore.ieee.org/document/6776520/
Empirical Research Guidelines
1. Experimental Context
2. Experimental Design
3. Data Collection
4. Analysis
5. Presentation of Results
6. Interpretation of Results
To be submitted
Example:
https://link.springer.com/chapter/10.1007%2F978-
3-319-23063-4_21C
1. Experimental Context
• research questions, hypotesis and variables (see Section 4
in the example)
Example:
https://link.springer.com/chapter/10.1007%2F978-
3-319-23063-4_21C
2. Experimental Design
• Subjects/objects, datasets, procedure and metrics
(see Section 4.1, 4.2 and 4.3 in the example)
Example:
https://link.springer.com/chapter/10.1007%2F978-
3-319-23063-4_21C
3. Data Collection
• Ensure that the data collection process is well
defined
• Monitor the data collection and watch for deviations
from the experiment design
4. Analysis
• Ensure that the collected data from the experiment
is analyzed correctly
• Monitor the data analysis and watch for deviations
from the experiment design
5. Presentation of Results
• Ensure that the reader of the results can
understand the objective, the process and
the results of experiment
6. Interpretation of Results
• Ensure that the conclusions are derived
merely from the results of the experiment
Presentation
• Present the following:
– The problem statement
– Background
– Review of the state of the art
– Identified gaps in the state of the art
– The expected contribution of the thesis given the identified gaps (objectives)
– Experimental design and research design
• Each student has a time-slot of 15 minutes: 12 minutes for the talk + 3 minutes
for Q&A
• After each session I will organize a short feedback session to give suggestions
to the presenters
60