the green lab - [01 c] empirical software engineering

1 Het begint met een idee

Empirical software engineering

Ivano Malavolta

Vrije Universiteit Amsterdam

Software engineering


- definition- dimensions- types of studies

2 Ivano Malavolta / S2 group / Empirical software engineering

Roadmap



Software engineering


Where is software today?


Software today


How “big” is software today?


Software today

http://www.informationisbeautiful.net/visualizations/million-lines-of-code

http://hbr.org/2010/06/why-dinosaurs-will-keep-ruling-the-auto-industry/ar/1

https://www.google.com/url?q=http://www.informationisbeautiful.net/visualizations/million-lines-of-code/&sa=D&ust=1477923773279000&usg=AFQjCNG18vlxT0bfK1UN_er6beJ9GZMo7g

https://www.google.com/url?q=http://www.informationisbeautiful.net/visualizations/million-lines-of-code/&sa=D&ust=1477923773280000&usg=AFQjCNEG3cRsowSb5_vBW-GQwFsPnggOqw


To DESIGN software- software development has to be a systematic activity

QUALITY assurance- we have to verify and validate our SW in order to make it something

people can rely on- we have to do it as soon as possible

ABSTRACTION- the principal instrument for managing complexity


Needs


The application of engineering to software

Field of computer science dealing with software systems that are:

– large and complex– built by teams– exist in many versions– last many years– undergo changes

Programming skills are not enough“Physicist example”

–7 Ivano Malavolta / S2 group / Empirical software engineering

Software engineering (SE)

Programmer: complete program

Software engineer: software component



A more precise definition of SE

"Software Engineering means application of a

systematic, disciplined, quantifiable approach

to development, operation and maintenance of

software [1]"

1. “IEEE Standard Glossary of Software Engineering Terminology,” IEEE std 610.12-1990, 1990.


Computer science- Computability, algorithms and complexity, programming

languages, data structures, databases, artificial intelligence, etc.

Software engineering- The APPLICATION of computer science, mathematics, project

management to build high quality software


Software engineering VS computer science


Scientific use of quantitative and qualitative data to- understand and- improve

software products and software development processes[Victor Basili]

Data is central to address any research question

Issues related to validity addressed continuously





Intuition

● It is an application of the scientific method

○ ask yourself a question

○ observe a phenomenon

○ formulate an hypothesis

○ build a prediction

○ setup an experiment

○ perform analysis on your results


Anecdotal evidence or “common-sense” often not good enough

- Anecdotes often insufficient to support decisions in the industry- Practitioners need better advice on how and when to use

methodologies

Evidence important for successful technology transfer- systematic gathering of evidence- wide dissemination of evidence


Why empirical studies?


● Tested on a single system (lack of generalization)● Data analysis not statistical ● No indications on how many runs were performed, error rate,

etc.


How NOT to do experiments


17


IE9

Chrome

FireFox

Opera

Safari


“In the lab” versus “in the wild” studies Qualitative versus quantitative studies

Primary versus secondary studies


Dimensions of empirical studies


Common “in the lab” methods– Controlled experiments– Literature reviews– Simulations

Common “in the wild” methods– Quasi-experiments– Case studies– Survey research– Ethnographies– Action research


“In the lab” VS “in the wild” studies



Examples


Qualitative researchstudying objects in their natural setting and letting the findings emerge from the observations

– inductive process– the subject is the person

Quantitative researchquantifying a relationship or to compare two or more groups with the aim to identify a cause-effect relationship

– fixed implied factors– focus on quantitative data → promotes comparison and statistical analyses


Qualitative VS quantitative studies



Example

They are complementary


Primary studiesempirical studies in which we directly make measurements or observations about the objects of interest, whether by surveys, experiments, case studies, etc.

Secondary studiesempirical studies that do not generate any data from direct measurements, but:

– analyze a set of primary studies– usually seek to aggregate the results from these in order to provide stronger forms of evidence about a phenomenon


Primary VS secondary studies



Examples



… and what about this?


Survey

Case study

Experiment


Types of empirical studies


Def: a system for collecting information from or about people to describe, compare or explain their knowledge, attitudes and behavior

Often an investigation performed in retrospect

Interviews and questionnaires are the primary means of gathering qualitative or quantitative data

Surveys are done by taking a sample which is representative from the population to be studied


Survey



Example


1. ADLs Identification– Identify preliminary set of ADLs– Systematic search

2. Planning the Survey3. Designing the survey

4. Analyzing the Data– vertical analysis (and coding) + horizontal analysis


Example


Def: an empirical enquiry to investigate one instance (or a small number of instances) of a contemporary software engineering phenomenon within its real-life context, especially when the boundary between phenomenon and context cannot be clearly specified

Observational study

Data collected to track a specific attribute or establishing relationships between different attributes

Multivariate statistical analysis is often applied30 Ivano Malavolta / S2 group / Empirical software engineering

Case study



Example


Def: an empirical enquiry that manipulates one factor or variable of the studied setting

1. Identify and understand the variables that play a role in software development, and the connections between variables

2. Learn cause-effect relationships between the development process and the obtained products

3. Establish laws and theories about software construction that explain development behaviour

32

Experiment


ExperimentModels key characteristics of a reality in a controlled environment and manipulating them iteratively to investigate the impact of such variations and get a better understanding of a phenomenon

LaboratorySimplified and controllable reality where the phenomenon under study can be manipulated

CONTROL - CONTROL - CONTROL


A more intuitive definition...


34

Example



What to choose?


This is called technology transfer


How to have an impact in reality?


Software is pervasive

Software engineering != computer science

You know the basics of empirical SE now

Don’t make claims you cannot eventually measure

Don’t think in black and white only– don’t divide the world in methods, analyses, case study, etc.– don’t be afraid to look also at other disciplines

→ we are software engineers in any case


What this lecture means to you?



Readings

Chapters 1, 2, 3Chapters 1 and 2

+ one article among the ones mentioned in the previous slides

the green lab - [01 c] empirical software engineering

Technology