Software Engineering Experimentation

An Experimental Framework

Jeff Offutt

http://www.cs.gmu.edu/~offutt/

© Jeff Offutt, 2005-2012 2

Experimental Framework• Victor Basili, Richard Selby and David

Hutchens, “Experimentation in Software Engineering”, IEEE Transactions on Software Engineering, SE-12(7):733–743, July 1986

• A little old but still relevant and helpful• Three objectives

1. Framework for performing experiments2. Classify and discuss experiments3. Problem areas and lessons learned

© Jeff Offutt, 2005-2012 3

Framework

Four categories1. Definition – Problem definition and

hypothesis2. Planning – Design3. Operation – Conduct test4. Interpretation – Hypothesis test,

disseminate

© Jeff Offutt, 2005-2012 4

Framework – 1. Definition

Motivation

(why)

Object

(what)

Purpose(what)

Perspective

(who)

Domain(which)

Scope(which)

UnderstandAssessManageEngineerLearnImproveValidateAssure

ProductProcessModelMetricTheory

CharacterizeEvaluatePredictMotivate

DeveloperModifierMaintainerProject managerCorp. managerCustomerUserResearcher

ProgrammerProgram / project

Single projectMulti-projectReplicated projectBlocked subject- project

© Jeff Offutt, 2005-2012 5

Framework – 2. PlanningDesign

(samples, control)

Criteria(what to measure)

Measurement(how to measure)

Experimental designs Incomplete block Completely random Randomized block Fractional factorialMultivariate analysis Correlation Factor analysis RegressionStatistical modelsNon-parametricSampling

Direct reflections of cost / quality Cost Errors Changes Reliability CorrectnessIndirect reflections of cost / quality Data coupling Information visibility Programmer comprehension Execution coverage Size Complexity

Metric definition Goal-question-metric Factor-criteria-metricMetric validationData collection Automatability Form design and testObjective vs. subjectiveLevel of measurement Nominal / classificatory Ordinal / ranking Interval Ratio

© Jeff Offutt, 2005-2012 6

Framework – 3. Operation

Preparation Execution Analysis

Pilot study Data collectionData validation

Quantitative vs. qualitativePreliminary data analysis Plots and histograms Model assumptionsPrimary data analysis Model application

© Jeff Offutt, 2005-2012 7

Framework – 4. Interpretation

Interpretation context

Extrapolation Impact

Statistical frameworkStudy purposeField of research

Sample representativeness

VisibilityReplicationApplication

© Jeff Offutt, 2005-2012 8

Framework Example DefinitionDefinition Element

Example

Motivation To improve unit testing

Object characterize and evaluate

Purpose the process of functional & structural testing

Perspective from the perspective of the developer

Domain: programmer

as they are applied by experienced programmers

Domain: program to unit-size software

Scope in a blocked subject-project study