software engineering experimentation an experimental framework jeff offutt offutt
Post on 27-Dec-2015
212 Views
Preview:
TRANSCRIPT
Software Engineering Experimentation
An Experimental Framework
Jeff Offutt
http://www.cs.gmu.edu/~offutt/
© Jeff Offutt, 2005-2012 2
Experimental Framework• Victor Basili, Richard Selby and David
Hutchens, “Experimentation in Software Engineering”, IEEE Transactions on Software Engineering, SE-12(7):733–743, July 1986
• A little old but still relevant and helpful• Three objectives
1. Framework for performing experiments2. Classify and discuss experiments3. Problem areas and lessons learned
© Jeff Offutt, 2005-2012 3
Framework
Four categories1. Definition – Problem definition and
hypothesis2. Planning – Design3. Operation – Conduct test4. Interpretation – Hypothesis test,
disseminate
© Jeff Offutt, 2005-2012 4
Framework – 1. Definition
Motivation
(why)
Object
(what)
Purpose(what)
Perspective
(who)
Domain(which)
Scope(which)
UnderstandAssessManageEngineerLearnImproveValidateAssure
ProductProcessModelMetricTheory
CharacterizeEvaluatePredictMotivate
DeveloperModifierMaintainerProject managerCorp. managerCustomerUserResearcher
ProgrammerProgram / project
Single projectMulti-projectReplicated projectBlocked subject- project
© Jeff Offutt, 2005-2012 5
Framework – 2. PlanningDesign
(samples, control)
Criteria(what to measure)
Measurement(how to measure)
Experimental designs Incomplete block Completely random Randomized block Fractional factorialMultivariate analysis Correlation Factor analysis RegressionStatistical modelsNon-parametricSampling
Direct reflections of cost / quality Cost Errors Changes Reliability CorrectnessIndirect reflections of cost / quality Data coupling Information visibility Programmer comprehension Execution coverage Size Complexity
Metric definition Goal-question-metric Factor-criteria-metricMetric validationData collection Automatability Form design and testObjective vs. subjectiveLevel of measurement Nominal / classificatory Ordinal / ranking Interval Ratio
© Jeff Offutt, 2005-2012 6
Framework – 3. Operation
Preparation Execution Analysis
Pilot study Data collectionData validation
Quantitative vs. qualitativePreliminary data analysis Plots and histograms Model assumptionsPrimary data analysis Model application
© Jeff Offutt, 2005-2012 7
Framework – 4. Interpretation
Interpretation context
Extrapolation Impact
Statistical frameworkStudy purposeField of research
Sample representativeness
VisibilityReplicationApplication
© Jeff Offutt, 2005-2012 8
Framework Example DefinitionDefinition Element
Example
Motivation To improve unit testing
Object characterize and evaluate
Purpose the process of functional & structural testing
Perspective from the perspective of the developer
Domain: programmer
as they are applied by experienced programmers
Domain: program to unit-size software
Scope in a blocked subject-project study
top related