the green lab - [02 a] the experimental process
TRANSCRIPT
Vrije Universiteit Amsterdam
Vrije Universiteit Amsterdam
Experiment principles
Vrije Universiteit Amsterdam
Image encoding: which algorithm should I
use to optimize battery usage?
○ PNG
○ JPEG
○ …
Vrije Universiteit Amsterdam
MAIN DRIVERS of experimentation: ● to be able to scientifically assess an idea● generalize
Vrije Universiteit Amsterdam
Theory
Observation
Cause EffectCausation
e.g. green encoding algorithm e.g. energy efficiency
Treatment Experiment Outcome
e.g. JPEG, PNG e.g. energy consumption per image
Vrije Universiteit Amsterdam
Terminology
Vrije Universiteit Amsterdam
● Dependent variables: quantities observed in the study
(a.k.a. response, output variables)
○ e.g. energy consumption, image quality
● Independent variables: quantities that we are able to
manipulate/control (a.k.a. input variables)
○ e.g. encoding algorithm, size of image, operating system
Vrije Universiteit Amsterdam
● Factor: an independent variable that we deliberately
manipulate/control
○ e.g. image encoding algorithm
● Treatment: a specific value of a factor
○ e.g. JPEG, PNG
Vrije Universiteit Amsterdam
● Subject: who applies the treatment
○ e.g. software developer, software architect, user
● Object: the receiver of the treatment
○ e.g. software application
Vrije Universiteit Amsterdam
● Trial: combination of subject, object and treatment
○ e.g. a software developer develops an app that
encodes images using the JPEG algorithm
● Experiment: a combination of several trials to observe the
effects of the treatments
○ e.g. ???
Vrije Universiteit Amsterdam
Vrije Universiteit Amsterdam
The experimental process
Vrije Universiteit Amsterdam
● Main idea behind the experiment
● The hypotheses must be stated clearly
○ Not formally, just clearly
Idea
Vrije Universiteit Amsterdam
Experiment scoping
● We scope the experiment by stating
the problem, objectives, and goals
● We will use the Goal-Question-Metric
(GQM) framework
Idea
Vrije Universiteit Amsterdam
● Define context
● Formulate hypotheses
● Identify input and output variables
● Design the study
● Instrumentation
● Analyze validity threats
Experiment scoping
Experiment planning
Idea
Vrije Universiteit Amsterdam
● Preparation
○ Guidelines, subjects training,
code instrumentation, ...
● Execution
○ aka measurements
collection
● Data Validation
Experiment scoping
Experiment planning
Idea
Experiment operation
Vrije Universiteit Amsterdam
● Understanding the data
○ descriptive statistics
○ exploratory data analysis
(e.g. box plots, scatter plots, … )
● Hypothesis testing
● Results interpretation
Experiment scoping
Experiment planning
Idea
Experiment operation
Analysis & interpretation
Vrije Universiteit Amsterdam
● Document the results
● Prepare replication package
● Sum up lessons learned
● Write down reflectionsExperiment
scoping
Experiment planning
Idea
Experiment operation
Analysis & interpretation
Presentation & package
Vrije Universiteit Amsterdam
●
○
○ …
●
○
■
Vrije Universiteit Amsterdam
Chapter 6
Vrije Universiteit Amsterdam
Some contents of lecture extracted from: ● Giuseppe Procaccianti’s lectures at VU