applying iterative design to the eco-feedback design process
DESCRIPTION
Although randomized controlled trials are the gold standard in evaluating the effectiveness of eco-feedback systems on reducing consumption behaviors, such trials are resource intensive and costly. As such, it is crucial that the intervention—the eco-feedback artifact—is well designed before effort is invested in a longitudinal study. In this talk, I will discuss the application of iterative design to eco-feedback systems. Iterative design is a design methodology based on a cyclic process of prototyping, user testing, and analysis, the results of which are then used to inform a new round of prototyping (and the cycle continues). Through an 18-month design process of a prototype eco-feedback display (Froehlich, 2011), I will describe how iterative design was used to evaluate and refine the aesthetic, usability, understandability, and educational potential of an eco-feedback system before a field deployment. I will highlight the role of massive online surveys in evaluating early eco-feedback design ideas and the role of in-home interviews in evaluating higher-fidelity (more refined) designs. Finally, I will close the talk with a discussion of low-cost methods to deploy and test eco-feedback designs in the field even when underlying resource sensing systems (e.g., smart meters) are unavailable. These methods can be used to evaluate how the eco-feedback system may fit into domestic space, explore differences in perspective and preference across household members, and evaluate how the system affects household dynamics (e.g., if the design provokes privacy concerns) before behavioral trials are conducted in earnest. Froehlich, J. (2011). Sensing and Feedback of Everyday Activities to Promote Environmental Behaviors. University of Washington Doctoral Dissertation 2011. http://www.cs.umd.edu/~jonf/publications.htmlTRANSCRIPT
Applying Iterative Design to the Eco-Feedback Design Process
Human Computer Interaction Laboratory
@jonfroehlich Assistant Professor Computer Science
Behavior, Energy, & Climate Change Conference Nov 13, 2012
eco-feedback sensing and visualizing behavior to reduce environmental impact
eco-feedback sensing and visualizing behavior to reduce environmental impact
“Getting the design right and the right design.”
— Bill Buxton
Sketching User Experiences
Before moving forward, I want to ask a question…
The following eco-feedback paper is missing something.
What is it?
Brandon et al., Reducing Household Energy Consumption: A Qualitative & Quantitative Field Study, Environmental Psychology 1999
Brandon et al., Reducing Household Energy Consumption: A Qualitative & Quantitative Field Study, Environmental Psychology 1999
½ paragraph description of
eco-feedback interface and no
screenshots in 11 page paper
Brandon et al., Reducing Household Energy Consumption: A Qualitative & Quantitative Field Study, Environmental Psychology 1999
Well, but that was 1999.
Grønhøj & Thøgersen, Feedback on Household Electricity Consumption: Learning & Social Influence Processes, IJCS2011
Grønhøj & Thøgersen, Feedback on Household Electricity Consumption: Learning & Social Influence Processes, IJCS2011
2 paragraph description of
eco-feedback interface and no
screenshots in 8 page paper
Where’s Waldo?
So, clearly a disciplinary divide…
psychologists
designers
engineers
economists
building scientists
others?
This oversight seems to reflect a lack of recognition
about the critical role that particular design
choices play in affecting behavior.
Jain et al., Assessing Eco-Feedback Interface Usage and Design to Drive Energy Efficiency in Buildings, Energy and Buildings 2012
Jain et al., Assessing Eco-Feedback Interface Usage and Design to Drive Energy Efficiency in Buildings, Energy and Buildings 2012
Perhaps because of the design de-emphasis, very few
papers discuss the design process that led to the
ultimate design artifact that was created and studied
This is abigproblem
How can we structure and support the design process to create and identify the most promising (and potentially most influential) aspects of an eco-feedback design robustly and in a cost-efficient manner?
Overarching Question
Ideation Data
Gathering
Ideation /
Sketch
Pilot
Studies Refinement
Formative
Evaluation Refinement
Small Field
Deployment(s) Refinement
Large
Randomized
Control Trial(s)
An Eco-Feedback Iterative Design Process
A/B Design
Ideas Pilot
Studies Refinement
A/B Testing
RCTs
Gunel, A. The Halo Effect: Using Behavior to Upgrade Technology,BECC2012
Ideation Data
Gathering
Ideation /
Sketch
Pilot
Studies Refinement
Formative
Evaluation Refinement
Small Field
Deployment(s) Refinement
Large
Randomized
Control Trial(s)
A/B Design
Ideas Pilot
Studies Refinement
A/B Testing
RCTs
An Eco-Feedback Iterative Design Process A/B Testing is Ultimate Playground
Ideation Data
Gathering
Ideation /
Sketch
Pilot
Studies Refinement
Formative
Evaluation Refinement
Small Field
Deployment(s) Refinement
Large
Randomized
Control Trial(s)
A/B Design
Ideas Pilot
Studies Refinement
A/B Testing
RCTs
A/B Testing is Ultimate Playground
Gunel, A. The Halo Effect: Using Behavior to Upgrade Technology,BECC2012
Ideation Data
Gathering
Ideation /
Sketch
Pilot
Studies Refinement
Formative
Evaluation Refinement Small Field
Deployment(s) Refinement
Large
Randomized
Control Trial(s)
A/B Design
Ideas Pilot
Studies Refinement
A/B Testing
RCTs
Evaluating early design ideas to prepare for
field deployments
An Eco-Feedback Iterative Design Process
Jon Froehlich
Leah Findlater
Marilyn Ostergren
Solai Ramanathan
Josh Peterson
Inness Wragg
Eric Larson
Fabia Fu
Mazhengmin Bai
Shwetak N. Patel
James A. Landay
Froehlich et al., The Design & Evaluation of Prototype Eco-Feedback Displays for Fixture-Level Water Usage Data, CHI2012
Ideation Data
Gathering
Ideation /
Sketch
Pilot
Studies Refinement
Formative
Evaluation Refinement
Small Field
Deployment(s) Refinement
Large
Randomized
Control Trial(s)
Goal: gather formative data and use as basis to create a set of early, promising designs
Inquiry Methods: ethnography, interviews, surveys, literature reviews
An Eco-Feedback Iterative Design Process
Ideation Data
Gathering
Ideation /
Sketch
Pilot
Studies Refinement
Formative
Evaluation Refinement
Small Field
Deployment(s) Refinement
Large
Randomized
Control Trial(s)
Goal: gather formative data and use as basis to create a set of early, promising designs
Inquiry Methods: ethnography, interviews, surveys, literature reviews
An Eco-Feedback Iterative Design Process
Informal interviews with water experts (e.g., SPU, Amy Vickers)
Literature review of water resource management, environmental psychology
Our own online survey of water usage attitudes & knowledge (N=656 respondents)
Respondents (N=651) dramatically underestimated the
amount of water used in common everyday activities.
toilet :
shower :
bath :
low-flow shower :
outdoor yard watering :
by 15%
by 30%
by 55%
by 60%
by 83% to 95%
underestimate
[Froehlich, UW PhD Dissertation, 2011]
Poor Water Literacy
Challenge: how can we use gathered data to inform our designs?
webpage in-home display
mobile phone app
wearable interface
custom display
manifestation
size small large
ambience not-ambient ambient
DISPLAY MEDIUM
update frequency real-time monthly or less
effort to access low high
spatial proximity to behavior
attentional demand glanceable high attention
co-located remote
INFORMATION ACCESS
user poll
interface customizability none high
degree of interactivity none high
INTERACTIVITY
user annotations user corrections user
additions
aesthetic pragmatic artistic
visual complexity simple complex
primary visual encoding textual graphical
data granularity coarse-grain fine-grain
time window <hour >year
DATA REPRESENTATION
primary view temporal spatial categorical
measurement unit resource metaphor cost environmental
impact time
temporal grouping ≤sec ≥year by hour by day by week by month
data grouping by
resource by consumption
category by
person by
time by
space
activity
by activity
target person
social- comparison
data sharing none everyone
private/ public
SOCIAL ASPECTS
household community state country
private public
available unavailable (see COMPARISON) degree of
actionability low high
decision support
personal-ization no personalization highly personalized
ACTIONABILITY/UTILITY
suggests actions
anomaly alerts
suggests purchase decisions
automation/ control no control system controls resource use
information intent Informs one action informs many actions
MOTIVATIONAL/PERSUASIVE STRATEGIES
persuasive tactics from psychology and applied social
psychology disciplines:
persuasive design persuasive technology
behavioral science/economics environmental psychology
game design social marketing
health behavior change
persuasive tactics include:
rewards punishment public commitment written commitment loss aversion kairos encouragement descriptive norms scarcity principle framing anchoring bias defaults
goal-setting narrative likeability reputation competition social proof authority emotional appeals door-in-face unlock features endowment effect collection building
comparison by time past projected
COMPARISON
comparison target self goal social
difficulty to reach comparison target easy hard
time window
time granularity
data grouping
data granularity
measurement unit
statistic
raw value average median mode other
comparison variables computation
@ this time [yest, last wk, mo, yr] [hrly, daily, wkly, monthly, yrly] over past [X] days this day type [weekday, weekend] this day of week (e.g., mondays)
social-comp. target
geographically proximal
selected social network
demographically similar
goal-setting strategy self-set externally-set system-set
Eco-Feedback Design Space
[Froehlich et al., HCIC2009; CHI2010; UW PhD Dissertation 2011]
My own experiences Existing frameworks (e.g., persuasive tech, health informatics,
other eco-feedback work, and infovis)
Psychology (e.g., environmental psychology, social
psychology, behavioral economics)
webpage in-home display
mobile phone app
wearable interface
custom display
manifestation
size small large
ambience not-ambient ambient
DISPLAY MEDIUM
update frequency real-time monthly or less
effort to access low high
spatial proximity to behavior
attentional demand glanceable high attention
co-located remote
INFORMATION ACCESS
user poll
interface customizability none high
degree of interactivity none high
INTERACTIVITY
user annotations user corrections user
additions
aesthetic pragmatic artistic
visual complexity simple complex
primary visual encoding textual graphical
data granularity coarse-grain fine-grain
time window <hour >year
DATA REPRESENTATION
primary view temporal spatial categorical
measurement unit resource metaphor cost environmental
impact time
temporal grouping ≤sec ≥year by hour by day by week by month
data grouping by
resource by consumption
category by
person by
time by
space
activity
by activity
target person
social- comparison
data sharing none everyone
private/ public
SOCIAL ASPECTS
household community state country
private public
available unavailable (see COMPARISON) degree of
actionability low high
decision support
personal-ization no personalization highly personalized
ACTIONABILITY/UTILITY
suggests actions
anomaly alerts
suggests purchase decisions
automation/ control no control system controls resource use
information intent Informs one action informs many actions
MOTIVATIONAL/PERSUASIVE STRATEGIES
persuasive tactics from psychology and applied social
psychology disciplines:
persuasive design persuasive technology
behavioral science/economics environmental psychology
game design social marketing
health behavior change
persuasive tactics include:
rewards punishment public commitment written commitment loss aversion kairos encouragement descriptive norms scarcity principle framing anchoring bias defaults
goal-setting narrative likeability reputation competition social proof authority emotional appeals door-in-face unlock features endowment effect collection building
comparison by time past projected
COMPARISON
comparison target self goal social
difficulty to reach comparison target easy hard
time window
time granularity
data grouping
data granularity
measurement unit
statistic
raw value average median mode other
comparison variables computation
@ this time [yest, last wk, mo, yr] [hrly, daily, wkly, monthly, yrly] over past [X] days this day type [weekday, weekend] this day of week (e.g., mondays)
social-comp. target
geographically proximal
selected social network
demographically similar
goal-setting strategy self-set externally-set system-set
Eco-Feedback Design Space
[Froehlich et al., HCIC2009; CHI2010; UW PhD Dissertation 2011]
behavioral
models
social
inputs
data
representation information
access
display
medium
comparison
actionability
motivational
strategies
eco-feedback design space
the
behavioral
models
social
inputs
data
representation information
access
display
medium
comparison
actionability
motivational
strategies
eco-feedback design space
the
social
inputs
data
representation information
access
display
medium
comparison
actionability
motivational
strategies
eco-feedback design space
the
behavioral
models
social
data
representation information
access
display
medium
comparison
actionability
motivational
strategies
eco-feedback design space
the
behavioral
models
inputs
behavioral
models
social
inputs
data
representation information
access
display
medium
comparison
actionability
motivational
strategies
eco-feedback design space
the
social
inputs
data
representation information
access
display
medium
comparison
actionability
motivational
strategies
eco-feedback design space
the
behavioral
models
inputs
social
behavioral
models
data
representation
information
access
display
medium
comparison
actionability
motivational
strategies
eco-feedback design space
the
pragmatic artistic aesthetic
simple complex visual complexity
coarse-grain fine-grain data granularity
primary view temporal spatial categorical
measurement unit resource cost metaphor environmental
impact time activity
< hour > year time granularity
inputs
social
behavioral
models
data
representation
information
access
display
medium
comparison
actionability
motivational
strategies
eco-feedback design space
the
pragmatic artistic aesthetic
simple complex visual complexity
coarse-grain fine-grain data granularity
primary view temporal spatial categorical
measurement unit resource cost metaphor environmental
impact time activity
< hour > year time granularity
coarse-grain fine-grain ≥neighbor- hood
home room fixture category
fixture ≤ valve activity
Data Granularity
Higher
fidelity
mockups
Sketches
Iterative Design Fidelities Prototype and Evaluate Across
a Fidelity Spectrum
Sketch Lo-to-Mid Fidelity
Mockup
Higher Fidelity
Mockup
Ideation Data
Gathering
Ideation /
Sketch
Pilot
Studies Refinement
Formative
Evaluation Refinement
Small Field
Deployment(s) Refinement
Large
Randomized
Control Trial(s)
Goal: gather formative data and use as basis to create a set of early, promising designs
Inquiry Methods: ethnography, interviews, surveys, literature reviews
An Eco-Feedback Iterative Design Process
Ideation Data
Gathering
Ideation /
Sketch
Pilot
Studies Refinement
Formative
Evaluation Refinement
Small Field
Deployment(s) Refinement
Large
Randomized
Control Trial(s)
Goal: test early design ideas, improve promising ideas, improve usability / aesthetic
Evaluation Methods: online interactive surveys, design probe-based interviews, lab studies
Challenge: the ultimate goal is to create a design that informs and, possibly, motivates
behavior. The former is easier to evaluate with these methods than the latter.
An Eco-Feedback Iterative Design Process
Online Survey Recruitment o Online postings and word-of-mouth
Survey Design o 63 questions (10 optional)
o Question and answer order
randomized when possible
Collected Data o 712 completed surveys
(651 from US or Canada)
o Nearly 6,000 qualitative responses
Our online interactive survey allowed us to
study a large N and gather both quantitative
and qualitative data
Comparisons were the most
uniformly desired pieces of
information of all the dimensions
Self-comparison
was most preferred
91%
Our in-home, design-probe interviews allowed us
to explore how the display was received by families
and how (and where) it fit in a domestic setting
In-Home Interviews Recruitment o Online postings and word-of-mouth
o Specifically recruited families
Interview Method o Semi-structured with two researchers
o 90-minutes, 3-phases
o Data coded by two researchers into themes
Participants o 10 households (20 adults)
o 11 female/9 male
o Diff. socio-economic backgrounds & occupations
o 18 had college degrees
Display Location Preferences
kitchen
near thermostat
high traffic areas
accessible when needed
Behavioral Lab Study
Ideation Data
Gathering
Ideation /
Sketch
Pilot
Studies Refinement
Formative
Evaluation Refinement
Small Field
Deployment(s) Refinement
Large
Randomized
Control Trial(s)
Integrate Findings & Revise Designs
A/B Design
Ideas Pilot
Studies Refinement
A/B Testing
RCTs
Place more emphasis on describing eco-feedback
designs and how design choices may affect behavior
in our research papers / white papers
Help generate reusable design knowledge by
including information not just on the final eco-
feedback design but the process used to achieve it
1
2
A Call
Froehlich et al., The Design and Evaluation of Prototype Eco-Feedback Displays for Fixture-Level Water Usage
Data, CHI2012, http://bit.ly/JonUMDPubs
Froehlich et al., The Design & Evaluation of Prototype Eco-Feedback Displays for Fixture-Level Water Usage Data, CHI2012
More detail and examples about the eco-feedback
design process here: http://bit.ly/jonUMD
Jon Froehlich, PhD Dissertation, 2011, http://www.cs.umd.edu/~jonf/publications.html
Jon Froehlich, Sensing and Feedback of Everyday Activities to Promote Environmental Behaviors, PhD Dissertation 2011
Jon Froehlich, Moving Beyond Line Graphs, BECC2010, http://www.cs.umd.edu/~jonf/talks.html
Applying Iterative Design to the Eco-Feedback Design Process
Human Computer Interaction Laboratory
@jonfroehlich Assistant Professor Computer Science
http://bit.ly/jonUMD [email protected]