living laboratories': rethinking ecological designs and experimentation in human-computer...
DESCRIPTION
HCI have long moved beyond the evaluation setting of a single user sitting in front of a single desktop computer, yet many of our fundamentally held viewpoints about evaluation continues to be ruled by outdated biases derived from this legacy. We need to engage with real users in 'Living Laboratories', in which researchers either adopt or create functioning systems that are used in real settings. These new experimental platforms will greatly enable researchers to conduct evaluations that span many users, places, time, location, and social factors in ways that are unimaginable before.TRANSCRIPT
EdH.ChiAreaManagerandSr.ResearchScientistPaloAltoResearchCenter
2009HCICWorkshopatYMCACamp,Colorado
Asafield,earlyfundamentalcontributionsfrom:– Computerscientistsinterestedinchangesinwaysweinteract
withinformationsystems– Psychologistsinterestedintheimplicationsofthesechanges
Combustible,because:– Computerscientistswanttocreategreattools,butdidn’tknow
howtomeasureimpact– Psychologistswanttogobeyondclassicalresearchofthebrain
andhumancognition
Example:– anenduringinterestin‘augmentinghumanintellect’:V.Bush,
Licklider,Engelbart,inturninspiringStuCard,AlanNewell,AlanKay,andmanyothers.
2/13/09 HCIC "Living Lab" 2
TheneedtoestablishHCIasascience– Adoptmethodsfrompsychology– Convenientandfitswellwithproblemsathand
Issuesaroundpersonalcomputing(WIMP)– Dualpurpose:understandnatureofhuman
behaviorandbuildupascienceofHCItechniques.
– GoodExamples:Fitts’Law,ModelsofHumanMemory,CognitiveandBehavioralModeling,InformationForaging
– StuartK.Card,WilliamK.English,andBettyJ.Burr(1978).Evaluationofmouse,rate‐controlledisometricjoystick,stepkeys,andtextkeysfortextselectiononaCRT.Ergonomics,21(8):601–613,1978.
2/13/09 3 HCIC "Living Lab"
Beyondauserinfrontofcomputer– Yetevaluationmethodsmostlystayedthesame– PerceivedCHIpapertemplateforacceptance
Manyproblemsdon’tfitthelaboratoryexperimentalmethodsanymore– Yesterday’sdiscussionaboutLargeDataandHCIwaslargelydrivenby
howHCIevaluationmethodsneedtochangetofitthewild
Thebestexamples:TrendsinSocialComputingandUbiCompforceustothinkaboutnewcontextofuse
2/13/09 HCIC "Living Lab" 4
Old Assumptions New Considerations Single display Multiple displays Knowledge work Games, communication, social apps Isolated worker Collaborative and social groups Stationary location Mobile and stationary Short task durations Short and long tasks, and tasks with
no time boundries Controllable experimental conditions Uncontrollable experimental conditions
2/13/09 5 HCIC "Living Lab"
ThinkaboutresearchthathasbeendoneonUIanimationsorflashingicons.
Fromvisualperception,weknowmotionintheperipheryismorenoticeablethaninthefovealregion[DaVinci].
2/13/09 HCIC "Living Lab" 6
EvaluationssurroundingmanyHCIsystemsforknowledgeworkfocusonproductivityincrease,butwhataboutfactorsforadoption?– Argument:ifnoproductivityincrease,thenadoptionis
irrelevant– Buttheoppositeargumentisjustasright:ifnoadoption,no
amountofproductivityincreaseshownisrelevant!
Academicresearchoftenfocusonproductivityimprovements,increasingtheperceivedgulfbetweentheivorytowerandthetrenches
AnExample:Colorcopierstudies
2/13/09 HCIC "Living Lab" 7
Artificialexperimentalsetupsareonlycapableoftellingusbehaviorsinconstrainedsituations– Ecologicalconsiderations– Hardtogeneralizetonewtaskcontexts(withinterruptions,other
tasks,othergoals,unfocusedattention,moredisplays)– Hardtogeneralizetoothertools,apps– Impossibletoanswerquestionsaboutaggregatebehaviorsofgroups
Exampleproblems:– Adoptionofmobiletechnology
iPhonesinJapan,single‐handedinput[PARC] BestsellingphonesinIndonesiacomeswithacompass[Bell]
– AggregatebehaviorofWikipediaorDelicioususers Bigdataanalysisofeditlogs
2/13/09 8 HCIC "Living Lab"
Wasacomputationalmolecularbiologist Analogy:Justasbiologistsworkonmodelplantsand
genomesinthelab,thistellsusjusthowitbehavesinanisolatedenvironmentundercontrolledconditions,butnothowtheplantwillbehaveintherealworld.
Biologistsdon’tjuststudymodelsinthelab,butinthewildalso.
2/13/09 HCIC "Living Lab" 9
ObservationalStudies– Ethnography– SocialTechnicalDesign– IterativeDesign– DiaryStudies– LongitudinalStudieswithsingleoutcome
Problems:– Samplingfromnon‐normaldistributions– Treatvariationsinsocialcontextsas‘noise’– Whataboutadoption?
MixedMethodsisnowpopulartopartiallyameliorate– Convergentmeasurestellsuswe’regettingclosertothetruth
2/13/09 10 HCIC "Living Lab"
– similartoVentureBusinessmentionedbyDavidMillen
Conductresearchonrealplatformsandservices– Nottoreplacecontrolledlabstudies– Expandourarsenaltocovernewsituations
Someprinciples:– Embeddedintherealworld– Ecologicallyvalidsituations– Embracethecomplexity– Relyonbig‐data‐sciencetoextractpatterns
Notfirsttosuggestthis:– S.Carter,J.Mankoff,S.KlemmerandT.Matthews.Exitingthecleanroom:On
ecologicalvalidityandubiquitouscomputing.HCIJournal,2008– EClass[Abowd],PlaceLab[Intille],PlasmaPoster[ChurchillandNelson],Digital
FamilyPortrait[Rowan,Mynatt]
2/13/09 11 HCIC "Living Lab"
Twodimensions– 1.Whetherthesystemisunderthecontroloftheresearcher– 2.Whetherthestudyisconductedinthelaborinthewild
System Control System Not in Control
Laboratory (1) Build a system, study in the Lab
(2) Adopt a system, study in the Lab
Wild (Real World)
(4) Build a system, release it, study in the Wild
(3) Adopt a system, study in the Wild
2/13/09 12 HCIC "Living Lab"
TraditionalApproach;Numerousexamples FavoredbyCHIreviewers Typicalsituationisthestudyofsomeinteractiontechnique
– Peninput,gestures,perceptionofsomevisualizeddata,readingtasks,mobiletextinput
Typicalmeasuresarequantitativeinnature– performanceintime,performanceinaccuracy,eyetracking,learning
measures,userpreferences
Issues:– Notalwaysecologicallyvalid– Hardtotakeallinteractionsintoaccount– Oftentime‐consuming;eventhoughwethoughtwecoulddoitfast.
2/13/09 13 HCIC "Living Lab"
Hardertofindintheliterature Oftencomparingagainstanoldersystemasbaseline Typicalcaseiscomparisonoftwosystems
– (onewebsitewithanother,onewordprocessorvs.another)– Whichhighlightingfeatureworksbetter– Twotextinputtechniqueonacellphone
Typicalmeasuresaresimilarto(1) Issues:
– Somesimilarissuesto(1)becauseit’sinlab– Systemfeaturenotincontrol,sonotabletocomparefairly,or
isolatethefeature
2/13/09 HCIC "Living Lab" 14
2/13/09 HCIC "Living Lab" 15
http://www.flickr.com/photos/crush_images/3245220458/
Realapplicationsinecologicalvalidsituations Realfindingscanbeappliedtoarunningsystem Impactofresearchismoreimmediate,sincesystemisalready
running Typicalcaseisloganalyticswithlargesubjectpools
– logstudiesofwebsites,realmobilecallingusages,websearchlogs,studiesofWikipediaedits.
Typicalmeasuresarestickiness,amountofactivity,clusteringanalysis,correlationalanalysis
Issues:– Factorsnotincontrol,findingsnotcomparable– Factorscannotbeisolated– Reasonsforfailureisoftenjustguesswork
2/13/09 HCIC "Living Lab" 16
Hypothesis:ConflictiswhatdrivesWikipediaforward. Howtostudythis?
– Tukeyparadigm– Getalargepaper,andplotthedamndata!
– DownloadedallofWikipediaandalloftherevisions– Hadoop/MapReduce,MySQL,etc.
2/13/09 HCIC "Living Lab" 17
2/13/09 18
60%
65%
70%
75%
80%
85%
90%
95%
100%
2001 2002 2003 2004 2005 2006
Perc
enta
ge o
f tot
al e
dits
Article
User
Article Talk
User Talk
Other
Maintenance
HCIC "Living Lab"
Group A
Group B Group C
Group D
Number of users in user group A B C Total
Users with Korean point of view 10 6 0 16
Users with Japanese point of view 1 8 7 16
Neutral or Unidentified 7 3 6 17 2/13/09 19 HCIC "Living Lab"
Mediators
Sympathetic to parents
Sympathetic to husband
Anonymous (vandals/spammers)
2/13/09 20 HCIC "Living Lab"
2/13/09 HCIC "Living Lab" 21
2/13/09 HCIC "Living Lab" 22
2/13/09 23 HCIC "Living Lab"
2/13/09 24 HCIC "Living Lab"
2/13/09 25 HCIC "Living Lab"
Hypothesis:SocialTaggingdoesn’tscaleovertime. Howtostudythis?
– Crawlasmuchtaggingdataaswecan.– Studythenoiseinthesystem.
– 40machinesfor3months
2/13/09 HCIC "Living Lab" 26
Topics
Users Documents
Tags
T1…TnEncodingDecoding
Noise
2/13/09 27 HCIC "Living Lab"
Concepts
2/13/09 HCIC "Living Lab" 28
Source: Hypertext 2008 study on del.icio.us (Chi & Mytkowicz)
29 2/13/09 HCIC "Living Lab"
Similarto(3),practicalforrunningsystems;ecologicallyvalid,impactcanbeimmediate.– Goodforcasesinwhicheconomicsmakessense[Google]– Changestosystemispossible;Factorscanbecontrolled.
TypicalcasemightbeA/Btesting,largesubjectpools Typicalmeasuresarebeingdeveloped
– Impactmeasures.Largevisit#andinterest(measuredbyblogposts?)NewBusinessinquiries?
– Usabilitymeasuresvs.Usefulnessmeasures
Issues:– Effortandresourcerequirementisdroppingbutstillsignificant– Hardforaresearchlabtotakeon
2/13/09 HCIC "Living Lab" 30
2/13/09 HCIC "Living Lab" 31
CollaborativeFiltering
TapestrysystematPARC
GroupLens 6kusers Ecosystemwith
passionateusers
2/13/09 HCIC "Living Lab" 32
2/13/09 HCIC "Living Lab" 33
2/13/09 34
HCIC "Living Lab"
2/13/09 35
HCIC "Living
Lab"
2/13/09 36
HCIC "Living
Lab"
2/13/09 37
Guide
Web
Howto
Tips Help
Tools
Tip
Tricks
Tutorial
Tutorials
Reference
Semantic Similarity Graph
HCIC "Living Lab"
2/13/09 HCIC "Living Lab" 38
2/13/09 HCIC "Living Lab" 39
Interactioncostsdeterminenumberofpeoplewhoparticipate
Surplusofattention&motivationatsmalltransactioncosts
Therefore… Importanttokeep
interactioncostslow
Cost of participation #
Peop
le w
illin
g to
pro
duce
for “
free
”
40 2/13/09 HCIC "Living Lab"
In situ tagging while reading – No new window – Clicking vs typing
Tagging + highlighting
41 2/13/09 HCIC "Living Lab"
Exitingthecleanroom[Carteretal.]– SparseData– CriticalMass– Dealingwithrealisticenvironments– Unobtrusivenessofmonitoring– Ambiguityanderrorofinferencing
2/13/09 HCIC "Living Lab" 43
Evaluationmethodsarein‐separablefromthekindsofscienceandmodelsthatcanbebuildinafield.
Platformadvancesenablerealtechnologyinsertionintorealworldsituationscheaperandmoremanageable.
2/13/09 44 HCIC "Living Lab"
Characteriza7on Models
PrototypesEvalua7ons
ResearchVision:Understandhowsocialcomputingsystemscanenhancetheabilityofagroupofpeopletoremember,think,andreason.
LivingLaboratory:Createapplicationsthatharnesscollectiveintelligencetoimproveknowledgecapture,transfer,anddiscovery.
http://asc‐parc.blogspot.comhttp://[email protected]
WikiDashboard MrTaggy SparTag.us