saving user attention in ubiquitous computing scaling task management in space and time joão pedro...
TRANSCRIPT
saving user attention in ubiquitous computing
Scaling Task Management
in Space and Time
João Pedro Sousa
March 28, 2005
committeeDavid Garlan, chairMahadev SatyanarayananReid SimmonsGregory Abowd, Georgia Tech
2
the problemusers work hardwhen moving tasks from one location to another
worse withdifferent platformsresource variation
save filesstop applications
remember tasktransfer datafind & restart appsreconstruct state
3
ideallyall migration and configuration handled automaticallywhen moving tasks from one location to another
leave
select task to resume
4
solutions
devices becomingmore powerful
cheaper
interconnected
expect computingto be availableeverywherejust like furniture
taskslast long time
recur
need to find& recoverpast tasks
daily activitieshandle many tasks
tasks requiremany services
desktop managers(Rooms, 1987)
?
?
reality
saving user attention in ubiquitous computing
Scaling Task Management
in Space and Time
João Pedro Sousa
March 28, 2005
committeeDavid Garlan, chairMahadev SatyanarayananReid SimmonsGregory Abowd, Georgia Tech
6
outline
the problemhow existing solutions fall short
new approach
summary of research
validation
related & future work
contributions
discussion
7
computers and info everywhere:heterogeneous devices and applications
Goal
support mobile users in their daily
tasks
Enable users to:carry out their tasks in diverse environmentstake full advantage ofthe available capabilities and resources
Reduce user overhead in managing heterogeneityresource variability
8
support mobile users in their daily
tasks
aren’t existing solutions good enough?
desktop managers
remote computing
mobile computing
code migration
internet suspend/resume
9
desktop managers
task support user overhead
coordinated use of services
handle several taskslimited number of tasksno support for finding past tasks
virtual desktops
assumes primary use of one machine
user mobility take full advantage
10
remote computing
thin clientseasy / no reconfiguration
user mobility user overhead
to deal with distinct devices,assume common set of shared capabilities
wasteful of local capabilities
not satisfactory for the user
bad connectivity may be a showstopper
task support take full advantage
11
mobile & wearable computing
people try to be self-sufficientby carrying personal devices
low overhead – as long aspersonal devices are enough
user mobility user overhead
no support for using other services in the vicinity
task support take full advantage
12
code migration
personalized applications andtheir data follow the user around
Ex: Gaia, ICrafter, Kimura,
Labscape
strong assumption about thecompatibility of the VMs
user mobility user overhead
deal with diverse target devices
enrich applications, or
reduce the set of assumptions(not satisfactory and wasteful)
task support take full advantage
(new apps)
13
internet suspend/resume
migrate contentsof the virtual memory
strong assumption about thecompatibility of the hardware
user mobility user overhead
no support for using other devices in the vicinity
no discrimination of which tasks to migrate
task support take full advantage
(no changes to apps)
14
good job at some of the requirementsbut a poor job at others…
new approach
aren’t existing solutions good enough?
code migration
remote computing
desktop managers
mobile computing
usermobility
reducedoverhea
d
tasksupport
take fulladvantag
e
internet S/R
15
new Approachsupport mobile users in office-like tasks by migrating high-level models of user tasks
infrastructure infrastructure
16
model user needs for each task
reactivate a task in different locationscoordinated use of services
abstract modelsmove to heterogeneous
environmentsmatch needs, e.g., user needs to edit text with local capabilities
model user preferences
take full advantagefind the best matchearly advice on task feasibilitydynamic adaptation to changes
automatic configuration reduce user overheadroutine configuration and reconfiguration
new Approachhighlights
new approachmodel migration
usermobility
useroverhea
d
tasksupport
take fulladvantag
e
17
outline
the problem
new approach
summary of researchthesischallenges
validation
related & future work
contributions
discussion
18
environmentset of suppliers, resources & materialsaccessible to a user
terminology
servicewhat users need
suppliersoftware/device offering services
resourceconsumed by supplierswhile delivering services
materialinformation asset
print, edit text, browse the web…
MS Word, Emacs…
CPU, battery, bandwidth…
files, data streams
19
thesis statement
enabling users to take full advantageof the services and resources of computing environments
reducing user overhead associated with configuring and managing those environments
High-level models of user tasks can be used to address the scalability of task management
in space, across heterogeneous environments
in time, enabling users to recover past tasks
while simultaneously
20
model tasks
usable on adaily basis
challenges
infrastructure infrastructure
describe andoperateon tasks
21
challengesmodeling user tasks
coordinated useof services
suspend/resume
find past tasks
task support model services & materials
user-perceived statewindow layout, cursors…
persistent semantic identitygoals, time & place, collaborators…
heterogeneity
dynamic change
environment
what users need, not which applications
user preferencesalternative service suppliersalternative configurations of servicesalternative QoS tradeoffs
27
modeling user tasksmodel services & materials
what users need,not which applications
user-perceived statewindow layout, cursors…
persistent semantic identitygoals, time & place, collaborators…
user preferences
28
modeling user tasksindependently of specific apps
model services & materials
what users need,not which applications
user-perceived statewindow layout, cursors…
persistent semantic identitygoals, time & place, collaborators…
user preferences
<auraTask id="34"> <preferences …/> <service …/> <material …/>
<configuration name="skip web" weight="0.8"> <service id="1"> <uses materialId="87"/> <uses materialId="45"/> </service> <service id="3"> <uses materialId="102"/> </service> </configuration> <configuration name="paper only" …/> <configuration name="full" …/></auraTask>
29
<auraTask id="34"> <preferences …/>
<service type="edit Text" id="1"> <settings> <format overtype="0"/> </settings> </service> <material id="87"> <state> <cursor position="31510"/> <scroll horizontal="0" vertical="7"/> <zoom value="140"/> <spellchecking enabled="1" language="English"/> <window height="500" width="600" xpos="20" ypos="100" mode="min"/>
</state> </material> <configuration …/></auraTask>
model services & materials
what users need,not which applications
user-perceived statewindow layout, cursors…
persistent semantic identitygoals, time & place, collaborators…
user preferences
modeling user tasksindependently of specific apps
30
model services & materials
what users need,not which applications
user-perceived statewindow layout, cursors…
persistent semantic identitygoals, time & place, collaborators…
user preferences
<task state="pending" id="34"> <description>
<name>conference paper</name> <notes>main result: scalability of task management in space and time</notes> <collabs>David;Vahe;</collabs> </description>
<history due="2/07/04" created="30/06/04 9:51 PM"> <accessed at="home" stop="30/06/04 10:32 PM" start="30/06/04 9:53 PM"/> <accessed …/> </history> <links>
<link label="previous conference" tId="14"/> </links></task>
modeling user tasksenable users to find past tasks
31
task feasibility
qpcU ,maxargp: choice suppliersq: QoS level
algorithm by Vahe Poladian joint paper in ICSE
model
services & materialswhat users need,not which applications
user-perceived statewindow layout, cursors…
persistent semantic identitygoals, time & place, collaborators…
user preferencesalternative service suppliersalternative configurationsalternative QoS tradeoffs
Uchoice
suppliers
achievable QoS
real number in [0,1]
utility functions
qpcUchtF Config ˆ,ˆˆˆˆ)(
modeling user tasksheterogeneity & change
32
infrastructure for task managementexploits these models
configure environmentfind the best match to user needs
resource allocation across applicationsutility-theoreticcomparison of alternatives
activate/deactivate suppliers
reconstruct user-perceived state
monitor & adapt to changesframework for adaptation
coordinate adaptation policies
<auraTask id="34"> <preferences …/>
<service type="edit Text" id="1"> <settings> <format overtype="0"/> </settings> </service> <material id="87"> <state> <cursor position="31510"/> <scroll horizontal="0" vertical="7"/> <zoom value="140"/> <spellchecking enabled="1" language="English"/> <window height="500" width="600"
xpos="20" ypos="100" mode="min"/> </state> </material> <configuration …/></auraTask>
c
d
c
s
Dd
wdQoS
Ss
wSupp dqhsphqpcU ˆ,
33
how to configuremonitor services and resourcesfind the best match to user needs
(utility-theoretic) comparison of alternativesresource allocation across applications
coordinate supplier activation
monitor failures and improvement opportunities
infrastructure for task managementseparate concerns
configure environmentfind the best match to user needs
resource allocation across applicationsutility-theoreticcomparison of alternatives
activate/deactivate suppliers
reconstruct user-perceived state
monitor & adapt to changesframework for adaptation
coordinate adaptation policies
what to configure formonitor & model user needs
coordinated use of services(utility-theoretic) user preferences
coordinate task suspend/resumereconstruct user-perceived statecoordinate adaptation policies
34
how to configuremonitor services and resourcesfind the best match to user needs
(utility-theoretic) comparison of alternativesresource allocation across applications
coordinate supplier activation
monitor failures and improvement opportunities
what to configure formonitor & model user needs
coordinated use of services(utility-theoretic) user preferences
coordinate task suspend/resumerecover user-perceived statecoordinate adaptation policies
Task
Managemen
t
Managed
Environme
nt
infrastructure for task managementseparate concerns
35
TaskManageme
nt
ManagedEnvironme
nt
Distributed File Access
TaskManageme
nt
ManagedEnvironme
nt
infrastructure for task managementuses models to configure environment
36
Environment
TaskManageme
nt
ManagedEnvironme
nt
Operating System
Prism
service
Supplier
App
EnvironmentManager
ContextObserver
UI
UI
infrastructure for task managementarchitectural framework
37
register <sModel>...
EM Supplier
newTaskcreatedTask <id>
budget <id,tModel>taskLayout <id,config>
setup <id, tModel>
taskLayout <id, config>
...
(re)configure...
change
Prism
taskLayout <id, config>
setup <id, tModel>
taskLayout <id, config>
...
taskGone <id>
disband <id>
(re)configure...
(re)configure...
change
infrastructure for task managementarchitectural framework
formal specification of the protocols
FSPdeadlock freerecovery from errors
Zedstate shared between Prism and EMfor each taskmodel of available services in the EM
38
infrastructure for task management implementation
Operating System
Prism
service
Supplier
App
EnvironmentManager
ContextObserver
UI
UI
Dis
trib
ute
d F
ile S
yst
em
PrismUser
TaskIdentity
TaskModels
Speakeasy
Focus
Task Manager
Dashboard LampJava + Swing
XML/TCP
39
Operating System
Prism
service
Supplier
App
EnvironmentManager
ContextObserver
UI
UI
Suppliers (several students) MS Word PowerPoint Excel Internet Explorer Media Player Babel Fish, Festival,
Emacs, Pangloss, Sphinx, Xanim
EM (Vahe Poladian)
infrastructure for task management implementation
40
model tasks
challenges
infrastructure infrastructure
describe andoperateon tasks
usable on adaily basis
41
are delays introduced by the infrastructurea source of overhead?
measure latency ofoperational Dashboard
after user authenticationread task directorysearch pending tasksevaluate task feasibility
search tasks using Lamp
suspend/resume tasksexperiments on a IBM ThinkPad 30 Windows XP, 1.6 GHz, 512 MB
42
latency introduced by infrastructureblends in with starting/stopping apps
~ 3sfor 10 tasks
< 1s
< 1s
operational Dashboardafter user authentication
read task directorysearch pending tasksevaluate task feasibility
search tasks using Lamp
suspend/resume tasks
44
outline
the problem
new approach
summary of research
thesis validationcost-benefit analysis
related & future work
contributions
discussion
45
thesis validation
enabling users to take full advantage
reducing user overhead
scalability of task management
in space, across heterogeneous environments
in time, enabling users to recover past tasks
while simultaneously
migrate models of user tasksexploited by task management infrastructure
persistent task identity + browser
find the best match between needs and capabilitiesutility-theoretic framework
?
46
thesis validationreducing user overhead
compare overheadwith vs. without the infrastructure
focus on c-taskssuspend/resume tasksdescribe tasksfind tasks
general model of overhead(independent of specific apps/OS)
count of user operations(coarse interactions: start apps, open files...)
47
reducing user overheadmodel overhead of each c-task
starting a task of preparing a presentation
without ops with ops overhead
– open app (slide editor)
– look for relevant papers– open app (text viewer) on the papers
– look for web pages
1
nm
k
– new task– new service (slides)– resume task– look for relevant papers– add the papers to the task– save task– look for web pages– add pages to the task– save task
1 1†
1n
m‡
1kn.
pages1
1
1
1
n. pages1
(†) For opening an application, users issue a command, click on some icon, etc.; for adding the slide editing service to the new task, users click one button and make a list selection.(‡) For adding the papers to the new task, users drag the papers from some file view in the operating system and drop them into the more area in the Dashboard.
overhead of defining a task is at most 2n+1n is the number of services/materials
operations put side by side are comparable
48
without ops with ops benefit
– find files and links– open files/links with apps– recover state of apps
1†
n. files/pagesn. files/pages‡
– find task in Dashboard– resume
11 n. files/pages – 1
n. files/pages
(†) Assuming that users have fresh on their minds where all the files are stored.(‡) Assuming recovering the user-perceived state of each service/file takes a single operation.
For instance, recovering the editing position on a text file.
resuming a task
benefit of resuming a task is at least 2n-1…
benefit of suspending a task is at least n-1
conservative estimates
reducing user overheadmodel overhead of each c-task
49
benefit of using the infrastructureadds up over time
-20
-10
0
10
20
30
40
50
def s r s r s r s r
be
ne
fit
5
4
3
2
1
co
st
# operations
# m
ate
ria
ls
accumulated benefit of using the infrastructure
(disregarding scalability in space and time)
time(c-task)
50
finding past tasksno significant differencewhen compared, e.g., with Google Desktop Search
benefit of using the infrastructureincreases with user mobility
unassisted configurationmuch harderdepending on user expertise
environment
heterogeneity
dynamic change
51
outline
the problem
new approach
summary of research
validation
related & future work
contributions
discussion
52
task management
task modeling
desktop managersone machine
Rooms (1987),…
teleport desktopsthin clients
X windows (1994)MS Easy Living
TM in smart spacesmigrate (new) apps
Kimura, ICrafter, Gaia, Labscape
user mobilitymigrate virtual memory
ISR
assistance with tasks
related work
guide usersin complex tasks
Adtranz (1998)Smart Homeselderly/health care
carry out taskson users’ behalf
autonomous agentsRADAR
(future work)
plans to be carried outby automated systems
robotics, agents, distributed systems
plans to be carried outby humans
business processesusage models
user needsin a computing environment
UI adaptationtask management
53
service awareness
context awareness
fault tolerancemodel availability
load balancingmodel QoS
model-based adaptationknown components
autonomic computingRainbow
service ontologyweb services
DAML/OWL, UDDI…
scoping discoveryis an open problem
resource awareness
related workadaptive systems
(future work)
fidelity-aware appsone applicationlocal policies
Odyssey/Chroma, Puppeteer…
context sensinglocation, activity…
CIS…
context-aware appsone applicationlocal rules
context-awaretask managementis an open problem
54
future workcomplementary areas
architectural & economic model-based adaptatione.g., Shang-Wen Cheng, Vahe Poladian
fidelity/resource-aware applicationse.g., Rajesh Balan
context awareness
distributed file systems
natural human-computer interfaces
incorporate results from
55
future worktask management
delegation of services/parts of tasks to humans framework to combine automatic configuration with human participation
proactive supplyforecast service demands, customize service offering to demand
scoping the environmentcombine demand preferences (familiar/desired suppliers)with supply offering, across administrative boundaries
describing task modelsframework that accommodates a range of domains
guidance on tasksreconcile guidance with freedom/serenity in carrying out tasks
56
approach infrastructure
contributions
frameworktask management
placeholder fortask knowledge
environment management
utility-theoreticframework forfinding best match
coordinatesadaptation at 3 levels
user tasksenvironment capabilitiesresources
demonstratesfeasibility of approach
usabilityby non-experts
adequate performancefor daily use
foundation forfurther research
well documented design
affordable to integrate legacy/new components
57
approachuses task models toscale task management
in space, across heterogeneous environmentsin time, enabling users to find past tasks
control over which tasks to carry out
accounts for user preferences
reconcilesreducing user overhead taking full advantage
contributions
58
requirement: taking full advantageartifact of today’s limitations?
mobile computing
low overhead – as long aspersonal devices are enough
user mobility user overhead
+ task support
59
the next big thing
computation gets physical
today
process information
support mobile users in their daily tasks
tomorrow, actuateworkplacehomemedicine…
models of user needs & preferences
60
solutions
devices becomingmore powerful
cheaper
interconnected
expect computingto be availableeverywherejust like furniture
taskslast long time
recur
need to find& recoverpast tasks
daily activitieshandle many tasks
tasks requiremany services
desktop managers(Rooms, 1987)
persistenttask identity
high-leveltask modelssupport mobility
reality
61
outline
the problem
new approach
related work
summary of research
validation
future work
contributions
discussion
62
limitations & assumptions
migrating/accessingtask models & materials
dependent on apps’APIs
good behavior wrt resource management
shared vocabulariesfor describing services
data format compatibility
63
task feasibility
qpcU ,maxargp: choice suppliersq: QoS level
algorithm by Vahe Poladian joint paper in ICSE
model
services & materialswhat users need,not which applications
user-perceived statewindow layout, cursors…
persistent semantic identitygoals, time & place, collaborators…
user preferencesalternative service suppliersalternative configurationsalternative QoS tradeoffs
Uchoice
suppliers
achievable QoS
real number in [0,1]
utility functions
qpcUchtF Config ˆ,ˆˆˆˆ)(
modeling user tasksheterogeneity & change
64
Uavailable suppliers
achievable QoS
real number in [0,1]
overall utility = supplier utility x QoS utility
combining utilities by multiplicationreflects and semantics
supplier utility = supplier 1 ut. x supplier 2 ut. x …QoS utility = QoS dim. 1 x QoS dim. x 2 …
65
Uavailable suppliersachievabl
e QoS
real number in [0,1]
model supplier preferencesby enumeration
supplier utility = supplier 1 ut. x supplier 2 ut. x …
prefer text editing supplier:
supports pointing device
accelerator keys WYSIYG
spell checking…
Word 1Emacs 0.6Vim 0.3other …none …
66
model QoS tradeoffsby good and bad thresholds
QoS utility = QoS dim. 1 x QoS dim. x 2 …
high accuracymediumlow
101
late
ncy
(s)
67
model QoS tradeoffsby good and bad thresholds
QoS utility = QoS dim. 1 x QoS dim. x 2 …
high accuracymediumlow
good
bad
101
late
ncy
(s)
68
model QoS tradeoffsby good and bad thresholds
QoS utility = QoS dim. 1 x QoS dim. x 2 …
20
high accuracy
1
mediumlow
late
ncy
(s)
good
bad
69
users understand & controlthreshold-based QoS tradeoffs
user study: 10 students
resourcesplentyscarce
tradeoff 1 tradeoff 2 tradeoff 3
completed experiments
recognized expected adaptation behavior(95% statistical significance)
all 10
joint work with Rajesh Balan & Vahe Poladian paper submission