responsive media
DESCRIPTION
Presentation of concepts and research around the idea of Responsive Media presented at the 2009 Workshop on Pervasive Advertising in Nara Japan.TRANSCRIPT
Responsive Media
Bo BegoleJames Glasnapp
Strategic review March 2009 parc confidential
Mixed-Initiative Interaction
Conventional systems: User initiates interaction and commands the system
Mixed-initiative: System sometimes initiates interaction with the user– You have mail.– Can I help you find that?– Here is something useful to you.
ResponsiveMirror
Responsive Technologies InformationRecommendation
ClothingRecognition
Psychographic Profiling
[ICDSC 2008][IUI 2008, HCII 2009][CHI 2008]Magitti
Related PARC Research:also:Human-Robot Interaction
Multi-party conversations
Camera-based tracking
MIT Media Lab
Microsoft Research
3
Business Marketplace
In-store signage– Traditional: Point-of-Purchase displays, shelf positioning,
packaging, store-handouts (coupons), specials (e.g., Kmart blue light), aisle coupons, loyalty programs (lower price)
– Emerging: digital kiosks, digital signage, directed audio Companies
– NewsAmerica leases store space and sells ad spaces to consumer packaged goods
Search Engine Marketing $13B to $26B in 2013
Advertisers pay more for personalization
Reactrix charged higher rates than static digital signage.Reactrix is just the tip of the iceberg.
4
Today we are just at the tip of the iceberg in
conversational
interaction
Today we are just at the tip of the iceberg in
conversational
interaction
Voice Systems
Robots
Media
Avatars
In the future we
will interact with all types of
technology as if they were social entities
In the future we
will interact with all types of
technology as if they were social entities
Service Agents
Marketing
Sales
Education
Therapy
Performance Coaching
In-Store Product Recommendations
PersonalProfile
What productIs she looking
at now?
Eye-contactsensors
TrackingSensors
Previouspurchases
Is she searchingor just browsing?
FloorSensors
Is this a groupor individual?
MotionSensors
Is sherushing ina hurry?
Items: x, y, z, ….
A blue blazer
browsing
Data TypeSensor
PerceptionPersonalized
Recommendation
Group
Question
DisplayImpulse-buy
itemsRushing
Similar items
Matchingskirts in the
store
Highlight gift items
Highlight new trendy fashions
Inferred User Goal
Interest Profile:Style, colors, price
range, etc.
Looking forBusiness clothes
Shopping for gifts
Wants to show“Fashion sense”
Needs to decidequickly
Web Shopping
today
Responsive Personalized Sales Promotions
Existing Research: Many indicators of a person’s engagement with media
Component technologies exist, but not integrated, not directed by behavioral models:- What are sequential structures of engaged interactions?- Which indicators are most predictive of engagement?- Can we predict disengagement before it happens?
[Vogel & Balakrishnan ‘04]
[Grauman et al. ‘01]
[Cohen et al. ‘03]
[Yu, Aoki, Woodruff, PARC ‘04]
vocal affect
proximity, orientation of head & body
[Daugman ‘94]
pupil dilation
eye blinksfacial affect
skin temperature
[Haro, Flickner, Essa 2000]
eye gaze
Engineering approach (Reactrix) currently achieves Phase 1 using disruptive techniques Phase 4 is the real value – requires recognizing human micro-behaviors Conversation and interaction analysis bring clarity to vague notions like “engagement”
– Detect, describe and model the structured organization of natural interaction– Create systems that interact and respond to individuals
Responsive and Personalized Public Information Display
[HCII 2009]
Interaction Structure of a Marketing Engagement1
– Approach (hook)– Assess– Relax– Describe– Benefit– How to buy– Reduce resistance– Incentive to act
Monitor &Re-engageas needed
[1Robert Prus, Making Sales]
Attract and maintain audience engagement Content follows interaction model toward
an objective: Marketing, Entertainment, Education, …
8
Improving social capability and interactive personalization
Making systems socially interactive Conversation analysis (CA) can build a more personalized, smooth interaction between technological
systems and humans
Interaction Analysis provides Technology designed using frameworks inspired by conversational structures
Previous research: Sotto Voce, Responsive Mirror, Human-Robot Interaction
Broad Applications of Conversational Responsiveness Any field with interactive features with customers: call centers and interactive voice responses to
improve voice interactions; games – making characters more interactive; mobile phone manufactures can make more use of conversational data (i.e., providing analysis of conversations to provide feedback); and automobile - design better audio-based interfaces
Linking research in human behavior to technology
design.
Sales Interaction ModelRepresenting Elements of Sellers’ Goals
Representation of dependencies and degree to which each sales goal has been achieved
[adapted from Making Sales, Robert Prus]
Engage Assess
Offer Service
Present Products
Generate Trust
Show Customer Need
Neutralize Reservations
ObtainCommitment
Maintain Trust
engaged engaged
low
engaged engaged
Not engaged
Maximize Trust
Appears uninterested
Psychographic Profiling through Clothes Recognition
Mens shirts: multiple features– Collar vs. crew neck– Short vs. long sleeve– Color, texture– Pattern, emblems
[Zhang, et al. IUI 2008]
What you wear says more about your tastes than demographics
Similarity: Example shirt matches
Shirt style classification Classes
SVM results
Class Collar Sleeve Button
T-shirt No Short No
Polo shirt Yes Short Half
Casual shirt Collar Short Full
Business shirt Collar Long Full
Classified as ->
T-shirt Polo Casual Business
T-shirt 80.8% 3.9% 15.4% 0%
Polo 16.7% 41.7% 8.3% 33.3%
Casual 0% 12.5% 50% 37.5%
Business 0% 5% 5% 90%
Overall accuracy: 72.7%
Sellers would approach someone wearing a T-shirt differently than someone wearing a Business shirt
Research Opportunities
Decision Engine & Objective Model• Select best abstract
response toward objective
Perception• Detect external cues that
indicate internal mental state
Composable Content• Content organized according to
abstract actions
Computer Vision• Robust algorithms to detect
specific behaviors• Measures of inaccuracy• Other Sensors
• Audio, thermal, pupil, etc.
Multimedia Data Structures• Efficient data structures for
realtime program re-composition
Ethnography• Internal user mental states• External behavioral cues• Abstract actions toward objectiveInteraction Engine• Develop realtime decision engine
Interdependencies Between Perception, Decision and Action Components
Decision Engine & Objective Model• Select best abstract
response toward objective
Perception• Detect external cues that
indicate internal mental state
Composable Content• Content organized according to
abstract actions
Computer Vision• Robust algorithms to
detect specific behaviors
• Measures of inaccuracy
Multimedia Data Structures• Efficient data
structures for realtime program composition
Ethnography• Identify user mental states• Identify external cues of
mental state• Identify abstract actions
leading to an objective
1. Decision engine and objective model depend on reliability of computer vision techniques.
2. Required computer vision depends on needs of decision engine and object model.
1. Structure of composable content framework depends on output of decision engine and object model.
2. Output of decision engine and object model should allow for realtime composition of content.
Responsive Interaction PlatformSensing of Environment
Image/VideoAnalyzer
AudioAnalyzer
Perception of Environment
Person ModelPerson Model
Person ModelModel of internal state
eye gazehand/body gesturesfacial expression
non-vocal soundsspeech
Emotional stateEnergy levelPatienceMental activity – thinking, confusionInterest levelAttitude toward informationHome position…
Interaction among peoplePositions and postures…
Decision EngineSelect “best” abstract action based on abstract state of environment and the objective. Use the framework of Partially Observable Markov Decision Processes (POMDP).
state of environment
Objective ModelThis is the objective function in the POMDP framework that defines what the “best” action is. Example Objectives: Increase brand awareness, Introduce new product, Direct sales to mobile device, Provide navigation information, …
Content Actuation EngineConvert abstract action to content segments.
Abstract action
Display Sound Ambient Motion, Lights
Promote Interest
Fast Animation Catchy music Movement, light flash
Gain trust Scenes of family life with product
Smooth music Non-distracting
… … … …
abstract action – e.g., Promote Interest, Gain Trust, Present
Product, make joke, …
actuator control
Sensor Analyzer
Interaction Engine
sensor features
Content Actuation
Interaction ModelThis is the sequencing structure in the POMDP framework that defines what stages the interaction should follow. E.g., Sales*:
• Approach• Assess• Relax• Pitch• Benefit• Reduce Resistance• Incentive to act
Interaction stages
Objective metrics
Summary
Mixed-Initiative Interaction generates new business opportunities Mixed-Initiative Interaction Engine
– Inference models to measure audience engagement» Identify the most predictive set of sensors and the cost tradeoffs
– Precise assessment metrics of content effectiveness– Engagement Detection
» Convert raw data to human-meaningful cues of engagement Dynamic content framework
– Maps abstract actions to content segments to achieve the objective– Tailorable to structure of engagements across multiple target domains
» Education, Training, Service, Sales, etc.
Far-reaching research and invention of next-generation interaction paradigm for media technologies– Displays, mobile device, speech conversation, etc.