big media: multimedia goes big data
TRANSCRIPT
Technische Universität Darmstadt
Prof. Dr. Max Mühlhäuser
BigMedia:Multimedia goes Big Data
Video as example & driver:
YouTube: 100 h/min.
Internet: >50%, > 40 XB/mo.
LTE advanced!
Cities: >500,000 surveillance cams
e.g., London: 6 months screening effort
Plus: zillions of sensors
Crowd: cf. foto heatmaps
BigMedia @ ISM 2014 © M. Mühlhäüuser 2
A Few BigMedia Facts
www.statista.com/chart/624 ( Cisco Visual Networking Index)
www.sightsmap.com
Multimedia & BigData: Do They Blend?
BigMedia @ ISM 2014 © M. Mühlhäüuser 3
processinput output
capture interacttransport store
process
multimedia
sense analyze
big data
Volume Velocity
Variety Veracity
conventionaldata processing
Unified BigMedia pipeline:1) capture/sense 2) transport 3) store 4) process 5) interact/analyze
BigMedia
1. CAPTURE/SENSE
BigMedia @ ISM 2014 © M. Mühlhäüuser 4
capture interacttransport store
processsense analyze
BigMedia
Sensors (CaptureDevices):
1. Hard:1a) „scene capturing“
Camara, Mike1b) „value sensing“
Accelerometer, …
2. Soft:2a) Documents
browse, view, edit2b) Software
contextualcommand-lvl. stroke-lvl. use
BigMedia @ ISM 2014 © M. Mühlhäüuser 5
2x2 Categories of Sensors
Events in-situ
w/ Smartphone
Desire: link to…
people on site
(+net)
CoStream:
BigMedia @ ISM 2014 © M. Mühlhäüuser 6
Crowd Media Example: CoStream
Login Awareness Streaming
Portrait upright
BigMedia @ ISM 2014 © M. Mühlhäüuser 7
CoStream Key Interaction Concepts
Portrait parallelto the ground
Landscape
• Rotate to watch• Tap to stream
Invitations, Sharing:• Push & Pull
(friend list video)• Rating (Like/Dislike)• Vibration to indicate
BigMedia is Multisensory Media AudioVideo + SocialMedia + Location/Motion/…
BigMedia @ ISM 2014 © M. Mühlhäüuser 8
Another Example 1st General Trend
+ conventional multimedia“media type”: tweetsDomain:Small-ScaleIncident Detection
BigMedia is Blended Media:
Generation (Sources): User Generated
Authoriative
Automatic
Consumption (Targets):@Site
@Net
@Home / @Mobile
BigMedia @ ISM 2014 © M. Mühlhäüuser 9
2nd General Trend
BigMedia is “Mass” Media: (user generated, automatic, authoritative) selection @ receiver?
social & personal pref’s
automatic (BigData)
or: “summaries” for massive reduction
e.g., emotion metering
e.g., hotspot / trend indicators
e.g., event analytics
BigMedia @ ISM 2014 © M. Mühlhäüuser 10
3rd General Trend
BigMedia means …
mass amounts of (streams of)
blended (-source/-target)
multi-sensory (hard + soft)
media
BigMedia @ ISM 2014 © M. Mühlhäüuser 11
Intermediate Summary
2. TRANSPORT
1. net/edge processing
(2./3. basically skipped)
BigMedia @ ISM 2014 © M. Mühlhäüuser 12
capture interacttransport store
processsense analyze
BigMedia
The Issue: Transport vs. Store&Process
BigMedia @ ISM 2014 © M. Mühlhäüuser 13
Reality: “Field” Stakeholders: “POI”Infrastructure: “Net”
capture interacttransport store
processsense analyze
?!
Cloud?
BigMedia @ ISM 2014 © M. Mühlhäüuser 14
Net/Edge Processing & Latency Needs
real scenery encounteredby mobile user (here: outdoor!)
real scenery captured by camera
processed: 3D, semantics, location, orientation, …
virtual scenery overlayed3D, in real time, as user moves
Dual Reality (DR)
Goal: dual reality DR (100% VR + 100% reality)
+ universal semantics, in- & outdoor, 3D overlay
Grand challenge:
universal semantics (“my phone can see!”)- geometry, location, domains …
3D robust real-time registration~today: popular buildings only
BigMedia @ ISM 2014 © M. Mühlhäüuser 15
Net/Edge Processing & Latency Needs
Junaio, Layar, Wikitude, TK
indoorexample
photo courtesy of TUD GRIS & FhG IGD
We are facing the age of latency DR (see above) + speech
federated interaction
realtime media stream analytics
Cloud: not enough!
add ‘local cloud’
required: mobility, handover, replication, self-X
driver: excess processing capacity @net
cf. Microsoft™ Cloudlets, Cisco™ Fog Computing
SWOT?
- pro: low latency, ‘cheap’ & ctx-aware processingownership@origin (see below)
- con: distributed processing
BigMedia @ ISM 2014 © M. Mühlhäüuser 16
Net/Edge Processing: Cloudlets
2. Resilient Networks:
3. Highly Adaptive i.e. Fluid Networks
BigMedia @ ISM 2014 © M. Mühlhäüuser 17
Further Key Transport (=Net) Issues
Smart GridIndustrial Facilities
Smart CitiesSmart Transport
Fluctuation:- Density
- Intensity- Mobility
3. STORE
1. The Forgotten Forgetting
2. Distributed Storage (&Processing) Privacy?
BigMedia @ ISM 2014 © M. Mühlhäüuser 18
capture interacttransport store
processsense analyze
BigMedia
BigMedia: always-on recording LiveLogs, cf. Microsoft™ Sensecam
Analogy:
cam+mike eyes+ears
brain ‚intelligent‘ processing
Open Issue: data organization user swamped
Google, FB … surrender, timeline
(cf. Gelernter‘s lifestream)
even less organization!
remember brain: associative & forgetful
BigMedia @ ISM 2014 © M. Mühlhäüuser 19
1. The Forgotten Forgetting
Eyes/ears Bionics:
auto-organize + auto-consolidate (‘forget’)
idea: reinforce retrieved data + ‘neighbors’
but: too many neighbors!
idea: leverage user interaction
1. Acquire: meta data
2. Interact: N-dim. space
3D visualization, user picks 3-of-N
neighbors: cf. user’s selection!
3. Consolidate:
‘fade’ least reinforced data
BigMedia @ ISM 2014 © M. Mühlhäüuser 20
1. The Forgotten Forgetting
approach
BigMedia @ ISM 2014 © M. Mühlhäüuser 21
Ad 2: Interlude
processing& storage
@net/edge
latency & bandwidth
limits
excesspower
@net/edge
AlwaysOn
/ LiveLogs
Problems, e.g.:how to process,
to archive, …
Further oppor-tunities, e.g.:privacy thruownership
BigMedia @ ISM 2014 © M. Mühlhäüuser 22
2. BigMedia Privacy
Privacy rights, laws, or desiresw.r.t. PII: personally identifyable info.
AnonymizationData Thrift
big data
collect more datadon’t throw
any data away
Prof. Narayanan, Princeton: essentially impossible … in a foolproof way w/o losing … utility of data1
1: Privacy and Security: Myths and Fallacies of “Personally Identifiable Information” .CACM 53 (6), 2010, pp. 23-25
1. Cyberphysical Spaces
2. Cyberphysical Humans
BigMedia @ ISM 2014 © M. Mühlhäüuser 23
2. BigMedia Privacy: Things are getting worse
Consent?
latent PII
photos © economist, siliconangle, ebiz-results, REX, Thinkstock/Ninell_art
remote processingvs. local data
“quantified self”& Assistence
anonymousstore?
BigMedia @ ISM 2014 © M. Mühlhäüuser 24
Challenge
tustedstore
inte
rfac
e
?
4. PROCESS
Just a brief recap / overview, for the sake of time
BigMedia @ ISM 2014 © M. Mühlhäüuser 25
capture interacttransport store
processsense analyze
BigMedia
Selected: 3rd Processing Category
BigMedia @ ISM 2014 © M. Mühlhäüuser 26
observablesperceivables
1. hard
2. soft
learning
person group popul.whohow
offline
realtime
conceivables
capture interacttransport store
processsense analyze
1 of 3 processcategories:
machine learning
trace movement intentionexample:
BigMedia pipeline unify&standardize 3 paradigms1. conventional & statistical processing
2. machine learning
3. crowd processing
Remember: processing @net/edge (Cloudlets …) requires
modularization, smooth mobility, appropriate algorithms & models, …
BigMedia @ ISM 2014 © M. Mühlhäüuser 27
Process Categories, @Net Processing
5. INTERACT/ANALYZE
technology proliferation new interaction concepts
Mobile - Natural - Large Scale
(many other trends ignored for the sake of time)
BigMedia @ ISM 2014 © M. Mühlhäüuser 28
/capture interact
transport storeprocesssense analyze
BigMedia
5A. MOBILE INTERACTION
Resizable Displays
AR DR Displays
On-Body Interaction
BigMedia @ ISM 2014 © M. Mühlhäüuser 29
Lab equipment:
targets user experiments& controlled user studies:UI concepts for devices-to-be!
BigMedia @ ISM 2014 © M. Mühlhäüuser 30
Resizable Displays (1): Rollable
‚display size dilemma‘: an innovation driver
Resizable Displays (1): Rollables, contd.
…
(this was just a selection)further UI concepts concern,
e.g.: semantic zoom,visual clipboard,
horizontal scroll, …
BigMedia @ ISM 2014 © M. Mühlhäüuser 31
OLEDs … thin devices folding?
The question, again: interaction concepts towards UX?
BigMedia @ ISM 2014 © M. Mühlhäüuser 32
Resizable Displays (2): Foldables
BigMedia @ ISM 2014 © M. Mühlhäüuser 33
FoldMe Design Space
PicoProjectors? use environment as display
daylight, power efficiency
collaboration? privacy?
empty space / wall
hand-held hand jitter
HOWEVER: Add depth cam tangible information space
BigMedia @ ISM 2014 © M. Mühlhäüuser 34
Resizable Displays (3): Pico Projectors
©Samsung (Galaxy Beam)
Google glass (note: full computer) hype & $$$
multimodal
‘app ready’ in 2 sec. (vs. 22)
Two sets of open issues:interaction concepts & experience1. degree of ‘AR’
2. advancement of vision & speech
3. direct manipulation!!
4. sharing experience?- SeeWhatISee sufficient?
5. acceptance as eyeware fashion
6. privacy
Note: DR-ready sophisticated glasses 2-6 remain!
BigMedia @ ISM 2014 © M. Mühlhäüuser 35
ARDR displays: Google Glass lessons
Google folks (today!): “no line-of-sight, no AR”
proliferation of technologies interaction concepts
1. rollable! promising
2. glass! heavy invest vs. open issues
3. pico projector: information spaces?
4. foldable: interaction concepts
not promoted doomed to fail?
BigMedia @ ISM 2014 © M. Mühlhäüuser 36
Future Mobile Displays: Summary
… 3D displays? games as driver
Use Palm-of-Hand (plus fingers) for interactiona) buttons
b) sliders
c) numbers,
d) ..
On-Body Interaction: Hand
a)
b)
8c)
BigMedia @ ISM 2014 © M. Mühlhäüuser 37
Wireless Sensor
Arc-shaped Board (12 Touch Points)
Interaction Techniques
Evaluation
here: control audio @ “human audio device ”
38
On-Body Interaction: Ear
BigMedia @ ISM 2014 © M. Mühlhäüuser
5B. MORE NATURAL INTERACTION
implicit
tabletop
paper like
spoken
printed tangible
BigMedia @ ISM 2014 © M. Mühlhäüuser 39
Basis: user(s)‘ pose, emotion, attitude, …
‚Interaction‘: appearance
actions
Example (1): CouchTV
BigMedia @ ISM 2014 © M. Mühlhäüuser 40
Implicit Interaction: Idea, Example 1
Implicit control
Implicit suggest
Implicit pause and record
Rollables again, but collaborative
bridge phone tabletop
Implicit interaction: auto-adapt UI to physical ‘connectedness’
BigMedia @ ISM 2014 © M. Mühlhäüuser 41
Implicit Interaction (2), collaborative setting
BigMedia @ ISM 2014 © M. Mühlhäüuser 42
Table Based Interaction: Concern
Desktop PC: isolated
immersive tabletop:social
table: social
digitize
tabletop: isolated
augment
awareness/accessibility: interactive halo & icon, gradual & remote access, ‘exposè’ organization: teleporting, hybrid piling / hiding , hybrid binding
BigMedia @ ISM 2014 © M. Mühlhäüuser 43
Table Based Interaction: ObjecTop
Table Based Interaction: PeriTop
add top projection & depth camera
BigMedia @ ISM 2014 © M. Mühlhäüuser 44
-digital info atop occluders-about object or else-low resolution OK here
Table Based Interaction: CoMAP
BigMedia @ ISM 2014 © M. Mühlhäüuser 45
46
Table Based Collaboration: Permulin
personal
shared
personalized
output
personalized
output
personalized
input
personal
shared
personalized
output
personalized
input
personal
sharedshared
personalized
output
personalized
input
personal
BigMedia @ ISM 2014 © M. Mühlhäüuser
…
47
Table Based Collaboration: Permulin
Divide View
Merge Views
Interaction Concepts Divide / Merge
BigMedia @ ISM 2014 © M. Mühlhäüuser
48
View of User A
View of User B
49
Table Based Collaboration: Permulin
SharePeek
Sharing &Peeking…
Interaction Conepts Share / Peek
BigMedia @ ISM 2014 © M. Mühlhäüuser
50
View of User A
View of User B
51
Table Based Collaboration: Permuli
Better Parallel WorkLarger Interaction Area
User A User B
…
Mutual AwarenessTruly Fluid Transition
Kinect for user recog-nition
3D Display
Multi-touchframe
Kinect for hand recog-nition
Modified3D shutterglasses
hardware setup today
… and tomorrow?
evaluation results
BigMedia @ ISM 2014 © M. Mühlhäüuser
Several projects atop Anoto ePen technology!
just one pen for ….
BigMedia @ ISM 2014 © M. Mühlhäüuser 52
Paper Like (1): Pen&Paper Computing
Paper table (& wall) hybrid: paper+table physical objects
hand writtenannotations
tagging usingmenue cards
printed userinterfaces
folders books
Paper Like (2): Paper like displays
BigMedia @ ISM 2014 © M. Mühlhäüuser 53
Multiple Paper-like Displays
?
Custom printed objects …
… if interactive(!): boost „tangible interaction“
… were demonstrated by U Saarbrücken
… and by TK (@TU Darmstadt)
… are still at an early stage
BigMedia @ ISM 2014 © M. Mühlhäüuser 54
Printed Tangible Interaction
a b c
d e
Spoken Interaction?Example Smart-Space Dialogs
Speak to smart spaces -homogeneous UI vs. heterogeneous devices
requires:1.context awareness
2.user awareness (multi-speaker!)
3.mike awareness (array … headsets)
plus (not included here):
federation w/ other modalities
openness (cf. Siri++: for app/service developers)
BigMedia @ ISM 2014 © M. Mühlhäüuser 55
C. LARGE SCALE INTERACTION
(just briefly touched here)
In-door: e.g., walls
Out-door: e.g., facades
Global: e.g., social or virtual
Overarching challenge:Collaboration of / with a large user base
BigMedia @ ISM 2014 © M. Mühlhäüuser 56
Still lots of passive walls
Interactive walls still inappropriate
Remember lesson learnt today:
novel technology new interaction concepts
Collaborative wall interaction:
only few concepts known, rarely in use
large walls: real estate reach
quest for mobile device federation!
BigMedia @ ISM 2014 © M. Mühlhäüuser 57
Wall size interaction
Interactive Walls & Rooms
QUT Brisbane “CUBE”
negative example
City Scale Example: Media Facades
2 approaches:
1. Indirect interaction input “abstract aggregation”
cf. emotion metering, hotspots, …
2.Temporal individual control often as “competition”
user creativity framed by app
BigMedia @ ISM 2014 © M. Mühlhäüuser 58
Global Interaction
Here, CoStream@Home: in-situ socialNet home
BigMedia @ ISM 2014 © M. Mühlhäüuser 59
Tv Broadcast
CoStream
User Generated Videos
Notifications
Friend List
by default“compressed” to vibration
Cheering Frustration Clapping
D. ORTHOGONAL ISSUES
(one slide for brevity)
BigMedia @ ISM 2014 © M. Mühlhäüuser 60
Collaboration, Federation, Intelligence
Collaboration: mentioned above
local distributed, team social
quest for research (cf. our ConCalls!)
Federated Interaction leverage multimodality
challenge: open ad-hoc federation, latency
Intelligent UIs: Proactivity: adjust UI implicitly + in advance
Intelligibility: UI explains itself & its reasoning
BigMedia @ ISM 2014 © M. Mühlhäüuser 61
Web basedfederated UIs
SUMMARY
BigMedia @ ISM 2014 © M. Mühlhäüuser 62
capture interacttransport store
processsense analyze
BigMedia
Capture / Sense:mass amounts of blended-source/-target multi-sensory media
Transport: processing @edge/net (Cloudlets) + fluid networks + (if 24/7) resilience
Store: (partial) user site storage: novel processing, ‘forgetting’ & privacy opportunities
Process: Unify three BigData/Media pipelines: conventional + ML + crowd processing selected challenge: processing @edge/net
Interact/Analyze: many challenges, selected: proliferating technologies + interaction concepts mobile: resizable displays, AR DR, on-body interaction
(more) natural: implicit, table based, paper like, spoken
large scale: walls, facades, social networks
collaboration, federation, intelligence as orthogonal aspects
BigMedia @ ISM 2014 © M. Mühlhäüuser 63
For Your Long Term Memory
imagine consequences on industry sectors:
Software industry (every app ready for 25 sets of interaction concepts?)
Media industry (OSN convergence done right?)
Telecom industry (Cloudlets embraced?)
Critical infastructures (500k surveillance cameras plus mobile reports?)
BigMedia @ ISM 2014 © M. Mühlhäüuser 64
Food for Smalltalk