1Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Emerging Topics on Personalized and LocalizedMultimedia Information Systems
November 3, 2014
Yi Yu, Roger Zimmermann
School of Computing, National University of Singapore
-- Exploring Interesting Aspects Hidden in Location-Aware Multimedia Data from Individual Level to Society Level --
2Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Personalized Multimedia Service
S. Ramanathan, P. V. RanganIEEE MultiMedia archive Volume 1 Issue 1, March 1994 Page 37-46
Content Analysis
User
Profile Construction
User’s interests
Relevance feedback
Content Selection
3Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Society-based User behaviors
User1, .…. , Usern
Profile Construction
User interests
Relevance feedback
Personalized Recommend.
Individual User
Participatory Sensing
Location Information
Multimedia Data
Aggregating data of more users
Evolved Framework of Personalized Multimedia Service
4Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
People Life & Mobile Technologies
5Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Smart Mobile Services
Understanding a user, his physical and mental state
6Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Smart Search
7Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
https://ginger.io/
Smart Well-Being
8Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Smart Watch
9Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
People with Data
10Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Data about People (1)
11Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Data about People (2)
12Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
What Will be Covered ?
Correlating between preference-aware activity dataLocation-aware user profilingPersonalized geo-fencingWeb-based map personalizationParticipatory Sensing of Venue via Multimedia Events
13Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Correlating between Preference-Aware Activities
From a user-centric point of view Extract activity data
• E.g., online presence, physical check-ins Category by semantic concepts
14Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Smartphone
appVideo/Audio Association
UGV with Soundtrack
Ready for Sharing
Geo-feat., GVisual feat., F
Raw UGV
SVMhmm
Training
Scene MoodRecognition
f (MG,G), f (MF, F) SoundtrackRecommendation
Training dataset with Geo-tagged video
External Data Source
MusicE.g.: Mp3 Files, Tags
Server side
Personalized Music Soundtrack
(1)
(2)
(5)
Rajiv Ratn Shah, Yi Yu, Roger Zimmermann, ACM MM’14
Personalized Video Soundtrack Recommendation
Moodannotation
Visual feat., FGeo-feat., G
ModelsMG, MF
mood
Listening history
Personalization
offline Online
(3)
(4)
http://www.comp.nus.edu.sg/~yuy/matsrc_build_Updated.zipCodes at
15Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Geo-sensor info recorded with video Geo-sensor info geo-category geo feature mood tags with geo-aware likelihood CG
Visual content Visual feature mood tags with visual-aware likelihood CF
Likelihoods of mood tags (CG and CF) Mood tags with large likelihoods scene mood C
Mood Recognition from Video
Geo feature G
Visual feature F
Late
fusio
n
SVMf (MF, F)
SVMf (MG,G) CG
CF
Scene mood C
Geosensor
info
Visual content
Geocategory
Visual feature extraction(e.g., color histogram)
APIBag ofword
Rajiv, Ratn Shah, Yi Yu, Roger Zimmermann, ACM MM’14
16Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Matching Songs with User Preference
Sweet
Funny
Sad
Songs
Songs
Songs
Music databaseWith mood-tag as index
Scene mood C
Soundtrackretrieval
Listeninghistory
Personalizationby correlating audio features
Initial song list User specific songs
Mood tagSongs
Rajiv, Ratn Shah, Yi Yu, Roger Zimmermann, ACM MM’14
17Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
What Will be Covered ?
Correlating between preference-aware activity dataLocation-aware user profilingPersonalized geo-fencing Personalization of web-based map generalizationParticipatory Sensing of Venue via Multimedia Events
18Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
User Preference Profiling: Background
User modelingAn understanding of a user ( characteristics,
preferences, needs)
Modeling user location historyProvide personalized servicesGeographic data semantic geo-category
• (e.g., coordinates bar, shopping mall)
19Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Location-aware user profiling (approaches)Term Frequency-Inverse Document Frequency (TF-IDF)Sparse Additive Generative model (SAGE)Latent Dirichlet Allocation (LDA)
Examples Social check-ins & location preferenceCheck-in patterns & shopping habitTopical diversity, geographical diversity interest diversity
• Tweet prediction• Location inference
User Preference Profiling : Background
20Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Jie Bao, et al ACM GIS’12Location history (document), categories (terms) User preference hierarchy (TF-IDF)
Category Name Number of sub-categories
Arts & Entertainment 17
College & University 23
Food 78
Great Outdoors 28
Home, Work, Other 15
Nightlife Spot 20
Shop 45
Travel Spot 14
Check-ins & Preference
Number of venue c’ being visited by u
Number of users visiting venue c’
w1×sim1(ua,ub)+w2×sim2(ua,ub)
Total similarity between a and b
'| { . : . '} | | |. log
| . | |{ : ' . } |cu v v c c Ru w
uV u c u C== ×
∈
21Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Top 9 categories or 410 sub-categoriesTrade area analysisNot effective or too high dimensional
User profilingHistogram of user check-ins (sub-categories)Latent Dirichelet Allocation (LDA)
• Assumption– document: a mixture of topics– Topic: probability of mentioning a word
• Goal: calculate proportion of documents by examining word distribution
Check-ins & Habit Pattern: Background
David M. Blei, et al JMLR’03
22Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Yan Qu, et al WWW’13
Check-ins & Habit Pattern Distribution of topics (users) User (document) Topic (term)
Store profile: histogram of topics Generated by all customers
23Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
For a term in a model Term frequency is Log-frequency is Get distribution via normalization Addition of several models
exp( )( | )exp( )
v v
v vv v
p v β φφβ φ
= =
++++=++
v
gv
uvv
gv
uvvguvp
)exp()exp()|( 0
00
φφφφφφφφφ
gv
uvvv ββββ ⋅⋅= 0
: Basic reference model: Difference between one model and the reference model
: Difference between another model and the reference model
Termfrequency
Change rate of term frequency
Change rate of term frequency
Preliminaries in Log Space
vβvv βφ log=
)( 00 βφ)( uu βφ)( gg βφ
24Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
SAGE models the difference from a background distribution in log-space Use to denote the background model Other components , used to model the differences from the background
model Sparsity-inducing for each specific model (difference of term frequency) Generative facets combination through simple addition in log space
Jacob Eisenstein, ICML’11
Sparse Additive Generative Model
0φ
0φ
uφ
gφ
Background
Topic
Perspective
+
SparsityOnly a few non-zero items
Term distribution
uφ gφ
25Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
A probabilistic model considers User locations, global topics, regional language models
SAGE for Tweet PredictionLiangjie Hong, et al WWW’12
Pick a region r
Pick a topic z
Generate tweet d
Background LM
Topic Region LM
: user dep. distr.over regions
: global distr. over regions
: user dependent distribution over topics
: global distribution over topics
: regional distribution over topics
: global distribution over terms
: region-dependence of terms: topic matrix, each row is a distribution over terms
26Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Location Prediction for New TweetBased on the words in the tweet and user informationMost proper region given
Pick a region r
Pick a topic z
New tweet
Find
regi
on
Topics
User info
Maximize likelihood
likelihood
r: user dep. distr.over regions
: global distribution over topics
: user dependent distribution over topics
: global distr. over regions
: regional distribution over topics
27Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
What Will be Covered ?
Correlating between preference-aware activity dataLocation-aware user profilingPersonalized geo-fencing Web-based map personalizationParticipatory Sensing of Venue via Multimedia Events
28Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
A virtual perimeterCoverage of a radio cell or Wi-Fi access pointOr manually specified geographic shapeDifferent shapes (e.g., circles, rectangles, polygons)
Basic ideaUsers enter or exit boundaries of areas send notification
Geo-Fence
29Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Personalized Geo-Fencing (1)
Location-aware social networks
30Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Personalized Geo-Fencing (2)Keep track of children
31Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Personalized Geo-Fencing (3)Home securityAwareness what is happening at your property
32Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Pairing points with polygons Scalability
• Big-data
Basic spatial predicatesINSIDE, WITHIN
A novel geo-fencing algorithmSimple but effective and efficientLSH + probing
Efficient Geo-Fencing Algorithm
Yi Yu, SuhuaTang, Roger Zimmermann, ACM SIGSPATIAL GIS’13
33Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Geo-Fencing: Points & PolygonsPoints Multiple instances
• A unique sequence number Moving
Polygons Multiple instances
• A unique sequence number Changing
Sequence numbers Same space & no overlapping A timestamp
sequence
34Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Geo-Fencing: Polygon Instances Two types With a single out-ring With a single out-ring & multiple inner-rings
-1.5 -1 -0.5 0 0.5 1 1.5
x 104
-1.5
-1
-0.5
0
0.5
1
1.5x 10
4ID=7, seq=[7,410293,419522,701191,734903]
1
2
3
4
5
Tips: Inner rings
• Inside outer-ring• Separated from
outer-ring
35Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
-1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3x 10
4
-1
-0.5
0
0.5
1
1.5
2
2.5
3x 10
4
1
2
34
5
6
789
10
11
12
13
14
15
X axis (m)
Y a
xis (
m)
Geo-Fencing: Polygons, Point, Edges
# edges of 15 polygons (200)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
285 255 235 196 264 250 240 239 226 226 242 153, 15
152, 20
250 217
36Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Geo-Fencing: INSIDECrossing number algorithm
Inside (number of intersections = odd) Checking each edge
Exploiting minimum bounding rectangle (point outside MBR is surely outside polygon)
LSH-based acceleration (point inside MBR)
MBR: minimum bounding rectangle
V1,9
V1,8
V1,11
V1,10
V1,7 V1,6
V1,4
V1,3
V1,2
V1,1
P V2,1V2,2
V2,3V2,4
V2,5
V1,5
37Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Geo-Fencing: WINTH
V1,9V1,8
V1,11
V1,10
V1,7 V1,6
V1,5
V1,4
V1,3
V1,2
V1,1
P
V2,1V2,2
V2,3V2,4
V2,5
dthdth
Point outside MBR but within a distance A rectangle centered at the point, edge length=2 * dth
• Non-overlap, point surely not WITHIN a distance
38Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Point file
Polygonfile
Polygon cache
MBRsin R-tree
Edges in LSH table
R-tree based pre-filtering
LSH-based INSIDE
detection
LSH-basedWITHINdetectionPo
lygo
n m
anag
emen
t
INSIDE result
WITHIN result
Pairing engine
Geo-Fencing: Scalable Framework A point inside MBR of a polygon Adapt to crossing number algorithm A probing scheme to lookup edges
Yi Yu, SuhuaTang, Roger Zimmermann, ACM SIGSPATIAL GIS’13
39Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Geo-Fencing: Point, Latest Instances of Polygons
-1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3x 10
4
-1
-0.5
0
0.5
1
1.5
2
2.5
3x 10
4
1
2
34
5
6
789
10
11
12
13
14
15
X axis (m)
Y a
xis (
m)
15 MBRs, three groups (an R-tree) MBR: minimum bounding rectangle
40Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
A separate hash table for each polygon
An edge within a bucket (x range within a bucket’s x sub-range)
Hash function T = (Xmax – Xmin) / N HashKey(x) = int ((x – Xmin)/T)
An edge (x1, y1)—(x2, y2) stored in buckets from key1 to key2
key1=HashKey(x1) key2=HashKey(x2)
Geo-Fencing: INSIDE Detection
B0 B1 B2 B3 BN-1
P
x0 x1 x2 x3 x4 xN
Xmin
Xmax
Yi Yu, SuhuaTang, Roger Zimmermann, ACM SIGSPATIAL GIS’13
41Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Geo-Fencing: WITHIN Detection
x0 x1 x2 x3 x4 xN
Buckets probed for WITHIN dth of Polygon
B0 B1 B2 B3 BN-1
dth
dth
dth
dth
III
III IV
P3dth dthPT
PL
PB
PRP2
WITHIN in three cases Inside polygon
Inside inner ring
Outside outer ring
Optimization Range of a point Divide outer area into 4
ranges Only check edges in the same
range as the point
P1
P1
P2P3
P3
probed
Yi Yu, SuhuaTang, Roger Zimmermann, ACM SIGSPATIAL GIS’13
42Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
What Will Be Covered ?
Correlating between preference-aware activity dataLocation-aware user profilingPersonalized geo-fencing Web-based map personalizationParticipatory Sensing of Venue via Multimedia Events
43Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Web-Based Map Personalization (1)Describing a part of the physical world for the user
44Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Web-Based Map Personalization (2)
Map customization: annotating personal maps with e.g., landmarks, routes, custom shapes
Map simplification: removing non-relevant details while retaining personalized details on small screens
45Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Map Customization
My Maps
46Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Map Simplification: Background (1)
Producing maps with less detail
Personal GIS, business mapping applications
Reducing data without losing general shape of map
http://openstreetmap.us/~migurski/streets-and-routes/
47Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Map Simplification: Background (2)
User preferences Specify his own constraining points to
control where to keep more details of his personal map
User 1 ( p1, p2, p3)
User 2 (p1, p3, p4, p5)
48Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Map Simplification: TaskAccess to linear geometries along with points
Reducing #vertices while preserving topological constraints
-8.1 -8.05 -8 -7.95 -7.9 -7.85
x 106
5.2
5.25
5.3
5.35
5.4
5.45
5.5
5.55
5.6
5.65x 10
6
Inputs: linear geometries (27) & constraining points (26)
49Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Topological relationships between original set of input linear geometries does not change after simplification
Relationship between constraining points and linear geometries before and after simplification does not change
Map Simplification: Two Constraints
http://mypages.iit.edu/~xzhang22/GISCUP2014/problem.php
(1) before
(1) after
(2) before
(2) after
50Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Douglas-Peucker algorithm (1973)
Visvalingam-Whyatt algorithm (1993)
Map Simplification: Two Algorithms
51Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
First and last points to be kept Point having maximum
perpendicular distance P4 in Fig. 1 is larger than the
tolerance
Whole linear geometry split into two parts at P4 Apply to linear geometry P2 and P7 have the maximal
perpendicular distance
(2) Tentative simplified segment
(1) Initial simplified line
(3) Final version of simplified line
P6P5
P7
P8
P3
P4
P2
P1
P6
P5
P7
P8
P3
P4
P2
P1P4
P6
P5
P7
P8
P3P2
P1
Tolerance
Douglas-Peucker Algorithm & Example (1973)
52Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Pseudo Codes for Douglas–Peucker Algorithm--using a given threshold tolerance--
function DouglasPeucker(PointList[], epsilon)// Find the point with the maximum distancedmax = 0, index = 1, end = length(PointList)for i = 2 to (end - 1) {
d = shortestDistanceToSegment(PointList[i], Line(PointList[1], PointList[end]))if ( d > dmax ) { index = i, dmax = d }
}if ( dmax > epsilon ) { // Recursive call
recResults1[] = DouglasPeucker(PointList[1...index], epsilon)recResults2[] = DouglasPeucker(PointList[index...end], epsilon)// Build the result list
ResultList[] = {recResults1[1...end-1] recResults2[1...end]}} else {
ResultList[] = {PointList[1], PointList[end]}}return ResultList[]
53Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Visvalingam-Whyatt Algorithm & Example (1993)
(2) Effective area for P6
(1) Effective area for P2(minimal )
(3) Simplified line with six vertices
P6P5
P7
P8
P3
P4
P2
P1
P6
P5
P7
P8
P3
P4
P2
P1
Original line
The triangle formed by original point and its immediate neighbors
Computing effective areas for all original points
P6
P5
P7
P8
P3
P4
P2
P1
Iteration, effective areas of points adjacent to the removed one are recalculated
54Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Pseudo Codes for ModifiedVisvalingam-Whyatt
ResultList = PointList.clone()for i=2 to i=length(ResultList)-1
Comp. effectiveArea[i] of ResultList[i], (triangle by ResultList[i-1, i, i+1])while length(ResultList) >number_to_keep
Find the point (minIndex) with least effective area// Remove the point with least effective areaif violate(PointList[minIdx], Constraint[])
effectiveArea[minIdx] = ∞else
update effectiveArea[minIdx-1], effectiveArea[minIdx+1]ResultList.remove(minIdx)
return ResultList
Iteratively remove the point with least effective areafunction VisvalingamWhyatt(PointList[], number_to_keep)
55Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Simplification
Before After
Maintain the topological consistency & satisfy point constraint
-8.1 -8.05 -8 -7.95 -7.9 -7.85
x 106
5.2
5.25
5.3
5.35
5.4
5.45
5.5
5.55
5.6
5.65x 10
6
-8.1 -8.05 -8 -7.95 -7.9 -7.85
x 106
5.2
5.25
5.3
5.35
5.4
5.45
5.5
5.55
5.6
5.65x 10
6
56Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
(a) User A ’s constraining points (b) User B ’s constraining points
-8.1 -8.05 -8 -7.95 -7.9 -7.85
x 106
5.2
5.25
5.3
5.35
5.4
5.45
5.5
5.55
5.6
5.65x 10
6
-8.1 -8.05 -8 -7.95 -7.9 -7.85
x 106
5.2
5.25
5.3
5.35
5.4
5.45
5.5
5.55
5.6
5.65x 10
6
A
B
ExampleRealizing user preference by setting
constraining points
57Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
What Will be Covered ?
Correlating between preference-aware activity dataLocation-aware user profilingPersonalized geo-fencingPersonalization of web-based map generalizationParticipatory Sensing of Venue via Multimedia Events
58Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Participatory Sensing
Concept of communities
Using personal mobile devices and web services to systematically explore interesting aspects
E.g., urban computing, public health, cultural identification, mobile multimedia computing
http://en.wikipedia.org/wiki/Participatory_sensinghttp://www.mobilizingcs.org/about/participatory-sensing
59Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Venue-Sensing Systems
E.g., Foursquare, Instagram
60Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
https://foursquare.com/v/the-metropolitan-museum-of-art/427c0500f964a52097211fe3/photos
Participatory Sensing of Venue via Multimedia Events
61Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Sensed Data in FoursquareVast volumes E.g., check ins, venue photos, venue tips
Valuable knowledge
62Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Associated with geographic category (e.g., beach, food)Leveraged for user activity analytics
Heterogeneous Information
63Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Implications of Sensed Data in FoursquareUser physical activities & online sharing
behaviors Personalized information & participatory
sensingApplied to research Location-aware recommendation
• E.g., providing media advertisement, travel plan
Urban computing• E.g., providing sustainability and outlook of
urban environment, people life quality, city planning, social sciences
64Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Check ins in FoursquareAn intersection of virtual social networks and
physical worldAs of December 2013, 45 million registered users
with 5 billion check-ins Spreading the world about their favorite spots
http://en.wikipedia.org/wiki/Foursquare
65Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Business storefronts and interiors (e.g., restaurant) and service contents
Photos in Foursquare
66Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Tips in Foursquare
67Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
#Venues #Users #Check-ins #Tips #Photos
LA 63,991 166,922 39,693,415 321,386 906,820
NYC 126,658 341,545 109,230,334 890,750 1,821,591
Empirical Observation of User Activities in Foursquare: Motivation
Yi Yu et al
Distribution and relationshipObserve interesting things behind data
Release data and source codes used in this study
68Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Empirical Observation of User Activities in Foursquare: Data Description
Yi Yu et al
A Foursquare venue A physical location
• E.g., Union Square Park (Outdoors & Recreation--Park), Times Square (Outdoors & Recreation--Plaza), John F. Kennedy International Airport (Travel & Transport--Airport)
Two regions (NYC & Los Angeles) 10 primary categories
• Arts & Entertainment, College & University, Event, Food, Nightlife Spot, Outdoor & Recreation, Professional & other places, Residence, Shop & Service, and Travel & Transport
Release data and source codes used in this study
69Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
1 2 3 4 5 6 7 8 9 100
0.2
0.4
Category index
Hist
ogra
m o
f #tip
NYCLA
1 2 3 4 5 6 7 8 9 100
0.2
0.4
Category index
Hist
ogra
m o
f #ph
oto
NYCLA
1 2 3 4 5 6 7 8 9 100
0.2
0.4
Category index
Hist
ogra
m o
f #ch
ecki
n
NYCLA
Distributions of tips, photos, and check-ins
1. Arts & Entertainment
2. College & University
3. Event
4. Food
5. Nightlife Spot
6. Outdoor & Recreation
7. Professional & other places
8. Residence
9. Shop & Service
10. Travel & Transport
newly added
Tow similarities: Similar in different regions
Similar trend in different categories
70Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Sun Mon Tue Wed Thu Fri Sat10
4
105
106
Day of week
Hist
ogra
m o
f #ph
oto
FoodShop & ServiceNightlife SpotProfessional & Other PlacesTravel & TransportArts & EntertainmentOutdoors & Recreation
Sun Mon Tue Wed Thu Fri Sat10
3
104
105
Day of week
Hist
ogra
m o
f #tip
FoodShop & ServiceNightlife SpotProfessional & Other PlacesTravel & TransportArts & EntertainmentOutdoors & Recreation
Dynamics of Sharing Activities in Foursquare
Similar trend: more popular at weekends
More professional activities on weekdays
Some differences
More tips on weekdays
More photos on weekend
71Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
100
101
102
103
104
10-5
100
(a) #tip per venue
NYCLA
100
101
102
103
104
10-5
100
(b) #photo per venue
CCD
F of
#ve
nue
NYCLA
100
101
102
103
104
10-2
100
(c) #checkin per venue
NYCLA
CCDF for #tips, #photos, and #checkinsat Venues in Foursquare
Almost half venues have only one tip while few venues have more than 100 tips
A common trend: only few venues have a large number of events
CCDF: Complementary cumulative distribution functions
72Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
100 10
1 102 10
3
100
101
10210
3
10-4
10-3
10-2
10-1
#Tip per user#Photo per user
Hist
ogra
m
0.02
0.04
0.06
0.08
0.1
0.12
0.14
Histogram of users in terms of <#tip, #photo>
100 10
1 102 10
3 104
10010
110210
3104
10-4
10-3
10-2
10-1
#Tip per venue#Photo per venue
Hist
ogra
m
0
0.02
0.04
0.06
0.08
0.1
0.12
Histogram of venues in terms of <#tip, #photo>
Different preferences: posting tips & photos
73Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
10-3
10-2
10-1
100
101
102
10-4
10-3
10-2
10-1
100
Inter-visit time (day)
CCD
F of
inte
r-vi
sit ti
me
TipPhoto
100
101
102
10-4
10-3
10-2
10-1
100
Inter-visit distance (km)CC
DF
of in
ter-
visit
dist
ance
TipPhoto
Distribution of inter-visit time and inter-visit distance
Inter-visit time: time interval between two successive events 50%: tips, 7.3 days, photos, 1.83 days; average: tips, 50.0days, photos, 17.3days
Inter-visit distance: distance between two venues successively visited) 50%: tips, 3.72km, photos, 4.03km; average: tips, 6.10km, photos, 6.67km
50%
74Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Different interests over different categories Variations of tips or photos per-category
A non-uniform distribution (a large fraction of visits in few categories)
Measuring user interest
Interest Focus/Entropy (1)
75Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Interest focus F: highest fraction of visits per category A high interest focus interest limited to a specific category
Interest entropy H: standard entropy Fraction of visits to a category as a probability Higher values (more uniform)
Interest Focus/Entropy (2)
max iki k
ikk
vFv
=
c1 c2 c3 c4 cN
−=k
ikiki ppH 2logik
ikikk
vpv
=
Probability density function
76Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Interest focus Nearly 50% users have an interest focus greater than 0.5 Many users have a primary interest
Interest entropy Only 20% users have an interest entropy of photo greater than 1bit The interest entropy of tips is lower
Interest Focus/Entropy (3)
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
Focus per user
CCD
F of
#us
er
TipPhoto
0 0.5 1 1.5 2 2.5 30
0.2
0.4
0.6
0.8
1
Entropy per user (bit)
CCD
F of
#us
er
TipPhoto
CCDF: Complementary cumulative distribution functions
0.5
77Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Initial Observations in Foursquare
Category dynamics among venue photo sharing, tip sharing, and check ins have analogous geo-temporal rhythms
Shared venue photos are highly relevant to food
Users prefer to share photos rather than tips
Venue photos are more important in promoting Venues
78Emerging Topics on Personalized and Localized Multimedia Information Systems in MM14 –Y. Yu, R. Zimmermann
Discussion on This Study of FoursquareThe potential applications Location recommendation Trip planning Media advertising Urban environment improvement City operation
Distributing our data and source codes http://www.comp.nus.edu.sg/~yuy/MyAnalysis.zip
Collecting data from broader regions More detailed usage of sensed data Prediction models