understanding contextual factors in location-aware multimedia messaging

Understanding Contextual Factors in Location-aware Multimedia Messaging!

Abdallah  “Abdo”  El  Ali  

Frank  Nack  

Lynda  Hardman  

Outline!

I.  Introduc*on  

II.  Prototype  

III.  Diary  Study  

IV.  Results  

V.  Conclusions  &  Future  Work  

2 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

  Ubiquitous  Compu*ng  promises  to  populate  our  daily  lives  with  specialized  ‘context-­‐aware’  services  that  enhance  our  experience  

    easier  

    friendlier  

    efficient  

  Loca*on-­‐awareness  (GPS)    Context-­‐awareness  (human  intent?)  

3

Motivation!

(Weiser, 1995)"

(Dey, 2002)"

Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

! Case Study: Location-aware Multimedia Messaging (LMM)!

       Geo-­‐tagged  

mul*media:  photos,  text,  video,  audio    

     Made  at  a  loca*on  and  viewed  at  that  loca*on    

      Assump:on:  LMMs  can  

reflect  cultural  aspects  of  people’s  experiences  and  make  them  visible  at  loca*ons  

       Window  into  experiences?      

4

e.g.,  Photos  of  sunset,  tagged  with  sunset,  taken  at  *me  t  indicates  X  was  appreciating something

Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

What is an Experience?!

  Memory  View:  Result  of  an  experience  process.  Experience  memory  consists  of  one  or  more  actors,  spa*otemporal,  social,  cogni*ve,  and  affec*ve  aspects  

   “Looking  at  these  photos  reminds  me  of  the  good  *mes  we  had  at  the  last  ICMI  conference  in  Beijing”  

   Post-­‐hoc  representa*on          Used  as  a  framework  to  understand  the  contextual  

factors  in  LMM      Emphasis  on  message  produc*on    

5

based on idea of episodic memory "(Tulving, 1993)

Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Questions!

1.  What  contextual  factors  are  involved  in  using  LMM  systems?  

2.  Can  these  factors  inform  the  study  and  design  of  future  LMM  systems?    

      Exploratory  study!  6 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Related Work!     GeoMedia  (Papliatseyeu  &  Mayora,  2008):         -­‐  permits  aaaching  mul*media  messages  (as  images,  audio  

  or    video)  to  loca*ons       -­‐  no  user  study  

  GeoNotes  (Persson  &  Fagerberg,  2002)  and  E-­‐graffi*  (Burrell  &  Gay,  2002):    

    -­‐  loca*on-­‐aware  systems  that  allow  users  to  leave  textual        messages  such  as  reminders  or  post-­‐it  notes  at  loca*ons  

    -­‐  extensively  studied  in  real-­‐world  usage  contexts       -­‐  focus  on  user  reac*ons  to  designed  systems       -­‐  tools  used  as  loca*on-­‐based  e-­‐mail  

  We  were  interested  in:       -­‐  whether  mul*media  messages  can  help  understand  experiences  

7 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Prototype!

      LMM  prototype  on  the  Android  mobile  device      

  Allows  annota*on  of  loca*ons  with  mul*media  messages  (drawings,  text,  photographs)  

Tool  to  acquaint  study  par*cipants  with  LMM  concept  

8 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Prototype: Message Creation!

Create     -­‐  Touch-­‐based  

drawing  (doodles)     -­‐  Wri*ng  text  

Snap       -­‐  Photographs  

9

Create at t1"

Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Engage at t0"

Prototype: Message Viewing!

  Explore       When  user  is  at  the  right  

posi*on  and  orienta*on,  s/he  can  view  the  message  

  Message  appears  as  an  Augmented  Reality  overlay  on  the  camera  view  

10

View at t2"

Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Diary Study !

  Longitudinal  (~1  week)  mul*modal  diary  method  

11

(Amin, 2009)"

Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Categorization Task!

   Needed  for  inter-­‐coder  reliability  (n=6)        

  Messages  categorized  according  to  both:  

    Domain:  What  is  the  message  about?                                          (e.g.,  entertainment,  architecture)  

      Task:  What  was  the  purpose  of  the  message?           (e.g.,  apprecia*on,  cri*cism)  

12 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Diary Study: Participants !

8  subjects  (6  m,  2  f)  aged  between  13-­‐27  (M=  23;  SD=  4.4)  

13 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Diary Study: Materials !

   8  custom-­‐designed  paper  diaries,  where  each  had  a  template  asking:  

  1.  Ques*ons  about  the  message  

  2.  Ques*ons  about  the  subject  and  her  context  

 Post-­‐study  interview  ques*ons  

14 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Diary Study: Materials(1) !

1.  Ques*ons  about  the  expression:          

    -­‐  Date         -­‐  Time       -­‐  Message  format  (drawing,  text,  photo,    -­‐  

  video,  audio  recording,  other)       -­‐  Title  of  message  

    -­‐  Public  or  private          

15 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Diary Study: Materials(2) !

2.  Ques*ons  about  the  subject  and  her  context:  

  Spa*otemporal  (Q1,  Q4)     Social  (Q5)     Affec*ve  (Q3)     Cogni*ve  aspects  (Q2,  

Q7,  Q8,  Q9,  Q10,  Q11)  

16 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Diary Study: Materials(3) !

      Interview  Ques*ons:    

          -­‐  Difficulty  filling  in  the  diary       -­‐  Media  preference              -­‐  Awareness  and  experience  of  past  week              -­‐  Desire  to  view  and  write  message  metadata  

           -­‐  Willingness  to  use  a  future  mobile  applica*on      

17 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Procedure!

                 Info  brochure      

                  LMM  Prototype  Demo    

          Diary  (2  mul*media  messages  per  day  for  1  week)    

        Post-­‐study  Interview    

18 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Results!

  Total  message  count:  110  

  Vo*ng  “winner-­‐takes-­‐all”  procedure  used  to  analyze  Categoriza*on  Task  results  (n=6)    

  If  equal  number  of  responses  to  two  dis*nct  categories  then  expression  classified  under  both    

      Categoriza*on  Task  results  directly  assimilated  into  diary  

study  results    

19 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Media Choice!

  “In  the  beginning,  it  was  photos,  and  during  the  week,  because  it  wasn't  that  interes*ng,  I  used  more  text"    

Text  (symbolic)  can  be  used  to  express  something  beyond  quali*es  of  the  loca*on  itself  

Songs  act  as  surrogates  for  the  memory  of  a  place  

20

Total = 114%"

21

Domain Ratings (N=6) for 110 Messages!

Total = 113%"

22

Task Ratings (N=6) for 110 Messages!

Spatiotemporal Aspects!

  Controlling  for  Public  Place,  many  messages  were  about  Ac*vity  Repor*ng  (39%)    

   microblogging  behavior  (e.g.,  Twiaer  feeds)    

  Most  Urban  expressions  fell  into  Aesthe*cs  (63%)  and  Apprecia*on  (49%)    

   *ght  correspondence  between  being  outdoors  and  aesthe*c  apprecia*on  

23

  Difference  between  public  and  private  messages  and  messages  made  alone  or  in  the  company  of  others    

   Messages  made  alone  were  also  made  public  (76%)        

24

Social Aspects!

Affective Aspects!

  Tendency  between  being  alone  and  nega*vely  valenced  mood  (60%)  

   Cathar*c  outlet  typical  of  web  2.0  social  behavior?    

25

(Russell, 1980)

Cognitive Aspects!

  Causal  rela*on  between  prior  ac*vity  &  message  crea*on:  35%  

  Interview:  Awareness  of  daily  environment?    

    All  reported  planning     behavior,  but  not  if  the  tool     is  embedded  in  daily  life  

26 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Viewing Metadata(1)!           Context  metadata  (mood,  companions,  event)  desired:  6  

subjects     Standard  metadata  (name,  date,  *me)  desired:  2  subjects      

  "Not  at  first  sight,  that  would  ruin  my  personal  view  of  their  message.  But  it  should  be  available  if  wanted...why  the  message  was  made,  what  did  the  person  want  to  express."  

27 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Viewing Metadata(2)!

  No*fica*on:       Filter  by  context:  7  subjects       Query:  1  subject         “If  I'm  walking,  then  I'd  like  to  search  myself,  but  if  I'm      

biking,  I'd  like  noEficaEon  of  what  there  is”      

   No*fica*on  should  depend  on  the  situa*on  subjects  are  in  to  avoid  interrup*on  

   Personaliza*on  best  considered  itself  context-­‐dependent?  

28 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Limitations!

      2  expressions  per  day  for  one  week  is  unnatural     Cogni*ve  effort    

  Availability  of  media  capture  devices     Instruc*ons  insufficient  

29 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions"

Implications!

  Predominant  domain  (Aesthe*cs,  Entertainment)  and  task  (Apprecia*on,  Ac*vity-­‐repor*ng)  categories  in  experience    capture  behavior  

  Applica*on  personaliza*on  (‘when’)  should  depend  on  and  adapt  to  the  user's  context  (‘what’)  

  Capturing  experiences  (memory  view)  vs.  the  experience  of  capture  (process/interac*on  view)  

30 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions!

Conclusions!

  Paper  diary  a  useful  low-­‐fidelity  mechanism  for  understanding  contextual  factors  in  LMM  

         Experience  memory  framework  not  very  useful  and  effortul  

to  analyze    clear  behavioral  paaerns  emerge  with  larger  datasets?    

31 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions!

Current / Future Work!

  Path  planning  &  POI  recommenda*on  service  based  on  social  media  content  (e.g.,  FlickR  photos)  

   Experience  as  Memory  View  

  Mul*modal  &  Crossmodal  feedback  in  place  and  path  recommenders  to  enhance  city  explora*on  

   Experience  as  Interac*on  View  

32 Introduction" " Prototype" " " Diary Study" " Results " " Conclusions!

Thanks  for  listening.  Ques:ons?  

(VISIT  ME  AT  MY  DOCTORAL  SPOTLIGHT  POSTER)

33 Website:  http://staff.science.uva.nl/~elali/

References!A.  Papliatseyeu  and  O.  Mayora  Ibarra.  Nailing  the  reality  with  GeoMedia:  loca*on-­‐

aware  mul*media  tags.  In  Proceedings  of  MobiMedia’08  Conference,  Oulu,  Finland,  July  2008.  ACM.  

A.  Amin,  S.  Townsend,  J.  Ossenbruggen,  and  L.  Hardman.  Fancy  a  drink  in  canary  wharf?:  A  user  study  on  loca*on-­‐based  mobile  search.  In  INTERACT  ’09:  Proceedings  of  the  12th  IFIP  TC  13  Interna*onal  Conference  on  Human-­‐Computer  Interac*on,  pages  736–749.  Springer-­‐Verlag,  2009.  

E.  Tulving.  What  is  episodic  memory?  Current  Direc*ons  in  Psychological  Science,  pages  67-­‐70,  1993.  

J.  Burrell  and  G.  K.  Gay.  E-­‐graffi*:  Evalua*ng  real-­‐world  use  of  a  context-­‐aware  system.  Interac*ng  with  Computers,  14(4):301–312,  2002.  

K.  Dey.  Understanding  and  using  context.  Personal  &  Ubiquitous  Compu*ng,  5(1):4–7,  2001.  

M.  Weiser.  The  computer  for  the  21st  century.  Human-­‐computer  interac*on:  toward  the  year  2000,  pages  933–940,  1995.  

P.  Persson  and  P.  Fagerberg.  Geonotes:  a  real-­‐use  study  of  a  public  loca*on-­‐aware  community  system.  Technical  Report,  2002.  

Russell  J.  A.  (1980).  A  circumplex  model  of  affect.  Journal  of  Personality  and  Social  Psychology,  39:1161–1178.  

34