intelligent interactive systems - ambient intelligence + mobile technology

Fly See Buy

Barry KolleeSelvi RatnasingamSylvia van SchieWouter StuifmeelRobert Jan Prick

INTELLIGENT INTERACTIVE SYSTEMS | OCTOBER 8TH, 2013

Introduction: Topics

- ambient intelligence

- mobile technology

Introduction: Topics

ambient intelligence (en.wikipedia, 7/9/2013):

“ambient intelligence (AmI) refers to electronic environments that are sensitive and responsive

to the presence of people”

Introduction: Topics

Mobile technology (en.wikipedia, 7/9/2013):

“a standard mobile device has gone from being no more than a simple two-way pager to being

a mobile phone, GPS navigation device, an embedded web browser and instant messaging

client, and a handheld game console”.

Introduction: Vision

Weiser, M. The Computer for the 21st Century, Scientific American (1991)

Introduction: Ambient Intelligence

Weiser, M. The Computer for the 21st Century, Scientific American (1991)

“Specialized elements of hardware and software, connected by wires, radio waves and

infrared, will beso ubiquitous that no one will notice their

presence”

Introduction: Mobile technology

“Little is more basic to human perception than physical juxtaposition, and so ubiquitous

computers must know where they are. If a computer knows merely what room it is in, it can adapt its behavior in significant ways….”

Weiser, M. The Computer for the 21st Century, Scientific American (1991)

Introduction: Vision

Weiser, M. The Computer for the 21st Century, Scientific American (1991)

Introduction: Goal

“When things disappear ….. we are freed to use them without thinking and so to focus on

new goals”

Weiser, M. The Computer for the 21st Century, Scientific American (1991)

The Big Data Challenge

● Lausanne Data Collection Campaign

● A large-scale mobile data resource

● ‘Privacy by design’

● Image logfiles

● Monitor entire smartphone (N95)

Laurila, J. K., Gatica-Perez, D., Aad, I., Blom, J., & Bornet, O.The mobile data challenge: Big data for mobile computing research. , http://privacybydesign.ca/

The Big Data Challenge

Laurila, J. K., et al.The mobile data challenge: Big data for mobile computing research.

The Big Data Challenge

● Semantic place prediction

● Next place prediction

● Demographic attribute prediction

Laurila, J. K., et al.The mobile data challenge: Big data for mobile computing research.

Applying to Fly See Buy system

“Each data type corresponds to a table in which each row represents a record such as a phone call or an observation of a WLAN access point. User IDs and

timestamps are the basic information for each record.“

Laurila, J. K., et al.The mobile data challenge: Big data for mobile computing research.

How long?

● Investigate properties of learning

● Predicting social and individual models

Altshuler, Y., (2012). Incremental learning with accuracy prediction of social and individual properties from mobile-phone data.

‘Reality mining’

A. Pentland, in The Global Information Technology Report 2008-2009 (World Economic Forum, Geneva, 2009)

Methodology

● Classifiers○ Personal properties (first level)○ Social links (life-partner?)

● Correlation amount of time vs. accuracy

Altshuler, Y., Aharony, N., Fire, M., Elovici, Y., & Pentland, A. S. (2012). Incremental learning with accuracy prediction of social and individual properties from mobile-phone data.

Methodology

● Android Application○ GPS○ Accelerometer○ Third Party application○ Cell tower ID’s○ WIFI LAN ID’s (proximity)

Altshuler, Y., Aharony, N., Fire, M., Elovici, Y., & Pentland, A. S. (2012). Incremental learning with accuracy prediction of social and individual properties from mobile-phone data.

Methodology

● Feature vector○ Location○ Sms-pattern○ Internet usage○ Call-pattern○ Phone applications○ Alarms

● Friends and family dataset (140 people)

Altshuler, Y., Aharony, N., Fire, M., Elovici, Y., & Pentland, A. S. (2012). Incremental learning with accuracy prediction of social and individual properties from mobile-phone data.

Conclusion

● Ethnicity 60%

● Is student

● Significant other 65 %

Altshuler, Y., Aharony, N., Fire, M., Elovici, Y., & Pentland, A. S. (2012). Incremental learning with accuracy prediction of social and individual properties from mobile-phone data.

Conclusion

Modeled using ‘gompertz function’

“At a moment I can say with an amount of certainty who you are.”

Laurila, J. K., Gatica-Perez, D., Aad, I., Blom, J., & Bornet, O.The mobile data challenge: Big data for mobile computing research.

AdNext: A Visit-Pattern-Aware Mobile Advertising System | for Urban Commercial Complexes

Keywords

● Mobile advertising

● Sequential visit patterns

● Prediction models

● Wi-Fi localization

● User survey

Kim, B., Ha, J., Lee, S., Kang, S., Lee, Y., Rhee, Y., . . . Song, J. (2011). AdNext: A visit-pattern-aware mobile advertising system for urban commercial complexes. Proceedings of the 12th Workshop on Mobile Computing Systems and Applications, Phoenix, Arizona. 7-12.

COEX Mall

● Largest commercial complex in South Korea

● 260 stores / 100.000 visitors per day

● Customer targeting

○ Spatial relevance

○ Temporal relevance

Image: Official Site of Korea Tourism http://www.visitkorea.or.

kr/enu/SI/SI_EN_3_1_1_1.jsp?cid=736121Kim, B. et al AdNext: A visit-pattern-aware mobile advertising system for urban commercial complexes. Proceedings of the 12th Workshop on Mobile Computing Systems and Applications, Phoenix, Arizona. 7-12.

AdNext System Architecture

Kim, B. et al AdNext: A visit-pattern-aware mobile advertising system for urban commercial complexes. Proceedings of the 12th Workshop on Mobile Computing Systems and Applications, Phoenix, Arizona. 7-12.

Collecting Place Visit History

● store-level localization accuracy

● identify users’ current location (using accelerometer)

● detect in/out time (location change validation)

Kim, B. et al AdNext: A visit-pattern-aware mobile advertising system for urban commercial complexes. Proceedings of the 12th Workshop on Mobile Computing Systems and Applications, Phoenix, Arizona. 7-12.

Next Visit Prediction Model 1/2

Bayesian Networks – Probabilistic graphical modelIt models the joint probability P(X, Y), where X represents features and Y represents labels. Main features:● visit place (P)● visit time (T)● visit duration (D)● gender (G) - static● age (A) - static

Kim, B. et al AdNext: A visit-pattern-aware mobile advertising system for urban commercial complexes. Proceedings of the 12th Workshop on Mobile Computing Systems and Applications, Phoenix, Arizona. 7-12.

Next Visit Prediction Model 2/2

Kim, B. et al AdNext: A visit-pattern-aware mobile advertising system for urban commercial complexes. Proceedings of the 12th Workshop on Mobile Computing Systems and Applications, Phoenix, Arizona. 7-12.

Accuracy

● Data collection

● Prediction accuracy

● Comparison

Kim, B.et al (2011). AdNext: A visit-pattern-aware mobile advertising system for urban commercial complexes. Proceedings of the 12th Workshop on Mobile Computing Systems and Applications, Phoenix, Arizona. 7-12.

Discussion

● Privacy concerns

● Energy consumption

Kim, B.,et al (2011). AdNext: A visit-pattern-aware mobile advertising system for urban commercial complexes. Proceedings of the 12th Workshop on Mobile Computing Systems and Applications, Phoenix, Arizona. 7-12.

Context-aware Media Agent for Public Spaces | Guide visitors in Museums

Authors: Ichiro Satoh

Satoh, Ichiro. Context-aware Media Agent for Public Spaces. Lansdale PA 19446: IOS Press, doi:10.3233/978-1-60750-606-5-407, 0. Print. Ichiro, Satoh, (2008) Context-Aware Agents to Guide Visitors in Museums, National Institute of Informatics, Tokyo, Japan Pages 441-455,

Portable terminals are not desirable

● requires end-users to carry players along

● require explicit input

● expensive devices to lend

● require regular maintenance

● interacting difficulties

● prevent visitors from focusing on the exhibits

Satoh, Ichiro. Context-aware Media Agent for Public Spaces. Lansdale PA 19446: IOS Press, doi:10.3233/978-1-60750-606-5-407, 0. Print. Ichiro, Satoh, (2008) Context-Aware Agents to Guide Visitors in Museums, National Institute of Informatics, Tokyo, Japan Pages 441-455,

Agent runtime system

Mobile agent

● annotation

● navigation

● user preference

Satoh, Ichiro. Context-aware Media Agent for Public Spaces. Lansdale PA 19446: IOS Press, doi:10.3233/978-1-60750-606-5-407, 0. Print. Ichiro, Satoh, (2008) Context-Aware Agents to Guide Visitors in Museums, National Institute of Informatics, Tokyo, Japan Pages 441-455,

User navigation patterns

Satoh, Ichiro. Context-aware Media Agent for Public Spaces. Lansdale PA 19446: IOS Press, doi:10.3233/978-1-60750-606-5-407, 0. Print. Ichiro, Satoh, (2008) Context-Aware Agents to Guide Visitors in Museums, National Institute of Informatics, Tokyo, Japan Pages 441-455,

Context-aware Annotation in Museum

Satoh, Ichiro. Context-aware Media Agent for Public Spaces. Lansdale PA 19446: IOS Press, doi:10.3233/978-1-60750-606-5-407, 0. Print. Ichiro, Satoh, (2008) Context-Aware Agents to Guide Visitors in Museums, National Institute of Informatics, Tokyo, Japan Pages 441-455,

Conclusion

This paper only described the design and implementation of an agent-based system for building and operating context-aware visitor-guide services in public museums.

Satoh, Ichiro. Context-aware Media Agent for Public Spaces. Lansdale PA 19446: IOS Press, doi:10.3233/978-1-60750-606-5-407, 0. Print. Ichiro, Satoh, (2008) Context-Aware Agents to Guide Visitors in Museums, National Institute of Informatics, Tokyo, Japan Pages 441-455,

Need a location-based sensor to track

the user.

Most used sensors

● RFID

● NFC

● WiFi

Probability kernel regression for WiFi localisation

How the KL-divergence kernel regression algorithm bridges the gap with other WiFi

localisation algorithm?

Mirowski, P., Whiting, P., Steck, H., Palaniappan, R., MacDonald, M., Hartmann, D., et al. (2012). Probability kernel regression for WiFi localisation. Journal of Location Based Services, 6(2), 81-100.

WiFi localisation based on

● Fingerprints

Mirowski, P., Whiting, P., Steck, H., Palaniappan, R., MacDonald, M., Hartmann, D., et al. (2012). Probability kernel regression for WiFi localisation. Journal of Location Based Services, 6(2), 81-100.

Fingerprint?

The received signal strength (RSS) values from multiple access points (AP) are collected at different reference points and these reference points with related RSS are

referred as fingerprints.

Mirowski, P., Whiting, P., Steck, H., Palaniappan, R., MacDonald, M., Hartmann, D., et al. (2012). Probability kernel regression for WiFi localisation. Journal of Location Based Services, 6(2), 81-100.

Pattern matching process

Apply the learned model to real-time RSS sample.

Mirowski, P., Whiting, P., Steck, H., Palaniappan, R., MacDonald, M., Hartmann, D., et al. (2012). Probability kernel regression for WiFi localisation. Journal of Location Based Services, 6(2), 81-100.

Fingerprint

Mirowski, P., Whiting, P., Steck, H., Palaniappan, R., MacDonald, M., Hartmann, D., et al. (2012). Probability kernel regression for WiFi localisation. Journal of Location Based Services, 6(2), 81-100.

Apply different statistical algorithm

● K-nearest neighbour● Artificial neural network

● KL-divergence kernel regression algorithm

Mirowski, P., Whiting, P., Steck, H., Palaniappan, R., MacDonald, M., Hartmann, D., et al. (2012). Probability kernel regression for WiFi localisation. Journal of Location Based Services, 6(2), 81-100.

Contrasting dataset

● Office building (with dense and repeated fingerprints)

● Auditorium (a large open space)

● Public space (with mixed layouts and heavy pedestrian traffic)

Mirowski, P., Whiting, P., Steck, H., Palaniappan, R., MacDonald, M., Hartmann, D., et al. (2012). Probability kernel regression for WiFi localisation. Journal of Location Based Services, 6(2), 81-100.

Challenges

● Device-independent measurements

● Conditional independence of the RSS from a single AP

● Choosing parameters, like N fingerprints, N RSS sample

Mirowski, P., Whiting, P., Steck, H., Palaniappan, R., MacDonald, M., Hartmann, D., et al. (2012). Probability kernel regression for WiFi localisation. Journal of Location Based Services, 6(2), 81-100.

Conclusion

Very flexible and generalise several existing WiFi localisation algorithm

Mirowski, P., Whiting, P., Steck, H., Palaniappan, R., MacDonald, M., Hartmann, D., et al. (2012). Probability kernel regression for WiFi localisation. Journal of Location Based Services, 6(2), 81-100.

Ambient system

● Project goals○ Personalised recommendations for venues

(shopping, dining, etc.) in an airport terminal environment

○ Recommendations are represented in an ambient system in the public space

○ More foot traffic to airport venues○ Increased revenue of venues

Points by Breakfast NY

How do you distinguishan ambient display from a regular

display?

Designing interactivity awareness for ambient displays

● Addresses particular aspects of interaction regarding ambient displays

● How can users make out which display is interactive, and which isn't?

● How can users tell which type of interface the ambient display exposes?

Vatavu, R. D. (2013). On designing interactivity awareness for ambient displays. Multimedia Tools and Applications, 1-22.

Interaction techniques

A. Using software installed on the mobile device

B. Natural interaction via pointing and gestures

C. Touch-based interaction

D. Combining mobile phones and gestures

Vatavu, R. D. (2013). On designing interactivity awareness for ambient displays. Multimedia Tools and Applications, 1-22.

Example of public display

Clear Channel gaat digitaal op Amsterdam CS. (2012, December 4). Clear Channel Hillenaar - Buitenreclame. Retrieved October 7, 2013, from http://www.clearchannel.nl/nl/Nieuws/Laatste-nieuws/Default.aspx?PageId=535

Study

● Study to investigate perception of interactivity of displays

● Design○ Repeated-measures design, 3 independent

variables:■ Display■ Interactivity■ Reachability

○ Dependent variable:■ Response

Vatavu, R. D. (2013). On designing interactivity awareness for ambient displays. Multimedia Tools and Applications, 1-22.

Test data

Vatavu, R. D. (2013). On designing interactivity awareness for ambient displays. Multimedia Tools and Applications, 1-22.

Test results

Vatavu, R. D. (2013). On designing interactivity awareness for ambient displays. Multimedia Tools and Applications, 1-22.

Test results

Vatavu, R. D. (2013). On designing interactivity awareness for ambient displays. Multimedia Tools and Applications, 1-22.

Test results

● Accurate responses: only 54.9%

● But: participants were more accurate than being wrong or undecided (33% wrong)

● Participants were prone to mark normal displays as ambient, and vice versa

Vatavu, R. D. (2013). On designing interactivity awareness for ambient displays. Multimedia Tools and Applications, 1-22.

Factors for successful ambient awareness

● Location

● Installation

● Reachability

● Content

Vatavu, R. D. (2013). On designing interactivity awareness for ambient displays. Multimedia Tools and Applications, 1-22.

Improving ambient awareness

● Clear symbols● Instruction video● Instruction manual, text● Metaphorical pictures

Vatavu, R. D. (2013). On designing interactivity awareness for ambient displays. Multimedia Tools and Applications, 1-22.

Conclusions

"How we should interact with ambient content is still to be answered"- Vatavu R. D.

● Perceptions of ambient displays are still sometimes wrong

● Designers need to think of clever implementation of the system based on the four factors

● Future work is still needed for better understanding

Vatavu, R. D. (2013). On designing interactivity awareness for ambient displays. Multimedia Tools and Applications, 1-22.

How do you recommend venues using input from a user and contextual circumstances?

Location-based recommendation system using bayesian user's preference model

● Goal:○ Reflecting individual preference at a proper time

● Map-based personalised recommendation system○ Recommended restaurants nearby○ User's preference modeled by Bayesian networks

● Input from user and contextual information

Park, M., Hong, J., & Cho, S. (2007). Location-based recommendation system using bayesian user's preference model in mobile devices.4611, 1130-1139

Bayesian model

● Set of variables and their conditional dependencies

Park, M., Hong, J., & Cho, S. (2007). Location-based recommendation system using bayesian user's preference model in mobile devices.4611, 1130-1139

System design

Park, M., Hong, J., & Cho, S. (2007). Location-based recommendation system using bayesian user's preference model in mobile devices.4611, 1130-1139

User input

● User profile○ Age, Gender, Blood type, Has car,

Income, Food preference● User request

○ Class, Mood, Price, Parking Area

Park, M., Hong, J., & Cho, S. (2007). Location-based recommendation system using bayesian user's preference model in mobile devices.4611, 1130-1139

Contextual input

● Web, system, GPS and application data will be preprocessed into context log

● Live information (location, weather, time)is taken intoaccount with userrequest

Park, M., Hong, J., & Cho, S. (2007). Location-based recommendation system using bayesian user's preference model in mobile devices.4611, 1130-1139

Output

● Top 2 recommended restaurants per typeof meal○ Based on personal

preferences andcontextualcircumstances

● At least one matchon personalpreference

Park, M., Hong, J., & Cho, S. (2007). Location-based recommendation system using bayesian user's preference model in mobile devices.4611, 1130-1139

Conclusions

● Result is a proposition of a BN-based recommendation system○ It reflects user's preference using a user profile and

context information

● Usability and additional stability tests are yet to be done

Park, M., Hong, J., & Cho, S. (2007). Location-based recommendation system using bayesian user's preference model in mobile devices.4611, 1130-1139

Debate

FOR OR AGAINST #1

The Fly See Buy, personal, efficient and preference based navigation application for airports is

something I would consider using.

GREEN RED( free beer afterwards )

FOR OR AGAINST #2

Navigation by ambient displays is a useful addition to a mobile interface.

GREEN RED

FOR OR AGAINST #3

If one uses a portable device for navigation, can we speak of a ambient system (definition Weiser).

GREEN RED

Final Discussion Question

Sign Post vs Mobile Device?The mobile device is used for preference input in both cases.