vision-based monitoring of people in private spaces

Francisco Flórez-Revuelta Digital Imaging Research Centre

Vision-based monitoring of people in private spaces

Presentation Architecture Human action recognition Optimisation Privacy by the context Discussion

Overview

Associate Professor (now on leave) at the University of Alicante (Spain) Member of the "Domotics and Ambient Intelligence" research group Multi-disciplinary research: computer vision, computational intelligence (artificial neural networks, evolutionary computation), home automation… Research on ICT for smart environments (services at home, system integration, efficient use of energy, ambient assisted living, person-environment interaction…) Development of person-environment interfaces for the interaction of people with their environment, paying particular attention to the support of the independent living of elderly and disabled people

My background

Joined Kingston University in 2011 as Marie Curie fellow The Robot Vision Team at the Digital Imaging Research Centre had a huge experience on video-surveillance Combine their expertise on video-surveillance with mine on ambient assisted living

My background

A study on people activity in private spaces: towards a multisensor network that meets privacy requirements Project funded by the European Commission – 2011-2013

Objectives Study, design and development of a distributed vision system for human motion analysis in a private environment Development of appropriate techniques for maintaining the privacy of the individuals being monitored Integration of the vision system with other sensors forming a perceptual, unobtrusive and interoperable network Integration and validation of the system in a smart home environment

Senior Researcher since 2013 Strengthen research on ambient assisted living and intelligent environments at Kingston University

Current position

Platform for self-assessment and efficient management for informal caregivers Project funded by the European Commission Ambient Assisted Living Joint Programme (call 5) – 2013-2015

Activities and events to be detected: Recognition of activities of daily living (ADL): wake-up, going to the toilet, sleeping… Activity level: light, moderate, intensive Location and tracking of the person inside the home Risky situations: fall detection, dangerous objects; fire, flood, gas alarms… Social activity: leaving home, receiving visits, use of telephone… Detection of basic and advanced activities: cooking, watching TV…

Architecture

Camera 1

Motion Detection

Human Behaviour Analysis

Multi-view Human Behaviour Analysis

Privacy

Reasoning System

Alarm Actuators

Camera 2

Motion Detection

Human Behaviour Analysis

Camera N

Motion Detection

Human Behaviour Analysis

...

Setup and Profiles DB (Activities, Inhabitants, Objects, ...)

Log

Event

Long-term analysis

...

...

Environmental Sensor Information

Caregiver

Human behaviour analysis

Feature extraction

Bag of key poses

Multiple views

Results

Weizmann  MuHAVi-­‐8  

MuHAVi-­‐14  

IXMAS  

Use of RGB-D cameras

Optimisation

We need to recognise the action/activity in order to apply an appropiate privacy filter A high success rate is compulsory

Evolutionary feature subset selection

Some features are not needed or they introduce noise Use of an evolutionary algorithm as a wrapper Chromosomes are encoded as binary vectors, each gene representing whether a feature is or is not used for recognition

Results

Weizmann  MuHAVi-­‐8  

MuHAVi-­‐14  

DHA  

IXMAS  

Evolutionary feature subset selection

Chromosome  

Normalisa=on  

Crossover  

Results

Results

Cooperative coevolutionary optimisation

Simultaneous selection of the training set, feature set and parameters of the clustering algorithm to improve the success rate

Cooperative coevolutionary optimisation

Cooperative coevolutionary optimisation

Cooperative coevolutionary optimisation

Privacy preservation

Privacy by the context Context:

identity of the subject appearance location on-going activity identity of the observer

Privacy filters

Current work

Fusion of multiple vision devices: multiple RGB-D devices, RGB and RGB-D cameras… Feature selection from a large bank of features Incremental and adaptive learning Continuous activity recognition Validation in real scenarios Monitoring people with all kind of sensors, fusion with environmental sensors Anything related to intelligent environments

Papers

Human activity recognition Chaaraoui, A.A.; Padilla-López, J.R.; Ferrández-Pastor, F.J.; Nieto-Hidalgo, M.; Flórez-Revuelta, F.: A Vision-based System for Intelligent Monitoring: Human Behaviour Analysis and Privacy by Context, Sensors, 14(5):8895-8925, 2014. Chaaraoui, A.A.; Flórez-Revuelta, F.: Adaptive Human Action Recognition With an Evolving Bag of Key Poses, IEEE Transactions on Autonomous Mental Development (accepted for publication) Chaaraoui, A.A.; Padilla-López, J.R.; Flórez-Revuelta, F.: Fusion of Skeletal and Silhouette-based Features for Human Action Recognition with RGB-D Devices, 3rd Workshop on Consumer Depth Cameras for Computer Vision (CDC4CV), ICCV 2013 Workshop, Sydney (Australia), 2013. Chaaraoui, A.A.; Climent-Pérez, P.; Flórez-Revuelta, F.: Silhouette-based Human Action Recognition using Sequences of Key Poses, Pattern Recognition Letters, 34(15):1799–1807, 2013. Chaaraoui, A.A.; Climent-Pérez, P.; Flórez-Revuelta, F.: An Efficient Approach for Multi-view Human Action Recognition Based on Bag-of-Key-Poses, Lecture Notes in Computer Science, 7559:29-40, 2012. Chaaraoui, A.A.; Climent-Pérez, P.; Flórez-Revuelta, F., A review on vision techniques applied to Human Behaviour Analysis for Ambient-Assisted Living. Expert Systems with Applications, 39(12): 10873-10888, 2012.

Papers

Evolutionary optimisation Chaaraoui, A.A.; Padilla-López, J.R.; Climent-Pérez, P.; Flórez-Revuelta, F.: Evolutionary joint selection to improve human action recognition with RGB-D devices, Expert Systems with Applications, 41(3):786-794, 2014. Chaaraoui, A.A.; Flórez-Revuelta, F.: Optimizing human action recognition based on a cooperative coevolutionary algorithm, Engineering Applications of Artificial Intelligence, Volume 31:116–125, 2014. Chaaraoui, A.A.; Flórez-Revuelta, F.: Human action recognition optimization based on evolutionary feature subset selection, Proceeding of the 15th Annual Genetic and Evolutionary Computation Conference, Amsterdam (The Netherlands), 2013. Climent-Pérez, P.; Chaaraoui, A.A.; Padilla-López, J.R.; Flórez-Revuelta, F.: Optimal Feature Selection for Skeletal Data from RGB-D Devices using a Genetic Algorithm, 11th Mexican International Conference on Artificial Intelligence, San Luis Potosí (Mexico), 2012.

Papers

Privacy preservation Padilla-López, J.R.; Chaaraoui, A.A.; Flórez-Revuelta, F.: Visual privacy by context: A level-based visualisation scheme (submitted to conference) Padilla-López, J.R.; Chaaraoui, A.A.; Flórez-Revuelta, F.: Visual privacy protection methods: a survey (submitted to journal) Padilla-López, J.R.; Flórez-Revuelta, F.; Monekosso, D.N.; Remagnino, P.: The "Good" Brother: Monitoring People Activity in Private Spaces, International Symposium on Distributed Computing and Artificial Intelligence (DCAI 2012), Salamanca (Spain), pp. 49-56, 2012.

Papers

Home automation and intelligent environments Ferrández-Pastor, F.J.; García-Chamizo, J.M.; Romacho-Agud, V.; Flórez-Revuelta, F.: Using the wavelet transform to disaggregate electrical power consumption into the major end-uses (submitted to journal) Ferrández-Pastor, F.J.; Nieto-Hidalgo, M.; García-Chamizo, J.M.; Flórez-Revuelta, F.: DAI Virtual Lab: a Virtual Laboratory for Testing Ambient Intelligence Digital Services, V Congreso Internacional de Diseño, Redes de Investigación y Tecnología para Todos (DRT4ALL 2013), Madrid (Spain), 2013. Climent-Pérez, P.; Chaaraoui, A.A.; Flórez-Revuelta, F.: Useful Research Tools for Human Behaviour Understanding in the Context of Ambient Assisted Living, International Symposium on Ambient Intelligence (ISAmI 2012), Salamanca (Spain), pp. 201-205, 2012. Nieto-Hidalgo, M.; Limiñana-Benito, A.; Torregrosa-Jorqués, R.; Padilla-López, J.R.; Cabo-Díez, M.; Ferrández-Pastor, F.J.; Romacho-Agud, V.; Climent-Pérez, P.; Chaaraoui, A.; Flórez-Revuelta, F.: DAI Virtual Lab: a virtual lab for the co-design, development and validation of AmI services and technologies, International Joint Conference on Ambient Intelligence (AmI-11), Amsterdam (Nederlands), 2011.

Francisco (Paco) Flórez-Revuelta F.Florez@kingston.ac.uk @fflorezrevuelta www.dtic.ua.es/~florez franciscoflorezrevuelta