georgea.tsihrintzis,ernestodamiani,mariavirvou,robertj ......smart innovation,systemsand...

George A. Tsihrintzis, Ernesto Damiani, Maria Virvou, Robert J. Howlett,

and Lakhmi C. Jain (Eds.)

Intelligent Interactive Multimedia Systems and Services

Smart Innovation, Systems and Technologies 6

Editors-in-Chief

Prof. Robert James HowlettKES InternationalPO Box 2115Shoreham-by-seaBN43 9AFUKE-mail: [email protected]

Prof. Lakhmi C. JainSchool of Electrical and Information EngineeringUniversity of South AustraliaAdelaide, Mawson Lakes CampusSouth Australia SA 5095AustraliaE-mail: [email protected]

Further volumes of this series can be found on our homepage: springer.com

Vol. 1. Toyoaki Nishida, Lakhmi C. Jain, and Colette Faucher (Eds.)Modeling Machine Emotions for Realizing Intelligence, 2010ISBN 978-3-642-12603-1

Vol. 2. George A. Tsihrintzis, Maria Virvou, and Lakhmi C. Jain (Eds.)Multimedia Services in Intelligent Environments –Software Development Challenges and Solutions, 2010ISBN 978-3-642-13354-1

Vol. 3. George A. Tsihrintzis and Lakhmi C. Jain (Eds.)Multimedia Services in Intelligent Environments –Integrated Systems, 2010ISBN 978-3-642-13395-4

Vol. 4. Gloria Phillips-Wren, Lakhmi C. Jain,Kazumi Nakamatsu, and Robert J. Howlett (Eds.)Advances in Intelligent Decision Technologies –Proceedings of the Second KES InternationalSymposium IDT 2010, 2010ISBN 978-3-642-14615-2

Vol. 5. Robert James Howlett (Ed.)Innovation through Knowledge Transfer, 2010ISBN 978-3-642-14593-3

Vol. 6. George A. Tsihrintzis, Ernesto Damiani,Maria Virvou, Robert J. Howlett,and Lakhmi C. Jain (Eds.)Intelligent Interactive Multimedia Systemsand Services, 2010ISBN 978-3-642-14618-3

George A. Tsihrintzis, Ernesto Damiani, Maria Virvou,Robert J. Howlett, and Lakhmi C. Jain (Eds.)

Intelligent InteractiveMultimedia SystemsandServices

123

Prof. George A. TsihrintzisDept. of Informatics

University of Piraeus

Piraeus 185 34, GreeceE-mail: [email protected]

Prof. Ernesto DamianiUniversita degli Studi di Milano

Dipto. Tecnologie dell´InformazioneVia Bramante, 65

26013 Crema, Italy

E-mail: [email protected]

Prof. Maria VirvouDept. of InformaticsUniversity of Piraeus

Piraeus 185 34, GreeceE-mail: [email protected]

Prof. Robert J. HowlettKES International

P.O. Box 2115, Shoreham-by-Sea

BN43 9AF, UKEmail: [email protected]

Tel.: +44 2081 330306

Mob.: +44 7905 987544

Prof. Lakhmi C. JainSchool of Electrical andInformation Engineering,

University of South Australia,

Adelaide,Mawson Lakes Campus,

South Australia SA 5095,AustraliaE-mail: [email protected]

ISBN 978-3-642-14618-3 e-ISBN 978-3-642-14619-0

DOI 10.1007/978-3-642-14619-0

Smart Innovation, Systems and Technologies ISSN 2190-3018

Library of Congress Control Number: 2010930914

c© 2010 Springer-Verlag Berlin Heidelberg

This work is subject to copyright. All rights are reserved, whether the whole or partof the material is concerned, specifically the rights of translation, reprinting, reuseof illustrations, recitation, broadcasting, reproduction on microfilm or in any otherway, and storage in data banks. Duplication of this publication or parts thereof ispermitted only under the provisions of the German Copyright Law of September 9,1965, in its current version, and permission for use must always be obtained fromSpringer. Violations are liable to prosecution under the German Copyright Law.

The use of general descriptive names, registered names, trademarks, etc. in thispublication does not imply, even in the absence of a specific statement, that suchnames are exempt from the relevant protective laws and regulations and thereforefree for general use.

Typesetting: Scientific Publishing Services Pvt. Ltd., Chennai, India.

Printed on acid-free paper

9 8 7 6 5 4 3 2 1

springer.com

Foreword

KES International (KES) is a worldwide organisation that provides a professional community and association for researchers, originally in the discipline of Knowl-edge Based and Intelligent Engineering Systems, but now extending into other related areas. Through this, KES provides its members with opportunities for publication and beneficial interaction.

The focus of KES is research and technology transfer in the area of Intelligent Systems, i.e. computer-based software systems that operate in a manner analogous to the human brain, in order to perform advanced tasks. Recently KES has started to extend its area of interest to encompass the contribution that intelligent systems can make to sustainability and renewable energy, and also the knowledge transfer, innovation and enterprise agenda.

Involving several thousand researchers, managers and engineers drawn from universities and companies world-wide, KES is in an excellent position to facili-tate international research co-operation and generate synergy in the area of artifi-cial intelligence applied to real-world ‘Smart’ systems and the underlying related theory.

The KES annual conference covers a broad spectrum of intelligent systems top-ics and attracts several hundred delegates from a range of countries round the world. KES also organises symposia on specific technical topics, for example, Agent and Multi Agent Systems, Intelligent Decision Technologies, Intelligent Interactive Multimedia Systems and Services, Sustainability in Energy and Build-ings and Innovations through Knowledge Transfer. KES is responsible for two peer-reviewed journals, the International Journal of Knowledge based and Intelli-gent Engineering Systems, and Intelligent Decision Technologies: an International Journal.

KES supports a number of book series in partnership with major scientific publishers.

Published by Springer, ‘Smart Innovative Systems and Technologies’ is the KES flagship book series. The aim of the series is to make available a platform for the publication of books (in both hard copy and electronic form) on all aspects of single and multi-disciplinary research involving smart innovative systems and technologies, in order to make the latest results available in a readily-accessible form.

The series covers systems that employ knowledge and intelligence in a broad sense. Its focus is systems having embedded knowledge and intelligence, which may be applied to the solution of world industrial, economic and environmental problems and the knowledge-transfer methodologies employed to make this happen effectively. The combination of intelligent systems tools and a broad range of applications introduces a need for a synergy of scientific and technological disciplines.

Foreword VI

Examples of applicable areas to be covered by the series include intelligent decision support, smart robotics and mechatronics, knowledge engineering, intel-ligent multi-media, intelligent product design, intelligent medical systems, smart industrial products, smart alternative energy systems, and underpinning areas such as smart systems theory and practice, knowledge transfer, innovation and enterprise.

The series includes conference proceedings, edited collections, monographs, handbooks, reference books, and other relevant types of book in areas of science and technology where smart systems and technologies can offer innovative solutions.

High quality is an essential feature for all book proposals accepted for the series. It is expected that editors of all accepted volumes take responsibility for ensuring that contributions are subjected to an appropriate level of reviewing process and adhere to KES quality principles.

Professor Robert J. Howlett Executive Chair, KES International

Visiting Professor, Enterprise: Bournemouth University United Kingdom

Preface

This volume contains the Proceedings of the 3nd International Symposium onIntelligent Interactive Multimedia Systems and Services (KES-IIMSS 2010).This third edition of the KES-IIMSS Symposium was jointly organized bythe Department of Informatics of the University of Piraeus, Greece and theDepartment of Information Technologies of the University of Milan, Italy inconjunction KES International.

KES-IIMSS is a new series of international scientific symposia aimed atpresenting novel research in the fields of intelligent multimedia systems rele-vant to the development of a new generation of interactive, user-centric ser-vices. The major theme underlying this year’s symposium is the rapid integra-tion of multimedia processing techniques within a new wave of user-centricservices and processes. Indeed, pervasive computing has blurred the tradi-tional distinction between conventional information technologies and multi-media processing, making multimedia an integral part of a new generation ofIT-based interactive systems.

KES-IIMSS symposia, following the general structure of KES events, aimat providing an internationally respected forum for presenting and publishinghigh-quality results of scientific research while allowing for timely dissemina-tion of research breakthroughs and novel ideas via a number of autonomousspecial sessions and workshops on emerging issues and topics identified eachyear.

KES-IIMSS-2010 co-located events include: (1) the International Work-shop Human-Computer Interaction in Knowledge-based Environments, (2)the International Workshop on Interactive Multimodal Environment and(3) two invited sessions respectively on Intelligent Healthcare InformationManagement, Pervasive Systems for Healthcare. KES-IIMSS-2010 is also co-located with the 2nd International Symposium on Intelligent Decision Tech-nologies (KES-IDT-2010).

VIII Preface

We are very satisfied of the quality of the program and would like to thankthe authors for choosing KES-IIMSS as the forum for presentation of theirwork. Also, we gratefully acknowledge the hard work of KES-IIMSS inter-national program committee members and of the additional reviewers forselecting the accepted conference papers.

General Co-chairsMaria Virvou

Ernesto DamianiGeorge A. Tsihrintzis

Executive ChairR. J. Howlett

Honorary ChairProf. Lakhmi C.Jain

Liaison Chair - Asia:Prof. Toyohide Watanabe , Nagoya University , Japan

Programme Coordinator:Dr. Marco Anisetti, University of Milan, Italy

General Track Coordinator:Dr. Valerio Bellandi, University of Milan, Italy

Contents

Adapting Spreading Activation Techniques towards a NewApproach to Content-Based Recommender Systems . . . . . . . . . 1Yolanda Blanco-Fernandez, Martın Lopez-Nores,Jose J. Pazos-Arias

A Framework for Automatic Detection of AbandonedLuggage in Airport Terminal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13Grzegorz Szwoch, Piotr Dalka, Andrzej Czyzewski

Modeling Student’s Knowledge on Programming UsingFuzzy Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23Konstantina Chrysafiadi, Maria Virvou

Camera Angle Invariant Shape Recognition in SurveillanceSystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33D. Ellwart, A. Czyzewski

Multicriteria-Based Decision for Services Discovery andSelection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41Younes El Bouzekri El Idrissi, Rachida Ajhoun, M.A. Janati Idrissi

Building a Minimalistic Multimedia User Interface forQuadriplegic Patients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53Constantinos Patsakis, Nikolaos Alexandris

Biofeedback-Based Brain Hemispheric SynchronizingEmploying Man-Machine Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 59Kaszuba Katarzyna, Kopaczewski Krzysztof, Odya Piotr,Kostek Bozena

Performance of Watermarking-Based DTD Algorithmunder Time-Varying Echo Path Conditions . . . . . . . . . . . . . . . . . . 69Andrzej Ciarkowski, Andrzej Czyzewski

X Contents

Applying HTM-Based System to Recognize Object inVisual Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79Hoai-Bac Le, Anh-Phuong Pham, Thanh-Thang Tran

Constant Bitrate Image Scrambling Method Using CAVLCin H.264 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91Junsang Cho, Gwanggil Jeon, Jungil Seo, Seongmin Hong,Jechang Jeong

Color Image Restoration Technique Using Gradient EdgeDirection Detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101Gwanggil Jeon, Sang-Jun Park, Abdellah Chehri, Junsang Cho,Jechang Jeong

Watermarking with the UPC and DWT . . . . . . . . . . . . . . . . . . . . . 111Evelyn Brannock, Michael Weeks

Building a Novel Web Service Framework – Through aCase Study of Bioinformatics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121Hei-Chia Wang, Wei-Chun Chang, Ching-seh Wu

An Architecture for Collaborative Translational ResearchUtilizing the Honest Broker System . . . . . . . . . . . . . . . . . . . . . . . . . 137Christopher Gillies, Nilesh Patel, Gautam Singh, Ishwar Sethi,Jan Akervall, George Wilson

Simulated Annealing in Finding Optimum Groups ofLearners of UML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147Kalliopi Tourtoglou, Maria Virvou

A Smart Network Architecture for e-Health Applications . . . . 157Abdellah Chehri, Hussein Mouftah, Gwanggil Jeon

A Music Recommender Based on Artificial ImmuneSystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167Aristomenis S. Lampropoulos, Dionysios N. Sotiropoulos,George A. Tsihrintzis

The iCabiNET System: Building Standard MedicationRecords from the Networked Home . . . . . . . . . . . . . . . . . . . . . . . . . . 181Martın Lopez-Nores, Yolanda Blanco-Fernandez,Jose J. Pazos-Arias, Jorge Garcıa-Duque

Multi-agent Framework Based on Web Service in MedicalData Quality Improvement for e-Healthcare InformationSystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191Ching-Seh Wu, Wei-Chun Chang, Nilesh Patel, Ishwar Sethi

Contents XI

Towards a Unified Data Management and Decision SupportSystem for Health Care . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205Robert D. Kent, Ziad Kobti, Anne Snowdon, Akshai Aggarwal

A Glove-Based Interface for 3D Medical ImageVisualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221Luigi Gallo

Open Issues in IDS Design for Wireless Biomedical SensorNetworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231Luigi Coppolino, Luigi Romano

Context-Aware Notifications: A Healthcare System for aNursing Home . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241Sandra Nava-Munoz, Alberto L. Moran, Victoria Meza-Kubo

Multimodality in Pervasive Environment . . . . . . . . . . . . . . . . . . . . 251Marco Anisetti, Valerio Bellandi, Paolo Ceravolo, Ernesto Damiani

Optimizing the Location Prediction of a Moving Patient toPrevent the Accident . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261Wei-Chun Chang, Ching-Seh Wu, Chih-Chiang Fang,Ishwar K. Sethi

An MDE Parameterized Transformation for Adaptive UserInterfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275Wided Bouchelligua, Nesrine Mezhoudi, Adel Mahfoudhi,Olfa Daassi, Mourad Abed

Agent Based MPEG Query Format Middleware forStandardized Multimedia Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . . 287Mario Doller, Gunther Holbling, Christine Webersberger

Query Result Aggregation in Distributed MultimediaDatabases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299Christian Vilsmaier, David Coquil, Florian Stegmaier,Mario Doller, Lionel Brunie, Harald Kosch

Sensor-Aware Web interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309Marco Anisetti, Valerio Bellandi, Ernesto Damiani,Alessandro Mondoni, Luigi Arnone

Author Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323

Adapting Spreading Activation Techniquestowards a New Approach to Content-BasedRecommender Systems�

Yolanda Blanco-Fernandez, Martın Lopez-Nores, and Jose J. Pazos-Arias

Abstract. Recommender systems fight information overload by selecting automat-ically items that match the personal preferences of each user. Content-based rec-ommenders suggest items similar to those the user liked in the past by resorting tosyntactic matching techniques, which leads to overspecialized recommendations.The so-called collaborative approaches fight this problem by considering the pref-erences of other users, which results in new limitations. In this paper, we avoidthe intrinsic downsides of collaborative solutions and diversify the content-basedrecommendations by reasoning about the semantics of the user’s preferences. Specif-ically, we present a novel domain-independent content-based recommendation strat-egy that exploits Spreading Activation techniques as the reasoning mechanism. Ourcontribution consists of adapting and extending the internals of traditional SA tech-niques in order to fulfill the personalization requirements of a recommender system.The resulting reasoning-driven strategy enables to discover additional knowledgeabout the user’s preferences and leads to more accurate and diverse content-basedrecommendations. Our approach has been preliminary validated with a set of view-ers who received recommendations of Digital TV contents.

1 Introduction

Recommender systems provide personalized advice to users about items they mightbe interested in. These tools are already helping people efficiently manage con-tent overload and reduce complexity when searching for relevant information. Thefirst recommendation strategy was content-based filtering [8], which consists of

Yolanda Blanco-Fernandez, Martın Lopez-Nores, and Jose J. Pazos-AriasDepartment of Telematics Engineering, University of Vigo, 36310, Spaine-mail: {yolanda,mlnores,jose}@det.uvigo.es� Work funded by the Ministerio de Educacin y Ciencia (Gobierno de Espaa) research

project TSI2007-61599, by the Consellera de Educacin e Ordenacin Universitaria (Xuntade Galicia) incentives file 2007/000016-0.

G.A. Tsihrintzis et al. (Eds.): Intel. Interactive Multimedia Systems & Services, SIST 6, pp. 1–11.springerlink.com c© Springer-Verlag Berlin Heidelberg 2010

2 Y. Blanco-Fernandez, M. Lopez-Nores, and J.J. Pazos-Arias

suggesting items similar to those the user liked in the past. In spite of its accuracy,this technique is limited due to the similarity metrics employed, which are basedon syntactic matching approaches that can only detect similarity between items thatshare all or some of their attributes [1]. Consequently, traditional content-based ap-proaches lead to overspecialized suggestions including only items that bear strongresemblance to those the user already knows (which are recorded in his/her profile).In order to fight overspecialization, researchers devised collaborative filtering [7]–whose idea is to move from the experience of an individual user’s profile to theexperiences of a community of like-minded users (his/her neighbors)–, and eventhey combined content-based and collaborative filtering in hybrid approaches [4].Although collaborative (and hybrid) approaches mitigate the effects of overspecial-ization by considering the interests of other users, they bring in new limitations,such as the sparsity problem (related to difficulties to select each individual’s neigh-borhood when there is no much knowledge about the users’ preferences), privacyconcerns bound to the confidentiality of the users’ personal data, and scalabilityproblems due to the management of many user profiles.

The contribution of our paper is a content-based strategy that, instead of consid-ering other individuals’ preferences, diversifies the recommendations by exploitingsemantic reasoning about the user’s interests, so that we overcome overspecializa-tion without suffering the intrinsic limitations of collaborative and hybrid solutions.A reasoning-driven recommender system requires three components: an ontologythat contains classes and properties referred to the semantic annotations of the avail-able items and their relationships; personal profiles that keep track the items the user(dis)liked along with ratings measuring his/her level of interest in them (typicallynegative values for unappealing items and positive values for interesting items);and a recommendation strategy that adopts reasoning techniques to infer knowl-edge about the user’s preferences, by uncovering semantic relationships betweenthe items registered in his/her profile and others formalized in the ontology. Forexample, if a TV viewer has enjoyed a program about keep-fit, a reasoning-drivenrecommender would exploit an ontology like that depicted in Fig. 1 to infer thats/he likes personal cares, thus being able to suggest a program about fashion likePersonal shopper tips.

The so-called Spreading Activation (SA) techniques are especially useful for thispurpose because they are able to efficiently manage huge knowledge networks (suchas ontologies) working as follows: first, these techniques activate a set of conceptsin the considered network; then, after a spreading process based on the relationshipsmodeled in the network, they select other concepts significantly related to thoseinitially activated. As per these guidelines, our idea is to harness SA techniquesas follows: the initially activated concepts would be the user’s preferences, whilethose finally selected would refer to the items recommended by our content-basedstrategy. For that purpose, our approach must: (i) identify the nodes and links tobe modeled in the knowledge network of each user (starting from the classes andproperties formalized in the ontology and from his/her profile), and (ii) define thespreading process aimed at processing the user network and selecting our content-based recommendations. In order to solve overspecialization, our spreading process

Adapting Spreading Activation Techniques towards a New Approach 3

Fig. 1 Subset of classes, properties and instances formalized in a TV ontology.

relies on a reasoning mechanism able to discover complex semantic associationsbetween the user’s preferences and the available items, which are hidden behind theknowledge formalized in the system ontology. Besides, this spreading process putsthe focus on the user’s interests, so that the resulting recommendations evolve ashis/her preferences change over time.

This paper is organized as follows: Section 2 describes the internals and limi-tations of existing SA techniques to be adopted in a recommender system. Next,Section 3 explains the improvements we propose in order to diversity traditionalcontent-based recommendations by a novel approach to SA techniques. Section 4describes a sample scenario to illustrate how our reasoning-driven strategy works.Section 5 summarizes some preliminary testing experiences and discusses scalability-related concerns. Finally, Section 6 concludes the paper and outlines possible linesof future work.

2 Spreading Activation Techniques

SA techniques are computational mechanisms able to explore efficiently hugegeneric networks of nodes interconnected by links. According to the guidelines es-tablished in [5], these techniques work as follows.


• Each node is associated to a weight (named activation level), so that the morerelevant the node in the network, the higher its activation level. Besides, each linkjoining two nodes has a weight, in a such way that the stronger the relationshipbetween both nodes, the higher the assigned weight. Initially, a set of nodes areselected and the nodes connected with them by links (named neighbor nodes) areactivated. In this process, the activation levels of the initially selected nodes arespread until reaching their neighbors in the network.

• The activation level of a reached node is computed by considering the levels ofits neighbors and the weights assigned to the links that join them to each other.Consequently, the more relevant the neighbors of a given node (i.e. the highertheir activation levels) and the stronger the relationship between the node andits neighbors (i.e. the higher the weights of the links between them), the morerelevant the node will be in the network.

• This spreading process is repeated successively until reaching all the nodes ofthe network. Finally, the highest activation levels correspond to the nodes thatare closest related to those initially selected.

We have identified two severe drawbacks that prevent us from exploiting theinferential capabilities of traditional SA techniques in our reasoning-driven recom-mendation strategy. These drawbacks lie within (i) the kind of links modeled inthe considered network, and (ii) the weighting processes of those links. On the onehand, the kind of the modeled links is closely related to the richness of the reason-ing processes carried out during the spreading process. These links establish paths topropagate the relevance of the initially activated nodes to other nodes closely relatedto them. This way, some nodes might never be detected if there are no links reach-ing them in the network. Existing SA techniques model very simple relationships,which lead to poor inferences and prevent from discovering the knowledge hiddenbehind more complex associations (see examples in [9, 10, 6, 11]). In other words, ifthe links model only simple relationships (like those detected by a syntax-driven ap-proach), the recommendations resulting from SA techniques would continue beingoverspecialized.

The second limitation of traditional SA approaches is related to the weightingprocesses of the links modeled in the network. According to the guidelines describedat the beginning of this section, these weights remain invariable over time, becausetheir values depend either on the existence of a relationship between the two linkednodes or on the strength of this relationship. This static weighting process is notappropriate for our personalization process, where it is necessary that the weightsassigned to the links of the user’s network enable to: (i) learn automatically his/herpreferences from the feedback provided after recommendations, and (ii) adapt dy-namically the spread-based inference process as these preferences evolve.

In the next section, we will explain how our reasoning-driven approach fightsabove limitations by extending traditional SA techniques so that they can be adoptedin a void-of-overspecialization content-based recommender system.


3 Our Content-Based Recommendation Strategy

First, it is necessary to delimit the knowledge network to be processed by our im-proved SA techniques. Starting from the system ontology, our strategy creates anetwork for each user by including both the his/her preferences and other concepts(nodes) strongly related to those interests. In order to identify those concepts, wefirstly locate in the domain ontology the items rated in the user’s profile. Next,we traverse successively the properties bound to these items until reaching newclass instances in the ontology, referred to other items and their attributes. In orderto guarantee computational feasibility, we have developed a controlled inferencemechanism that progressively filters the instances of classes and properties that donot provide useful knowledge for the personalization process: Specifically, as newnodes are reached from a given instance, we firstly quantify their relevance for theuser by an index named semantic intensity. In order to measure the semantic in-tensity of a node n, we take into account various ontology-dependent pre-filteringcriteria (detailed in [3]), so that the more significant the relationship between agiven node and the user’s preferences, the higher the resulting value. Next, the nodeswhose intensity indexes are not greater than a specific threshold are disregarded, sothat our inference mechanism continues traversing only the properties that permit toreach new nodes from those that are relevant for the user.

The following step in our strategy consists of processing the user network by SAtechniques. In this regard, we have extended the existing approaches by overcom-ing the limitations pointed out in Section 2. On the one hand, our approach extendsthe simple relationships adopted by traditional SA techniques by considering boththe properties defined in the ontology and a set of semantic associations (whichwill be categorized in Section 3.1) inferred from them. This rich variety of relation-ships permit to establish links that propagate the relevance of the items selected bythe pre-filtering phase, leading to diverse enhanced recommendations. On the otherhand, to fulfill the personalization requirements of a recommender system, our linkweighting process depends not only on the two nodes joined by the considered link,but also on (the strength of) their relationship to the items defined in the user profile,as we will describe in Section 3.2. This way, the links of the network created for theuser are updated as our strategy learns new knowledge about his/her preferences,leading to tailor-made recommendations after the spreading process.

3.1 Semantic Associations

Once the nodes related to the user’s interests (and the properties linking them toeach other) have been selected, our strategy infers semantic associations between theinstances referred to items. Specifically, we have borrowed from [2] the followingsemantic associations:

• ρ-path association. In our approach, two items are ρ-pathAssociated when theyare linked by a chain or sequence of properties in the ontology (e.g. in Fig. 1, it


is possible to trace a sequence between the movies Chocolat and Paris, je t’aimethrough the instance referred to their starring actress Juliette Binoche).

• ρ-join association. Two items are ρ-joinAssociated when their respective at-tributes belong to a common class in the domain ontology (e.g. the tourism doc-umentary about Toulouse depicted in Fig. 1 and the movie Paris, je t’aime wouldbe associated through the common class France cities).

• ρ-cp association. Two items are ρ-cpAssociated when they share a common an-cestor in some hierarchy defined in the ontology (e.g., all the movies depicted atthe top of Fig. 1 are associated by the ancestor Fiction Contents).

Our strategy harnesses the knowledge learned from the semantic associationsin order to draw new links in the user’s network, which improves the reasoningcapabilities of traditional SA techniques. Specifically, we incorporate a new linkfor each semantic association discovered between the items defined in the user’snetwork. We call real links to those referred to property instances formalized in theontology, and virtual links to the ones corresponding to the semantic associationsinferred from it.

3.2 Weighting of Links in the User’s Network

Before selecting our content-based recommendations by the spreading activationprocess, it is necessary to weigh the links modeled in the user’s network. Insteadof considering that the weight of a link between two nodes depends only on thestrength of their mutual relationship, our approach imposes two constraints on thelinks to be weighed.

• First, given two nodes joined by a link, we consider that the stronger the (seman-tic) relationship between the two linked nodes and the user’s preferences, thehigher the weight of the link. To measure how relevant a node is for a user, weconsider either the rating of this node in his/her profile (if the node is known bythe user) or the value of the semantic intensity of this node (otherwise).

• Second, the weights are dynamically adjusted as the user’s preferences evolveover time, thus offering permanently updated content-based recommendations.

Besides, the weights assigned to the virtual links are lower than those set for thereal links. The intuition behind this idea is that the relationship existing betweentwo nodes joined by a real link is explicitly represented in the system ontology bymeans of properties, while the relationship between two nodes joined by a virtuallink has been inferred by a reasoning-driven prediction process.

3.3 Selection of Recommendations: Our Spreading ActivationProcess

For the selection of the items finally recommended to the user, we use an improvedspreading activation mechanism. Firstly, we activate in the user’s network the nodes


referred to the item defined in his/her profile, and assign them an initial activa-tion level equal to their respective ratings. Next, the activation levels of the user’spreferences are propagated through his/her network by using an iterative algorithm,which activates all the nodes in parallel in each iteration. Specifically, the algorithmcomputes the activation level of each node in the user’s SA network by adding thecontribution from all of its neighbor nodes. This contribution considers both theactivation level of each neighbor node and the weight of the link (real or virtual)joining it to the considered node. For that reason, the more relevant the neighbors ofa node (i.e. higher activation levels) and the stronger the relationships among themand the considered node (i.e. higher weights of links), the more significant this nodewill be for the user.

According to the guidelines of traditional SA techniques (see Section 2), once thespreading process has reached all the nodes in the user’s network, the highest activa-tion levels correspond to items meeting two conditions: (i) their neighbor nodes arealso relevant for the user and (ii) they are closely related to the user’s interests. Forthat reason, these nodes identify the items finally suggested by our content-basedstrategy.

4 Example of Reasoning-Driven Content-BasedRecommendations

This section presents a sample scenario in the scope of Digital TV, where we rec-ommend programs to a user U who has enjoyed the comedy romance Chocolatstarring Juliette Binoche, the documentary The Falklands crisis: the untold story,and the program Toulouse in a nutshell about the main tourist attractions of thisFrench city.

First, our strategy selects in the TV ontology depicted in Fig. 1 the instancesthat are relevant for U by considering his/her personal preferences. After inferringsemantic associations among them, we create the user network depicted in Fig. 2.

Chocolat

Toulouse in

a nutshell

The Falklands crisis:

an untold story

The English

patient

Paris, je t’aime

Juliette

Binoche

Paris

Toulouse

World War

IIFalklands

War real links

virtual links

rdf:typeOf

U’s preferences

Recommendations

Content attributes

Fig. 2 Network used by SA techniques to select content-based recommendations for U


As represented in this network, we have selected class instances that share com-mon ancestors with U’s preferences:

• First, the nodes Paris, je t’aime and The English patient are included in U’s net-work because they are Comedy and Romance movies (respectively) just like thefilm Chocolat the user has appreciated. According to what we explained in Sec-tion 3.1, both ancestors lead us to inferring the following associations betweenU’s preferences and the nodes of his/her network: ρ-cpAssociated (Chocolat,Paris, je t’aime) and ρ-cpAssociated (Chocolat, The English patient).

• Second, the nodes World War II (which is the topic of the movie The Englishpatient) and Paris (which is the city where the movie Paris, je t’aime is settled)are relevant for U because this user has liked other instances belonging to theirclasses in the ontology (specifically, Falklands war belonging to the War conflictsclass, and Toulouse belonging to France cities). These common classes permit todiscover the following associations: ρ-joinAssociated (The Falklands crisis: anuntold story, The English patient) and ρ-joinAssociated (Toulouse in a nutshell,Paris je t’aime).

Once the links of U’s network have been weighted, we process the representedknowledge by our SA techniques in order to select content-based recommendations.After spreading the activation levels of U’s preferences until reaching all the nodesin his/her network, our strategy suggests the TV programs with the highest levels.As per Section 3.3, these programs receive links from other contents which are ap-pealing to the user. This way, our content-based approach suggests to U the moviesThe English patient and Paris, je t’aime by exploiting the associations inferred be-tween these contents and U’s preferences.

• The English patient: The activation level of this movie gets higher thanks to thelinks from two programs relevant for U : the documentary The Falklands crisis:an untold story and the film Chocolat starring Juliette Binoche. The war topicturns the documentary into a program appealing to the user, whereas Chocolat isrelevant because it involves his/her favorite actresses.

• Paris, je t’aime: As shown in Fig. 2, Chocolat and Toulouse in a nutshell injectpositive weights in Paris, je t’aime node. Both programs are specially relevantfor U , thus increasing the activation level of Paris, je t’aime, a movie the usermay appreciate due to two reasons: (i) his/her favorite actress takes part in it, and(ii) the movie is set in a city of France, a country that seems to be interesting forU in view of the documentary about customs and tourist attractions U has liked.

To conclude, consider the following situation: a program in U’s network receiveslinks from nodes referred to contents the user has rated negatively. Here the weightsof the links would be very low, which contributes to decrease the activation levelof the program after the spreading process. This reveals an important benefit of ourstrategy, which is also able to identify contents that must not be recommended tothe user because they are (semantically) associated to programs U did not like inthe past.


5 Experimental Evaluation

To validate our proposal we have developed a prototype of TV recommender systemthat works with an ontology containing about 50,000 nodes referred to specific TVprograms and their semantic attributes. The knowledge formalized in our ontologywas queried by an OWL-specific API (Application Programming Guide) providedby Protg1, a free open-source tool that includes mechanisms to create, view andquery the classes, properties and specific instances formalized in OWL2 ontologies.Besides, we have developed a tool (named Reasoning Inspector) that permits to un-derstand the kind of semantic associations that lead to our diverse content-based rec-ommendations. In order to implement this tool, we have used an ontology-viewingplugin provided by Protg (TGVizTab3) to create and browse interactively genericgraphs.

5.1 Preliminary Testing Experiences

Our tests involved 150 users (recruited from among our undergraduate students,their relatives and friends), who provided us with both their initial preferences andtheir relevance feedback about our reasoning-based recommendations. For thosepurposes, the users accessed a Web form where a list of 200 TV programs wasshown, which were classified into a hierarchy of genres to facilitate browsing tasks.The users identified the contents they liked and disliked by assigning specific ratingsto each TV program4.

The information about the users’ preferences was processed by our validationtool, which was in charge of modeling the users’ profiles and running our reasoning-based strategy. The list of suggested TV programs was e-mailed and feedback aboutthese recommendations was requested to each viewer. Next, our validation tool up-dated the users’ profiles according to their relevance feedback, and the content-basedstrategy was executed again to check that the offered recommendations adapted asthe users’ preferences evolve over time. The processes of sending recommendationsand acquiring relevance feedback were repeated during one week, and convenientlymonitored by our Reasoning Inspector. After the 7-days testing period, a question-naire was e-mailed to each user, asking about his/her perception of our personal-ization services. Most users (78%) rated as very positive or positive the diversityof our reasoning-based recommendations, whereas only 12% of them remained in-different towards the received suggestions. Nearly all the users noticed the diversenature of our recommendations. In fact, many users (about 76%) told us that theydid not know some of the suggested TV programs; however, they admitted that theway to relate the programs to their personal preferences was really “ingenious”,

1 http://protege.stanford.edu/2 Web Ontology Language: http://www.w3.org/TR/owl-features/.3 See http://users.ecs.soton.ac.uk/ha/TGVizTab/ for details.4 Note that these TV programs were shown with a brief synopsis, in such a way that the

users could rate even programs they did not known.


“peculiar but appropriate” and even “intelligent”. Lastly, from the questionnaires,we also discovered that most of the users (84%) would be willing to pay a smallfee for receiving our recommendations, which evidences the interest of our content-based approach.

5.2 Computational Viability and Scalability-Related Concerns

We have defined some optimization features aimed at ensuring scalability and com-putational feasibility of our reasoning-driven content-based strategy. Firstly, due tothe iterative nature of the algorithm used in our spreading process, our strategy canreturn suboptimal solutions to guarantee fast responses to the users practically inreal-time. Besides, our implementation works with a master server that shares outthe computational burden among a set of slaves personalization servers, which re-turn content-based recommendations by running our strategy based on SA tech-niques and accessing a database that lodges the system ontology and users’ profiles.

Our third feature consists of distributing the tasks involved in our strategy amongseveral servers: the ontology server updates the items in the ontology and computesoff-line parameters that can be reused as new users log into the recommender sys-tem, while the profiles server updates the users’ profiles by adding new preferencesand ratings for items. Finally, we maintain multiple instances of the profiles serversand ontology servers. Besides, in order to avoid bottlenecks when accessing the on-tology and users’ profiles, each instance of these servers works with a replica of thesystem database.

6 Conclusions and Further Work

This paper fights the overspecialized nature of traditional content-based recommen-dations, which include only items very similar to those the user already knows(mainly due to the adoption of syntactic matching techniques). The novelty is thatwe overcome this limitation without considering the preferences of other individu-als, which was the solution proposed so far in literature at expenses of causing othersevere drawbacks.

Our recommendation strategy harnesses the benefits of semantic reasoning overan underlying ontology as a means to discover additional knowledge about the userpreferences, enabling to compare them to the available items in a more effectiveway. This way, instead of suggesting items very similar to those the user liked inthe past, our strategy recommends items semantically related to his/her preferences.For that purpose, we have extended existing semantic reasoning mechanisms, sothat they can be adopted in a personalization scenario where the focus is put onthe user’s preferences. Specifically, we have described how semantic associationsand SA techniques fit together in our content-based recommendation strategy: theassociations help to diversify the recommendations because they discover hiddenrelationships between the user’s preferences and the available items, while our im-proved SA techniques enable (i) to process efficiently the knowledge learned from


those associations, and (ii) to evolve the recommendations as the user’ preferenceschange. Our contribution is generic and can be reused in multiple contexts, becom-ing an easy-to-adopt starting point to implement diverse personalization services.

As future work, we plan to carry out a quantitative evaluation driven by accu-racy metrics such as MSE, Hit Rate, recall and precision. Besides assessing ourreasoning-driven personalization capabilities, we will exploit the data gathered froma greater number of users in order to compare our approach against existing collab-orative and hybrid works in terms of performance and personalization quality.

References

1. Adomavicius, G., Tuzhilin, A.: Towards the next generation of recommender systems: asurvey of the state-of-the-art and possible extensions. IEEE Transactions on Knowledgeand Data Engineering 17(6), 739–749 (2005)

2. Anyanwu, K., Sheth, A.: ρ-Queries: enabling querying for semantic associations on theSemantic Web. In: Proceeding of the 12th International World Wide Web Conference(WWW 2003), pp. 115–125 (2003)

3. Blanco-Fernandez, Y., Pazos-Arias, J.J., Gil-Solla, A., Ramos-Cabrer, M., Lopez-Nores,M.: A flexible semantic inference methodology to reason about user preferences inknowledge-based recommender systems. Knowledge-Based Systems 21(4), 305–320(2008)

4. Cornelis, C., Lu, J., Guo, X., Zhang, G.: One-and-only item recommendation with fuzzylogic techniques. Information Sciences 177(1), 4906–4921 (2007)

5. Crestani, F.: Application of Spreading Activation techniques in information retrieval.Artificial Intelligence Review 11(6), 453–482 (1997)

6. Huang, Z., Chen, H., Zeng, D.: Applying associative retrieval techniques to alleviatethe sparsity problem in collaborative filtering. ACM Trans. on Inform. Systems 22(1),116–142 (2004)

7. Liu, D., Lai, C., Lee, W.: A hybrid of sequential rules and collaborative filtering forproduct recommendation. Information Sciences 179(20), 3505–3519 (2009)

8. Pazzani, M., Billsus, D.: Content-based recommendation systems. In: Brusilovsky, P.,Kobsa, A., Nejdl, W. (eds.) Adaptive Web 2007. LNCS, vol. 4321, pp. 325–341.Springer, Heidelberg (2007)

9. Rocha, C., Schawabe, D., Poggi, M.: A hybrid approach for searching in the SemanticWeb. In: Proceedings of 13th International World Wide Web Conference (WWW 2004),pp. 74–84 (2004)

10. Stojanovic, N., Struder, R., Stojanovic, L.: An approach for ranking of query results inSemantic Web. In: Fensel, D., Sycara, K., Mylopoulos, J. (eds.) ISWC 2003. LNCS,vol. 2870, pp. 500–516. Springer, Heidelberg (2003)

11. Troussov, A., Sogrin, M., Judge, J., Botvich, D.: Mining socio-semantic networks usingspreading activation techniques. In: Proceedings of the 8th International Conference onKnowledge Management and Knowledge Technologies, pp. 8–16 (2008)

A Framework for Automatic Detection ofAbandoned Luggage in Airport Terminal

Grzegorz Szwoch, Piotr Dalka, and Andrzej Czyzewski

Abstract. A framework for automatic detection of events in a video stream transmit-ted from a monitoring system is presented. The framework is based on the widelyused background subtraction and object tracking algorithms. The authors elaboratedan algorithm for detection of left and removed objects based on morphological pro-cessing and edge detection. The event detection algorithm collects and analyzesdata of all the moving objects in order to detect events defined by rules. A systemwas installed at the airport for detecting abandoned luggage. The results of the testsindicate that the system generally works as expected, but the low-level modules cur-rently limit the system performance in some problematic conditions. The proposedsolution may supplement the existing monitoring systems in order to improve thedetection of security threats.

1 Introduction

Automatic event detection in surveillance systems is becoming a necessity. Enor-mous amount of video cameras used in facilities such as shopping malls, publictransport stations and airports makes it impossible for human operators to watchand analyze all video streams in the real time. Such systems are typically used onlyas a forensic tool rather than a preventive or interceptive tool. Therefore, it is easy tomiss harmful activities like theft, robbery, vandalism, fight or luggage abandonmentas well as frequent events that may be dangerous, like unauthorized presence in re-stricted areas. In the last few years many publications regarding automatic videosurveillance systems have been presented. These systems are usually focused ona single type of human activity. Events regarding human behavior may be dividedinto three main groups. The first group contains activities that does not involve otherpersons or objects such as loitering [1] or sudden human pose changes like going

Grzegorz Szwoch, Piotr Dalka, and Andrzej CzyzewskiGdansk University of Technology, Multimedia Systems Department 80-233 Gdansk,Poland, Narutowicza 11/12e-mail: [email protected]

G.A. Tsihrintzis et al. (Eds.): Intel. Interactive Multimedia Systems & Services, SIST 6, pp. 13–22.springerlink.com c© Springer-Verlag Berlin Heidelberg 2010

14 G. Szwoch, P. Dalka, and A. Czyzewski

from standing to lying down that might indicate a pedestrian collapse [2]. The sec-ond group includes neutral human interactions like walking together, approaching,ignoring, meeting, splitting [3] and violent ones, such as fist fighting, kicking orhitting with objects [4]. The last group contains human activities that are relatedto the environment. This includes intrusion or trespassing [5], wrong direction ofmovement [6], vandalism [7] and luggage abandonment. Determining object sta-tionarity and finding an object left behind (e.g. backpack or briefcase) is a criticaltask that leads to safety and security of all public transport passengers or shoppingmall customers. Abandoned luggage detection was the main topic of PETS (Perfor-mance Evaluation of Tracking and Surveillance) Workshop in 2006. The majorityof papers presented there employ background subtraction to detect foreground ob-jects that are classified as newly appeared stationary objects using simple heuristics[8] or Bayesian inference framework [9]. Other methods regarding this topic maybe found in the literature. Spagnolo et al. classify objects as abandoned or removedby matching the boundaries of static foreground regions [10]. Another solution di-vides a video frame into blocks that are classified as background and foreground;non-moving foreground block is assumed to be stationary [5]. This method is robustagainst frequent dynamic occlusions caused by moving people. This paper presentsa framework for detection a wide range of events in video monitoring systems, us-ing rules defined by the system operator. The framework is based on the widely usedbackground subtraction and object tracking algorithms and adds procedures devel-oped especially for the presented system, as described in Section 2. An exampleapplication of the system at the airport for detection of abandoned luggage is brieflypresented in Section 3 and the paper ends with conclusions and indication of areasfor the future developments.

2 Framework Design and Implementation

2.1 System Overview

The framework for automatic event detection is composed of several modules, asdepicted in Fig. 1. The low-level algorithms extract information on moving objectsfrom camera images. The detailed explanation of these algorithms lies beyond thescope of this paper, hence only the general idea is described here. Moving objectsare detected in the camera frames using the background modeling method, basedon Gaussian Mixture Model [11] (five distributions were used for each pixel). Theresults of background modeling are processed by detecting and removing shadowpixels (basing on the color and luminance of the pixels) and by performing morpho-logical operations on the detected objects in order to remove small areas and to fillholes inside the objects. Next, movements of the detected objects (blobs) are trackedin successive image frames using a method based on Kalman filters [12]. A state ofeach tracked object (tracker) in each frame is described by an eight-element vectordescribing its position, velocity and change in the position and velocity. The stateof each Kalman filter is updated for each image frame, so the movement of each

A Framework for Automatic Detection of Abandoned Luggage 15

Fig. 1 General block diagram of the proposed framework for the automatic event detection

object is tracked continuously. The next blocks of the analysis system are designedfor the purpose of the presented framework. The task of the classification module isto assign detected moving objects (human, luggage, etc.). An event detection mod-ule analyses the current and the past states of objects and evaluate rules in order tocheck if any of the defined events occurred. The results of event detection are sent tothe framework output for further processing (visualization, logging, camera steer-ing). The remaining part of this paper discusses relevant modules of the frameworkin detail.

2.2 Resolving the Splitting Trackers

The main problem that had to be resolved in the object tracking procedure im-plemented in the discussed framework was handling of ’splitting objects’, e.g. ifa person leaves their luggage and walks away. In this situation, the tracker that wasassigned to a person carrying a luggage has to track further movements of the sameperson and a new tracker has to be created for the luggage. A following approachis proposed for this task. First, groups of matching trackers and blobs are formed.Each group contains all the blobs that match at least one tracker in the group andall the trackers that match at least one blob in the group. The match is defined as atleast one pixel common to the bounding boxes of the blob and the tracker. Withina single group, blobs matching each tracker are divided into subgroups (in somecases a subgroup may contain only one blob) separated by a distance larger than thethreshold value. If all the blobs matching a given tracker form a single subgroup,the state of the tracker is updated with the whole blob group. If there is more thanone subgroup of blobs matching the tracker, it is necessary to select one subgroupand assign the tracker to it. In order to find the sub-group that matches the tracker,two types of object descriptors are used - color and texture. Color descriptors arecalculated using a two-dimensional chrominance histogram of the image represent-ing the object. The texture descriptors are calculated from the same image, usinga gray level co-occurrence matrix [13]. Five texture descriptors are used (contrast,energy, mean, variance and correlation) for three color channels and four directionsof pixel adjacency, resulting in a vector of 60 parameters describing single object’sappearance. In order to find a subgroup of blobs matching the tracker, a vector of


texture descriptors for the tracker DT is compared with a corresponding vector DBcalculated for the subgroup of blobs, using a formula:

ST = 1− 1N

N

∑i=1

DTi−DBi

max(DTi,DBi)(1)

where N is the number of elements in both vectors. The final similarity measure isa weighted sum of texture similarity ST and color histogram similarity SC (equal toa correlation coefficient of histograms of a tracker and a group of blobs):

S = WT ST +WCSC (2)

where the values of weighting coefficients (WT = 0.75,WC = 0.25) were found em-pirically. The subgroup of blobs with the largest S value is used to update the state ofthe tracker. After each tracker in the group is processed, the remaining (unassigned)blobs are used to construct new trackers. As a result, in case of a person leaving aluggage, the existing tracker follows the person and a new tracker is created for aleft luggage.

2.3 Detection of Left or Removed Objects

For the purpose of event detection, each tracker has to be assigned to a proper class(human, vehicle, luggage, etc.). In the test system intended to work at the airport,simplified classification is used: each object is classified either as a human or as aluggage, basing on analysis of objects velocity and their size and shape variabil-ity. It is assumed that luggage (as a separate object) remains stationary and do notchange its dimensions significantly (some fluctuations in size and shape of objectsare inevitable due to inaccuracies of the background subtraction procedure). In or-der to increase the accuracy of this simple classifier, a number of past states of theobject are taken into account, together with its current state. The averaged parame-ters (speed, changes in size and shape) are compared with thresholds; if their typicalvalues are exceeded, the object is classified as human, otherwise it is qualified as aluggage. In the further stages of system development, the classification procedurewill be expanded so that more object classes will be defined. The main problemin this approach is that due to the nature of the background subtraction algorithm,leaving an object in the scene causes the same effect as removing an object that wasa part of the background (e.g. a luggage that remained stationary for a prolongedtime). In both cases a new tracker is created, containing either a left object or a’hole’ in the background (remaining after the object was taken). The system has todecide whether the detected object was left or taken. This is achieved by examiningthe content (image) of the newly created tracker. It is expected that edges of the leftobject are located close to the blob’s border, while no distinct edges are present incase of the taken object (provided that a background is sufficiently smooth). Theproposed procedure works as follows. First, the grayscale image B of the object(blob) and its mask M (having non-zero values for pixels belonging to the blob


and zero values otherwise) are processed by the Canny detector in order to find theedges. The results of edge detection in the mask (EM) is processed by morphologicaldilation in order to increase a detection margin:

EMd = EM⊕SE (3)

where SE is a 7 x 7 structuring element. Next, the result of dilation is combined withEB - the result of edge detection in the image B:

R = EMd ∩EB (4)

and the resulting image is dilated using the same structuring element:

Rd = R⊕SE (5)

Finally, a measure of difference between the object and the background is calculatedas:

D =NR

NM(6)

whereNR = CNZ(Rd), NM = CNZ(EMd) (7)

and CNZ() is a function that counts the number of non-zero pixels in the grayscaleimage. If the blob represents a left object, D is expected to be significantly largerthan for a removed object. Therefore, the analyzed object is classified as a left oneif D > Tcl or as a taken object otherwise, where Tcl is a detection threshold forclassification. A proper selection of the threshold allows for accurate detection oftaken and left objects regardless of the background (which may also contain edges)and errors in background subtraction. Fig. 2 presents an example of the proceduredescribed above, for left and removed object cases (Tcl = 0.6). It may be seen thatthe proposed procedure allows for proper distinction of taken and left objects basingon the D measure value.

Fig. 2 Example of detection of left and taken objects using the procedure described in thetext.


2.4 Event Detection

The event detection module utilizes data acquired from previous modules in order todetect events defined with rules. Detected events may refer to simple cases (objectentering or leaving an area, crossing a barrier, stopping, moving, etc) as well as tomore complex situations (abandoned luggage, theft and others). In this section ofthe paper, detection of an abandoned luggage scenario is used as an example. Theevent detector stores a history of each tracker states (in the experiments, last fivestates were used). The object state contains all the parameters needed for event de-tection (position of an object, its velocity and direction of movement, type, values ofdescriptors, etc.). Event rules are tested against all past states. If the rule is fulfilledfor a defined number of states (e.g. three of total five), an event is detected. Thisapproach allows for time-filtering of instantaneous events, reducing the number offalse-positive decisions. The framework allows an user to add new rules; each ruleis analyzed in parallel based on the same tracker data. An example rule for detectionof an abandoned luggage may be formulated in plain English as follows: if an objectof type ’human’ leaves an object of the type ’luggage’, the human moves away fromthe luggage at the distance d and does not approach the luggage for the time periodt, an event ’abandoned luggage’ is detected. An implementation of this rule in theevent detector is presented in Fig. 3. Only the objects classified as left luggage areprocessed by the rule. For each frame, distance d between the luggage and its ’par-ent’ (person that left the luggage) is calculated. If d exceeds the threshold Td (or theparent left the screen) for a time period Tt , an alarm is sent to the system output. Ifthe person goes back to the luggage (d < Td) before the time Tt passes, the counter tis reset. The rule may be extended with additional conditions, e.g. the event may bedetected only if the person and/or the luggage are detected in the defined area or ifthe person crosses the defined barrier. The main idea of the event detection modulepresented here remains valid for more complex scenarios. Other rules (stolen object,loitering, etc.) may operate in a similar manner.

3 Test Results

The framework for automatic event detection described in Section 2 was imple-mented in C++ programming language, using an OpenCV library [14] for perform-ing low-level image operations and implementing Kalman filters. The first version ofthe system for detection of an abandoned luggage is currently tested at the Poznan-Lawica airport in Poland. A camera is mounted in the arrivals hall, at the height of3.20 meters, overlooking the area in which most of the abandoned luggage caseswere observed. The area visible by the camera is calibrated using the Tsai’s method[15] in order to analyze distances between objects in meters instead of pixels. Thesystem runs on the PC with Fedora Linux operating system, 2.5 GHz quadcore pro-cessor and 4 GB of memory and is able to process 10 video frames per second inreal time (resolution 1600 x 900 pixels, fps limit imposed by the camera). The thor-ough evaluation of accuracy of the proposed abandoned luggage detector requires


Fig. 3 Block diagram of the detector of abandoned luggage described in text.

the ground data which has not been collected yet. Therefore, an extensive quanti-tative analysis of the system performance remains to be evaluated in future. Basedon initial tests and visual evaluation of results provided by the system, it may bestated that the accuracy of the detection is satisfactory in good and moderate condi-tions (stable light, moderately busy scene). An example of proper detection of theabandoned luggage is presented in Fig. 4. The lightning conditions in this exampleare not optimal, therefore results of background subtraction and object tracking areinaccurate to some degree. However, the algorithms described in this paper gener-ally work as expected. The tracker representing the person walking away from theluggage (Fig. 4a and 4b) matches the two blobs separated by distance larger thanthe threshold (30 pixels). Similarity factors calculated for the tracker and both blobs(using Eq. 1 for texture similarity ST, correlation coefficient for color histogramsimilarity SC and Eq. 2 for the total similarity) are: ST = 0.83,SC = 0.86,S = 0.84for the tracker compared with the ‘person blob’ and ST = 0.51,SC = 0.99,S = 0.64for the tracker compared with the ‘luggage blob’. Although the result calculated forcolor histograms is invalid in this case, the final similarity measure is correct andallows for proper assignment of the existing tracker to the person and a new trackeris created for the left luggage (Fig. 4c). This new tracker is correctly classified as aleft object, basing on the edge measure calculated using Eq. 6: D = 0.71 (Tcl = 0.6).


Fig. 4 Example result of tests of the automatic detector of the abandoned luggage, per-formed at the Poznan-Lawica airport: (a) person with luggage, (b) person leaving luggage, (c)new tracker created for the abandoned luggage, (d) event rule matched - abandoned luggagedetected.

After the distance between the person and the luggage exceeds the defined thresh-old (d = 3m) for a defined time period (t = 15s), the event detector detects that theluggage was abandoned (Fig. 4d) and sends an alarm to its output. Conditions thatmay cause event misdetection by the system are related mainly to the inaccuracy ofthe background subtraction module due to the changes in lighting. In the presentedexample, sunlight falling through glass walls of the hall causes reflections on thefloor that disturbed the background subtraction procedure and resulted in creationof numerous false trackers. Another problem is related to object tracking in case ofcrowded scenes, with a large number of moving objects overlapping each other fora prolonged time. In such situations, the number of erroneous decisions made bythe tracking procedure increases significantly. As a result, the event detector is fedwith invalid data and fails to provide expected results. This kind of problems will beaddressed in the future research.

georgea.tsihrintzis,ernestodamiani,mariavirvou,robertj ......smart innovation,systemsand...

Documents