A Review of Content-Based Image Retrieval Systems

Colin C. Venters and Dr. Matthew Cooper

Manchester Visualization Centre Manchester Computing

University of Manchester

2

“Ours is a visual age.”

E. H. Gombrich, 1982 The Image and The Eye: Further Studies in the Psychology of Pictorial Representation

3

The JISC Technology Applications Programme is an initiative of the Joint Information Systems Committee of the Higher Education Funding Councils. For more information contact: Tom Franklin JTAP Programme Manager Manchester Computer Building University of Manchester Manchester M13 9PL Email: t.franklin@man.ac.uk JTAP URL: www.jtap.ac.uk

4

Keywords

Colour

Content-Based Image Retrieval

Database

Digital Image

Image

Image Retrieval

Information Retrieval

Multimedia

Non-Alphanumeric Information

Pictorial Database

Query by Visual Example

Shape

Similarity

Spatial Relationship

Texture

Visual Information Retrieval

5

Table of Contents Keywords..........................................................................................................................................................................4 Acknowledgements.........................................................................................................................................................7 List of Tables ....................................................................................................................................................................8 Executive Summary ........................................................................................................................................................9 Abbreviations.................................................................................................................................................................11 Copyright........................................................................................................................................................................13 Disclaimer.......................................................................................................................................................................14 1 Introduction..........................................................................................................................................................15 2 Methods.................................................................................................................................................................18

2.1 Retrieval Effectiveness...............................................................................................................................18 2.2 Heuristic Evaluation ..................................................................................................................................19

3 Commercial Content-Based Image Retrieval Systems....................................................................................20 3.1 Excalibur Visual RetrievalWare SDK, Excalibur Corp..........................................................................21

3.1.1 Matching Features ............................................................................................................................22 3.1.2 Image Formats...................................................................................................................................22 3.1.3 Hardware & Software Requirements.............................................................................................22 3.1.4 Price ....................................................................................................................................................22 3.1.5 Excalibur Image DataBlade .............................................................................................................23

3.2 ImageFinder ................................................................................................................................................24 3.2.1 Feature Matching ..............................................................................................................................25 3.2.2 Image Formats...................................................................................................................................26 3.2.3 User Interface ....................................................................................................................................26 3.2.4 Hardware & Software Requirements.............................................................................................27 3.2.5 Price ....................................................................................................................................................27

3.3 IMatch, MWLabs ........................................................................................................................................28 3.3.1 Matching Features ............................................................................................................................28 3.3.2 Image Formats...................................................................................................................................30 3.3.3 User Interface ....................................................................................................................................30 3.3.4 Hardware & Software Requirements.............................................................................................32 3.3.5 Price ....................................................................................................................................................32

3.4 QBIC (Query By Image Content) Development Kit, IBM Corp. ..........................................................33 3.4.1 Matching Features ............................................................................................................................34 3.4.2 Image Formats...................................................................................................................................34 3.4.3 Hardware & Software Requirements.............................................................................................35 3.4.4 Price ....................................................................................................................................................35

3.5 VIR Image Engine & Image Read/Write Toolkit, Virage Inc...............................................................36 3.5.1 Matching Features ............................................................................................................................37 3.5.2 Image Formats...................................................................................................................................38 3.5.3 Hardware & Software Requirements.............................................................................................38 3.5.4 Price ....................................................................................................................................................39 3.5.5 Visual Information Retrieval Data Cartridge, Oracle ..................................................................39

4 Prototype Research Systems...............................................................................................................................40 4.1 AMORE........................................................................................................................................................40 4.2 ARTISAN (Automatic Retrieval of Trademark Images by Shape ANalysis).....................................40 4.3 BlobWorld ...................................................................................................................................................41 4.4 C-BIRD (Content-Based Image Retrieval)...............................................................................................41 4.5 CAETIIML ...................................................................................................................................................41 4.6 CANDID (Comparison Algorithm for Navigating Digital Image Database) ....................................41 4.7 Charmer .......................................................................................................................................................42 4.8 Circus (Content-based Image Retrieval and Consultation User-centered System)...........................42 4.9 Compass (COMPuter Aided Search System) .........................................................................................42 4.10 Diogenes ......................................................................................................................................................42 4.11 DrawSearch .................................................................................................................................................43 4.12 ImageRETRO ..............................................................................................................................................43 4.13 ImageRover .................................................................................................................................................43

6

4.14 ImageScape..................................................................................................................................................43 4.15 IQUEST ........................................................................................................................................................44 4.16 LCPD (The Leiden 19th Century Portrait Database) .............................................................................44 4.17 MARS (Multimedia Analysis and Retrieval System) ............................................................................44 4.18 MIDSS (Multi-Resolution Image Database Search System) .................................................................44 4.19 NeTra ...........................................................................................................................................................44 4.20 Photobook....................................................................................................................................................45 4.21 PicSOM (Picture Self-Organising Maps) .................................................................................................45 4.22 PicToSeek.....................................................................................................................................................45 4.23 SaFe(Spatial and Feature Query System) ................................................................................................45 4.24 SQUID (Shape Queries Using Image Databases)...................................................................................46 4.25 Surf Image ...................................................................................................................................................46 4.26 Synapse ........................................................................................................................................................46 4.27 Viper (Visual Information Processing for Enhanced Retrieval)...........................................................46 4.28 VisualSEEk ..................................................................................................................................................47 4.29 WebSEEk .....................................................................................................................................................47

5 Discussion .............................................................................................................................................................48 5.1 System Features ..........................................................................................................................................48 5.2 Feature Matching & Retrieval Effectiveness...........................................................................................48 5.3 Usability.......................................................................................................................................................49

6 Conclusions...........................................................................................................................................................50 7 References .............................................................................................................................................................51 8 Appendix A: General System Information.......................................................................................................53 9 Appendix B: Feature Extraction Methods ........................................................................................................56 10 Appendix C: Retrieval Results ......................................................................................................................59

10.1 Global Colour Retrieval.............................................................................................................................59 10.2 Local Colour Retrieval ...............................................................................................................................60 10.3 Global Colour Retrieval.............................................................................................................................61 10.4 Local Colour Retrieval ...............................................................................................................................62 10.5 Global Colour Retrieval.............................................................................................................................63 10.6 Local Colour Retrieval ...............................................................................................................................64 10.7 Texture Retrieval ........................................................................................................................................65 10.8 Texture Retrieval ........................................................................................................................................66 10.9 Texture Retrieval ........................................................................................................................................67

7

Acknowledgements We would like to thank JISC for funding this project under a JISC Technology Applications Program award. We are indebted to the following companies and individuals that devoted their valuable time to assist with this study: Attrasoft, David Baxter, Andy Black, David Blaney, Tom Broom, C&C Research Laboratory, Dr. Chad Carson, Professor Shih-Fu Chang, Dr. Scott Cohen, Dr. Eugenio. Di Sciascio, Excalibur Technologies Corp., Dr. David Forsyth, Dr. Theo Gevers, Qin He, Dr. D. P. Huijsmans, Professor Thomas S. Huang, IBM Almaden Research Centre, Illustra Information Technologies, Informix Software Ltd., Dr. Eleni Kaldoudi, Professor Toshikazu Kato, Patrick Kelly, Professor Ze-Nian Li, Ted Loewenberg, Dr. B. S. Manjunath, Professor Raghavan Manmatha, Thomas P. Minka, Dr. Sougata Mukherjea, Dr. Chahab Naster, Ginger Ogle, Professor Mark H. Overmars, Zoran Pecenovic, Dr. Dragutin Petkovic, John Pickford, Gina Porter, Professor Thierry Pun, Professor Erkki Oja, Mike Rhodes, roz Software Systems, Dr. Stan Sclaroff, Professor Martin Vetterli, Jeroen Vendrig, Virage, Inc., Professor Wayne Wolf, Mario M. Westphal, Professor Clement T. Yu, Xiang Zhou; Dr. Xiaoming Zhu. Stimuli Archive Images provided courtesy of Michael J. Tarr, Brown University, Providence, USA. Thanks also to Jonathan Edwards and Jonathan Riley, Institute for Image Data Research.

8

List of Tables Figure I: Birth of Venus by Sandro Botticelli ..............................................................................15 Figure II: The Query Image Set.....................................................................................................18 Figure III: Usability Heuristics......................................................................................................19 Figure IV: The Retrieval Results from an ImageFinder Search................................................25 Figure V: The ImageFinder User Interface..................................................................................26 Figure VI: ImageFinder Heuristic Evaluation Results. .............................................................27 Figure VII: The IMatch User Interface. ........................................................................................30 Figure VIII: The IMatch Search Results User Interface. ............................................................31 Figure IX: The IMatch Hearistic Evaluation Results. ................................................................31 Figure X: Image Read/Write Toolkit Compression Mode & Colour Space...........................38 Figure XI: Matching & Retrieval Features...................................................................................48

9

Executive Summary This report documents a six month investigation into content-based image retrieval (CBIR) software. The study forms part of a joint venture between Manchester Visualization Centre and the Institute for Image Data Research, which aims to investigate the feasibility of content-based image retrieval for the UK Higher Education Community. The project is funded through a JISC Technology Applications Programme (JTAP) award. The primary aim of this phase of the project was to undertake a review of currently available CBIR software in order to make informed recommendations for Phase III of the project. The report compliments the review of CBIR, Phase I, conducted by the Institute for Image Data Research. The study involved the identification, acquisition, installation, analysis and testing of a number of available CBIR applications. The original intention was to systematically analyse each system within a laboratory setting to establish its functionality, effectiveness and usability. Due to a number of factors this was not possible for all the systems identified. This was primarily a direct result of the application type and associated development time required to implement each system. The project was limited to content-based image retrieval systems that retrieved static image data. Over the course of the investigation, 74 systems were identified, which included systems both past and present. These were a combination of prototype research systems, database management systems (DBMS), software development kits (SDK), ‘turnkey’ systems, and World Wide Web (WWW) image search engines. While the identification of CBIR systems was very encouraging, attempts at acquisition proved disappointing. Of the identified systems, 9 in total were acquired:

♦ ARTISAN ♦ Excalibur Visual RetrievalWare SDK ♦ ImageFinder ♦ IMatch ♦ Informix Internet Foundation 2000 Server & the Excalibur Image Datablade ♦ Oracle 8i Enterprise Server and the Virage VIR Image Data Cartridge ♦ Photobook ♦ QBIC Development Kit ♦ Virage VIR Image Engine SDK and Image Read/Write Toolkit

The majority of the identified systems were research prototypes. Prototype research systems exist primarily to test the feature matching algorithms being developed by the research community. These systems are generally not available for public use or were not shipped in a suitable form to have been included in this review without significant development work. As a result, the functionality of the research prototypes is outlined. No attempt was made to assess their retrieval effectiveness or usability. Several of the prototype research systems are available over the web and URLs are listed for the web-based demonstrator.

10

The report provides a summary of the functionality for the following applications:

♦ Excalibur Visual RetrievalWare SDK, Excalibur Corp. ♦ ImageFinder, Attrasoft ♦ IMatch, MWLabs ♦ QBIC Development Kit, IBM Corp. ♦ Virage VIR Image Engine SDK and Image Read/Write Toolkit, Virage Inc.

General system information and matching features are described, and are collated in a matrix, Appendix A and Appendix B, for all identified systems. The amount of publicly accessible information varied significantly between systems and this is reflected in the information collated. The results of a small-scale feature matching and retrieval experiment for three applications are also documented: ImageFinder, IMatch and QBIC. The purpose of the test was to provide a general indication as to the initial effectiveness of the systems matching features and retrieval capabilities. The tests were not rigorous or scientifically controlled retrieval experiments and must not be regarded as an indicator as to the systems overall effectiveness. The experiments were specifically designed to test whether the matching features in the application achieved their intended purpose by identifying similar images to the query image based on the matching feature employed. Due to the widely recognised difficulties of assessing the effectiveness of retrieval results readers are left to draw their own conclusions. A sample of the retrieval output is documented in Appendix C. The image dataset used in the retrieval experiment was comprised of several distributed image datasets, which are freely available to the academic community, and image datasets bundled with ImageFinder and QBIC. The datasets were merged and converted to produce a set of 24-bit colour and greyscale JPEG files with no additional compression. The retrieval test verified that generally the matching features behaved predictably in terms of their functionality. Comparison of the retrieval results suggests that there are varying degrees of retrieval precision and recall between the CBIR applications. Detailed knowledge of the data set used in the retrieval experiment indicates that the degree of retrieval precision and recall, and the reliability of the system are in general highly questionable. Two systems were subject to a heuristic evaluation, ImageFinder and IMatch. The evaluation involved the examination of the user interface to judge its compliance with recognized and widely accepted usability principles. The results suggest that there is considerable scope for improvement for both systems. Content-based image retrieval potentially provides new opportunities to extend and enhance the constraints and limitations imposed by the traditional information retrieval paradigm on image collections. The number of CBIR systems is extremely encouraging. Nevertheless, there are still a significant number of open research issues to be addressed if this technique is to prove fruitful. The current impasse with regards to the efficacy of the retrieval techniques being developed and the need to develop suitable evaluation frameworks and benchmarks is now critical.

11

Abbreviations ABS: Attrasoft Boltzmann Machine AOI: Area of Interest API: Application Programming Interface APRP: Adaptive Pattern Recognition Processing BMP: Bit Mapped CMYK: Cyan, Magenta, Yellow, Black DBMS: Database Management System DDIF: Digital Document Interchange Format DK: Development Kit EMF: Enhanced Meta File GIF: Graphics Interchange Format GUI: Graphical User Interface JPEG: Joint Photographic Experts Group LZW: Lempel-Ziv-Welch MAC: MacPaint MDI: Multiple Document Interface PCT: Macintosh PICT PCX: Zsoft Paintbrush PPM: Portable Pixelmap PNG: Portable Network Graphics PSD: Photoshop Graphic Format QDK: QBIC Development Kit

12

QVE: Query by Visual Example RAM: Random Access Memory RDMS: Relational Database Management System RGB: Red, Green Blue RLE: Run Length Encoding SCT: SciTex Continuous Tone SDK: Software Development Kit SQL: Structured Query Language TIFF: Tagged Image File Format TGA: Truevision Targa URL: Universal Resource Locator VM: Virtual Machine WMF: Windows Meta File WWW: World-Wide-Web

13

Copyright Excalibur SDK© and Visual RetrievalWare© is copyright of Excalibur Technologies Corp.®; ImageFinder© is copyright of Attrasoft®; IMatch© and the IMatch Image Retrieval Engine© are copyright of Mario M. Westphal, 1998-2000; Informix Internet Foundation 2000© is copyright of Informix; The Excalibur Image datablade© is copyright of Excalibur Technologies Corp.® and Informix; Microsoft®, Internet Explorer®, Windows®, and the Windows logo® are registered trademarks of Microsoft Corporation in the United States and Other Countries; Oracle 8i© and the Virage VIR Image Data Cartridge© are copyright of Oracle Corp; QBIC© is copyright of IBM Corp.®; UNIX® is a registered trademark in the United States and other countries, exclusively licensed through X/Open Company, Ltd.; VIR Image Engine©, Virage Core Library©, and the Image Read/Write Toolkit© are copyright of Virage, Inc.®. Virage® is a registered trademarks of Virage, Inc. VIR Image Engine®, Virage Core Library®, and the Virage logo® are registered trademarks of Virage, Inc; The Stimuli© image dataset is copyright Michael J. Tarr and Brown University, 1997, 1998; The graphics interchange format, GIF©, is copyright by CompuServe, Inc. GIF® is a registered trademark of CompuServe, Inc.; Parts of the JPEG image read-write software© is copyright 1991-95 by Thomas G. Lane. This software is based in part on the work of the independent JPEG group. All trademarks are the property of their respective owners. All rights reserved.

14

Disclaimer The opinions expressed in this report are solely those of the authors. While every precaution has been taken in the preparation of this document, the authors, Manchester Visualization Centre, Manchester Computing, and the University of Manchester assume no responsibility for errors or omissions. No liability is assumed for damages resulting from the use of the information contained herein.

15

1 Introduction The field of image retrieval has been an active research area for several decades and has gained steady momentum in recent years as a result of the dramatic and unparalleled increase in the volume of digital images [1]. Volk-Beard [2] cites the example of a medical-image radiology workstation. Based on initial calculations it was estimated that a single 14”x17” radiography could be digitised at 22000 12-bit pixels (6MB approx). A similar investigation by Pizer [3] indicated that 24000 12-bit pixels (24MB approx) was a more realistic estimation. This combined with other factors such as an examination requiring several radiographs, and the average number of examinations per year left Volk-Beard to conclude that an image database for a typical 500-bed hospital would require at least 15 terabytes storage per year [2]. Image production and use now routinely occurs across a broad range of disciplines and subject fields e.g. art galleries and museum management; architectural and engineering design; interior design; remote sensing and earth resource management; geographic information systems; scientific database management; weather forecasting; retailing, fabric and fashion design; trademark and copyright database management; law enforcement and criminal investigation; picture archiving and communication systems. Despite the technological advances in image data capture and storage, the expertise and techniques for effective image retrieval has not kept pace with the technology of image production [4]. The ability to effectively retrieve non-alphanumeric data is a complex and multifaceted issue. While it is generally feasible to state what an image consists of in terms of the objects it contains, one of the main difficulties arises from the subjective, individual interpretation of the 'non-verbal symbolism' of an image [5]. For example, Botticelli’s depiction of the Birth of Venus [6]. See Figure I.

Figure I: Birth of Venus by Sandro Botticelli

16

The figures depicted in the picture are Venus, Pomona, and the Zephyrs. The figure on the right is the nymph Pomona, a descendent from the ancient goddess of fruit trees. The figures on the left are the Zephyrs, gods of the west wind. The figure in the centre is Venus, the Roman goddess of fertility. While some degree of background knowledge and subject expertise is required on the reader’s part to identify the characters depicted in the painting it is generally possible to state what the picture contains. What cannot be clearly stated with any certainty is what the picture means, and whether it means anything at all. Zollner [6] comments that few details of the picture have been interpreted either coherently or convincingly. Work by Enser [7] and Keister [8] also illustrates that description and meaning of a visual image is open to individual interpretation. Earlier work by Sunderland [9] suggests that the ability to interpret visually oriented material is subject to a number of external factors which compound the problem further e.g. age, gender, social grouping. Initial solutions resulted in the development of a number of image retrieval systems, which followed the traditional information retrieval paradigm [10]. Early work by Chang [11] is an example of this approach. It is widely recognised that this method imposes severe limitations on the utility of an image collection as an information resource [7]. The inherent difficulties and limitations of applying this paradigm to collections of images has been outlined and discussed in some depth by a number of commentators who emphasise specific areas of contention [12-20]. In recent years, developments have focused on the retrieval of images by their content. The origins of the technique have been attributed to the early experiments conducted by Kato [21] into the automatic retrieval of images by colour and shape feature [22]. Content-based image retrieval involves a direct matching operation between a query image and a database of stored images. The three most common image matching features are colour, shape and texture. The process involves computing a feature vector for the unique characteristics of the image. Similarity is computed by comparing the feature vectors of the images. The result of this process is a quantified similarity score that measures the visual distance between the two images represented by the feature vectors. Queries are expressed through visual examples, query by visual example (QVE), which can either be formulated by users or selected from randomly generated image sets. Feature characteristics of the query image can be specified and weighted against each other. Several methods have been proposed to support this interaction [10, 23]. However, research to date is contradictory and the validity of the interaction methods continue to remain untested with real user populations [4, 24, 25]. This analysis of the feature characteristics permits the objective analysis of pixel distribution measurements from the raw sensory input data [10]. A considerable amount of research in this area has been directed at developing and advancing robust retrieval algorithms for CBIR and this has been achieved with limited success [26, 27]. A number of recent publications have surveyed the feature matching techniques utilised in the process of content-based image retrieval and readers are directed to those publications for detailed descriptions [1, 10, 22]. Nevertheless, the difficulties involved in developing effective and robust image retrieval systems are problematic e.g. image encoding, storage, compression, transmission, display, shape description and matching [16].

17

The principal aim of this phase of the project was to undertake a review of currently available content-based image retrieval (CBIR) systems in order to make informed decisions and recommendations for Phase III of the project. This report documents a six month review of content-based image retrieval software. The investigation forms part of a broader study, which aims to investigate the feasibility of content-based image retrieval for the UK Higher Education Community. The report is divided into several sections: Section 2 outlines the methods employed in the study; Section 3 summaries the functionality of five commercial CBIR applications: Excalibur Visual RetrievalWare SDK, ImageFinder, IMatch, QBIC DK, and the Virage VIR Image Engine SDK; Section 4 highlights the functionality for the increasing number of prototype research systems; Section 5 discusses the findings of the investigation.

18

2 Methods This section outlines the methods employed to assist in determining retrieval effectiveness and the usability of the systems included in the study.

2.1 Retrieval Effectiveness The purpose of the retrieval experiment was to provide an indication to the initial effectiveness of the systems matching features. Functional testing, also known as black-box testing, was employed as the framework to test the effectiveness of the matching features [28]. This method is a software engineering technique, traditionally employed in defect testing, where the system is considered a ‘black-box’ whose behaviour can be only be determined by analysing its input and related output operations. See Sommerville for a full description of the method [28]. All test were performed on a Dell Precession Workstation 220 with the following hardware components:

♦ 533Mhz Intel Pentium III microprocessor with 256K Cache ♦ 128MB ECC RDRAM ♦ Windows NT4 Service Pack 6(a) ♦ Diamond nVidia TNT2 (2x AGP) Video Card 32MB RAM at 1600*1200

resolution in True Colour Mode. ♦ 10.2GB ATA-66 EIDE HD with a seek time of 7.200rpm

Before starting the retrieval test the system was rebooted. This provided full system resources to each product with an empty cache and clean paging file. The default matching feature settings were used for all the applications involved in the test. The image dataset used in the retrieval experiment contained 3000 static images and comprised of several distributed image datasets located at the Stimuli archive, and image sets bundled with ImageFinder and QBIC. The datasets consisted of range of 24-bit or 8-bit colour and greyscale images in a variety of image formats e.g. GIF, JPEG, PICT, and TIFF. The datasets were merged and converted to produce a set of 24-bit colour and greyscale JPEG files with no additional compression. This set was further reduced to a random selection of 1000 images as a result of the maximum number of images that the ImageFinder application could currently handle i.e. 1000. A set of images was specifically selected as the query images. This selection was based on their visual properties and characteristics. See Figure II.

Chemical Structure

Fractal Hands Beth Gibbons

Rock Face Rocks

Figure II: The Query Image Set

19

2.2 Heuristic Evaluation Usability heuristics were used to assess the usability of the systems [29]. This technique is a usability engineering method for identifying usability problem with the design of a user interface. See Nielsen for a full description of the technique [30]. The evaluation involved the examination of the user interface against a framework of ten heuristics to judge its compliance with recognized usability principles i.e. the “heuristics.” See Figure III. The heuristics were measured on a eight point rating scale, -3, -2, -1, 0, +1, +2, +3, NA, centering the scale at zero as an appropriately neutral value.

Heuristic

Visibility of system status Does the system keep users informed about what is going on, through appropriate feedback within reasonable time?

Match between system and the real world

Does the system speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms? Does the system follow real-world conventions, making information appear in a natural and logical order?

User control and freedom Does the system allow users to correct mistakes? Are there clearly marked “emergency exits” to leave the unwanted state without having to go through an extended dialogue? Does the system support undo and redo?

Consistency and standards

Do users have to wonder whether different words, situations, or actions mean the same thing? Does the system follow accepted platform conventions?

Error prevention

Has the system been designed to prevent problems from occurring in the first place?

Recognition rather than recall

Does the system make objects, actions, and options visible? Are instructions for use of the system visible or easily retrievable whenever appropriate.

Flexibility and efficiency of use

Does the system support features, which will speed up the interaction for the expert user i.e. does the system cater for both inexperienced and experienced users? Does the system allow users to tailor frequent actions?

Aesthetic and minimalist design

Do dialogues contain information that is irrelevant or rarely needed?

Help users recognize, diagnose, and recover from errors

Are error messages expressed in plain language, precisely indicate the problem, and constructively suggest a solution?

Help documentation

Is there system documentation? Is the documentation easy to search, focused on the user's task, list concrete steps to be carried out, and not too complex?

Figure III: Usability Heuristics

20

3 Commercial Content-Based Image Retrieval Systems The following subsections provide a summary of the functionality for five commercial content-based image retrieval applications:

♦ Excalibur Visual RetrievalWare SDK ♦ ImageFinder ♦ IMatch ♦ QBIC DK ♦ Virage VIR Image Engine SDK.

The systems are listed in alphabetical order. For commercial purposes, the individuals and companies responsible for the development of Excalibur Visual RetrievalWare SDK, ImageFinder, IMatch, and the VIR Image Engine SDK, reserved the right to withhold specific information relating to what the underlying matching algorithms were. As a result, no attempt has been made to speculate as to what these may be. QBIC has been reported in several journals and conference proceedings and readers are directed to those papers for more detailed descriptions of the underlying approach, techniques, algorithms, methods and development history of QBIC [31-34]. Three systems, ImageFinder, IMatch and QBIC, were subjected to a small-scale retrieval experiment. Two systems, ImageFinder and IMatch, were subjected to a heuristic evaluation. Due to the nature of the Excalibur Visual RetrievalWare SDK and the Virage VIR Image Engine SDK, both were excluded from the retrieval and usability tests. QBIC was not subject to the usability test. QBIC (QDK), the Excalibur Visual RetrievalWare SDK, and the Virage VIR Image Engine SDK are not turnkey applications they are software toolkits, which provide a set of tools for building content-based image retrieval applications. The QBIC development kit is shipped with a web-based demonstrator, which allowed the matching features supported by QBIC to be tested. The Informix Internet Foundation 2000 Server incorporates the image retrieval technology of Excalibur Corp. and Virage Inc. in the form of image Datablades to extend its functionality. The foundations of the Datablades are the Excalibur Visual RetrievalWare SDK and the Virage VIR Image Engine SDK respectively. Similarly, Oracle 8i Enterprise Server incorporates the Virage VIR Image Data Cartridge. The foundation of the Data Cartridge is the Virage VIR Image Engine SDK. As a result of the extensive similarities that exist between these products and the SDK's from which they were developed, they are discussed in the context of their originating technology.

21

3.1 Excalibur Visual RetrievalWare SDK, Excalibur Corp. The Excalibur Visual RetrievalWare Software Developers Kit (SDK) is an open application development environment distributed by Excalibur Corp. The SDK is a developer's toolkit, which provides a set of tools for building content-based image applications. The SDK includes three API's, C, C++ and Java, for image processing, a Tcl/Tk interpreter, sample programs, source code and reference documentation. The toolkit includes several examples that demonstrate what can be built using the SDK. These programs could serve as the foundation for building more sophisticated applications. The underlying technology of Visual RetrievalWare, APRP™ (Adaptive Pattern Recognition Processing), is a self-organizing system that automatically indexes the binary patterns in digital data and creates a self-optimised pattern-based memory for the native content of the data. APRP™ is modelled on the way biological systems use neural networks to process information. Several enhancements have been made to the current release of the SDK, including support for Java, enhanced colour, shape and texture feature extractors, support for MPEG-1 and MPEG-2 video format and multi-threaded search capabilities for multi-processor computers. The toolkit contains a range of built-in functions that create and maintain generic feature vector databases. The SDK has been specifically designed to allow users to develop their own domain specific feature extractors. The toolkit currently includes built-in support for a generalized feature vector extractor whose algorithm is based on a loose model of a biological retina. The algorithm extracts three characteristics from an image, colour, shape and texture, and then produces a feature vector according to the exact form of the image's constituent pixels. The feature vectors are stored in a binary index. Images are matched by comparing the indexes of the images. The feature vector index is optimised for speed. The algorithms utilized by Visual RetrievalWare are designed to be executable in parallel, which can have an adverse effect on the processing time. A web-based demo of Excalibur’s Visual RetrievalWare is available from: vrw.excalib.com The SDK ships with several example programs to demonstrate how feature extractors can be used to create and search the indexes. The sample programs include:

♦ CST (Colour, Shape, Texture): a colour, shape, and texture application that enables images to be indexed, searched and retrieved using the three feature characteristics of an image as the search criteria.

♦ Kanji: a Kanji character recognition demo. ♦ Show Image: an image database manager that indexes images by their content and

provides access to many of the image processing functions.

22

3.1.1 Matching Features The Excalibur Visual RetrievalWare SDK supports several matching features:

3.1.1.1 Colour The colour function analyses the global distribution of colour within the entire image for both the dominant colour and the variation of colour throughout the entire image independent of its location. The position of the colour has no value.

3.1.1.2 Shape The shape function measures the relative orientation, curvature, and contrast of lines in the image. The colour, position and absolute orientation of the lines have no value. This attribute is most effective when the lines are crisp and clean.

3.1.1.3 Texture The texture function analyses areas for periodicity, randomness, and roughness of fine-grained textures in images. The colour, position and absolute orientation of the features have no value.

3.1.2 Image Formats The SDK has been specifically designed to process two-dimensional, bi-tonal, greyscale, and colour image data from several common industry-standard image formats: BMP, DDIF, GIF, JFIF, PNG, PPM and TIFF. The software's JFIF support is based in part on the work of the Independent JPEG Group.

3.1.3 Hardware & Software Requirements The CD-ROM distribution of the Excalibur Visual RetrievalWare SDK contains the binaries for use with HP-UX, IRIX, DEC OSF1, Solaris, and Windows operating systems.

3.1.4 Price The Excalibur Visual RetrievalWare Software Developers Kit retails at £50,000 and incurs an additional deployment cost. Contact Excalibur Corp., www.excalib.com, for the latest pricing information.

23

3.1.5 Excalibur Image DataBlade The Excalibur Image DataBlade module extends the retrieval capabilities of Informix Internet Foundation 2000 Server. The DataBlade module combines the Excalibur image technology with the Informix Universal Data Option to store, retrieve, and search images in a RDBMS database. With the Excalibur Image DataBlade module and the Universal Data Option it is possible to use SQL statements to store images in a database and retrieve them with a content-based search. The DataBlade is co-developed and co-supported by Informix and Excalibur. The DataBlade is a product of the Excalibur Visual RetrievalWare SDK. The DataBlade is sold and distributed by Intraware and retails at $1,875.00. Contact Intraware, www.intraware.com, for the latest pricing information.

24

3.2 ImageFinder ImageFinder, developed by Attrasoft, is a Windows-based content-based retrieval system. Attrasoft produce three image retrieval products ImageFinder, Internet ImageFinder and ImageHunt. The underlying technology of each product is identical with the only significant difference between them being the design of their graphical user interfaces (GUI). An evaluation copy of all the software can be downloaded from: www.attrasoft.com. The install procedure for the software is an extremely simple process, which can be handled by most competent users. The foundation of Attrasoft's image retrieval software is PolyNet a derivative of Attrasoft's first generation neural network simulation software, Boltzmann Machine (ABM). PolyNet simulates two types of neural networks to optimise recall performance: Polytomous Hopfield Model and Polytomous Boltzmann Machine. The Hopfield network is a completely connected and recurrent neural network. The Boltzmann Machine is a special type of neural network where each neuron configuration has a certain probability to appear. The dynamics of a neural network can be expressed in terms of energy functions. The energy function of the Hopfield network is determined by exemplar patterns. The network weights are fixed by the patterns and there is no means of having the network learn the weights through exposure to them. The Hopfield network finds local minima of the energy function. A recognised problem with the Hopfield network is that it suffers from ill-defined global optimisation properties, which can hinder recall performance. The Boltzmann machine extends the capabilities of the Hopfield network by introducing an algorithm for the adaptive determination of the weights. The Boltzmann machine combines the Hopfield network architecture with a process developed in the field of statistical physics, simulated annealing. Annealing is the process where metals are often cooled very slowly, which allows them time to arrange themselves into a stable, structurally strong, low energy configuration in order to avoid the metastable states produced by quenching. This process gives the system the opportunity to jump out of local minima with a reasonable probability while the temperature is still relatively high. With simulated annealing, the network operates at a high temperature and arrives at the equilibrium state for that temperature. By decreasing the temperature without destroying the equilibrium state, the network is far more likely to arrive at a global minimum and increases the accuracy of global error minimisation, thereby improving recall performance. This process has been deployed as the core of Attrasoft Image retrieval software. In PolyNet, only two neuron states are allowed: ground state and excited state. To train the network, training data is provided in the form of key images or key segments. Key segments are selected areas of interest (AOI) within an image. If no AOI is specified the whole image is the AOI. The training patterns influence the network. Every time the network analyses the training pattern, the network stores the information by modifying the neuron synaptic connections. Modifying the values of the connections represents a learning process as neural networks learn their environment by changing their internal connections. Over a period of time the synaptic connections hold certain values. These values represent the neural network's memory and it can be used to perform certain tasks.

25

The PolyNet software performs two basic tasks: classification and pattern recognition. Classification is concerned with finding and allocating pattern classes. Pattern recognition is concerned with the pattern identification given its classification or pattern segment.

3.2.1 Feature Matching Retrieval is based on complete or incomplete pattern matching. To identify objects within an image ImageFinder simulates the mechanics of the human visual system by disregarding the image background and focusing on the foreground objects. This is achieved by setting background filters, which corresponds to the global distribution of background colour in the image. The demonstration system supported several background filters including black, white and several combinations of RGB. The retrieval results are output to Microsoft Internet Explorer, which includes information regarding image path, file name and similarity score. See Figure IV. Individual similarity scores are computed to indicate the similarity between the query image and the retrieved images. Similarity scores are expressed as integers. The higher the number, the greater the similarity to the original query. The retrieval output is not sorted in rank order of similarity. This makes it extremely difficult to identify the most similar images especially if the retrieval output contains a large number of similar images.

Figure IV: The Retrieval Results from an ImageFinder Search.

26

ImageFinder is linear under Microsoft's Java Virtual Machine (VM) and runs at a constant speed, regardless of the number of images in the data set. For example, one image takes one unit of time. A typical retrieval speed is the in order of 1 image per second. The maximum number of images that can be searched is currently limited to 1,000 files. Attrasoft will provide custom built solutions for larger image repositories. Custom built versions of the software can also be extended to include, alternative background filters, specific symmetries, and other image formats.

3.2.2 Image Formats ImageFinder currently supports three common industry-standard image formats: BMP, GIF, and JPEG. Custom built solutions can be extended to include other image formats.

3.2.3 User Interface The user interface of ImageFinder is small and compact. The functions of the system are assessable through several buttons and one combobox. System feedback is output to a small textbox. The path of key segments and the search directory are specified in textboxes. See Figure V

Figure V: The ImageFinder User Interface.

27

Figure VI summarises the results of the heuristic evaluation for the ImageFinder user interface. It should be stressed that the version used in this study was only a demonstration system. It is unclear whether the user interfaces for commercially available releases of the software differ significantly.

Heuristic -3 -2 -1 0 +1 +2 +3 NA

Visibility of system status Match between system and the real world User control and freedom Consistency and standard Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help users recognize, diagnose, and recover from errors Help and documentation

Figure VI: ImageFinder Heuristic Evaluation Results.

3.2.4 Hardware & Software Requirements ImageFinder requires the minimum hardware and software requirements:

♦ Intel Pentium II, 400 MHz ♦ WIN 95/98/NT/2000 ♦ 32 MB RAM ♦ Microsoft Java VM ♦ Microsoft Internet Explorer 5

Internet Explorer 5 is required because of the Microsoft Java virtual machine (VM). An older Microsoft Java VM will significantly reduce the performance of the software. It is highly recommended that searches are conducted on the highest specification machine available. For example, a search with an Intel Pentium II Class Processor with the following hardware components took 5 hours 47mins to search through 1000 images:

♦ Intel Pentium II 260MHz ♦ Windows 98 (2nd Edition) ♦ 64MB RAM ♦ 4MB Creative Labs graphics card

3.2.5 Price The current release of ImageFinder retails at $999.00. Contact Attrasoft for the latest for pricing information: www.attrasoft.com

28

3.3 IMatch, MWLabs IMatch, developed by Mario M. Westphal, is a shareware utility for the Windows operating system, which allows users to perform a variety of content-based image retrieval methods on their datasets. IMatch allows users to create multiple databases for their image collections. The databases contain the feature vectors of the images and other related image information such as the path of the image. Each IMatch database consists of two main files: *.imd and *.imi. The main database file is *.imd. The matching data is contained in the *.imi file. A third file can be created for thumbnails, *.imt. This file contains thumbnail representations of all the images in the database. This adds the facility to use the database browser and speeds up the display of images in the results window. It is highly recommended that a thumbnail database is created. It should be noted that creating a thumbnail database requires additional hard disk space. All three file extensions, *.imd, *.imi, *.imt, are unique to the IMatch system. The maximum database size is only limited by the available hard disk space on the users machine. The overall run-time performance and the match precision of the software is affected by databases containing more than 100,000 images. Once a database has been created, it can be queried by colour, texture, and shape matching facilities. A free 30-day evaluation copy is available for users to assess the software's capabilities. The evaluation version is fully functional and will display some additional messages when heavy use is made of IMatch or used beyond the evaluation period. The download package includes the following components:

♦ IMatch executable for Windows 95/98/NT/2000 ♦ IMatch Image Matching Engine (32-Bit DLL) ♦ Complete online manual and context-sensitive help ♦ Full install and uninstall feature

The evaluation copy can be downloaded from: www.mwlabs.de/download.htm. Both the install and uninstall procedure for the software are an extremely simple process, which can be handled by most competent users.

3.3.1 Matching Features IMatch supports several CBIR matching methods to help assist in the identification of similar images: colour similarity, colour and shape (Quick), colour and shape (Fuzzy) and colour percentage and distribution. Each method has strengths and weaknesses. Users are encouraged to experiment extensively with each matching feature in order to determine which method is most effective for the retrieval of specific query types. The following sections outline the basic retrieval features supported in IMatch.

29

3.3.1.1 Colour Similarity This feature matches images based on the global distribution of colour. This method was generally accurate and effective in retrieving images with similar global distributions of colour or grey scale levels.

3.3.1.2 Colour and Shape (Quick) The colour and shape matching feature identifies specific shapes, shape distributions and colour distributions in an image and orders the result set based on the overall distance. This method is significantly faster than the Colour and Shape (Fuzzy) method, but can provide less accurate results.

3.3.1.3 Colour and Shape (Fuzzy) This variant of the colour and shape algorithm performs additional steps to identify objects in the source image. As a result, it can find similar images that the Colour and Shape (Quick) variant may not find. However, this method can also lead to some very unexpected and unpredictable matches that bear little or no resemblance to the original query image. It is suggested that the Hue and Colour contribution sliders are adjusted to emphasize the importance of the features in the query image.

3.3.1.4 Colour Percentage & Distribution IMatch allows users to specify both global and local distributions of colour in an image. The colour percentage matching feature allows users to specify the overall percentage of a specific colour in an image. Each cell in the grid represents 5% of the image. The order of the colours in the grid is significant and influences the outcome of the search results. Cells on the left have a higher precedence. This method was generally accurate in retrieving images with similar global distributions of colour or grey scale levels. The colour distribution matching feature allows users to specify the distribution and location of colour in an image. This feature is similar to the percentage match, however, it not only examines the actual colours in an image, but also considers the colour location. Again, this method was generally effective in retrieving images with similar local distributions of colour or grey scale levels.

3.3.1.5 CRC Checksum, Duplicate Scanner and Fuzzy Filename IMatch also supports non-CBIR features to identify images: CRC Checksum, Duplicate Scanner, and Fuzzy Filename. The CRC Checksum feature identifies duplicate images in the database that are binary identical to a given query image. This method is very fast and retrieves only binary identical images in the image database. The Duplicate Scanner feature finds duplicate images, even if images have been resized, cropped or saved in different file formats.

30

This method is extremely useful for scanning database for duplicate images or to match whole folders with new images against the database. The fuzzy filename feature searches for images with similar file names to the name of the original query image. The fuzzy matching algorithm used in this method retrieves image file names that are similar to the original file name.

3.3.2 Image Formats IMatch supports several common industry-standard image formats: BMP, EMF, JPEG, PCT, PNG, TGA, TIF, and WMF. The GIF image file format is not supported in IMatch due to LZW licensing problems with UNISYS.

3.3.3 User Interface The primary user interface of IMatch is small and compact. The functions of the system are accessible through several buttons and ComboBox’s. Search parameters can be applied and adjusted through a variety of checkboxes and sliders. The user interface is ordered in logical steps so that users are guided through the process of database creation, population and matching. See Figure VII.

Figure VII: The IMatch User Interface.

Search results from the system are output to a second user interface in rank order. See Figure VIII. The left hand section of the user interface contains a hierarchical view of the retrieved images. Each image is displayed with its full name and path. A similarity indicator bar highlights the similarity between the retrieved image and the query image.

31

The more green displayed in the bar, the closer the image is to the original. Thumbnails for all matching images are displayed in the right hand section of the user interface. The original image is generally displayed in the top left-hand corner surrounded by a red border depending on the sort order selected. A coloured status bar is displayed next to each image, which indicates if the image is available (Green), orphaned (Red) or modified (White). From this GUI users can re-query the database using the default feature matching settings and parameters or perform simple image management operations e.g. cut, copy, paste, delete, move, and rename.

Figure VIII: The IMatch Search Results User Interface.

Figure IX summarises the results of the heuristic evaluation for the IMatch.

Heuristic -3 -2 -1 0 +1 +2 +3 NA

Visibility of system status Match between system and the real world User control and freedom Consistency and standard Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help users recognize, diagnose, and recover from errors Help and documentation

Figure IX: The IMatch Hearistic Evaluation Results.

32

3.3.4 Hardware & Software Requirements IMatch requires the minimum hardware and software requirements:

♦ 80486 PC ♦ 64 MB RAM ♦ Windows 95 (Service pack 1) ♦ Windows 98 ♦ Windows NT 4 (Service pack 3)

3.3.5 Price A free 30-day evaluation copy is available for users to assess the software's capabilities. An evaluation copy can be downloaded from: www.mwlabs.de/download.htm. After the 30-day trial period has expired users are required to purchase a license. The IMatch license costs $50.00 per user. A new version of IMatch is due for release in the Summer of 2000.

33

3.4 QBIC (Query By Image Content) Development Kit, IBM Corp. QBIC, developed by IBM, Almaden Research Centre, is an open framework and developing technology, which can be utilised for both static and dynamic image retrieval [31]. QBIC has undergone several iterations since it was first reported [32]. QBIC allows users to graphically pose and refine queries based on multiple visual properties including colour, shape and texture. QBIC is shipped as a software development kit, QBIC Development Kit (QDK). The QDK includes C++ libraries and header files, 4 command-line executable modules, source code, a demonstrator application built using the QBIC API, and a mini web server used by the QBIC demonstrator application, 33 images and reference documentation. The QBIC API computes, stores, and retrieves the data in a database and its catalogues. A database is a named collection of data e.g. Oracle or any other DBMS. A catalogue is a named set of tables within a database. The API has two levels of functionality: command-line executables and C++ abstract and fully implemented classes. The set of command-line executables, QbMkDbs, QbQBE, QbMkThmb, and QbDumpDb, are used for database population, querying, thumbnail generation, and dumping the contents of feature data from database files. It is possible to build a complete application using these command-line programs. The programs can be run in a command shell, or called from an application using system calls. The QDK provides a high-level wrapper class as an interface to the command-line executables, QbicWrapClass, making the functions accessible directly from within C/C++ code. The API also contains several advanced features to increase the speed of a search and to improve feature extraction. These advanced features are only available by special license agreement. QBIC supports several query types: simple, multi-feature, and multi-pass. A simple query involves only one feature. For example, identify images that have a colour distribution similar to the query image. A complex query involves more than one feature, which can take the form of a multi-feature or a multi-pass query. For example, identify images that have similar colour and texture features. With the multi-feature query the system searches through the different types of feature data in the database in order to identify similar images. All feature classes have equal weightings during the search, and all feature tables are searched in parallel. In contrast, with a multi-pass query the output of an initial search is used as the basis for the next search. The system reorganizes the search results from a previous pass based on the "feature distances" in the current pass. For example, identify images that have a similar colour distribution, and then reordering the results based on colour composition. With multi-feature and multi-pass queries, users can weight features to specify their relative importance. QBIC technology has been incorporated into several IBM software products, including DB2 Image Extender and Digital Library. For more information about QBIC, visit the web site at: wwwqbic.almaden.ibm.com.

34

3.4.1 Matching Features QBIC supports several matching features including colour, shape, and texture. Each method is implemented as a C++ class.

3.4.1.1 Global Colour The global colour function computes the average RGB colours within the entire image for both the dominant colour and the variation of colour throughout the entire image. Similarity is based on the three average colour values.

3.4.1.2 Local Colour The local colour function computes the colour distribution for both the dominant colour and the variation for each image in a predetermined 256 colour space. Image similarity is based on the similarity of the colour distribution.

3.4.1.3 Shape The shape function analyses images for combinations of area, circularity, eccentricity, and major axis orientation. All shapes are assumed to be non-occluded planar shapes allowing each shape to be represented as a binary image. It is unclear whether the current release of the QDK supports shape retrieval.

3.4.1.4 Texture The texture function analyses areas for global coarseness, contrast and directionality features.

3.4.2 Image Formats QBIC supports the following image formats: BMP, GIF, JPG/JPEG, PGM, PPM, TIF/TIFF, and TGA. In addition to the image formats listed, the QbGenericImageDataClass, can also read a QBIC Picker Image Description string either in memory or in a file. Currently, this method only recognizes rectangles. The description string is used heavily in the QBIC demonstrator.

35

3.4.3 Hardware & Software Requirements The binaries shipped with the QBIC development kit run on the following platforms:

♦ AIX 4.1 ♦ Linux 2.0.30 ♦ Solaris 2.6 ♦ Windows 95/98/NT/2000 ♦ Macintosh OS8 on a PowerPC

The development kit is distributed as a compressed file. Gunzip or a similar utility is required for uncompressing the files. For further queries regarding technical information contact Dr. Xiaoming Zhu, xzhu@almaden.ibm.com, at Almaden Research Centre, IBM.

3.4.4 Price The QDK is available for download with a free 90-day trial license. The files can be downloaded from: www6.software.ibm.com/dl/qbic/qbicn-p. For licensing information, contact Ted Loewenberg, tedl@almaden.ibm.com, at Almaden Research Centre, IBM.

36

3.5 VIR Image Engine & Image Read/Write Toolkit, Virage Inc. The VIR Engine and Image Read/Write Toolkit were developed by Virage Inc. The VIR Image Engine is a set of libraries, an object library with header file interfaces, which provide an ANSI C interface for application developers. The source code is delivered as a static or dynamic linkable library on several platforms. The VIR Image Engine analyses and compares the feature vectors and computes a similarity score. The Image Read/Write Toolkit is a C API that reads, writes and creates thumbnails for a variety of industry standard image files formats and some basic tools for converting colour space and other format attributes. The VIR Image Engine is the underlying technology driving the image search facilities at the AltaVista website: www.altavista.com The underlying philosophy behind the VIR Image Engine is that it is a tool that developers can utilize to develop or extend an applications retrieval functionality to include content-based image retrieval features. The Engine is designed to be applied in a range of diverse application domains and systems scenarios. As a result, the VIR Image Engine makes no assumptions about the application or system components in which it may be embedded. There is no knowledge of dependencies on file systems, database interfaces, user interfaces, network interfaces etc. It is a pure algorithmic API that can be employed by the application developer in any way desired. The VIR Image Engine examines pixels in the image and analyses it with respect to several primitives. These visual attributes include: colour, texture, and structure. The image characterization is derived from these features. Images that have been analysed can then be compared mathematically to determine their similarity score. Images are analysed once, and the primitive data is then used for comparison. The VIR Image Engine executes two primary functions within an application: image analysis and image comparison. Image analysis of an image is performed once and produces a feature vector of information. The feature vectors are mathematical representation of the visual content of the image and are approximately 1KB. The image comparison procedure takes two feature vectors, a set of weights for the visual primitives, and compares them using similarity operators. The result of this process is a quantified similarity score that measures the visual distance between the two images represented by the feature vectors. The scores are normalized between ranges of 0-100 and are independent of the dataset of images being compared.

37

3.5.1 Matching Features The visual analysis of images is performed by a set of primitives. Each primitive is responsible for extracting a visual property and providing the similarity operator for that property. There are four basic primitives: Global Colour, Local Colour, Structure, and Texture.

3.5.1.1 Global Colour The global colour function analyses the distribution of colour within the entire image for both the dominant colour and the variation of colour throughout the entire image independent of its location.

3.5.1.2 Local Colour The local colour function represents the distribution of colours in terms of localized colour and the spatial match-up of colour between two images.

3.5.1.3 Structure The structure function is used to determine large-scale shapes, which appear in the image. A known problem with this feature is that it faces problems with lighting and occlusion, and relies heavily on shape characterization techniques, rather than local shape segmentation methods.

3.5.1.4 Texture The texture function analyses areas for periodicity, randomness, and roughness of fine-grained textures in images. It is extremely sensitive to high-frequency features within the image.

38

3.5.2 Image Formats The Image Read/Write Toolkit supports several common industry-standard image formats: BMP (OS/2, Windows), GIF, JPEG, MAC, PCD, PCX, PICT, PNG, PSD, RLE, SGI, SCT, TGA, TIFF. Figure X outlines the compression mode and colour space for image file formats supported by the Image Read/Write Toolkit.

Compression Colour Mode None RLE LZW JPEG Bitmap Greyscale Indexed RGB CMYK BMP OS/2 RW R - - RW - RW RW - BMP Windows

RW R - - RW - RW RW -

GIF - - RW - - RW RW - - JPEG - - - RW - RW - RW RW MAC - RW - - RW - - - - PCD R - - - - - - R - PCX - RW - - RW RW RW RW - PI CT RW RW - - - - - RW - PNG RW - - - RW RW RW RW - PSD RW RW - - RW RW RW RW RW RLE RW RW - - RW RW RW RW - SGI RW RW - - RW RW - RW - SCT RW - - - - R - R RW TGA RW RW - - RW RW RW RW - TIFF RW RW RW - RW RW RW RW -

Figure X: Image Read/Write Toolkit Compression Mode & Colour Space

3.5.3 Hardware & Software Requirements The CD-ROM distribution contains the binaries of the VIR Image Engine and Image Read/Write Toolkit for HP-UX, IRIX, DEC OSF1, Solaris, and Windows NT. 35 megabytes of hard disk space are required for installation. The UNIX packages require the crypt and tar commands.

39

3.5.4 Price The VIR Visual Information Retrieval (VIR) Engine SDK retails at $30,000, and incurs an additional deployment cost of $16,500 per CPU. The VIR Visual Information Retrieval (VIR) Image Read/Write Toolkit costs $12,000 and incurs an additional deployment cost of $6,000 per CPU. Contact Virage Inc., www.virage.com, for the latest for pricing information.

3.5.5 Visual Information Retrieval Data Cartridge, Oracle While the VIR Image Engine and Image Read/Write Tool are intended for application developers, Virage has integrated this technology by collaborating with RDBMS vendors to extend their products retrieval capabilities. The VIR Image Engine is deployed as a data cartridge for Oracle 8i Server (Enterprise Edition), a datablade module for Informix Internet Foundation 2000 server, as well as being integrated into databases from Sybase, Object Design, and Objectivity. The Data Cartridge extends the retrieval capabilities of Oracle8i Image Cartridge to include content-based retrieval. The Oracle8 Image Data Cartridge provides native image data type support for the Oracle8i Server. The cartridges support two-dimensional static images in common industry-standard image file formats and compression schemes, and allows the images to be stored in-line or in external flat file repositories. The Visual Information Retrieval Data Cartridge includes the four basic primitives of the VIR Image Engine SDK: Global Colour, Local Colour, Structure, and Texture. When comparing images the cartridge provides control over the relative importance of each primitive. When matching, users can weight the importance of each visual attribute, and the cartridge calculates a measure of the similarity between two images for each attribute. Similarly users have the option to assign a threshold for determining whether or not two images are similar, or can derive rankings of the comparisons to identify how similar each image is to a query image. The following image formats are supported: BMP, CALS, GIF, JFIF, PCX, PICT, Sun Raster, Targa, TIFF. The Data Cartridge requires the Oracle8i Server combined with Oracle8i Object Option. Contact Oracle Corp. for the latest pricing information: www.oracle.com.

40

4 Prototype Research Systems The vast majority of content-based image retrieval systems are prototype research systems developed in universities and research laboratories. These systems exist primarily to test experimental retrieval algorithms. In general, the systems are not available or distributed in a suitable form to be evaluated without significant development work. As a result, the functionality of the research prototypes is summarised. No attempt was made to assess retrieval effectiveness or usability due to the experimental nature of the systems. The systems are listed in alphabetical order. Several of the research prototypes are freely available upon request or available as web-based demonstrators.

4.1 AMORE Amore, developed at the C&C Research Lab, NEC USA, is a World-Wide Web image search engine. The system allows the retrieval of images from the Web based on either keywords or the specification of a similar image or a combination of the two. AMORE supports colour and shape feature extraction. It is currently unclear whether the system supports relevance feedback. Users control the search by indicating how important colour and shape similarity should be in the results, and whether the results should be visually or semantically similar to the search query. Contact Kyoji Hirata, hirata@ccrl.sj.nec.com, at the C&C Research Lab, NEC USA for further information regarding the AMORE system. A WWW demonstration of the system can be found at: www.ccrl.com/amore.

4.2 ARTISAN (Automatic Retrieval of Trademark Images by Shape ANalysis)

The ARTISAN system, developed at the University of Northumbria at Newcastle, is a shape retrieval system, which performs both shape analysis, and provides both example-based similarity retrieval and partial shape matching. ARTISAN automatically identifies shape features in an image. The system subdivides each image into components, regions of similar intensity lying within a close boundary. These are categorized into families based on proximity or shape similarity, using principles derived from Gestalt psychology. ARTISAN calculates a variety of shape measures from these families and stores them in its database. Similar shape features are extracted from query images at search time, allowing the system to identify similar images by matching shape feature values. Similar images are returned and displayed on the screen in rank order of similarity. The effectiveness of ARTISAN system has been evaluated using 24 queries put to a collection of over 10,000 images from the UK Trademarks Registry. The evaluation produced encouraging retrieval results, demonstrating the overall feasibility of their approach to shape retrieval. Contact Dr John P. Eakins, john.eakins@unn.ac.uk, at the Institute for Image Data Research, University of Northumbria at Newcastle for further information regarding the ARTISAN system.

41

4.3 BlobWorld The BlobWorld system, developed at the University of California, Berkeley, supports colour, shape, spatial, and texture matching features. The system automatically segments each image into regions, which correspond approximately to objects or parts of objects in an image. The system allows users to view the results of the segmentation of both the query image and returned results to highlight how the segmented features have influenced the retrieval results. The system allows querying at a localised level of objects rather than global image properties by retrieving image regions that correspond approximately to objects. The BlobWorld system forms part of the Berkley Digital Library Project. Contact Dr. Chad Carson, carson@eecs.berkeley.edu, at the University of California, Berkeley for further information regarding the BlobWorld system. A WWW demonstration of Blobworld can be found at: elib.cs.berkeley.edu/photos/blobworld.

4.4 C-BIRD (Content-Based Image Retrieval) The C-BIRD system, developed at the Vision & Media Laboratory, Simon Fraser University, supports both colour, shape, and texture matching features. Contact Professor Ze-Nian Li, li@cs.sfu.ca, at the Vision & Media Laboratory, Simon Fraser University, for further information regarding C-BIRD. A WWW demonstration of the system can be found at: jupiter.cs.sfu.ca/cbird.

4.5 CAETIIML The CAETIIML system, developed at Princeton University, supports colour, shape, and texture matching features. Contact Professor Wayne Wolf, wolf@ee.princeton.edu, at Princeton University, for further information regarding CAETIIML.

4.6 CANDID (Comparison Algorithm for Navigating Digital Image Database)

The CANDID system, developed at Los Alamos National Laboratory, supports query by example of static images, allocating either an entire image or a specific region of interest with a global signature. The global signature represents texture, shape and colour matching features. Features are calculated at every pixel location in the image. A probability density function is then computed that describes the distribution of these features and is used as the content signature for the image. The system matches signatures with a distance measure to determine image similarity. Images are ranked according to their global signature. The CANDID approach has been tested retrieving pulmonary CT images, and multispectral satellite imagery. The evaluation produced encouraging retrieval results, demonstrating the overall feasibility of their approach to image retrieval. Contact Patrick Kelly, kelly@lanl.gov, at the Los Alamos National Laboratory, Los Alamos for further information regarding CANDID.

42

4.7 Charmer The Charmer system was developed at the Computer Vision Group, University of Geneva. Contact Professor Thierry Pun, thierry.pun@cui.unige.ch, at the Computer Vision Group, University of Geneva for further information regarding Charmer.

4.8 Circus (Content-based Image Retrieval and Consultation User-centered System)

The CIRCUS system, is the result of a collaboration between the Laboratoire de communications audiovisuelles (LCAV), Ecole Polytechnique Fédérale de Lausanne (EPFL), and the Computer Vision Group University of Geneva. The system supports colour, shape and texture matching features. Contact Zoran Pecenovic, zoran.pecenovic@lcav.de.epfl.ch, at the Laboratoire de communications audiovisuelles (LCAV), Ecole Polytechnique Fédérale de Lausanne (EPFL), for further information regarding Circus. A WWW demonstration of the system can be found at: lcavwww.epfl.ch/~zpecenov/CIRCUS/Demo.html.

4.9 Compass (COMPuter Aided Search System) The Compass system, developed at ITC-IRST Centre for Scientific and Technological Research, is an image retrieval system designed for distributed image databases. The system supports colour, spatial, and texture matching features. Contact Ornella Mich, mich@itc.it, at ITC-IRST for further information regarding the Compass system. A WWW demonstration of the system can be found at: compass.itc.it/demos.html.

4.10 Diogenes The Diogenes system, developed at the Department of Electrical Engineering and Computer Science, University of Illinois at Chicago, is an image search robot that uses multiple evidence combination techniques to identify images of people on the web. The system combines face recognition and metadata techniques in order to identify images. Diogenes employs Dempster-Shafer evidence combination based on automatic object recognition and dynamic local uncertainty assessment to account for uncertainty in retrieval. A comparative evaluation between Diogenes and several other well known commercial and research prototype produced encouraging retrieval results for the image type, and demonstrating the overall feasibility of their approach. Contact Yuksel Alp Aslandogan, yuksela@hotmail.com, at the Department of Electrical Engineering and Computer Science, University of Illinois at Chicago for further information regarding Diogenes.

43

4.11 DrawSearch The DrawSearch system, developed at the Department of Electrical and Electronics Engineering, Politecnico di Bari, supports colour, shape and texture matching features. The overall project is aimed at investigating techniques and tools to increase user's interactivity in query by image content over the WWW. Contact Dr. Eugenio Di Sciascio, disciascio@poliba.it, at the Department of Electrical and Electronics Engineering, Politecnico di Bari, for further information regarding DrawSearch. A WWW demonstration of the system can be found at: deecom03.poliba.it/DrawSearch.

4.12 ImageRETRO The ImageRETRO system, developed at the Department of Computer Science, University of Amsterdam, is an implementation of the Filter Image Browsing concept originally developed by Jeroen Vendrig. Users search through the image collection by choosing example images. The system computes a similarity ranking on image features and disregards the least similar images after each iteration until the user is satisfied with the set of images left in the retrieval loop. Contact Jeroen Vendrig, vendrig@wins.uva.nl, at Department of Computer Science, University of Amsterdam for further information regarding ImageRETRO. A WWW demonstration of the system can be located at: carol.wins.uva.nl/~vendrig/imageretro.

4.13 ImageRover The ImageRover system, developed at the Computer Science Department, Boston University, is a World Wide Web image search system that combines textual and visual statistics in a single index for content-based search of a WWW image database. Textual statistics are captured in a vector using a technique called latent semantic indexing (LSI). Similarly, visual statistics are captured in a feature vector using colour and orientation histograms. Users initially specify keywords to describe the desired images, and then refine the query through relevance feedback. The current implementation of the system includes sub-modules for the analysis of colour, orientation and word associations. Contact Dr. Stan Sclaroff, sclaroff@cs.bu.edu, at the Computer Science Department, Boston University, for further information regarding ImageRover. A WWW demonstration of the system can be located at: www.cs.bu.edu/groups/ivc/ImageRover.

4.14 ImageScape The ImageScape system, developed at the Institute of Advanced Computer Science, Leiden University, is a World Wide Web image search system. Contact Dr. Michael S. Lew, mlew@wi.leidenuniv.nl, at the Institute of Advanced Computer Science, Leiden University, for further information regarding ImageScape. A WWW demonstration of the system can be located at: www.wi.leidenuniv.nl/home/lim/image.scape.html.

44

4.15 IQUEST The IQUEST system, developed at the Laboratoire de Bases de Données, Ecole Polytechnique Fédérale de Lausanne supports both colour, shape, spatial and texture matching features for the retrieval of static images. Contact Mohamed Tahar Meharga, mohamed.meharga@epfl.ch, at the Laboratoire de Bases de Données, Ecole Polytechnique Fédérale de Lausanne, for further information regarding IQUEST.

4.16 LCPD (The Leiden 19th Century Portrait Database) LCPD, developed at the Computer Science Department, Leiden university, supports shape and texture matching features for the retrieval of greyscale images. Contact Dr D. P. Huijsmans, huijsman@wi.leidenuniv.nl, at the Computer Science Department, Leiden University, for further information regarding LCPD. A WWW demonstration of the system can be located at: www.wi.leidenuniv.nl/~huijsman.

4.17 MARS (Multimedia Analysis and Retrieval System) The MARS system, developed at the Beckman Institute for Advanced Science and Technology, University of Illinois at Urbana-Champaign, supports combinations of colour, shape, spatial layout, and texture matching features. Contact Professor Thomas S. Huang, 1huang@ifp.uiuc.edu, for further information regarding the MARS system.

4.18 MIDSS (Multi-Resolution Image Database Search System) The MIDSS system was developed at the School of Electrical and Computer Engineering, Purdue University. Contact Professor Charles A. Bouman, bouman@ecn.purdue.edu, at the School of Electrical and Computer Engineering, Purdue University for further information regarding MIDSS.

4.19 NeTra The NeTra system, developed at the Department of Electrical and Computer Engineering, University of California Santa Barbara, supports colour, shape, spatial layout and texture matching features in segmented image regions to search and retrieve similar regions from an image database. Contact Dr. B. S. Manjunath, manj@ece.ucsb.edu, for further information regarding NeTra. A WWW demonstration of the system can be located at: maya.ece.ucsb.edu/Netra.

45

4.20 Photobook The Photobook system, developed at the Massachusetts Institute of Technology Media Laboratory, supports colour, shape and texture matching features. The system computes features vectors for the image characteristics, which are then compared to compute a distance measure utilizing one of the systems matching algorithms, including euclidean, mahalanobis, divergence, vector space angle, histogram, fourier peak, wavelet tree distances and user-defined matching algorithms via dynamic code loading. Photobook is freely available to the academic community. Contact Thomas P. Minka, tpminka@media.mit.edu, at the Massachusetts Institute of Technology Media Laboratory for further information regarding Photobook.

4.21 PicSOM (Picture Self-Organising Maps) The PicSOM system, developed at the Laboratory of Computer and Information Science, Helsinki University of Technology, is an image browsing system based on the Self-Organizing Map(SOM). The system utilizes a hierarchical version of the SOM neural algorithm, Tree Structured Self-Organizing Map (TS-SOM), as the method for retrieving similar images from a set of reference images. The system adapts to the user's preferences by returning images from those SOMs where their responses have been most densely mapped. Contact Professor Erkki Oja, erkki.oja@hut.fi, at the Laboratory of Computer and Information Science, Helsinki University of Technology for further information regarding PicSOM. A WWW demonstration of the system can be located at: www.cis.hut.fi/picsom/ftp.sunet.se/990909160216.html.

4.22 PicToSeek PicToSeek, developed at the Department of Computer Science, University of Amsterdam, uses photometric colour and geometric invariant indices. Invariant features are extract from each image in the database and are matched with the invariant feature set derived from the query image. Contact Dr. Theo Gevers, gevers@wins.uva.nl, at the Department of Computer Science, University of Amsterdam for further information regarding PicToSeek. A WWW demonstration of the system can be located at: carol.wins.uva.nl/~gevers/

4.23 SaFe(Spatial and Feature Query System) The SaFe system, developed at the Department of Electrical Engineering, Columbia University, supports spatial matching features. It provides a general framework for searching and comparing images by the spatial arrangement of regions or objects. Objects or regions are user assigned and can be allocated properties of spatial location, size, and colour. The system can also resolves spatial relationships by allowing users to position objects relative to each other in a query. Contact, Professor Shih-Fu Chang, sfchang@ee.columbia.edu, for further information regarding SaFe. A WWW demonstration of the system can be located at: disney.ctr.columbia.edu/safe

46

4.24 SQUID (Shape Queries Using Image Databases) The SQUID system, developed at the Centre for Vision, Speech, and Signal Processing, Department of Electronic and Electrical Engineering, University of Surrey, is a shape retrieval system. Images are matched by comparing curvature zero-crossing contours in a Curvature Scale Space image. Contact Sadegh Abbasi, s.abbasi@ee.surrey.ac.uk, at Centre for Vision, Speech, and Signal Processing, Department of Electronic and Electrical Engineering, University of Surrey, for further information regarding SQUID. A WWW demonstration of the system can be located at: www.ee.surrey.ac.uk/Research/VSSP/imagedb/demo.html.

4.25 Surf Image The Surf Image system, developed by IMEDIA at INRIA, the French National Institute for Research in Computer Science and Control, supports colour, shape, spatial location, and texture matching features. Contact Dr. Chahab Nastar, chahab.nastar@inria.fr, further information regarding Surf Image. A WWW demonstration of the system can be located at: www-rocq.inria.fr/cgi-bin/imedia/surfimage.cgi.

4.26 Synapse The Synapse system was developed at Centre Intelligent Information Retrieval, University of Massachusetts. Contact Professor Raghavan Manmatha, manmatha@cs.umass.edu, at the Centre Intelligent Information Retrieval, University of Massachusetts for further information regarding Synapse. A WWW demonstration of the system can be located at: cowarie.cs.umass.edu/~demo/Demo.html.

4.27 Viper (Visual Information Processing for Enhanced Retrieval) The Viper system, developed at the Computer Vision Group, University of Geneva, supports a variety of colour and texture matching features, organised using an inverted file approach. The system supports relevance feedback, where images are marked by the user as either relevant (R) or non-relevant (NR). Contact Professor Thierry Pun, thierry.pun@cui.unige.ch, at the Computer Vision Group, University of Geneva for further information regarding Viper. A WWW demonstration of the system can be located at: cuiwww.unige.ch/~vision/demos.html

47

4.28 VisualSEEk VisualSEEK, developed at the Department of Electrical Engineering, Columbia University, supports colour and spatial location matching features. The system uses a novel approach for region extraction and representation based upon colour set back-projection where salient colour regions from images are automatically extracted. VisualSEEk supports several strategies for computing queries that specify the colours, sizes and arbitrary spatial layouts of colour regions, which include both absolute and relative spatial locations. Contact, Professor Shih-Fu Chang, sfchang@ee.columbia.edu, for further information regarding VisualSEEk. A WWW demonstration of the system can be located at: disney.ctr.columbia.edu/VisualSEEk

4.29 WebSEEk WebSEEK, developed at the Department of Electrical Engineering, Columbia University, is a content-based image and video catalog and search tool for the World Wide Web. The system supports colour, spatial layout and texture matching features. The system collects images and videos using several autonomous Web robots, which automatically analyse, index, and assign subject classes to the images. Currently 650,000 images and 10,000 videos have been catalogued. The system supports relevance feedback. Contact, Professor Shih-Fu Chang, sfchang@ee.columbia.edu, for further information regarding WebSEEk. A WWW demonstration of the system can be located at: disney.ctr.columbia.edu/WebSEEK

48

5 Discussion

5.1 System Features The systems outlined in Section 3 support the most common content-based image retrieval related matching features: colour, shape, and texture. All three of the major commercial vendors incorporate both global and local colour, shape and texture. It is unclear whether IBM still support shape retrieval in QBIC. IMatch extends the local colour feature matching technique and combines it with a shape matching feature. Readers are left to draw their own conclusion as to whether this actually improves the local colour matching feature. See Appendix C. A similar feature is now included within the Microsoft’s Clipart facility, which is part of the Office 2000 suite. Preliminary test with this feature indicate that considerable improvement is required. The retrieval output was very poor, as generally neither colour nor shape even vaguely resembled the query image. ImageFinder takes a more traditional approach to image retrieval, which is firmly rooted in the field of computer vision and pattern recognition. Attrasoft combine their approach with neural network technology, PolyNet. Excalibur adopts a similar approach with APRP™. Whether combing the retrieval techniques with neural networks is effective remains to be substantiated as no evaluation results have been publicly reported to date. Figure XI summarises the matching and retrieval features supported in each of the five commercially available systems.

System Global Colour

Local Colour

Shape Spatial location Texture Relevance Feedback

Rank Output

Excalibur SDK - ImageFinder - IMatch QBIC QDK - VIR SDK -

Figure XI: Matching & Retrieval Features.

It is not surprising that there are broad and overlapping similarities between the commercial and research prototype systems outlined in Section 4 in terms of the features that they support. Colour, shape and texture are generally well represented across the board of research prototype systems. Spatial relationship of objects is insufficiently represented both by the commercial vendors and the research community. What is not clear is whether there are significant differences in the efficacy between the similar matching features, and if so, what these differences are. There is a clear need to investigate this area in greater detail to determine the robustness of similar methods.

5.2 Feature Matching & Retrieval Effectiveness Three systems, ImageFinder, IMatch and QBIC, were subjected to a small-scale retrieval experiment. The trial was designed to test whether the matching features accomplished the tasks that they had been designed for i.e. did global distribution matching features return a set of results that were broadly similar to the query in terms of global colour distribution. Effectiveness of a system is inherently dependent on a users subjective interpretation of the results and other interrelated factors.

49

Several of the retrieval results are documented in Appendix C and readers are left to draw their own conclusions. The test indicated that the systems generally produced encouraging retrieval results based on the matching feature selected. All systems matched the query image with the exact stored image. In a number of instances the results were extremely questionable and unpredictable. Comparison of the retrieval results suggests that there are widely and varying degrees of retrieval precision and recall between the CBIR applications. Explicit knowledge of the data set used in the retrieval experiment indicates that the degree of retrieval precision and recall is extremely dubious. For example, there were several known items within the dataset that were expected to be retrieved and ranked higher than their rank order despite perceptual similarities between images. There was also significant difference in the ranking of the retrieved images. What is evident is that there is a heavy responsibility brought to bear on the user with regards to understating why the system is returning a specific set of results based on the parameters supplied to the system. The BlobWorld system, elib.cs.berkeley.edu/photos/blobworld, bridges this gap by allowing users to view the results of the segmentation of both the query image and returned results to highlight how the segmented features have influenced the retrieval results. Nevertheless, the majority of the systems provide no or little indication as to why certain images have been returned and it is highly likely that this burden will have a negative impact on a users perception of the systems retrieval effectiveness.

5.3 Usability Two systems underwent a heuristic evaluation, ImageFinder and IMatch [30]. The evaluation involved the examination of the user interface against a framework of ten heuristics to judge its compliance with recognized usability principles. The results indicated that overall IMatch reached a minimal acceptable level of usability without placing undue stress on the user. Novice users would have few problems installing, configuring and running the application. The user interface guides the user through a logical sequence of events in order to use the system reducing the cognitive load placed on the user. There are minor problems associated with the IMatch interfaces. The most pressing issue is the way the two user interfaces are implemented. The system has two related but separate interfaces, one for query formulation and another for displaying the retrieval results. The system would benefit greatly from a Multiple Document Interface (MDI) environment to improve the overall usability of the system. This issue has been resolved in the next release of IMatch. There are a small number of icons in the retrieval results user interface used to perform simple image management operations that do not comply with standard Windows icons for similar tasks e.g. copy and delete. While this is not a serious problem again, it would reduce the cognitive load on the user. In contrast, ImageFinder scored badly in all categories. It should be stressed that the version used in this study was only a demonstration system. It is unclear whether the user interfaces for the commercially available releases of the software differ significantly. Generally, greater thought needs to be applied to the overall design of the interface, particular to the layout of the interface and the objects used to implement features. For example, the sensitivity and blurring features should be implemented as ComboBox’s and not buttons. The usability of computer-based systems is a widely recognised issue and there is considerable scope for improvement in both systems.

50

6 Conclusions The dramatic rise in the volume of digital images has exacerbated the complex issues surrounding the effective retrieval of both dynamic and static images. This has proved a catalyst for resurgence in the field of visual information retrieval. Recent developments in the field have focused on the retrieval of images by image content, content-based image retrieval. This report reviewed the functionality, retrieval effectiveness and usability of several commercial CBIR systems:

♦ Excalibur Visual RetrievalWare SDK, Excalibur Corp. ♦ ImageFinder, Attrasoft ♦ IMatch, MWLabs ♦ QBIC, IBM Corp. ♦ Virage VIR Image Engine SDK and Image Read/Write Toolkit, Virage Inc.

Over the course of the investigation, a surprising number of CBIR systems were identified. The majority of the identified systems were research prototypes. There are distinct similarities between the commercial and prototype systems in terms of the features that they support. Colour, shape and texture are the three most common matching feature characteristics. What is not apparent is whether there are considerable differences between similar features, and if so, what those differences are. The effectiveness of any information retrieval system is dependent on several factors. The retrieval test verified that generally the matching features behaved predictably in terms of their functionality despite varying and questionable degrees of accuracy. Detailed knowledge of the data set used in the retrieval experiment suggests that the degree of retrieval precision and recall, and the reliability of the system are in general highly questionable. What was evident is that a heavy and unnecessary burden of responsibility is brought to bear on the user with regards to understanding the systems functionality. It is highly probable that this burden will have a severe negative effect on a users perception of the systems retrieval effectiveness. The two systems subjected to the heuristic evaluation suggest that there is considerable scope for improvement for both systems. Content-based image retrieval potentially provides new and exciting alternatives to the constraints and limitations imposed by the traditional information retrieval paradigm. The number of active research systems is encouraging and reflects the current and increasing interest in the field of content-based image retrieval. Nevertheless, there are still a significant number of open research issues to be addressed if this technique is to prove fruitful. The current impasse with regards to the efficacy of the retrieval techniques being developed and the need to develop suitable evaluation frameworks and benchmarks is now critical.

51

7 References

1. Rui, Y. and Huang, T. S. Chang, S-F., Image Retrieval: Current Techniques, Promising Directions, and Open Issues. Journal of Visual Communication and Image Representation, 10, (1), 1999, pp. 39-62.

2. Beard, D. V., Computer Human Interaction for Image Information Systems. Journal of the American Society for Information Science, 42, (8), 1991, pp. 600-608.

3. Pizer, S. M. and Beard. D. V., Medical Image Workstation: State of Science and Technology. The Journal of Digital Imaging, 2, 1989, p. 185-193.

4. Mostafa, J. and Dillon, A. Design and Evaluation of a User Interface Supporting Multiple Image Query Models. In: Proceedings of the 59th Annual Conference of the American Society for Information Science: Global Complexity, Information, Chaos and Control. Baltimore, Maryland, USA, American Society for Information Science. 1996.

5. Hidderly, R.. and Rafferty, P. Democratic Indexing: An Approach to the Retrieval of Film. In: Proceedings of Library and Information Studies: Research and Professional Practice. 1997. Queen Margaret College, Edinburgh, Taylor Graham. 1997.

6. Zollner, F., Botticelli Images of Love and Spring. Prestel-Verlag. 1998. 7. Enser, P. G. B., Query Analysis in a Visual Information Retrieval Context. Journal of

Document and Text Management, 1, (1), 1993, pp. 25-52. 8. Keister, L. H., User Types and Queries: Impact on Image Access Systems, Challenges in

Indexing Electronic Text and Images, R. Fidel, Editor. American Society for Information Science. 1994, p. 7-19.

9. Sunderland, J. Image Collections. Art Libraries, 7, (2), 1982. 10. Del Bimbo, A., Visual Information Retrieval. Morgan Kaufmann. 1999, pp. 1-270. 11. Chang, C. C., and Lee, S. Y., Retrieval of Similar Pictures on Pictorial Databases. Pattern

Recognition, 24, (7), 1991, pp. 675-680. 12. Cawkell, A. E., Picture Queries and Picture Databases. Journal of Information Science, 19,

1993, pp. 409-423. 13. Chang, S-K., and Hsu, A., Image Information Systems: Where do we go from Here? IEEE

Transactions on Knowledge and Data Engineering, 4, (5), 1992, pp. 431-442. 14. Cochrane, C., An Overview of Trends in the Collection and Use of Moving Images an the

United Kingdom. Journal of Documentation, 49, (3), 1993, pp. 278-291. 15. De Marsicoi, M., and Cinque, L. and Levialdi, S., Indexing Pictorial Documents by their

Content: A Survey of Current Techniques. Image and Vision Computing, 15, 1997, pp. 119-141.

16. Eakins, J.P. Pictorial Information Systems: Prospects & Problems. In: Proceedings of the 14th British Computing Society Information Retrieval Specialist Group Research Colloquium. Lancaster, England, UK, April 1992.

17. Fidel, R.. Challenges in Indexing Electronic Text and Images. ASIS Monograph Series. American Society for Information Science. 1994, pp. 3-306.

18. Holt, B. J., and Hartwick, L. Visual Image Retrieval for Applications in Art and Art History. In: Proceedings of Storage and Retrieval for Image and Video Databases II. 1994. San Jose, California, USA, SPIE, 1994.

19. Shatford, S., Analyzing the Subject of a Picture: A Theoretical Approach. Cataloging & Classification, 6, (3), 1986, pp. 39-62.

20. Svenonius, E., Access to Nonbook Materials: The Limits of Subject Indexing for Visual and Aural Languages. Journal of the American Society for Information Science, 45, (8), 1994, pp. 600-606.

21. Kato, T. Database Architecture for Content-Based Image Retrieval. In: Proceedings of Society of the Photo-Optical Instrumentation Engineers: Image Storage and Retrieval, 1662. 1992. San Jose, California, USA, SPIE, 1992.

52

22. Eakins, J. P., and Graham, M. E., Content-Based Image Retrieval: A Report to the JISC Technology Applications Programme. Institute for Image Data Research: Newcastle-Upon-Tyne. 1999, pp. 1-63.

23. Jain, R. Visual Information Retrieval in Digital Libraries. In: Clinic on Library Applications of Data Pricessing: Digital Image Access & Retrieval. Graduate School of Library and Information Science, University of Illionois at Urbana-Champaign, 1997.

24. Batley, S. Visual Information Retrieval: Browsing Strategies in Pictorial Databases. In: 12th International Online Information Meeting: ONLINE88 Information. London, England, Learned Information, 1988.

25. Bird, C. L., and Griffiths, E. E., User Interfaces for Content-Based Image Retrieval. In: IEE Colloquium on Intelligent Image Databases. London, Institute of Electrical Engineers. 1996.

26. Barros, J. E., French, J. C., Martin, W. N., and Kelly, P. M. System for Indexing Multispectral Satellite Images for Efficient Content-Based Retrieval. In: Proceedings of Storage and Retrieval for Image and Video Databases III. San Jose, California, USA, SPIE, 1995.

27. Eakins, J. P., Graham, Margaret E. and Boardman, J. M. Evaluation of a Trademark Image Retrieval System. In: Proceedings of the 19th British Computing Society Information Retrieval Specialist Group Research Colloquium. Robert Gordon University, Aberdeen, Scotland, UK, 1997.

28. Sommerville, I., Software Engineering. 5th Edition. International Computer Science Series, Addison-Wesley. 1997, p. 742.

29. Molich, R., and Nielsen, J., Improving a Human-Computer Dialogue. Communications of the ACM, 33, (3), 1990, p. 338-348.

30. Nielsen, J., Usability Engineering. AP Professional. 1993, pp. 1-362. 31. Niblack, W., Barber, R., Equitz, W. Flickner, M. Glasman, E. Petkovic, D. Yanker, P.

Faloutsos, C. Taubin, G. The QBIC Project: Querying Images By Content Using Color, Texture and Shape. In: Proceedings of Storage and Retrieval for Image and Video Databases. San Jose, California, USA, SPIE, 1993.

32. Niblack, W., Zhu, X. Hafner, J. L. Breuel, T. Ponceleon, D. B. Petkovic, D. Flickner, M. D. Upfal, E. Nin, S. I. Sull, S. Dom, B. E. Yeo, B-L. Srinivasan, S. Zivkovic, D. and Penner, M. Updates to the QBIC System. In: Proceedings of Storage and Retrieval for Image and Video Databases VI. San Jose, California, USA, SPIE, 1997.

33. Flickner, M., Sawhney, H. Niblack, W. Ashley, J. Huang, Q. Dom, B. Gorkani, M. Hafner, J. Lee, D. Petkovic, D. Steele, D. and Yanker, P., Query by Image and Video Content: The QBIC System. IEEE Computer, 1995, pp. 23-32.

34. Flickner, M., Sawhney, H. Niblack, W. Ashley, J. Huang, Q. Dom, B. Gorkani, M. Hafner, J. Lee, D. Petkovic, D. Steele, D. and Yanker, P., Query by Image and Video Content: The QBIC System, Intelligent Multimedia Information Retrieval, The MIT Press. 1997, pp. 7-22.

8 Appendix A: General System Information

System Name

Commercial Vendor

Academic Research

Active Application Web Demo

Code Programming Language

Operating System

Image Formats

Price CommerciallyAvailable

Acoi Yes AMORE Yes No Yes Yes Yes No C, Java, Perl Gif; JPEG;

PPM Yes

ARTHUR

ARTISAN No Yes Yes Yes No No C++ WIN 95/98/NT Tiff N/A No

ART MUSEUM

ASSERT

Attrasoft ImageFinder

Yes No Yes Yes Yes No Java WIN 95/98/NT BMP; Gif; JPEG

$999.00 Yes

Altavista AV Photo Finder

Yes No Yes No Yes No N/A No

Blobworld No Yes Yes No Yes Yes HTML, C, Matlab,

JavaScript

Unix JPEG No

BIS

C-BIRD No Yes Yes No Yes Java, Perl

CAETIIML No Yes Yes Yes No Yes C, Perl Unix JPEG

CANDID No Yes No No No No

CHABOT No Yes No Yes No Yes C, TCL/TK Unix Gif; JPEG N/A No

Charmer No Yes Yes

Circus No Yes Yes Yes

Compass No Yes Yes Yes Yes No C/C++, TCL/TK Unix; WIN 95/98/NT

Gif; JPEG N/A Yes

CORE

Cypress No Yes No Yes Yes Yes C, Perl Unix Gif; JPEG N/A No

Diogenes No Yes Yes Yes Yes No C, Perl Unix Gif; JPEG N/A No

DrawSearch No Yes Yes Yes Yes Yes C/C++, Java Unix; WIN 95/98/NT

BMP; JPEG; PPM

Free Yes

Excalibur CST Yes No Yes Yes No

Excalibur Visual RetrievalWare SDK

Yes No Yes No Yes Yes C/C++, Java, TCL/TK

Unix; WIN 95/98/NT

BMP; DDIF; Gif; JPEG; PNG; PPM;

Tiff

£50.000 Yes

FIBSSR

FourEyes No Yes Yes Yes Yes Yes C, Tcl Unix JPEG; PPM; SGI; Tiff

No

I2Cnet No Yes No

__________________________________________________________ JISC Technology Applications Programme (JTAP): Project # 647

54

System Name

Commercial Vendor

Academic Research

Active Application Web Demo

Code Programming Language

Operating System Image Formats

Price ComA

ICARS

Illustra Yes No No Yes No No C Unix; WIN NT Gif; JPEG; Tiff

On request Now

Image Retrieval & Database Navigation Demo

ImageFish

ImageMiner No Yes Yes No Yes No C/C++, Java Unix; WIN 95/96/NT Tiff

ImageRETRO No Yes Yes Yes Yes On Request Java Unix, WIN 95/98/NT

Gif Non

ImageRover No Yes Yes Yes Yes No C/C++, Perl Unix; Unix; WIN 95/96/NT

Gif; JPEG; Tiff

ImageScape No Yes Yes

ImageSearch No Yes

IMatch Yes No Yes Yes No No C++ WIN 95/98/2000/NT BMP; EMF; JPEG; PCT; PNG; TGA; TIFF; WMF

Informix Internet Foundation 2000 & Image Datablades

Yes No Yes No Yes No C/C++, HTML Java, Perl, SQL, TCL/TK

Unix, WIN 95/98/NT

BMP; FIT; Gif; JFIF

(JPEG); PDA; PNG; TIFF

On request

IQUEST No Yes Yes Yes No No Java, Javascript,

PHP3, PostgreSQL

Unix Gif; Tiff N/A

IRIS No Yes No No

LCPD No Yes Yes Yes Yes No C, Perl Unix Gif; JPEG; Tiff

Mac-Hermes

MARS No Yes Yes No No Yes C/C++, Java Unix, WIN 95/98/NT

BMP; Gif; JPEG; PPM

MIDSS No Yes Yes

MiRRor

Monet

NeTra No Yes Yes Yes

OGDEN

__________________________________________________________ JISC Technology Applications Programme (JTAP): Project # 647

55

System Name

Commercial Vendor

Academic Research

Active Application Web Demo

Code Programming Language

Operating System

Image Formats

Price CommerAvail

Photo Database Management System

Photobook No Yes Yes No Yes Yes C, Tcl Unix JPEG; PPM; SGI; Tiff

Free to Academia

N

Picasso

PicDB Yes No Yes Yes No No

PicSOM No Yes Yes Yes Yes No C++, Perl Unix BMP; Gif; JPEG; PNG; PPM; Tiff

N/A N

Piction

PictureFinder No Yes Yes Yes No On Request C/C++ WIN 95/98/NT BMP; Gif; Tiff

QBIC Yes No Yes Yes Yes Yes C/C++ MAC; Unix; WIN 95/98/NT

BMP; Gif; JPEG; PPM;

Tiff

Ye

SaFe No Yes Yes

SAFARI No Yes No No No C/C++ N

SCORE No Yes No Yes No No C Unix Gif; JPEG N/A N

Search by Content

No Yes Yes

SEDL No Yes No No

Shoebox

SQUID

Surf Image No Yes Yes No Yes No C/C++, TCL/TK Unix; WIN NT Gif; JPEG; PPM; Tiff

N/A N

SWIC No Yes No Yes Yes No C Unix BMP; Gif; JPEG; Tiff

N/A N

Synapse No Yes Yes Yes

TAKE

TRADEMARK No Yes

Viper No Yes Yes Yes

VIR Image Engine

Yes No Yes No - Yes C Unix; WIN 95/98/NT

$30.000 Ye

VisualSEEk No Yes Yes Yes Yes

WebSEEk No Yes Yes Yes

Zomax PicToSeek

No Yes Yes Yes Yes No Java

9 Appendix B: Feature Extraction Methods

System Name Colour Shape Spatial Texture Rank Output

RelevanceFeedback

Acoi AMORE Yes Yes No No

ARTHUR

ARTISAN No Yes No No Yes No

ART MUSEUM

ASSERT

Attrasoft ImageFinder Yes Yes No Yes No No

Altavista AV Photo Finder Yes Yes Yes Yes Yes

Blobworld Yes Yes Yes Yes Yes Yes

BIS

C-BIRD Yes Yes Yes

CAETIIML Yes No Yes Yes No No

CANDID Yes Yes No Yes Yes

CHABOT Yes

Charmer

Circus Yes Yes Yes

Compass Yes No Yes Yes Yes Yes

CORE

Cypress Yes

Diogenes No Yes No No Yes No

DrawSearch Yes Yes No Yes

Excalibur CST Yes Yes Yes

Excalibur Visual RetrievalWare SDK Yes Yes No Yes

FIBSSR

FourEyes

I2Cnet

ICARS

Illustra

Image Retrieval & Database Navigation Demo

ImageFish

ImageMiner

57

ImageRETRO Yes Yes

ImageRover Yes No Yes Yes Yes Yes

ImageScape

ImageSearch

Imatch Yes Yes No Yes Yes Yes

Informix Internet Foundation 2000 & Image Datablades

Yes Yes No Yes Yes Yes

IQUEST Yes Yes Yes Yes No Yes

IRIS

LCPD No Yes No Yes Yes

Mac-Hermes

MARS Yes Yes Yes Yes Yes

MIDSS

Mirror

Monet

NeTra Yes Yes Yes Yes

OGDEN

Photo Database Management System

System Name Colour Shape Spatial Texture Rank Output

RelevanceFeedback

Photobook Yes Yes Yes Yes

Picasso

PicDB Yes Yes Yes Yes

PicSOM Yes Yes No Yes Yes Yes

Piction

PictureFinder

QBIC Yes Yes Yes Yes Yes No

SAFE No Yes Yes No Yes Yes

SAFARI No Yes No No Yes No

SCORE No No No No Yes Yes

Search by Content

SEDL Yes Yes

Shoebox

SQUID No Yes No No

Surf Image Yes Yes Yes Yes Yes Yes

SWIC No Yes No No Yes No

58

Synapse Yes

TAKE

TRADEMARK

Viper Yes Yes Yes

VIR Image Engine Yes Yes No Yes Yes

VisualSEEk Yes Yes Yes Yes Yes

WebSEEk Yes Yes Yes Yes Yes

Zomax PicToSeek Yes

59

10 Appendix C: Retrieval Results

10.1 Global Colour Retrieval

QBIC IMatch ImageFinder

60

10.2 Local Colour Retrieval

QBIC IMatch ImageFinder

N/A

N/A

N/A

N/A

N/A

N/A

N/A

61

10.3 Global Colour Retrieval

QBIC IMatch ImageFinder

62

10.4 Local Colour Retrieval

QBIC IMatch ImageFinder

N/A

N/A

N/A

N/A

N/A

N/A

N/A

63

10.5 Global Colour Retrieval

64

10.6 Local Colour Retrieval

QBIC IMatch ImageFinder

N/A

N/A

N/A

N/A

N/A

N/A

N/A

65

10.7 Texture Retrieval

QBIC IMatch ImageFinder

N/A

N/A

N/A

N/A

N/A

N/A

N/A

66

10.8 Texture Retrieval

QBIC IMAtch ImageFinder

N/A

N/A

N/A

N/A

N/A

N/A

N/A

67

10.9 Texture Retrieval

QBIC IMatch ImageFinder