1

Image Browsing for Feature-based Products

Christopher C. Yang*,†, Sai Ho Kwok**, and Milo Yip* *: Department of Systems Engineering and Engineering Management

The Chinese University of Hong Kong **: Department of Information and Systems Management The Hong Kong University of Science and Technology

ABSTRACT

In the context of product search in information intermediary or infomediary, text- and nevigation-based searching mechanisms such as keyword search are usually adopted [13]. Google [5], WebSeer [4], and Alta Vista Photo Finder [1] are some prominent examples. However, such search mechanisms are not efficient for feature-based products and the major problem is that the feature-based products are difficult to be described with textual expression. A potential candidate for the search of feature-based products is query-by-example (QBE). However, our study reveals that QBE is not an ideal searching method for feature-based products. This paper proposes an image browsing technique for the search of feature-based products in infomediary. The image browsing technique allows the users to access feature-based products through a two-dimensional map constructed with self organizing map (SOM) technique. The technique overcomes the problem of describing feature-based products. Simple view and pick operations can drive the user to the desired group of products. A task-based user evaluation was conducted to examine the usability of the proposed technique and the experimental results show that the proposed browsing technique is more practical and efficient compared with QBE.

1. INTRODUCTION

Information intermediary or infomediary is likely to claim a major segment of the revenue of e-commerce transactions. Infomediaries are e-commerce companies leveraging the Internet to unite buyers and suppliers in a single efficient virtual marketplace to facilitate the consummation of a transaction. Most of the search facilities in the infomediaries are text- and navigation-based, for example by keyword and category searches. The search facility can handle most of the products, but it is inefficient to handle feature-based products, such as textile and shoes with different colors and textures. For example, it is not easy to describe the pattern and color of a T-shirt using textural expression. Image retrieval has been dominated by query-by-example (QBE) searching techniques. However, QBE is not an ideal searching method for a very large feature-based product database in infomediaries due to the following reasons.

1) It is not user-friendly to support product images with combinations of complicated features. For example, to describe the shape of a T-shirt logo (or sport shoe pattern), a buyer may be required to use a drawing board to depict the shape feature of the product.

2) Based on the principle of QBE, a visually similar product example should be identified from the image database before performing the search for the wanted product. The QBE becomes inefficient when the actual product is, to a certain extend dissimilar to other products in the database and the database is very large. It is because it could be difficult to identify a similar product image.

3) There may not be any relevant product image given a query example. Image browsing technique could be an alternative and better solution to this application. Database browsing requirements differ from those of querying: database items should be organized and presented to the user in

† yang@se.cuhk.edu.hk; phone (+852) 2609 8239; fax (+852) 2603 5505; http://www.se.cuhk.edu.hk/~yang/; Department of Systems Engineering and Engineering Management, The Chinese University of Hong Kong, Shatin, Hong Kong SAR, China

2

such a way that exploration of content is possible. Applying image browsing in infomediaries for feature-based product search, buyers can explore all available feature-based products in the database. Further searches can be simply by clicking relevant images presented in the screen of product exploration. The feature-based product search can be improved by using information fed by the user, also known as “relevant feedback”. The relevant feedback modifies the distance of the metric space using information derived from the interaction of the use with the system. Thus, image browsing can provide an overview of product images in the database to the user so the user can begin the search by clicking relevant images with similar product features in the overview. Even though the features of a product are greatly dissimilar to that of other products in the database, the product must be summarized in the overview. In this paper, we propose an image browsing technique for infomediaries. The image browsing technique is based on Self Organizing Map (SOM). It automatically organizes the color product images in a two-dimensional map. Similar product images are grouped into a region with the same label on the map. Users may explore the relevant product images by browsing the neighborhood of an interested region. Using such browsing technique, users no longer need to specify the query by an example, which may not be available at the beginning of the searching process. The SOM provides an outline of the available product images on the large image database and support browsing and navigation to explore the product images. We implemented an image-based retrieval system with the proposed image browsing technique for textile industrial products. Both system performance and user evaluation were examined and presented in this paper.

2. CONTENT-BASED IMAGE RETRIEVAL According to Smith [12], content-based image retrieval (CBIR) is defined as given that an image database D of N images and a feature dissimilarity function d(I,J) between image I and image J, the objective is to retrieve the Ncutoff images (J ∈ D) with the lowest d(I,J) to the query image I. Query by example (QBE) is the most traditional approach. The success of QBE greatly depends on how the initial query image can represent the information needs of the users. In a large-scale image database, it is problematic to identify a set of initial images that contains relevant images to represent the user information needs. It is referred as the page zero problem by La Cascia et al. [2]. Some other more advanced CBIR system allows user to submit queries by specifying the image features, such as sketches or selected segmented image regions. QBIC [3] developed by IBM Almaden Research Center, Photo book [14] developed by MIT Media Lab’s, and NETRA [9] developed by UCSB, are some prominent examples. However, such queries are too complicated for non-professional to formulate. In this paper, we propose to adopt SOM to develop a two-dimensional map as an image-browsing tool. Features of the images for the feature-based products, such as textile products, are first extracted and analyzed. After training by SOM, similar products are grouped together in the neighborhood regions and the most representative image will label the region. In the next few sub-sections, we shall briefly describe the techniques of feature extraction and analysis, and SOM for image classification. 2.1 Image Feature Extraction Various sorts of features have been studied extensively as visual feature is the foundation of all kinds of applications of CBIR. Most work in this area concentrated on finding the best features and indexing methods to represent the similarities among images. The simplest form of visual features is directly based on pixel values of the image [6]. However, this type of visual feature is very sensitive to noise and changes in brightness, hue, and saturation. Besides, it is not invariant to spatial transformations such as translation and rotations. As a result, CBIR systems that base on pixel values do not generally having satisfactory results. So many researches have put effort on computing useful characteristics from images using image processing and computer vision techniques. The general-purpose features in CBIR include chromatic, texture, shape and structure. Other features are specific to the application domains. They require some special knowledge and constrains on the database. For example, facial CBIR systems require techniques widely studied in image processing for face recognition. In this paper, we are concentrated on general-purpose features. The representation of content of an image I is usually compiled into a d-dimensional feature vector fI:

3

+

+−

= jWxyxyxg

yxyx

πσσσπσ

221exp

21),( 2

2

2

2

( )TId

IIII ffff K321=f (1) The dimensionality d of the feature vector directly affects the performance of image query. A typical value of d in the context CBIR is of the order 100 [11]. In the simplest form of query processing without indexing, it requires O((Nd)2) computations to compare each element of all vector pairs, where N is number of the images. In this paper, chromatic and texture features are adopted and the extraction of image features are depicted in Figure 1.

RGB

L

HS

Textural Analysis

Chromatic Analysis

Image Feature

Figure 1. The information flow of the process for image representation. 2.1.1 Chromatic Feature As illustrated in Figure 1, color is first transformed from RGB values to LHS values. For computer applications, color is usually described in terms of the RGB color coordinate system. However, the RGB color coordinate system does not model the human perception of color. Applying image processing techniques in the RGB system will often produce color distortion and artifacts. In measuring color similarity, a color coordinate system that describes hue and saturation, such as LHS, is more appropriate [15,16]. Humans recognize color by hue and saturation, and therefore, H and S values are utilized for chromatic analysis. A two-dimensional histogram with one axis for hue and another for saturation is used to represent the chromatic features. The formulation of HS histogram is:

(2) where H and S are the hue and saturation channels, and N is the number of pixels in the image. The hue component varies from 0 to 360 degrees and the saturation component varies from 0 to 1. In order to build the histogram, the hue and saturation values are quantizied into several levels. We have chosen 10 levels for each component, so that a 10 by 10 two-dimensional histogram h[10,10] for each image is built in our system. 2.1.2 Textural Feature Gabor filter [10] is adopted to represent the textural feature. The computation of Gabor filter as textural feature is given as follows: A two-dimensional Gabor function can be written as:

(3)

A self-similar filter dictionary can be obtained as a mother Gabor Wavelet G(x, y) by appropriate dilations and rotations of Eq. (3) as: (4)

where

),Pr(],[, sShHNshHistogram sh ==⋅=

),( yxmS

mn GaG θθ−=

integers aren m, 1,a

)/cos()()/sin()()/sin()()/cos()(

2 / 1) - (w wside2, / 1)-(h hsideimage, of width wimage, ofheight h

>

−+−−=−+−=

====

KnwsideyKnhsidexKnwsideyKnhsidex

y

x

ππθππθ

4

[ ])1)(1()1)(1(01010000 ... −−−−= KSKSf σµσµσµ

Given an image with luminance ( )yxI , , a Gabor decomposition can be obtained by multiplying the luminance by the magnitude of the Gabor Wavelet:

(5)

The mean and standard deviation of the magnitude of the transform coefficient are used to represent the texture feature for classification and retrieval purpose:

(6)

(7)

The Gabor feature vector is constructed by using mnµ and mnσ as feature components:

(8)

where S is number of scales and K is number of orientation. In the following experiments, we use S=3 and K=4. 2.2 Image browser by SOM SOM [7,8] is a powerful tool for categorization and classification that involves clustering or grouping items of similar nature. Continuous-valued vectors that represent the chromatic and textural features are presented sequentially to the map in time without specifying the desired output. After presenting sufficient input vectors, network connection weights will specify cluster or vector centers that sample the input space such that the point density function of the vector centers tends to approximate the probability density function of the input vectors. Moreover, the connection weights will be organized such that the topologically close nodes are sensitive to inputs that are physically similar. The outline of SOM is presented as below: Initialize input nodes, output nodes, and connection weights Use the chromatic feature vector and textural feature vector as the input vector with N (= N1 + N2) elements and create a two-dimensional map (grid) of M output nodes (for example, a 10 by 10 map has 100 nodes). Initialize weight vector of each output node, wj(0), to small random values. Present each color image in random order Represent each image by a vector of its chromatic and textural features and present it to the system in random order. Compute distances between each input vector and each output node's weight vector Distance, Dij, between input vector i and output node's weight vector j is

where xik(t) is the kth element of input vector i at time t and wik(t) is the kth element of output node's weight vector j at time t. Updating weights of the winning output node and its neighbors to reduce their distance The winning output node has the minimum Dij. The winning output node and its neighboring nodes will then be updated as follows: where η(t) is the learning factor and h(t) is the neighborhood function. Label regions Train the network through repeated presentation of all color images in the database until it converges. Label the output node, j, by the most similar image, i. Map the images to the labeled regions For each image in the database, map it to the node that has the minimum distance with the corresponding labeled image.

( ) ( ) 22,, rGiGyxIyxW mnmnmn +=

( )wh

dxdyyxWmnmn ⋅

= ∫∫ ,µ

( ) ( )( ) dxdyyxyxW mnmnmn2

,,∫∫ −= µσ

( )∑ −

=−= 1

02)()(N

k jkikij twtxD

( ))()()()()()1( twtxthttwtw jkikxjkjk i−+=+ η

5

An example of an image browsing map constructed by SOM is shown on Figure 2.

Figure 2. Example of an image browsing map constructed by SOM.

3. EXPERIMENT A task-based user evaluation was conducted to compare the performance of the image browsing map and Query-By-Example (QBE) systems. 34 undergraduate students participated in the evaluation session. A set of 496 textile images was used as the corpus. Each subject was asked to accomplish two tasks. In the first task, subjects were asked to describe the kind of textile pattern they are interested. The image browsing map or the QBE system was then provided to them to search for the images that match the described patterns. There is no limitation on time and the number of images to be found. In the second task, an image of textile pattern was given to the subject. The subject was then asked to use the image browsing map or QBE system to search the given image. The time taken to complete the task was recorded. A maximum of one minute is restricted for the second task. Figures 3 and 4 show the user interface of the experiment system for the first task. In Figure 3, user-defined textile pattern is listed on the left column, the image browsing map is presented next to the description. The subjects may click on a region of the browsing map to open the images mapped to the region on a new window. The subjects may click as many images as they like to select the images that meet the description. In Figure 4, the user-defined textile pattern is listed on the left column similar to Figure 3. The subjects may select an image listed in the window to submit as a query for QBE and revise their query as many times as possible. Figures 5 and 6 are similar to Figures 3 and 4 correspondingly except that Figures 5 and 6 are the interfaces for the second task. The target image to be search is presented on the left column and similar interfaces for image browsing map and QBE are presented on the right hand side.

Figure 3. Image browsing map of the experiment system for Task 1.

6

Figure 4. QBE user interface of the experiment system for Task 1.

Figure 5. Image browsing map of the experiment system for Task 2.

Figure 6. QBE user interface of the experiment system for Task 2. For the first task, the experimental results show that subjects were able to retrieve more images that match the description by using image browsing map. However, it took more time for the subjects to use the image browsing map to finish the task. By observations, the subjects were able to find matching images by exploring the categorized images presented on the browsing map. Although more time is taken to complete the task by

7

image browsing map, subjects indeed are able to explore more relevant images. On the other hand, subjects found it difficult to identify a representative image to submit as query for QBE. Exploration is limited by QBE. Users may lose interest to explore if no additional relevant images are found in a few queries. For the second task, the experimental results show that more subjects were able to accomplish the task within the given 1 minute time. By observations, if a representative image could be identified as the query of QBE, subjects were able to retrieve the target image as fast as using the image browsing map. However, if a representative image could not be identified, the subjects were not able to retrieve the target image within the one-minute limitation. Therefore, the failure rate is much higher when QBE is used.

4. CONCLUSION This paper addresses the problem of search for feature-based products in infomediary. This problem has not been openly addressed and discussed in the literature. This paper presents the motivations and related issues about the problem. Our findings reveal that the commonly used content-based image retrieval technique, query-by-example (QBE) is not efficient and desirable for the search of feature-based products. We propose an image browsing technique to resolve the problem. The image browsing technique presents the product images in a two-dimensional map based on their image features. The map is constructed using self organizing map. The users may perform a search task by simply viewing and picking images from the map. A task-based user evaluation was conducted and the experimental results evidence that the image browsing approach is much more efficient and practical, in particular when the representative image is not found in the database. The proposed image browsing technique can also be useful in other applications with content-based searching facility. One of our on-going researches is about the search feature for photographic pictures, in which various image features are studied and applied in the SOM representation of the image map. The primary results confirm that the browsing approach is also well received by the subjects.

ACKNOWLEDGEMENT This project is supported by the Earmarked Grant for Research of the Hong Kong Research Grants Council, CUHK 7034-98E.

REFERENCES 1. AltaVista Photo Finder, http://image.altavista.com/ 2. M. La Cascia, S. Sethi, and S. Sclaroff., “Combining textual and visual cues for content-based image

retrieval on the world wide web”, Proc. IEEE Workshop on Content-Based Access of Image and Video Libraries, pages 24--28, Santa Barbara, California, 1998. IEEE Computer Society.

3. M. Flickner, H. Sawhney, and W. Niblack et al., “Query by image and video content: The QBIC system,” IEEE Computer, September 1, 1995, 23--32.

4. C. Frankel, M. Swain, and V. Athitsos, Webseer: An Image Search Engine for the World Wide Web, Technical Report TR-96-14, CS Department, Univ. of Chicago, 1996.

5. Google, http://www.google.com/ 6. A. Gupta, R. Jain, “Visual information retrieval,” Comm. Assoc. Comp. Mach., vol. 40, no. 5, pp. 70-79,

May 1997. 7. T. Kohonen, “Self-Organization and Associative Memory”, Springer Series in Information Science, vol. 8,

1984. 8. T. Kohonen, “Self-Organizing Maps”, Springer Series in Information Science, vol. 30, 1995. 9. W. Y. Ma and B. S. Manjunath., “NETRA: A toolbox for navigating large image databases”, IEEE

International Conference on Image Processing, 1997. 10. B. S. Manjunath and W. Y. Ma, “Texture Features for Browsing and Retrieval of Image Data”, IEEE

Transactions on Pattern Analysis and Machine Intelligence, Vol. 8, No. 18, pp. 837-842, August 1996. 11. Y. Rui, T.S. Huang and S-F. Chang, “Image Retrieval: Current Techniques, Promising Directions, and

Open Issues,” Journal of Visual Communication and Image Representation, v 10, n 1, p 39-62, March 1999.

12. J. R. Smith, “Integrated Spatial and Feature Image Systems: Retrieval, Analysis and Compression”, PhD thesis, Graduate School of Arts and Science, Columbia University, 1997.

8

13. M. Swain, “Searching for Multimedia on the World Wide Web”, Technical Report CRL 99/1, Cambridge Research Laboratory, http://crl.research.compaq.com/publications/techreports/abstracts/99_1.html, 1999.

14. A. Pentland, R. Picard, and S. Sclaroff, “Photobook: Content-based Manipulation of Image Databases”, SPIE Storage and Retrieval for Image and Video Databases II, number 2185, San Jose, CA., February 1994.

15. C. C. Yang, M. K. Yip, “Quantization of Color Histograms using GLA,” Proceedings of the SPIE International Conference on Electronic Imaging and Multimedia Technology, Photonics Asia, Shanghai, China, October 14-18, 2002.

16. C. C. Yang, and M. C. Chan, “Color Image Retrieval Based on Textural and Chromatic Features,” Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, Tokyo, Japan, October 12-15, 1999.