image hub explorer: evaluating representations and

Multimedia Tools and Applications manuscript No.(will be inserted by the editor)

Image Hub Explorer:

Evaluating Representations and Metrics for

Content-based Image Retrieval

and Object Recognition

Nenad Tomasev · Dunja Mladenic

Received: date / Accepted: date

Abstract We present a novel tool for image data visualization and analysis, Im-age Hub Explorer. It is aimed at developers and researchers alike and it allows theusers to examine various aspects of content-based image retrieval and object recog-nition under different built-in metrics and models. Image Hub Explorer providesthe tools for understanding the distribution of influence in the data, primarily byexamining the emerging hub images. Hubness is an aspect of the well-known curse

of dimensionality that hampers the effectiveness of many information systems.Its consequences were thoroughly examined in the context of music/audio searchand recommendation, but not in case of image retrieval and object recognition.Image Hub Explorer was made with the goal of raising awareness of the hub-ness phenomenon and offering potential solutions by implementing state-of-the-arthubness-aware metric learning, ranking and classification methods. Various visu-alization components allow for a quick identification of critical issues and we hopethat they will prove helpful in working with large image datasets. We demonstratethe effectiveness of the implemented methods in various object recognition tasks.

Keywords: Image retrieval, visualization, hubness, object recognition,k-nearest neighbors, machine learning

1Published in Multimedia Tools and Applications, c©Springer Verlag.DOI: 10.1007/s11042-014-2254-1The original article is available at: http://link.springer.com/article/10.1007\%2Fs11042-014-2254-1

Nenad Tomasev · Dunja MladenicArtificial Intelligence LaboratoryJozef Stefan InstituteJamova 39, Ljubljana, SloveniaTel.: + 386 1 477 3528Fax.:+ 386 1 477 3851E-mail: [email protected], [email protected]

1 *

2 Nenad Tomasev, Dunja Mladenic

1 Introduction

Large quantities of digital image data are generated daily. Its volume has beenincreasing over the years, as more people were starting to use the mobile cameradevices, as well as upload and share their images online across a wide range ofdatabases and services. A large amount of image data is also captured by remotesensors in various monitoring systems.

Visualization plays an essential role in examining large image databases. Ithelps with detecting and eliminating errors in the data, as well as discoveringpatterns that help in improving system performance. This paper proposes a newimage data visualization tool aimed at feature representation evaluation from theperspective of content-based image retrieval and recommendation, Image Hub Ex-plorer (Figure 1). Over the years, many data visualization tools have been devel-oped and we will review other existing approaches in more detail in Section 2.1.

Fig. 1 The Image Hub Explorer experimentation process. Different pairs of metrics and repre-sentations are compared on a given dataset by considering their influence on the kNN topologyof the data. Various analytic functions are used to determine the potential impact of the changein kNN structure on image retrieval (IR) and object recognition (OR) system performance.

Image Hub Explorer has been designed to help with analyzing close-to-scale-free distributions of image relevance in k-nearest neighbor graphs of large processedimage datasets.

Power law degree distributions arise frequently in many real-world influenceand interaction networks, like social networks [4] or protein-protein interactionnetworks [39]. However, it was surprising to see that the k-nearest neighbor (kNN)graphs of many types of intrinsically high-dimensional data tend to exhibit similar

Image Hub Explorer was presented as a demo paper at the demo track of the ECML/PKDD2013 conference in Prague. A short paper that was published in the conference proceedings isavailable at: http://ailab.ijs.si/nenad tomasev/files/2013/08/ecmlImageHubExplorer2013.pdf

Title Suppressed Due to Excessive Length 3

scale-free properties [43–45]. This phenomenon is known as hubness [43] and isconsidered to be an aspect of the well-known dimensionality curse [6].

Hubness is an important property of high-dimensional data, as many recom-mendation and/or retrieval systems are built around the basic idea of returning a”top-k” relevance set for individual queries. Even if the ranking is not explicitlybased on a standard metric (like Euclidean/Minkowski or cosine), the task can stillbe interpreted as returning the k-nearest neighbors for the query item. Therefore,phenomena related to the kNN graph are likely to be of interest when analyzingsystem performance.

In information retrieval systems, hubs are the examples that the system regardsas most relevant for the queries, on average. A skewed distribution of relevancecan be quite detrimental for retrieval quality, as rarely retrieved items are under-utilized in the system and some items can even be entirely ignored and neverreturned as query results.

Hubness was first described in context of music recommendation [3]. Somesongs were being very frequently recommended by the systems, even when therewas no discernable semantic correlation to the queries. Such query results wereacting as noise and were clearly detrimental for the overall recommendation qual-ity. The existence of these hub songs was initially conjectured to be a consequenceof using inappropriate feature representations and similarity measures. It was laterdetermined to be a direct consequence of high intrinsic data dimensionality [44]and has since been observed in text mining [34,53], time series and sensor data [46,9], as well as images [54].

It is possible to choose the representation and metric in such a way that reducesthe overall impact of emerging hubs [19,51], but it is not possible to avoid the issuealtogether. The initial analysis of hubness in content-based object recognition [54]revealed that different local image feature types exhibit different susceptibilitiesto hubness and, depending on the domain and context, some might be preferredto others as they induce a more semantically correct k-nearest neighbor topology.Dimensionality reduction techniques can only eliminate hubness if they also inducesignificant information loss by mapping the data onto a space of a significantlylower dimensionality than the intrinsic dimensionality of the data [42].

Strictly speaking, semantic inconsistencies in kNN topologies in image dataare not a direct consequence of hubness, as they are believed to be caused primar-ily by the semantic gap between low-level feature representations and the actualperceived semantics of the data, that the features fail to capture in its entirety.However, lowering the data hubness by hubness-aware metric learning has beenshown to be highly beneficial in music recommendation and retrieval [51], despitethe fact that a similar semantic gap can be said to exist in audio data as well [12].This implies that there might be potential benefits to taking data hubness intoaccount when designing content-based image retrieval and recommendation sys-tems.

Despite the proven significance of hubness for high-dimensional data analy-sis [42], no tools were available that would enable researchers to interactivelyexamine hubness in their data and employ hubness-aware learning approaches inorder to mitigate its negative effects. This has lead us to design Image Hub Ex-plorer as a tool for examining hubness in image data, while evaluating differentimage feature representations. The system architecture and use cases will be dis-cussed in more detail in Section 4. An overview of the related work in the field


of image data visualization is given in Section 2, while Section 3 introduces thereader to the phenomenon of hubness in more detail and lays foundations for thediscussion in the rest of the paper.

1.1 Contributions

In this paper, we present an image collection visualization and experimentationsoftware named Image Hub Explorer. Its main objective lies in enabling users(present or future system developers) to quickly detect various types of emerginghubs among the images and pinpoint the principal gradients of misclassificationand the major semantic singularities among the queries.

All visualization panels integrate the underlying hubness information in vari-ous ways. Image Hub Explorer is based on the Hub Miner library (http://ailab.ijs.si/nenad_tomasev/hub-miner-library/) and implements several state-of-the-art hubness-aware metric learning techniques [57,51], hubness-aware classifi-cation methods [43,59,58,56], standard kNN baselines [18,29,64,52] and a recentlyproposed query result re-ranking procedure [53].

Image Hub Explorer also offers a novel feature assessment tool that exam-ines the usefulness of individual visual words and outlines the “good” and “bad”textural regions in the images.

Image Hub Explorer is the first interactive graphical tool for examining hubnessin image data and data in general. Most of its functionality does not assume aspecific underlying representation, so it is possible to use the system not only forimages, but also for analyzing other media types.

Our experiments demonstrate that learning from image hubness can be usedfor improving system performance and we hope that Image Hub Explorer willprove useful in future system design.

2 Related Work

2.1 Image Data Visualization

Similarity-based image data visualization is frequently used in practice [36,37,16].The ImagePlot tool is a typical example (http://flowingdata.com/2011/09/18/explore-large-image-collections-with-imageplot/). Most approaches focuson exploring different ways of performing similarity-preserving projections of thedata onto the plane, as well as the selection strategies that determine which imagesare to be shown. Images can sometimes be projected onto a pre-defined grid [67].Similarity search and visualization can both be based either on low-level imagefeatures or on the inferred higher-level semantic concepts [68]. Domain knowledgecan also be incorporated in the system [11]. Hierarchical systems that allow for amulti-faceted view of the data under varying levels of granularity are also avail-able [55]. More complex search and browsing systems focus on specific users andtry to tailor the sets of displayed images based on learned user preferences [41,71].Large-scale browsing tools sometimes incorporate different types of projections inorder to better represent the data [50]. Finally, it is possible to visualize different


information sources at the same time, by projecting the aligned textual data alongwith the images [24].

Apart from visualization, many systems have been developed for improvingimage search. Content-based systems rely on exploiting different types of imagefeatures [23,48,65] and similarity measures [30,31] to improve the overall searchperformance. Content-based systems are widely used, as assigning reliable meta-data to large image datasets remains non-trivial. Relevance feedback is sometimesincorporated in the content-based image retrieval systems in order to improve theirperformance [61,75,47]. As the emphasis has recently shifted towards scalabilityin order to handle very large datasets, research is also being done on integratingvarious hashing and indexing approaches for fast and scalable image search [62,76], as well as different indexing techniques.

The existing systems allow for quick browsing through large image collections,but they offer no support for examining image hubs and the skewed distribution ofinfluence. This is not surprising, since hubness is a newly discovered phenomenonand has only recently become a subject of independent study.

2.2 Image Feature Representation

Since Image Hub Explorer enables researchers to evaluate various image represen-tations from the perspective of content-based image retrieval and object recog-nition, here we briefly review some standard and modern approaches to imagefeature extraction and representation learning.

In many applications, the image feature extraction process is based on detect-ing a set of distinctive keypoints in the images and computing the descriptorsof their normalized neighborhoods. Different keypoint localization and descriptorextraction strategies yield different local image feature types. Some common ex-amples include SIFT [33], SURF [5], BRIEF [10] and ORB [49] features. Whilethese local features can be used to match corresponding regions across images,they are also frequently used to determine a global feature representation of theentire image and this is typically achieved by some form of vector quantizationor clustering. Images can then be represented as bags of visual words, somewhatsimilar to how textual documents are often handled, but with a much smallervocabulary and lower sparsity.

Representing images as bags of visual words disregards a lot of structural in-formation present in the images and poses a problem of overcoming the semanticgap between the low-level descriptors and the semantics present in the scene, typesof objects and how they relate to each other. This has lead researchers to exploreother ways of extracting image representations and calculating image similarity.Sparse coding attempts to learn a basis set capturing high-level semantics in thedata and learn the sparse coordinates in terms of the basis set [74]. Hierarchicalsparse coding makes it possible to achieve good recognition performance on fea-tures automatically extracted from image pixels, without using any hand-craftedfeature descriptors [69]. Instead of having an unstructured image representation,it is possible to establish a hierarchy of concepts with associated attributes andthis has been shown to be beneficial and lead to improvements in object recog-nition performance [72]. Deep representations in form of convolution activation


features have also recently been shown to outperform many other state-of-the-artapproaches in computer vision [15].

Due to a large number of existing feature types and implementations, ImageHub Explorer was designed to be mostly feature-independent and can thereforebe used to evaluate image data hubness in all of the above mentioned featurerepresentations. The details will be discussed in Section 4.

3 The Emerging Hubs and the Skewed Distribution of Relevance

3.1 Basic Concepts

Let us introduce some formal notation prior to proceeding with the discussion. LetD = (X,Y ) = (x1, y1), (x2, y2), . . . (xN , yN ) be the data representation, where X

denotes the feature vectors and Y ∈ {1 . . . C} the class labels. A k-neighborhoodDk(xi) is defined around each point xi ∈ D as a set of its k-nearest neighbors.The degree of neighbor points in the kNN graph is given as their total occurrencecount Nk(xi), as per Equation 1.

Nk(xi) = |xj : xi ∈ Dk(xj)| (1)

Data points that have high occurrence counts exhibit the highest influenceon the kNN learning or retrieval/recommendation process. This distribution ofinfluence assumes a long-tailed shape in intrinsically high-dimensional data [43],as suggested by Figure 2.

Def.Hubness of a particular datasetD is defined as the third standard moment(skewness) of the neighbor occurrence degree distribution and is denoted by SNk

and calculated according to Equation 2.

SNk =1n

∑ni=1(Nk(xi)− k)3

( 1n

∑ni=1(Nk(xi)− k)2)3/2

(2)

In principle, high hubness has been shown to hamper various practical machinelearning and information retrieval approaches and a low skewness value is prefer-able. Values of SNk that exceed 1 are considered to be high, by consensus [42].

The term “hubness” will also sometimes be used for individual data points,denoting their neighbor occurrence frequency Nk(xi).

As a consequence of hubness, most data points end up being orphans or anti-hubs, as they are either not retrieved at all or retrieved very rarely by the system.The k-neighbor sets are instead dominated by a small number of very frequentneighbor points, hubs. The skewed distribution of relevance in the model entailsan information loss. Ideally, we would like the system to be able to retrieve allthe items from the database that are semantically relevant for the given queries.Unfortunately, some items are never retrieved, so their utility remains limited andnot fully exploited. This can be viewed as another consequence of the semanticgap, as the perceived semantic relevance does not correspond well to the relevanceimplied by the computational models.

As discussed before, not all neighbor occurrences in top-k result sets are desir-able or beneficial. Sometimes there are semantic inconsistencies as certain neighborpoints are observed in result sets of queries that are not semantically related to


Fig. 2 The change in the neighbor occurrence distribution shape with increasing dimension-ality, in case of Gaussian mixture data. The increasing skewness results in most data pointsbecoming orphans and a small number of hubs in the long tail of the distribution dominatesthe analysis.

those particular items. As no feature representation or metric is perfect, this isquite common in practice. These inconsistencies can easily be measured whenworking with labeled data. The total neighbor occurrence frequency can be de-composed into “good” and “bad” partial occurrence frequencies based on labelmatches/mismatches between neighbor points. This is shown in Equation 3, whereGNk represents good hubness and BNk bad hubness.

Nk(xi) = GNk(xi) +BNk(xi),

GNk(xi) = |xj : xi ∈ Dk(xj) ∩ yi = yj |,

BNk(xi) = |xj : xi ∈ Dk(xj) ∩ yi 6= yj |.

(3)

Therefore, good occurrences are those that are well aligned with class affilia-tion and bad occurrences are those where points from different classes occur asneighbors. Hubness-aware methods often go a step further by building completeclass-conditional neighbor occurrence models, based on the occurrence count de-composition shown in Equation 4.

Nk(xi) =

C∑

c=1

Nk,c(xi),

Nk,c(xi) = |xj : xi ∈ Dk(xj) ∩ yj = c|.

(4)

3.2 Susceptibility to Hub-centered Noise

In some real-world networks, the presence of hubs can increase robustness to ran-dom noise [73], but this also makes scale-free networks much more vulnerable tohub-centered inaccuracies. Small changes in the initial conditions can sometimessubstantially harm system performance. We will illustrate this problem by consid-ering the image processing example outlined in Figure 3 and described in detailin [54].


Fig. 3 The emergence of top 5 major hubs on the iNet3Err dataset [54]. Under the particularchoice of feature representation (SIFT [33] bag of visual words) and metric (Manhattan), noisyfeature vectors that resulted as errors in the feature extraction pipeline ended up becomingthe major hubs in the data, with increasing dimensionality of the codebook. Their influencewas highly detrimental, as most of their occurrences induced label mismatches.

Figure 3 shows the emergence of 5 major hubs on the iNet3Err quantizedSIFT [33] representation along with the nature of their influence. The imageswere taken as a 3-class subset from the public ImageNet repository [14] (http://www.image-net.org/). In order to determine the optimal bag of visual words vo-cabulary dimensionality (codebook size), a series of experiments had been run [54].Quite unexpectedly, system performance deteriorated so much that the basic 5-NNclassifier performed worse than zero-rule (assigning all examples to the majorityclass by default) for the 1000-dimensional case. Subsequent analysis has deter-mined the cause of this pathological behavior to lie in the emergence of severalextremely bad pervasive hub images.

In this particular case, the image hubs were erroneously represented by zero-vectors as a result of an I/O error in the feature extraction pipeline. The Man-hattan distance from a zero vector to any given quantized image representationremains constant, regardless of the codebook size. At the same time, the distancesbetween pairs of images increase on average with increasing dimensionality, causingthe zero vectors to become major hubs in the data.

The particular error was easily corrected by re-running the feature extractioncomponent for the images in question and updating the extraction code. It canalso be argued that carefully designed image processing systems ought to performrun-time consistency checks to ensure valid data representation. However, thisexample shows the potential danger that lies hidden in the hubness of the data.Only 5 detrimental hub images had rendered the kNN-based object recognitioncomponent effectively useless on a 2731 image dataset.

In general, there is no way of knowing the localization of hub images in thefeature space and there is no way to ensure that mislabeled images or noisy rep-resentations would not end up being hubs in the resulting kNN topology. Moreimportantly, high bad hubness is not exclusively a consequence of some sort oferrors contained in the data. Many real-world datasets have been shown to exhibithigh bad hubness [43][42] under standard feature representations and metrics.


3.3 Origins of Hubness and Advances in Hubness-aware Learning

It was demonstrated that intrinsically high-dimensional data with finite and well-defined means has a certain tendency for exhibiting hubness [43–45] and thatchanging the similarity measure can only reduce, but not entirely eliminate theproblem. Boundary-less high-dimensional data does not necessarily exhibit hub-ness [32], though this case does not arise often in practical applications. It canbe said that hubness is expected to arise with increasing intrinsic dimensionality,under certain reasonable assumptions. Yet, as demonstrated in [32], it is possibleto generate certain types of synthetic data that would not exhibit hubness evenin high dimensions. When present in the data, hubness emerges partly due todistance concentration [21], as it becomes more difficult to distinguish betweenrelevant and irrelevant results for any given query.

For reasons outlined above, the focus is slowly shifting towards a hubness-awarealgorithm design, where the algorithms are robust to underlying data hubness. Theidea is to learn the neighbor occurrence distribution on the training data and infermodels that help with interpreting the semantics of neighbor occurrences on unseenexamples. Hubness-aware algorithms have been proposed for clustering [60], datareduction [9], classification [59][56][58], metric learning [51][57], document retrievaland ranking [53].

4 Image Hub Explorer

Despite the recent advances in hubness-aware algorithm design, there existed nopublicly available tools that would allow the developers to test and evaluate thesehypotheses on their own data and in their own systems.

We have developed Image Hub Explorer in order to enable other researchersto perform an in-depth analysis of the distribution of influence in their own data,based on the currently available state-of-the-art hubness-aware methods.

A demo video of Image Hub Explorer usage is available at: http://youtu.be/LB9ZWuvm0qw and a detailed usage guide with documentation is going to beavailable along with the other resources at: http://ailab.ijs.si/tools/image-hub-explorer/.

Image Hub Explorer embeds 4 types of functions in its graphical user interface:

– Visualize large image collections and explore global data properties.– Detect and examine the centers of influence in each class.– Interpret the observed similarities by feature assessment.– Solve the detected issues by selecting the best feature representation and met-

ric and by utilizing the available hubness-aware approaches.

Image Hub Explorer allows the users to experiment with different pairs ofmetrics and feature representations and to assess the consequences of the structuralchange in the kNN topology and the distribution of influence in the data. Thesechanges might either increase or decrease the performance of both image retrieval(IR) and object recognition (OR) systems.


4.1 System Architecture

Image Hub Explorer tool is based on the Hub Miner library (http://ailab.ijs.si/nenad_tomasev/hub-miner-library/), a recently developed java library op-timized for working with k-nearest neighbor methods and learning under the as-sumption of hubness in intrinsically high-dimensional data . The Hub Miner libraryconsists of more than 100.000 lines of code and over 500 java classes. It supportsboth dense and sparse data representations and also includes a pipeline for gen-erating a bag of visual words representations for image data based on local imagedescriptors, like SIFT features [33]. Multi-threading is well supported, so manyanalytic tasks can run in parallel.

There are also a few external dependencies. Multi-dimensional scaling is per-formed by the MDSJ library developed at the University of Konstanz [40]. Graphdrawing is performed by the JUNG library (http://jung.sourceforge.net/).Charts that are used to illustrate certain data properties are displayed via JFreeChart(http://www.jfree.org/jfreechart/).

Image Hub Explorer GUI offers several different views of the data, tailored fordifferent steps in the analytic process. We will examine each data view separatelyin Section 4.3. These views all hold the references to the same underlying set ofdata structures and are updated automatically if some re-calculations occur thatchange some of the shared objects.

The objects that are shared among the views include the currently selectedimage, browsing history, the primary and secondary distance matrices, featurerepresentations (if available), the list of kNN graphs over a range of differentneighborhood sizes, as well as k-dependent lists of hubness-related statistics andcharts. The actual images are loaded in batches from the disk when needed, inorder to reduce the overall memory consumption.

The organization of the content in individual views is performed by severalcustomized JPanel classes that are used for an interactive display of image content.There is a clear separation between the visual components and the underlyingalgorithmic implementations, as the Image Hub Explorer GUI does not in itselfcontain any explicit data mining code. All modeling is performed by invoking theappropriate classes and methods in the underlying Hub Miner library.

4.2 Test Data

We have used the Image Hub Explorer system to visualize and analyze several pub-licly available image datasets. We will demonstrate its functionality in Section 4.3on the examples taken from the Leeds Butterfly dataset [63] (http://www.comp.leeds.ac.uk/scs6jwks/dataset/leedsbutterfly/). We have also analyzed theimage data of the 17 flowers dataset [38], Caltech101 [17], Essex face database(http://cswww.essex.ac.uk/mv/allfaces/faces96.html) and several subsets ofthe ImageNet repository [14].

In our experiments, we have used the bag-of-visual-words representations basedon several standard local image feature types. While the classification experiments

The Hub Miner library is currently being fine-tuned and documented and we intend torelease it as open-source in mid-2014.


in Section 5 have been performed on quantized SIFT feature representations [33],use cases of Image Hub Explorer in Section 4 have also been demonstrated onSURF [5], BRIEF [10] and ORB [49] quantized representations. The analysis wouldrun similarly for other local feature types as well. Feature extraction was performedvia OpenCV (http://opencv.org/) and the quantized representation was gener-ated by using the processing pipeline in the Hub Miner library. A 400-dimensionalcodebook was obtained by K-means++ clustering [2] on a random sub-sample offeatures taken from all the images. Different codebooks were generated for differentimage datasets.

4.3 Visualization and Interactive Analysis

Image Hub Explorer has four main screens: Data Overview, Class View, NeighborView and Search (Figure 4). The Feature Assessment panel can be invoked forindividual images through the menus above. We will examine each system functionindividually, grouped by the views they are available from.

4.3.1 Representations and Metrics

The purpose of Image Hub Explorer is to allow for experimentation with dif-ferent feature representations and metrics. In order to make this possible, mostof its design is representation-independent. An explicit feature representation isnot required for most of the supported analytic tasks. The exceptions are search(Section 4.3.6) and feature assessment (Section 4.3.5). All other functions requiremerely the distance matrix along with the list of class assignments and possiblythumbnails and images for visualization.

This was done in order to avoid the potential difficulties with having to ex-plicitly factor in every possible feature type, as that would have raised certainissues. For instance, a user would not be able to experiment with a feature typethat is not explicitly supported. However, since most of Image Hub Explorer isrepresentation-independent, such problems can not arise.

Even though the analysis is based on the k-nearest neighbor sets and the dis-tance matrix, the users are not required to calculate those objects themselves.If a feature representation is provided in one of the supported input formats(ARFF [26], CSV, TSV), many primary and secondary metrics are available andboth the distance matrix and the kNN lists can be quickly calculated by the systemby invoking the respective multi-threaded methods.

The following frequently used primary metrics are built in: Manhattan, Eu-clidean, cosine, Tanimoto, symmetrized Kullback-Leibler divergence, Bray-Curtisand Canberra [42].

The distance concentration phenomenon is a well-known aspect of the dimen-sionality curse and many standard metrics concentrate and can not be reliablyused for making distinctions between relevant and irrelevant points for queries onintrinsically high-dimensional data [1,21].

The use of secondary metrics can help in mitigating the arising difficulties.The idea is to take the original primary distance matrix on input and learn abetter distance model. Image Hub Explorer offers 5 different secondary metrics:


simcoss [27], simhubs [57], mutual proximity (MP) [51], NICDM [28] and localscaling [70].

Mutual proximity [51] and simhubs [57] are two recently proposed hubness-aware metric learning approaches. MP is based on estimating the pairwise prob-abilities of points becoming each other’s nearest neighbors. On the other hand,simhubs introduces weights based on the neighbor occurrence self-information andthe reverse neighbor set homogeneity into the standard shared neighbor distanceframework (simcoss).

Any primary distance measure can be used to learn a secondary distance model,so it is possible to learn 35 different secondary models for any given feature rep-resentation.

4.3.2 Data Overview Screen

The Data Overview screen offers a high-level overview of the data and its mainproperties under the current feature representation and metric.

Fig. 4 The Data Overview screen of Image Hub Explorer: Visualizing the major image hubsvia multi-dimensional scaling.

The Projection Panel shows a 2D visualization of the image data and allowsthe users to browse through the central data points. Images are projected onto theviewing panel by a multi-dimensional scaling (MDS) [8] procedure. An examplecan be seen in Figure 4.

Two things differentiate our approach from the data overview approaches inother similar visualization tools: the selection of representatives and the back-

ground landscape.Image Hub Explorer calculates the total occurrence frequency of each image

and selects a certain number of hubs for display in the Projection Panel. Therefore,


only the most influential images are shown, those that have the potentially highestimpact on system performance.

The background landscape is calculated based on the average good and bad

hubness of different regions in the projected feature space. Naturally, the greencolor corresponds to good hubness and the red one to bad hubness. The landscapeis generated in two steps. The first step is a sort of a Gaussian blur, implementedefficiently, as in [20]. The panel is split into buckets by a grid and each image isassigned to its bucket according to the (x, y) coordinates obtained by applyingthe MDS. Each pixel is assigned its good hubness weight wG,k and bad hubnessweight wB,k. Within each bucket B, the weight of each pixel is determined by therule given in Equation 5.

wG,k(x, y) =∑

Ij∈B

GNk(Ij) · e−σ((x−xj)

2+(y−yj)2)

wB,k(x, y) =∑

Ij∈B

BNk(Ij) · e−σ((x−xj)

2+(y−yj)2)

(5)

If either of the two weights are non-zero for a given pixel (i.e. the bucketcontains some points), the green component of the RGB representation of thecolor in the pixel is given by g(x, y) and the red component as its complementr(x, y), as per Equation 6.

g(x, y) = 255 ·wG,k(x, y)

wG,k(x, y) + wB,k(x, y)

r(x, y) = 255− g(x, y)

(6)

After this initial stage, a two pass box blur is performed in order to furthersoften the landscape. Box blur sets the color of each pixel in the image to be theaverage color of its neighboring pixels. It is a low pass convolution filter.

One such landscape is generated for each neighborhood size k, as it depends ongood and bad hubness that are k-dependent quantities. One of the main featuresof the application is the slider-selector for neighborhood size, which allows theuser to quickly change among different k-values and observe the differences in allquantities and all tabular views of the application.

All images are shown within the frames that are colored according to theirclass. This makes distinguishing between different classes easier for small displayedthumbnails in various screens. All images in all the views can be selected by mouseclicks and in those cases a full image is shown in the appropriate place.

The quantities that are shown on the Data Overview tab (Figure 4) are asfollows: data size, the number of classes, neighbor occurrence frequency distri-bution skewness (hubness), neighbor occurrence frequency distribution kurtosis,entropy of the direct and reverse neighbor sets, skewness of the entropy distribu-tion, percentage of points that occur at least once as neighbors, percentages ofhubs, orphans and regular points, degree of the major hub in the data and thepercentage of label mismatches in k-neighbor sets (bad hubness). The neighboroccurrence frequency distribution is also given in a separate plot below for easierinterpretation.


The users can easily export the lists of hubness-related statistics for the entirerange of neighborhood sizes and analyze them more thoroughly. Figure 5 showsthe probability that an image is retrieved for a range of possible result set sizes. Itcan be seen that approximately 15% of quantized SIFT images are not retrievedeven once in top-10 result sets. In fact, less than 50% of quantized SIFT images areretrieved more than 5 times for k = 10. If the distribution of neighbor occurrenceswere Gaussian, we would expect about 50% of images to be retrieved at least k

times, but this is not the case and this is a consequence of hubness. This is evenmore pronounced in the corresponding BRIEF quantized representation, whereabout 20% of the images are not retrieved at least once in top-50 result sets, whichis a significant information loss. Best retrieval ratio on this particular dataset isachieved when using the ORB quantized feature representation.

Fig. 5 The probability that an image is retrieved at least once in a top-k result set, acrossseveral quantized feature representations.

The overall skewness of the neighbor occurrence distribution decreases withincreasing neighborhood sizes. However, here it remains non-negligible (higherthan 1) even for k = 50, as can be seen in Figure 6. There is an apparent differ-ence in the induced hubness among different feature representations and on thisdata the quantized BRIEF feature representation achieves the highest hubnessand the quantized ORB feature representation achieves the lowest hubness. Thiscorresponds well to the observed retrieval probabilities that have previously beenexamined in Figure 5.

The proportion of label mismatches in k-nearest neighbor sets is rather high onthis particular dataset, for all examined feature representations, under standardmetrics. Figure 7 shows the bad hubness of the data, as it slowly increases with in-creasing k-values. The quantized BRIEF feature representation again achieves theleast desirable performance on this data, while the ORB representation achievesthe lowest bad hubness scores over the entire range of examined neighborhoodsizes.

These examples illustrate how this panel of Image Hub Explorer can be usedto quickly compare the general utility of different image feature representations,before delving into a deeper analysis. While it is true that these general overviews


Fig. 6 Skewness of the neighbor k-occurrence distribution, across several quantized featurerepresentations. All of the compared representations exhibit high hubness, as their SNk ≥ 1.

Fig. 7 The probability that an image induces a label mismatch in a top-k result set, acrossseveral quantized feature representations.

can be automatically generated without user intervention (and thus, without aneed for a GUI component), it is still useful to include such functions in the tool,in order to support various types of analysis simultaneously.

4.3.3 Class View

The Class View (Figure 8) enables the users to inspect different classes separately,as well as compare them, based on their point type distributions. Image HubExplorer generates lists of major hubs, good hubs and bad hubs for each imageclass. Beneficial and detrimental centers of influence can therefore be very easilydetected, selected and examined.

Within each class, we can distinguish between different types of points [35]:safe points are those that are easily properly classified by kNN methods, as theyare contained in class interiors. Borderline points exist in the borderline regionsbetween different classes and are much more difficult to handle. Rare points and


Fig. 8 The Class View of Image Hub Explorer: Examining point type distributions and centersof influence for each class separately.

outliers have most of their neighbors belonging to different classes and are ex-tremely difficult for labeling.

An example can be seen in Figure 9, where the point type distributions ofthree different image classes are compared, based on the results shown in the ClassView. In this particular case, images of Heliconius erato seem to be much moredifficult to handle than those of Danaus plexippus, in all examined feature types.However, there are notable differences in class difficulty between different featuretypes. While most instances of Danaus plexippus images are interior class pointsin the quantized SIFT representation, most instances of Danaus plexippus imagesare rare points in the examined quantized BRIEF feature representation. On theother hand, Heliconius charitonius images are best handled in the quantized SURFrepresentation, while the images of Heliconius erato seem to be best handled inthe quantized ORB feature representation.

Some pairs of classes are more difficult to distinguish than others and this canbe analyzed by considering the class-to-class k-neighbor occurrence matrix, whichis shown on the right side of the Class View. It is displayed in form of a table andthe color of individual cells is determined by the amplitudes of pairwise influences.Clear green colors correspond to high values on the diagonal, where intra-classneighbor occurrence frequencies are aggregated. On the other hand, clear red cellscorrespond to the principal gradients of misclassification. Once the critical pairsof classes are determined, users can inspect the major bad hubs of those classesand their occurrence profiles in order to get a better interpretation of the semanticbreaches.


(a) SIFT (b) SURF

(c) BRIEF (d) ORB

Fig. 9 A comparison of point type distributions between three different classes, across severalquantized feature representations.

4.3.4 Neighbor View

The Neighbor View allows the user to pinpoint the critical subsets of points andexamine the nature of their influence. A screenshot is given in Figure 10.

Any selected image can be inserted into the local visualized subgraph of thekNN graph of the data, for the given neighborhood size. Moving the k-selectionslider in the Data Overview screen automatically updates the graphs, so it ispossible to examine the k-dependent changes in the local topology.

Apart from adding images one by one, the interface also supports an optionof adding all neighbors of any selected image, as well as all of its reverse nearestneighbors. Directed edges represent neighbor relations. The weights on the edgescorrespond to the distance between the selected points in the selected metric.

The neighbor k-occurrence profile for the selected image is shown as a coloredpie chart in the upper right corner of the view. The current lists of k-nearestneighbors and reverse k-nearest neighbors are shown below.

The Neighbor View helps in visualizing the influence of hub points, as shownin Figure 11, where one bad hub image is shown, along with a set of its reversek-nearest neighbors. In this case the Artogeia rapae image that is shown in themiddle acts as a neighbor only to points that are not from its own class (species),which is obviously detrimental to kNN-based analysis. The occurrence frequencyof all images varies across different feature representations and here we can see


Fig. 10 The Neighbor View of Image Hub Explorer: Exploring the nearest neighbor (NN)and reverse nearest neighbor (RNN) lists and visualizing local kNN subgraphs.

that an image that is a medium-sized hub in one feature representation can bea regular point in another feature representation and an orphan in yet anotherfeature representation.

While some of the previously outlined Image Hub Explorer functions couldhave been made accessible without a graphical interface, the Neighbor View offersan easy way to browse through the k-nearest neighbor graph in order to examinethe structure of the induced relevance more closely and detect possible issues onthe image level. Coupled with the information from the hub lists in the ClassView, this can lead to a semi-supervised detection of mislabeled and noisy imagerepresentations, as well as key problems with the feature design, after the visualword distributions in the images are more closely analyzed, which is the topic ofSection 4.3.5.

4.3.5 Feature Visualization and Assessment Panel

Not all features are equally informative. Image Hub Explorer enables the usersto estimate the usefulness of different visual words in the context of content-based object recognition from images, based on local image features. The currentimplementation of the feature assessment panel makes it possible to assess featureutility in the commonly used quantized bag-of-visual-words representations. Othertypes of feature representations would need to be assessed separately.

Let Q = {q1,q2, . . .qdc} be a dc-dimensional vocabulary of visual words ob-

tained by vector quantization in the descriptor feature space. Each image Ij can berepresented as a set of local image features FIj = {fj

i : i = 1 . . . NIj}. Each local

descriptor fji is mapped onto its closest concept vector from the codebook vocab-

ulary Q. A bag of visual words representation for the image is then computed as a


(a) SIFT (b) SURF

Fig. 11 An example of a bad hub in the quantized SIFT feature representation, a detrimentalcenter of influence. Neither of the reverse neighbors of the selected image belongs to the sameclass as the image itself, so its occurrences induce label mismatches and are semanticallyinconsistent. The same image has an equally inconsistent occurrence profile in the quantizedSURF feature representation, but it is not a hub there, as it does not occur very often. On theother hand, the displayed image never occurs as a neighbor in the quantized BRIEF featurerepresentation, for the same neighborhood size of k = 5.

histogram of visual words’ occurrence counts, so that xip = |fj

i : qp = NNQ(fji)|

is the pth component of the quantized image feature vector.Define η(qp) =

∑Ni=1 xi

p as the total occurrence count of the pth visual wordover the entire set of images. This can be decomposed into the class-conditionaloccurrence counts, as follows: η(qp) =

∑Cc=1 ηc(qp), where ηc(qp) =

∑yi=c xi

p.Visual words that occur with almost equal frequency across a wide set of differentcategories are not very informative for object recognition, as they do not carrymuch discriminative information. The most informative visual words are thosethat occur almost exclusively within the images from a single category. This wasthe motivation for defining the following codebook goodness scores:

g(qp) =maxc∈C ηc(qp)

η(qp)(7)

Within the Image Hub Explorer, users can inspect individual visual words andtheir class-conditional occurrence profiles, that are displayed in form of pie charts.

More importantly, Image Hub Explorer offers a possibility to visualize thedistribution of informativeness on each image individually. Figure 12 shows onesuch example. A grid is superimposed on the image region, dividing it into aset of rectangular boxes. The average goodness of all the local descriptors foundwithin a box is used to determine its average color on the display. The landscapeof informativeness is then generated by the same approach that was discussed inSection 4.3.2 for generating the MDS projection landscape, including the multiple


passes of convolution filters. The green color is used to denote regions with highdiscriminative information content and the red one for the regions that do notcontribute to object recognition.

(a) A regularly displayed selected image. (b) An overall visualization of the criticalfeature regions.

(c) A visualization of a single visualword, one that is most beneficial for ob-ject recognition of this image type.

Fig. 12 Individual visual words are displayed on top of the selected image and colored accord-ing to their overall usefulness and semantic consistency. This helps in identifying the criticalregions in the images, those that contribute to making good class distinctions and those thatrepresent textural patterns that might occur in many different image classes.

Figure 12 shows how the feature assessment and visualization componentsworks for SIFT features in case of recognizing Danaus plexippus butterfly spec-imens. The textural regions around the black veins on the butterfly’s wings arejudged to be the most informative by the system. This is indeed a highly distinctivefeature of the particular species. Similarly, for Heliconius charitonius the systemdetermines that the white stripes on otherwise black butterfly’s wings carry highlydiscriminative visual information.

Different datasets and quantized representations have a different distributionof goodness/badness among visual words. The entropy distribution of codebookoccurrences also varies. Figure 13 shows the codebook entropy distribution forthe Leeds Butterfly dataset, as well as iNet3 (see Section 4.2 for more detail).Most visual words in the studied data representations are not very discriminativeas features, as they have a high average occurrence entropy. In such cases, properdata preprocessing and feature selection/weighting are an important step in systemdesign.


(a) Leeds Butterfly Dataset. (b) iNet3.

Fig. 13 Distribution of visual word occurrence entropies, based on labels of images wherethey appear as features.

4.3.6 Search and Ranking

Image Hub Explorer system allows the users to query the image database by newimages. It extracts the features for the selected image, generates the bag-of-visual-words representation based on the loaded codebook and retrieves the top-k resultset from the database, a set of most similar images based on the currently selectedmetric. The Search View is shown in Figure 14.

Fig. 14 The Search screen of Image Hub Explorer. Apart from supporting the basic queryfunctionality, the system offers label suggestions based on the output of several kNN classifi-cation models, as well as a hubness-aware secondary re-ranking procedure.

Several k-nearest neighbor models are trained on the data and employed in or-der to determine the label of the query image: kNN [18], FNN [29], NWKNN [52],


AKNN [64], hw-kNN [43], h-FNN [59], HIKNN [56] and NHBNN [58]. HIKNN,h-FNN, NHBNN and hw-kNN are the hubness-aware kNN classifiers designedspecifically for handling intrinsically high-dimensional data. The k-nearest neigh-bor methods are not the only approach to object recognition, but some recentresults suggest that good results can be achieved by employing the k-nearest neigh-bor methodologies [7].

The inclusion of various classification models allows the users not only to de-termine the labels of the query images, as these are sometimes known in advance,but also to compare how different classification approaches handle certain types ofpoints. This makes it possible to select the most appropriate approach for futuredeployment in the IR/OR system.

It is important to rank the retrieved images in such a way that best reflectsthe underlying semantics, i.e. so that the most relevant images are shown first.However, due to the hubness phenomenon, many results emerge that act as noiseand reduce the system performance. A hubness-aware self-adaptive secondary re-ranking has recently been used in report retrieval for semi-automatic bug duplicatedetection [53]. We have adapted the approach to the general case of re-rankingarbitrary result sets by removing the temporal component from the original model.The idea is simple: take the original ranking that was based on some similaritymeasure and use the prior hubness information to re-calculate the similarities.The new similarity scores are then re-sorted and a new ranking is reached. This isshown in Equation 8.

simH(x, xi) =GNk(xi)

Nk(xi)· sim(x, xi) (8)

Re-calculating the similarities based on Equation 8 results in increasing thedistance between the query image and the bad image hubs, those images whoselabels often differ from the labels of their reverse k-neighbors on the training data.

4.4 System Applicability

4.4.1 Data Domains

Many features of Image Hub Explorer do not assume any specific underlying fea-ture type and can work with arbitrary data points, not just images. If the imagesare not available, the system prints out filled rectangles with object names shownin the middle instead. Furthermore, most functions operate even if the feature rep-resentation itself is not provided. What is required in that case is only the distancematrix and the class affiliation information.

If the visualization is performed on image data, as is the primary purpose ofthe system, the distances can still be calculated from other aligned feature typesinstead. For example, the distances can be obtained from image captions andembedding paragraphs, if these are provided. This sort of meta-data is frequentlyavailable in systems that work with online image search and retrieval, as they fetchthe images from web pages that also have some correlated textual content.

The list of supported algorithms is bound to change, as it will be further extended infuture versions.


4.4.2 Supported Image Feature Representations

While most functions of Image Hub Explorer operate on generic feature representa-tions and can therefore handle generic image feature representations as well, somefunctions have been designed with a specific class of image feature representationsin mind, namely the quantized bag-of-visual-words representations that have beenderived from bags of local image features. SIFT [33], SURF [5], BRIEF [10] andORB [49] quantized feature representations have been examined in this paper,though it is possible to use other local image feature types as well.

Image Hub Explorer components that do not support generic feature repre-sentations and currently assume that the images are represented via quantizedlocal feature representations are the feature visualization and assessment panel(Section 4.3.5) and the query component (Section 4.3.6). The feature visualiza-tion and assessment component requires a visual word vocabulary to be loadedand it examines the utility of individual visual words. The query component, onthe other hand, allows the users to query the database with new images and ittherefore needs to know which features to extract and how to prepare the imagerepresentation. The query component could, in principle, be extended to extractand use any image feature type. However, including many feature type extractorsfrom many libraries would significantly increase the number of dependencies of thetool, which would make it more difficult to build and deploy. The default featureextraction support is therefore limited, but can be extended by the system users,since Image Hub Explorer is an open-source tool and the code is distributed alongwith the binaries.

4.4.3 Scalability

Calculating all the kNN sets is the most computationally intensive task in theImage Hub Explorer tool. The Hub Miner library currently implements the ap-proximate divide and conquer method based on recursive Lanczos bisection [13].The time complexity of the procedure is Θ(dn1+τ ), where τ ∈ (0, 1] reflects thequality of the approximation. Earlier experiments have shown that the resultingkNN graphs can be effectively used for neighbor occurrence modeling in hubness-aware methods [56].

Another possibility is to use locality sensitive hashing (LSH) [25] in order toachieve the desired speed-up. This would allow the analysis to be carried out onvery large image datasets.

Scalability is especially important as a typical use case where the capabilities ofImage Hub Explorer are fully utilized would be to generate and compare a series offeature representations generated with different parameters and possibly differentfeature types. This is only possible if the individual iterations in the evaluationprocess are effective enough to take no more than some reasonable amount of timewith regards to the time frame of the project. Therefore, the feature extractionand representation learning process itself can become a potential obstacle in verylarge datasets. In these cases, random sub-sampling should be considered in orderto expedite the process, unless this leads to under-representing some classes in thedata in those cases when the number of classes is exceedingly large.

It should be noted, though, that even in such very large scale problems thatprevent experimentation with a large number of feature extraction and represen-


tation learning methods it is still possible to use Image Hub Explorer to comparethe distribution of induced relevance under different implemented primary andsecondary metrics and to experiment with the implemented hubness-aware metriclearning approaches in order to see whether they are capable of generating kNNtopologies of higher semantic consistency.

Metric learning approaches implemented in Image Hub Explorer can also betime consuming if derived from the exact kNN sets, but simcoss [27] andsimhubs [57] can actually be computed from the approximate kNN sets as well,with similar quality. Mutual proximity [51] can also be calculated approximatelyfrom a subset of the distances and such approximate implementations are availablein the underlying Hub Miner library.

Finally, certain large scale image datasets might pose a memory issue in thedefault implementation that loads all the image thumbnails during workspace ini-tialization. However, this can easily be modified so that the thumbnails are loadedonly when they need to be displayed, which would slightly reduce the responsive-ness of some graphical components, but would enable the users to handle evenlarger datasets.

5 Hubness-aware Classification in Object Recognition

In this Section we demonstrate how hubness-aware approaches can be used to im-prove the effectiveness of k-nearest neighbor classification in object recognition, byevaluating a series of hubness-aware implementations that are available in ImageHub Explorer. We have compared the classification accuracy of the following ap-proaches on a series of object recognition tasks: kNN [18], FNN [29], hw-kNN [43],h-FNN [59], HIKNN [56], NHBNN [58] and RRKNN. Re-ranked kNN (RRKNN)is a regular kNN classifier that uses the hubness-aware re-ranking [53] available inImage Hub Explorer (as described in Section 4.3.6) and re-ranks the top-k resultsprior to making a classification decision, which is then based on the top rankedsubset of the original k-nearest neighbor set. This embedded implementation rep-resents a way of assessing the utility of the proposed ranking, as it has not beenused before in the context of image ranking and/or object recognition. By default,the smaller neighborhood that the classification is based upon in RRKNN is takenas the half of the original kNN set.

The classification approaches were evaluated on a selection of image datasets:several subsets of the ImageNet repository [14], the Leeds Butterfly dataset [63],the 17 flowers dataset [38], Caltech101 [17], and the Essex face database (http://cswww.essex.ac.uk/mv/allfaces/faces96.html), as mentioned previously inSection 4.2. Each image was represented as a 400-dimensional SIFT bag-of-visualwords extended by a 16-dimensional global color histogram. A summary of themain data properties is given in Table 1

A skewness value (SNk) exceeding 1 is an indicator of high data hubness ac-cording to the established conventions [42] and many image datasets seem to ex-hibit significant hubness under this quantized feature representation. This resultsin some unexpectedly frequent neighbor points. For example, in the Caltech101dataset the most frequent 5-neighbor hub occurs in 29.1% of all 5-NN sets. Theonly examined image dataset that does not exhibit any visible hubness is the EssexFaces image database, as the overall skewness of k-neighbor occurrences is merely


Table 1 The summary of the image datasets. Each dataset is described both by a set of basicproperties (size, number of features, number of classes) and some hubness-related quantities fortwo different neighborhood sizes, namely: the skewness of the k-occurrence distribution (SNk

),the percentage of bad k-occurrences (BNk), the degree of the largest hub-point (maxNk).

Data set size d C SN5BN5 maxN5 SN10

BN10 maxN10

iNet3 2731 416 3 8.38 21.0% 213 6.19 21.9% 294iNet4 6054 416 4 7.69 40.3% 204 6.32 41.8% 311iNet5 6555 416 5 14.72 44.6% 469 11.87 46.5% 691iNet6 6010 416 6 8.42 43.4% 275 6.23 45.5% 384iNet7 10544 416 7 7.65 46.2% 268 6.72 48.4% 450

iNet3Imb 1681 416 3 3.48 17.2% 75 2.50 17.9% 106iNet4Imb 3927 416 4 7.39 38.2% 191 6.11 39.8% 288iNet5Imb 3619 416 5 9.35 41.4% 258 8.09 43.1% 401iNet6Imb 3442 416 6 4.96 41.3% 122 4.33 43.0% 200iNet7Imb 2671 416 7 6.44 42.8% 158 4.90 44.5% 231

LButterfly 832 416 10 2.75 62.4% 58 2.66 65.9% 10417Flowers 1360 416 17 2.19 60.5% 45 2.33 66.1% 85Caltech101 9144 416 102 60.81 78.6% 2664 51.44 79.6% 3012FacesEssex 2965 416 151 0.19 5.6% 16 0.41 16.7% 39

AVG 4395.35 416 23.57 10.32 41.68% 358.29 8.58 44.34% 471.14

0.19. This is due to the nature of the dataset, as all of the people that had theirpictures taken where photographed in front of very simple backgrounds, in orderto reduce the noise that might have otherwise resulted if the backgrounds wereallowed to vary.

The implemented hubness-aware approaches were compared with the baselinekNN. The comparisons were performed for neighborhood sizes k = 5 and k =10. The results are shown in Table 2. All experiments were performed as 10-times 10-fold cross-validation. The corrected re-sampled t-test was used to test forstatistical significance. All algorithms were run with the default parameter options,as proposed in the original papers.

The evaluation reveals that taking hubness into account helps with k-nearestneighbor classification in object recognition. RRKNN and HIKNN seem to be mostpromising among the examined classification approaches on this batch of objectrecognition tasks. Hubness-aware k-nearest neighbor classification approaches haveclearly outperformed the standard kNN baseline.

A comparison between the achieved macro-averaged F1-score [66] shows thateven the simple hubness-aware re-ranking scheme [53] helps with improving theF1

M and achieves good performance in class-imbalanced classification.

Not all images are equally difficult for object recognition and hubness-awarek-nearest neighbor classification improves the precision of classifying difficult ex-amples, those that lie far from class interiors. This can be seen in Figure 16.

Hubness-aware learning is not limited only to k-nearest neighbor classification,as hubness in intrinsically high-dimensional data has a geometric interpretation,due to the fact that hubs tend to lie closer to local cluster means. Due to acommon cluster assumption violation in high-dimensional data, these hub pointsneed not be central to a particular class, but can also lie in a borderline regionbetween different categories. It has been shown that these bad hubs often act assupport vectors in SVM-s [44,22]. This property can be exploited for hubness-


Table 2 Classification accuracy of different hubness-aware classification methods for k = 5and k = 10, on various image datasets represented as SIFT bag of visual words. Scores that arestatistically significantly better/worse (p < 0.01) than kNN are denoted by ◦/•, respectively.The smaller and larger neighborhood sizes used in RRKNN are given in brackets. The bestresult in each line is given in bold.

(a) k = 5

Data set kNN RRKNN(5,10) hw-kNN h-FNN NHBNN HIKNNiNet3 75.8 ±1.8 82.4 ±1.8◦ 79.6 ±2.2◦ 82.2 ± 1.4◦ 81.7 ±1.6◦ 82.3 ± 1.6◦iNet4 59.7 ±1.2 65.6 ±1.5◦ 61.0 ±1.5 64.5 ± 1.4◦ 64.7 ±1.3◦ 64.7 ± 1.5◦iNet5 50.6 ±1.3 61.9 ±1.5◦ 52.7 ±1.9• 61.4 ± 1.0◦ 61.8 ±1.1◦ 60.6 ± 1.1◦iNet6 66.0 ±1.2 68.8 ± 1.3◦ 67.0 ±1.2• 68.5 ± 1.2◦ 69.4 ±1.3◦ 69.9 ±1.1◦

iNet7 47.7 ±1.8 60.0 ±1.0◦ 52.6 ±1.9◦ 58.1 ± 2.0◦ 58.3 ±1.0◦ 56.4 ± 1.2◦iNet3Imb 80.4 ±1.9 89.4 ±1.7◦ 84.5 ±2.2◦ 87.6 ± 1.6◦ 85.0 ±1.5◦ 88.3 ± 1.5◦iNet4Imb 65.5 ±1.7 70.3 ±1.4◦ 76.6 ±1.5◦ 69.8 ± 1.3◦ 69.3 ±1.7◦ 70.2 ± 1.3◦iNet5Imb 61.5 ±1.8 65.8 ±1.8◦ 62.3 ±1.7 64.7 ± 1.7◦ 63.8 ±1.5◦ 65.2 ± 1.7◦iNet6Imb 67.3 ±1.6 70.5 ± 1.9◦ 68.0 ±1.8 70.8 ±1.6◦ 68.4 ±1.7 70.1 ± 1.7◦iNet7Imb 58.1 ±2.0 64.3 ±2.2◦ 62.2 ±2.3◦ 64.1 ± 2.1◦ 63.4 ±1.9◦ 63.9 ± 2.2◦LButterfly 48.4 ±3.3 52.2 ±3.9◦ 50.2 ±3.6 47.9 ± 3.5 45.9 ±3.6• 51.2 ± 3.4◦17Flowers 49.6 ±2.8 53.6 ± 3.0◦ 52.1 ±2.9 53.2 ± 3.1◦ 49.3 ±2.9 55.9 ±3.0◦

Caltech10126.1 ±1.5 35.2 ±1.1◦ 31.9 ±1.0◦ 34.2 ± 1.1◦ 32.1 ±1.1◦ 32.0 ± 0.9◦FacesEssex97.6 ±0.6 96.1 ± 0.7• 97.1 ±0.6• 97.6 ± 0.6 97.3 ±0.7 97.9 ±0.5AVG 61.02 66.86 64.13 66.04 65.02 66.32

(b) k = 10

Data set kNN RRKNN(5,10) hw-kNN h-FNN NHBNN HIKNNiNet3 77.4 ±1.7 82.4 ± 1.8◦ 80.4 ±2.4◦ 81.2 ± 2.0◦ 82.2 ±1.7◦ 83.3 ±1.8◦

iNet4 62.3 ±1.3 65.6 ± 1.5◦ 62.9 ±1.7 64.1 ± 1.6◦ 64.3 ±1.4◦ 66.5 ±1.6◦

iNet5 52.0 ±1.2 61.9 ± 1.5◦ 54.1 ±2.7 61.7 ± 1.5◦ 62.2 ±1.3◦ 62.8 ±1.4◦

iNet6 68.5 ±1.4 68.8 ± 1.3 68.7 ±1.4 67.5 ± 1.5 70.0 ±1.1◦ 70.7 ±1.4◦

iNet7 48.8 ±0.9 60.0 ±1.0◦ 54.9 ±2.4◦ 57.8 ± 1.9◦ 58.5 ±1.0◦ 59.3 ± 1.1◦iNet3Imb 83.4 ±1.9 89.4 ±1.7◦ 85.8 ±2.6◦ 86.9 ± 1.9◦ 82.8 ±1.8 88.7 ± 1.6◦iNet4Imb 68.0 ±1.5 70.3 ±1.4◦ 68.4 ±1.8 68.5 ± 1.5 66.5 ±1.7• 70.3 ±1.6◦

iNet5Imb 62.8 ±1.8 65.8 ±1.8◦ 63.0 ±1.9 64.4 ± 1.9 60.6 ±1.6• 65.0 ± 2.0◦iNet6Imb 68.4 ±1.6 70.5 ± 1.9◦ 69.1 ±1.7 70.5 ± 1.7◦ 67.6 ±1.7 71.4 ±1.8◦

iNet7Imb 59.6 ±2.2 64.3 ± 2.2◦ 64.3 ±2.4◦ 62.6 ± 2.1◦ 62.9 ±2.2◦ 65.1 ±2.2◦

LButterfly 51.4 ±3.5 52.2 ± 3.9 54.0 ±3.7 49.5 ± 3.9 47.4 ±3.7• 54.5 ±3.7◦

17Flowers 50.3 ±2.7 53.6 ± 3.0◦ 52.4 ±2.8◦ 51.5 ± 2.7 49.4 ±2.8 56.5 ±2.9◦

Caltech10128.8 ±1.1 35.2 ± 1.1◦ 34.0 ±1.1◦ 35.4 ±1.1◦ 34.4 ±1.0◦ 35.0 ± 1.1◦FacesEssex95.3 ±0.9 96.1 ± 0.7◦ 94.1 ±0.9• 95.5 ± 1.0 96.0 ±0.8 96.8 ±0.7◦

AVG 62.62 66.86 64.72 65.51 64.63 67.56

Fig. 15 The macro-averaged F1-score (F1M ) of 5-HIKNN, 10-HIKNN, RRKNN(5,10), 5-NN

and 10-NN, averaged over all examined image datasets. Re-ranking improves the classificationperformance over both kNN baselines.

aware instance selection and instance weighting for SVM classification. Similarinstance weighting approaches have been shown to improve boosting of CARTdecision trees, by focusing on regular points instead of hubs in early boostingiterations [44]. These implementations are currently not a part of Image Hub


Fig. 16 Precision of 5-HIKNN, 10-HIKNN, RRKNN(5,10), 5-NN and 10-NN on differenttypes of examples, averaged over all examined image datasets. Re-ranking and hubness-awarevoting improve the classification precision on outliers and isolated, difficult points. All methodsachieve comparably good performance within the class interiors.

Explorer and Hub Miner, though we intend to include various types of classifiersin future updates. These initial experiments show that hubness-aware learningrepresents a promising direction for classification in intrinsically high-dimensionaldata and object recognition from images in particular and that it should be takeninto consideration in future recognition systems.

6 Conclusions and Future Work

Image Hub Explorer is a novel tool for evaluating the utility of various imagefeature representations and metrics, from the perspective of associated hubness andthe distribution of relevance in top-k result sets in content-based image retrievaland classification.

Image data exhibits substantial hubness in various standard quantized featurerepresentations. The semantic gap that exists between the low-level feature rep-resentations and the actual perceived semantics of the images can be potentiallyemphasized by data hubness, as hubness increases the potential scope of errorpropagation. This is why it is important to be able to estimate the consequencesof hubness in practical applications.

Image Hub Explorer allows the users to quickly pinpoint the centers of influencewithin their image data and carefully examine how hubness affects the quality ofimage retrieval and object recognition. The system design allows the developersto experiment with different representations and metrics and find the potentialsolutions for the arising problems.

Image Hub Explorer is the first interactive graphical tool for examining hubnessin image data and data in general, introducing several novelties. A novel way forselecting image representatives for visualization is employed along with a novelhubness-based way of generating the background for the data projection panel.In individual image analysis, a novel feature assessment and visualization tool is


proposed, that allows the users to detect the most discriminative textural regionsin the images, under the current semantic context.

Image Hub Explorer implements many state-of-the-art hubness-aware methodsfor metric learning and classification, as well as a secondary re-ranking procedurefor improving the semantic consistency of the image query result sets.

We have demonstrated the potential usefulness of taking image hubness intoaccount by an evaluation of several state-of-the-art hubness-aware classificationapproaches on a series of object recognition tasks. The comparisons have confirmedthat the hubness-aware method substantially outperform the kNN baseline on theanalyzed image datasets. Most improvements have been shown to stem from abetter handling of non-central and rare types of images, which is important inclass-imbalanced classification.

We intend to further extend the current version of the software by fully sup-porting text-to-image and image-to-text search via the kernel canonical correlationanalysis (KCCA), as well as integrate more local image feature types in the fea-ture extraction pipeline. Also, we intend to perform an in-depth user study in thefuture, once enough people start using the system - and exploit the feedback inorder to improve the overall design and functionality of Image Hub Explorer.

Acknowledgements This work was supported by the Slovenian Research Agency, the ICTProgramme of the EC under XLike (ICT-STREP-288342), and RENDER (ICT-257790-STREP).

References

1. Aggarwal, C.C., Hinneburg, A., Keim, D.A.: On the surprising behavior of distance met-rics in high dimensional spaces. In: Proceedings of the 8th International Conference onDatabase Theory (ICDT), pp. 420–434. ACM, New York, NY, USA (2001)

2. Arthur, D., Vassilvitskii, S.: k-means++: The advantages of careful seeding. In: Proceed-ings of the 18th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pp.1027–1035. SIAM, Philadelphia, PA, USA (2007)

3. Aucouturier, J., Pachet, F.: Improving timbre similarity: How high is the sky? Journal ofNegative Results in Speech and Audio Sciences 1 (2004)

4. Axelsen, J.B., Bernhardsson, S., Rosvall, M., Sneppen, K., Trusina, A.: Degree landscapesin scale-free networks. Physical Review E - Statistical, Nonlinear and Soft Matter Physics74(3 Pt 2), 036,119 (2006)

5. Bay, H., Tuytelaars, T., Van Gool, L.: Surf: Speeded up robust features. In: A. Leonardis,H. Bischof, A. Pinz (eds.) Computer Vision ECCV 2006, Lecture Notes in Computer

Science, vol. 3951, pp. 404–417. Springer Berlin Heidelberg (2006)6. Bellman, R.E.: Adaptive control processes - A guided tour. Princeton University Press,

Princeton, New Jersey, U.S.A. (1961)7. Boiman, O., Shechtman, E., Irani, M.: In defense of nearest-neighbor based image classi-

fication. In: Proceedings of the Conference on Computer Vision and Pattern Recognition,pp. 1–8. IEEE, New York, NY, USA (2008)

8. Borg, I., Groenen, P.: Modern Multidimensional Scaling: Theory and Applications.Springer-Verlag, Berlin, Germany (2005)

9. Buza, K., Nanopoulos, A., Schmidt-Thieme, L.: Insight: efficient and effective instanceselection for time-series classification. In: Proceedings of the 15th Pacific-Asia conferenceon Advances in knowledge discovery and data mining - Volume Part II, PAKDD’11, pp.149–160. Springer-Verlag, Berlin, Germany (2011)

10. Calonder, M., Lepetit, V., Strecha, C., Fua, P.: Brief: Binary robust independent elemen-tary features. In: K. Daniilidis, P. Maragos, N. Paragios (eds.) Computer Vision ECCV2010, Lecture Notes in Computer Science, vol. 6314, pp. 778–792. Springer Berlin Heidel-berg (2010)


11. Camargo, J., Caicedo, J., Gonzlez, F.: Kernel-based visualization of large collections ofmedical images involving domain knowledge (2009)

12. Celma, O., Herrera, P., Serra, X.: Bridging the music semantic gap. In: 1st Internationalconference on Semantics And digital Media Technology (SAMT). Athens, Greece (2006)

13. Chen, J., ren Fang, H., Saad, Y.: Fast approximate kNN graph construction for highdimensional data via recursive Lanczos bisection. Journal of Machine Learning Research10, 1989–2012 (2009)

14. Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: A Large-ScaleHierarchical Image Database. In: CVPR09 (2009)

15. Donahue, J., Jia, Y., Vinyals, O., Hoffman, J., Zhang, N., Tzeng, E., Darrell, T.: Decaf: Adeep convolutional activation feature for generic visual recognition. CoRR abs/1310.1531(2013)

16. Eler, D., Nakazaki, M., Paulovich, F., Santos, D., Oliveira, M., Neto, J., Minghim, R.:Multidimensional visualization to support analysis of image collections. In: ComputerGraphics and Image Processing, 2008. SIBGRAPI ’08. XXI Brazilian Symposium on, pp.289–296 (2008)

17. Fei-Fei, L., Fergus, R., Perona, P.: Learning generative visual models from few trainingexamples: An incremental bayesian approach tested on 101 object categories (2004)

18. Fix, E., Hodges, J.: Discriminatory analysis, nonparametric discrimination: consistencyproperties. Tech. rep., USAF School of Aviation Medicine, Randolph Field (1951)

19. Flexer, A., Gasser, M., Schnitzer, D.: Limitations of interactive music recommendationbased on audio content. In: Proceedings of the 5th Audio Mostly Conference: A Conferenceon Interaction with Sound, AM ’10, pp. 13:1–13:7. ACM, New York, NY, USA (2010)

20. Fortuna, B., Grobelnik, M., Mladenic, D.: Visualization of text document corpus. Infor-matica pp. 497–502 (2005)

21. Francois, D., Wertz, V., Verleysen, M.: The concentration of fractional distances. IEEETransactions on Knowledge and Data Engineering 19(7), 873–886 (2007)

22. Georgios, K.: Investigating the Impact of Hubness on SVM Classifiers. University of theAegean, Lesvos, Greece (2011)

23. Gong, Y.: Advancing content-based image retrieval by exploiting image color and regionfeatures. Multimedia Systems 7(6), 449–457 (1999)

24. Gonzalez, J.C.J.C.F.: Multimodal visualization based on non-negative matrix factoriza-tion. In: Proceedings of the ECDL Conference, pp. 429–432 (2010)

25. Haghani, P., Michel, S., Aberer, K.: Distributed similarity search in high dimensions us-ing locality sensitive hashing. In: Proceedings of the 12th International Conference onExtending Database Technology: Advances in Database Technology, EDBT ’09, pp. 744–755. ACM, New York, NY, USA (2009). DOI 10.1145/1516360.1516446

26. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The wekadata mining software: An update. SIGKDD Explorations 11 (2009)

27. Houle, M.E., Kriegel, H.P., Kroger, P., Schubert, E., Zimek, A.: Can shared-neighbordistances defeat the curse of dimensionality? In: Proc. of the 22nd int. conf. on Scientificand statistical database management, SSDBM’10, pp. 482–500. Springer-Verlag (2010)

28. Jegou, H., Harzallah, H., Schmid, C.: A contextual dissimilarity measure for accurate andefficient image search. In: Computer Vision and Pattern Recognition, pp. 1–8. IEEE, NewYork, NY, USA (2007). DOI 10.1109/CVPR.2007.382970

29. Keller, J.E., Gray, M.R., Givens, J.A.: A fuzzy k-nearest-neighbor algorithm. IEEE Trans-actions on Systems, Man and Cybernetics pp. 580–585 (1985)

30. Li, B., Chang, E., Wu, Y.: Discovery of a perceptual distance function for measuring imagesimilarity. Multimedia Systems 8(6), 512–522 (2003)

31. Liu, J., Li, Z., Lu, H.: Sparse semantic metric learning for image retrieval. MultimediaSystems pp. 1–9 (2013)

32. Low, T., Borgelt, C., Stober, S., Nrnberger, A.: The hubness phenomenon: Fact or artifact?In: C. Borgelt, M.n. Gil, J.M. Sousa, M. Verleysen (eds.) Towards Advanced Data Analysisby Combining Soft Computing and Statistics, Studies in Fuzziness and Soft Computing,vol. 285, pp. 267–278. Springer Berlin Heidelberg (2013)

33. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Jour-nal of Computer Vision 60(2), 91–110 (2004)

34. Nanopoulos, A., Radovanovic, M., Ivanovic, M.: How does high dimensionality affect col-laborative filtering? In: Proceedings of the third ACM conference on Recommender sys-tems, RecSys ’09, pp. 293–296. ACM, New York, NY, USA (2009)


35. Napierala, K., Stefanowski, J.: Identification of different types of minority class examplesin imbalanced data. In: E. Corchado, V. Snasel, A. Abraham, M. Wozniak, M. Graa, S.B.Cho (eds.) Hybrid Artificial Intelligent Systems, Lecture Notes in Computer Science, vol.7209, pp. 139–150. Springer-Verlag, Berlin / Heidelberg, Germany (2012)

36. Nguyen, G.P., Worring, M.: Similarity based visualization of image collections. In: In IntlWorksh. Audio-Visual Content and Information Visualization in Digital Libraries (2005)

37. Nguyen, G.P., Worring, M.: Interactive access to large image collections using similarity-based visualization. J. Vis. Lang. Comput. 19(2), 203–224 (2008)

38. Nilsback, M.E., Zisserman, A.: Automated flower classification over a large number ofclasses. In: Proceedings of the Indian Conference on Computer Vision, Graphics andImage Processing (2008)

39. Patil, A., Kinoshita, K., Nakamura, H.: Hub promiscuity in protein-protein interactionnetworks. International Journal of Molecular Sciences 11(4), 1930–1943 (2010). DOI10.3390/ijms11041930

40. Pich, C.: Mdsj: Java library for multidimensional scaling (version 0.2) (2009). URL http://www.inf.uni-konstanz.de/algo/software/mdsj/

41. Porta, M.: Browsing large collections of images through unconventional visualization tech-niques. In: Proceedings of the working conference on Advanced visual interfaces, AVI ’06,pp. 440–444. ACM, New York, NY, USA (2006)

42. Radovanovic, M.: Representations and Metrics in High-Dimensional Data Mining. Iz-davacka knjizarnica Zorana Stojanovica, Novi Sad, Serbia (2011)

43. Radovanovic, M., Nanopoulos, A., Ivanovic, M.: Nearest neighbors in high-dimensionaldata: The emergence and influence of hubs. In: Proceedings of the 26th InternationalConference on Machine Learning (ICML), pp. 865–872. Morgan Kaufmann, San Francisco,CA, USA (2009)

44. Radovanovic, M., Nanopoulos, A., Ivanovic, M.: Hubs in space: Popular nearest neighborsin high-dimensional data. Journal of Machine Learning Research 11, 2487–2531 (2010)

45. Radovanovic, M., Nanopoulos, A., Ivanovic, M.: On the existence of obstinate resultsin vector space models. In: Proceedings of the 33rd Annual International ACM SIGIRConference on Research and Development in Information Retrieval, pp. 186–193. ACM,New York, NY, USA (2010)

46. Radovanovic, M., Nanopoulos, A., Ivanovic, M.: Time-series classification in many intrinsicdimensions. In: SDM, pp. 677–688. SIAM, Philadelphia, PA, USA (2010)

47. Rashedi, E., Nezamabadi-pour, H., Saryazdi, S.: Information fusion between short termlearning and long term learning in content based image retrieval systems. MultimediaTools and Applications pp. 1–24 (2013)

48. Rashedi, E., Nezamabadi-Pour, H., Saryazdi, S.: A simultaneous feature adaptation andfeature selection method for content-based image retrieval systems. Knowledge-BasedSystems 39, 85–94 (2013)

49. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: Orb: An efficient alternative to sift orsurf. In: Computer Vision (ICCV), 2011 IEEE International Conference on, pp. 2564–2571(2011)

50. Schaefer, G.: A next generation browsing environment for large image repositories. Mul-timedia Tools and Applications 47(1), 105–120 (2010)

51. Schnitzer, D., Flexer, A., Schedl, M., Widmer, G.: Local and global scaling reduce hubsin space. Journal of Machine Learning Research pp. 2871–2902 (2012)

52. Tan, S.: Neighbor-weighted k-nearest neighbor for unbalanced text corpus. Expert Systemswith Applications 28, 667–671 (2005)

53. Tomasev, N., , Leban, G., Mladenic, D.: Exploiting hubs for self-adaptive secondary re-ranking in bug report duplicate detection. In: Proceedings of the ITI conference, ITI 2013.SRCE, Zagreb, Croatia (2013)

54. Tomasev, N., Brehar, R., Mladenic, D., Nedevschi, S.: The influence of hubness on nearest-neighbor methods in object recognition. In: Proceedings of the 7th IEEE InternationalConference on Intelligent Computer Communication and Processing (ICCP), pp. 367–374.IEEE, New York, NY, USA (2011)

55. Tomasev, N., Fortuna, B., Mladenic, D., Nedevschi, S.: Ontogen extension for exploringimage collections. In: Proceedings of the 7th IEEE International Conference on IntelligentComputer Communication and Processing (ICCP) (2011)

56. Tomasev, N., Mladenic, D.: Nearest neighbor voting in high dimensional data: Learningfrom past occurrences. Computer Science and Information Systems 9, 691–712 (2012)


57. Tomasev, N., Mladenic, D.: Hubness-aware shared neighbor distances for high-dimensionalk-nearest neighbor classification. Knowledge and Information Systems pp. 1–34 (2013).DOI 10.1007/s10115-012-0607-5

58. Tomasev, N., Radovanovic, M., Mladenic, D., Ivanovic, M.: A probabilistic approach tonearest neighbor classification: Naive hubness bayesian k-nearest neighbor. In: Proceedingof the CIKM conference, pp. 2173–2176. ACM, New York, NY, USA (2011)

59. Tomasev, N., Radovanovic, M., Mladenic, D., Ivanovic, M.: Hubness-based fuzzy measuresfor high-dimensional k-nearest neighbor classification. International Journal of MachineLearning and Cybernetics (2013). DOI 10.1007/s13042-012-0137-1

60. Tomasev, N., Radovanovic, M., Mladenic, D., Ivanovic, M.: The role of hubness in clus-tering high-dimensional data. IEEE Transactions on Knowledge and Data Engineering99(PrePrints), 1 (2013). DOI 10.1109/TKDE.2013.25

61. Tronci, R., Murgia, G., Pili, M., Piras, L., Giacinto, G.: Imagehunter: A novel tool forrelevance feedback in content based image retrieval. In: C. Lai, G. Semeraro, E. Vargiu(eds.) New Challenges in Distributed Information Filtering and Retrieval, Studies in Com-

putational Intelligence, vol. 439, pp. 53–70. Springer Berlin Heidelberg (2013)62. Wang, J., Kumar, S., Chang, S.F.: Semi-supervised hashing for scalable image retrieval.

In: Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pp.3424–3431 (2010)

63. Wang, J., Markert, K., Everingham, M.: Learning models for object recognition fromnatural language descriptions. In: Proceedings of the British Machine Vision Conference.BMVA Press, London, UK (2009)

64. Wang, J., Neskovic, P., Cooper, L.N.: Improving nearest neighbor rule with a simpleadaptive distance measure. Pattern Recognition Letters 28, 207–213 (2007)

65. Wang, X.Y., Zhang, B.B., Yang, H.Y.: Content-based image retrieval by integrating colorand texture features. Multimedia Tools and Applications 68(3), 545–569 (2014)

66. Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques,Second Edition (Morgan Kaufmann Series in Data Management Systems). Morgan Kauf-mann Publishers Inc., San Francisco, CA, USA (2005)

67. Yamaoka, S., Doerr, K.U., Kuester, F.: Visualization of high-resolution image collectionson large tiled display walls. Future Generation Computer Systems 27(5), 498 – 505 (2011)

68. Yang, J., Fan, J., Hubball, D., Gao, Y., Luo, H., Ribarsky, W.: Semantic image browser:Bridging information visualization with automated intelligent image analysis. In: Proc.IEEE Symposium on Visual Analytics Science and Technology (2006)

69. Yu, K., Lin, Y., Lafferty, J.: Learning image representations from the pixel level via hierar-chical sparse coding. In: Computer Vision and Pattern Recognition (CVPR), 2011 IEEEConference on, pp. 1713–1720 (2011)

70. Zelnik-manor, L., Perona, P.: Self-tuning spectral clustering. In: Advances in Neural Infor-mation Processing Systems 17, pp. 1601–1608. MIT Press, Cambridge, MA, USA (2004)

71. Zha, Z.J., Yang, L., Mei, T., Wang, M., Wang, Z., Chua, T.S., Hua, X.S.: Visual querysuggestion: Towards capturing user intent in internet image search. ACM Transactions onMultimedia Computing and Communication Applications 6(3), 13:1–13:19 (2010)

72. Zhang, H., Zha, Z.J., Yang, Y., Yan, S., Gao, Y., Chua, T.S.: Attribute-augmented se-mantic hierarchy: Towards bridging semantic gap and intention gap in image retrieval.In: Proceedings of the 21st ACM International Conference on Multimedia, MM ’13, pp.33–42. ACM, New York, NY, USA (2013)

73. Zhao, J., Xu, K.: Enhancing the robustness of scale-free networks. Computing ResearchRepository (2009)

74. Zheng, M., Bu, J., Chen, C., Wang, C., Zhang, L., Qiu, G., Cai, D.: Graph regularizedsparse coding for image representation. Image Processing, IEEE Transactions on 20(5),1327–1336 (2011)

75. Zhou, Z.H., Chen, K.J., Dai, H.B.: Enhancing relevance feedback in image retrieval usingunlabeled data. ACM Trans. Inf. Syst. 24(2), 219–244 (2006)

76. Zhu, X., Huang, Z., Cheng, H., Cui, J., Shen, H.T.: Sparse hashing for fast multimediasearch. ACM Transactions on Information Systems 31(2), 9:1–9:24 (2013)

image hub explorer: evaluating representations and

Documents