Semantic Image Browser

Parker Dunlap11/15/2013

Semantic image analysis techniques can automatically detect high level content of images

Lack of intuitive visualization and analysis techniques

Introduction

Allow users to effectively browse/search in large databases

Allow analysts to evaluate their annotation process through interactive visual exploration

Goal of the Semantic Image Browser

Target search◦ User knows exactly what they want, a precise

image Search by association

◦ Find interesting things related to certain image Category search

◦ Retrieve images that are representative of a certain class

Common Tasks of Image Exploration

Semantic contents of images are more useful for image exploration than low level features but in most large scale image collections (internet) semantics are usually not described

This has given rise to techniques that enable automatic annotation of images according to their semantic concepts

Problem

Contains semantic image classification process that automatically annotates large image collections

Contains coordinated visualization techniques that allow interactive exploration

Contains visualization techniques that allow analysts to evaluate and monitor annotation process

Semantic Image Browser (SIB)

1. Annotation Engine2. Image Browsing Interface3. Visual Image Analysis

SIB Implementation

Abstract Image content by detecting underlying salient objects (distinguishable regions)

Associate salient objects with corresponding semantic objects according to their perceptual properties

Keywords for semantic objects are used to annotate the image

Annotation Engine

Highlighted regions are salient objects detected and associated with

semantic object “sand field”

Goal is to bridge the gap between low-level visual features and high-level semantic concepts

Annotation engine has set of predefined salient objects and functions to detect them from images◦ Uses techniques like image segmentation and

SVM classifiers

Bridge the semantic gap

Annotation engine assigns a semantic concept to the data based on semantic content◦ Sand, Field, Water → Seaworld◦ Flowers, Trees → Garden

The Data

Image overview using MDS ◦ Use the annotations to calculate distance matrix

and input into MDS algorithm Distance between each pair of images in the content

space◦ Algorithm outputs a 2D position for each image

based on similarity with other images

Image Browsing Interface

Maps image miniatures onto the screen based on their content similarities

Similar images placed closer to each other Goal of MDS is to map some high

dimensional data into lower dimension (in our case 2D)◦ To learn more about MDS see MDS Overview

Multi-Dimensional Scaling (MDS) image view

Visually represents the contents of the entire collection of images

Correlations of different contents and detailed annotations are displayed

Interactively exploring large datasets with real time response (high dimensionality)

Value and Relation (VaR) content view

Block of pixels to represent images contents Each image is mapped to a pixel whose

color indicates if the image contains/doesn’t contain the content for that block

Pixel representing the same image is the same for all blocks

Allows us to observe content of image collection by scanning labels of the blocks

VaR (Cont.)

Can see correlations among the contents Can also select images to see them

highlighted in the view Position of the blocks are determined by

similarity with neighboring contents Pixels are generally created in a spiral

arrangement starting from the center and moving out

VaR (cont.)

VaR (cont.)

Pixel order can greatly effect the looks of VaR view

VaR (cont.)

To increase scalability, interface users miniature versions of images◦ High res original pictures would increase load

times Load image miniatures as textures objects

in OpenGL◦ Allows all interactions to be done in real time

Image Browsing Interface (cont.)

To reduce clutter in the MDS overview, the system provides many interactions◦ Reordering◦ Dynamic Scaling◦ Relocation◦ Distortion◦ Showing Original Image◦ Zoom◦ Pan

Interactions in MDS display

Reordering◦ Randomizing order of all images allows each

frame to have an equal probability of being visible◦ User can also explicitly bring certain image to the

front by selecting it Dynamic Scaling

◦ Interactively reduce image miniature size to reduce overlap or increase image size to examine detail

Interactions in MDS (cont.)

Relocation◦ Manually change position of individual image by

dragging and dropping Distortion

◦ Enlarge size of certain image(s) while retaining size of all others

Interactions in MDS (cont.)

Interactions in MDS (cont.)

Showing Original Image◦ Actual image (instead of scaled down image used

by OpenGL) opens at full resolution in new window

◦ Only loaded when requested to save space/time Zoom/Pan

◦ Zoom in/out and pan left/right

Interactions in MDS (cont.)

Can use multiple techniques at once to achieve some goal◦ Use Dynamic Scaling with zooming in to examine

local details with less clutter

Interactions in MDS (cont.)

Selection◦ Interactively select a sample image to see similar

images in display◦ Can change similarity threshold via a slider to

increase/decrease number of results Sorting

◦ Images can be sorted by concepts or similarity to selected image

Interactions in MDS (cont.)

Inspired by rainfall animation Correlations between image of interest and

other images are modeled through an animation

Focus image is on the bottom (ground) and the other images fall to the ground (rain) at accelerations related to their similarity

Rainfall Mode

Search for images with/without certain content

Reduce a selected subset by requiring images must/not contain certain content

Increase selected subset by adding new images

All these functions done by clicking on images while holding certain function key

Offers many similar interactions as MDS as well

Interaction in VaR display

Interaction in VaR display (cont.)

Interaction in VaR display (cont.)

Each image has its visual representations in both MDS and VaR views

Selected images are highlighted in both views Can use appropriate view as needed

◦ MDS to select image based on relationship to sample image

◦ VaR to select image based on content Common strategy is to start from VaR and

switch to MDS after number of images has been greatly reduced

Putting it together

We can use the MDS and VaR views to see how well the annotations of images correspond to their actual content

Select “red-flower” images from VaR view and verify using MDS view to see if the images are actually red flowers

If automatic annotation makes a mistake, user can manually annotate image to fix it

Annotation Analysis

VaR display also shows the reliability of the annotation by surrounding it with a colored frame◦ Green is safe to use, Yellow means lower reliability

measure Reliability measure can be determined from

annotation process or manually set up by analysts

Annotation Analysis (cont.)

Comparison of SIB to the sequential thumbnail view from Microsoft Explorer

Modes used in Microsoft Explorer◦ Random Explorer – images are randomly sorted◦ Sorted Explorer – images are sorted according to

semantic concepts generated by the classification process

User Study

10 participants from varying fields Each subject used both Sorted Explorer and

SIB◦ Random Explorer was only tested on 3

participants since expected results were so low Participants given 3 tasks to perform on 2

data sets◦ 180 second timeout window

User Study (cont.)

1. Presented with a particular image and asked to search for it from the 1100 images in the data set

2. Asked to find images containing particular features (sand, water, sky, etc…)

3. Asked to approximate what proportion of the images in the dataset contained particular contents (% that contain mountains)

The Tasks

Random Explorer◦ 2/9 trials failed◦ 81 seconds was average time with 29 seconds

standard deviation Sorted Explorer

◦ 2/30 trials failed◦ 29 seconds was average time with 20 seconds

standard deviation SIB

◦ 6/30 trials failed◦ 45 seconds was average time with 26 seconds

standard deviation

Results for Task 1

Failure in SIB was due to inaccuracy in the annotation process

SIB tended to be slower than Sorted Explorer because content names could be confusing◦ This advantage will decrease as the data set

grows because Explorer provides no overview model

Task 2 had similar results to Task 1 Task 3 was where SIB became dominant

Results for Task 1

Positive feedback for SIB Enjoyed Search by content feature the most Enjoyed MDS overview over Windows

explorer to see entire collection of images at once

Suggested side-by-side views, example image next to blocks in VaR view

User Review

Semantic Image Browser was introduced that attempts to bride information visualization with automatic image annotation

MDS image layout that groups images based on semantic similarities

VaR content display to represent large image collections

Conclusion

Questions?

Semantic Image Browser: Bridging Information Visualization with Automated Intelligent Image Analysis

Value and Relation Display for Interactive Exploration of High Dimensional Datasets

MDS Overview

Resources