SUN Database: Large-scale Scene Recognition from Abbey to Zoo

Jianxiong Xiao *James Haysy Krista A. Ehinger Aude Oliva Antonio Torralba

Massachusetts Institute of Technology *Brown University

CVPR 2010.

Outline

• Introduction• A Large Database for Scene Recognition• Human Scene Classification• Computational Scene Classification• Scene Detection• Conclusion

Introduction

• We seek to quasi-exhaustively determine the number of different scene categories with different functionalities.

• We measure how accurately humans can classify scenes into hundreds of categories.

• We evaluate the scene classification performance of state of the art algorithms and establish new bounds for performance on the SUN database and the 15 scene database.

• We study the possibility of detecting scenes embedded inside larger scenes.

A Large Database for Scene Recognition

• We selected from the 70,000 terms of all the terms of WordNet that described scenes, places, and environments.

• Only color images of 200 × 200 pixels or larger were kept.• Dataset reaches 899 categories and 130,519 image. And we use 397

well-sampled categories in the following evaluation.

Human Scene Classification

• Experiment on Amazon’s Mechanical Turk.• We group the 397 scene categories in a 3-level

tree.

Computational Scene Classification

• Image Features and Kernels– GIST : the filters are Gabor-like filters tuned to 8

orientations at 4 different scales.– HOG2x2 : gives a 31-dimension descriptor for each node of

the grid. Then, 2×2 neighboring HOG descriptors are stacked together to form a descriptor with 124 dimensions.

– Dense SIFT、 LBP、 Sparse SIFT 、 histograms、 SSIM、 Tiny Images、 Line Features、 Texton Histograms、 Color Histograms、 Geometric Probability Map、 Geometry Specific Histograms.

• Experiments and Analysis

Scene Detection

• Seeing Scenes in Scenes

• Multiscale scanning window approach to find sub-scenes. (1, 0.65. 0.42)

• Test Set and Evaluation Criteria– We use 24 of the 398 well-sampled SUN

categories.– In every photo we trace the ground truth spatial

extent of each sub-scene.– area(Bp ∩ Pgt) / area(Bp) T = 15%≧

Conclusion

• We have proposed a quasi-exhaustive dataset of scene categories (899 environments).

• Using state-of-the art algorithms for image classification, we have achieved new performance bounds for scene classification.

• We introduced a new task of scene detection within images.

Thank you !