computer vision, part 2 object recognition and scene “understanding”

Computer Vision, Part 2

Object recognition and scene “understanding”

• What makes object recognition a hard task for computers?

HMAX Riesenhuber, M. & Poggio, T. (1999),

“Hierarchical Models of Object Recognition in Cortex”

Serre, T., Wolf, L., Bileschi, S., Risenhuber, M., and Poggio, T. (2006),“Robust Object Recognition with Cortex-Like Mechanisms”

• HMAX: A hierarchical neural-network model of object recognition.

• Meant to model human vision at level of “immediate recognition” capabilities of ventral visual pathway, independent of attention or other top-down processes.

• Also called “Standard Model” (because it incorporates the “standard model” of visual cortex)

• Inspired by earlier “Neocognitron” model of Fukushima (1980)

General ideas behind model

• “Immediate” visual processing is feedforward and hierachical: low levels detect simple features, which are combined hierarchically into increasingly complex features to be detected

• Layers of hierarchy alternate between “sensitivity” (to detecting features) and “invariance” (to position, scale, orientation)

• Size of receptive fields increases along the hierarchy

• Degree of invariance increases along the hierarchy

The HMAX model for object recognition(Riesenhuber, Poggio, Serre, et al.)

The HMAX model for object recognition(Riesenhuber, Poggio, Serre, et al.)

Image (gray-scale)

S1 layerEdge detectors

The HMAX model for object recognition(Riesenhuber, Poggio, Serre, et al.)

Image (gray-scale)

S1 layerEdge detectors

The HMAX model for object recognition(Riesenhuber, Poggio, Serre, et al.)

Image (gray-scale)

C1 layerMax over local S1 units

Layers alternatebetween“specificity”and“invariance”over position, scale, orientation

S1 layerEdge detectors

The HMAX model for object recognition(Riesenhuber, Poggio, Serre, et al.)

Image (gray-scale)

C1 layerMax over local S1 units

S2 layerPrototypes

(small image patches)

Layers alternatebetween“specificity”and“invariance”over position, scale, orientation

S1 layerEdge detectors

The HMAX model for object recognition(Riesenhuber, Poggio, Serre, et al.)

Image (gray-scale)

C1 layerMax over local S1 units

S2 layerPrototypes

(small image patches)

C2 layerMax activation over each

prototype Layers alternatebetween“specificity”and“invariance”over position, scale, orientation

S1 layerEdge detectors

The HMAX model for object recognition(Riesenhuber, Poggio, Serre, et al.)

Image (gray-scale)

C1 layerMax over local S1 units

S2 layerPrototypes

(small image patches)

C2 layerMax activation over each

prototype

Classification layerObject or image classification

Layers alternatebetween“specificity”and“invariance”over position, scale, orientation

S1 layerEdge detectors

The HMAX model for object recognition(Riesenhuber, Poggio, Serre, et al.)

Image (gray-scale)

C1 layerMax over local S1 units

S2 layerPrototypes

(small image patches)

C2 layerMax activation over each

prototype

Classification layerObject or image classification

Layers alternatebetween“specificity”and“invariance”over position, scale, orientation

Job of HMAX is toproduce a higher-level representation of an image thatwill be useful for classification.

S1 layerEdge detectors

4 orientations, 16 scales

Image (gray-scale)

Etc.: 16 scales

One S1 receptive field:

MAX MAX

S1 layerEdge detectors

4 orientations, 16 scales

C1 layerMax activation over local S1 units (local position, scale)

4 orientations, 8 scales

Image (gray-scale)

S2 layerCalculate similarity to

prototype (radial basis function)4 orientations, 8 scales

…

C1 layerMax activation over local S1 units (local position, scale)

4 orientations, 8 scales

S2 unit: Calculate similarity to prototype for each “pooled” position in C1 layer.

S2 layerCalculate similarity to

prototype (radial basis function)4 orientations, 8 scales

…

Prototypes(~1000, chosen from image collection,

translated to C1 features)

C1 layerMax activation over local S1 units (local position, scale)

4 orientations, 8 scales

S2 unit: Calculate similarity to prototype for each “pooled” position in C1 layer.

S2 layerCalculate similarity to

prototype (radial basis function)4 orientations, 8 scales

…

Prototypes(~1000, chosen from image collection,

translated to C1 features)

C1 layerMax activation over local S1 units (local position, scale)

4 orientations, 8 scales

S2 unit: Calculate similarity to prototype for each “pooled” positionin C1 layer.

Similarity: Radial basis function:

€

S2 i = exp −γ X − Pi2

( )

S2 layerCalculate similarity to

prototype (radial basis function)4 orientations, 8 scales

…

C2 layerMax activation over

position, orientation, scale

S21 S22 …

MAX(1 value)

MAX(1 value)

…

C2 layerMax over position, orientation, scale

.11 .78 … .32

Support Vector Machineclassification(e.g., dog / not dog)

Streetscenes “scene understanding” system(Bileschi, 2006)

Use HMAX + SVM to identify object classes: Car, Pedestrian, Bicycle, Building, Tree

How Streetscenes Works(Bileschi, 2006)

1. Densely tile the image withwindows of different sizes.

2. C1 and C2 features are computed in each window.

3. The features in eachwindow are given as inputto each of five trained support vector machines

4. If any return a classification with score above a learned threshold, that object is said to be “detected” .

…

Object detection (here, “car”) with HMAX model (Bileschi, 2006)

Sample of results from HMAX model

(Serre et al., 2006)