Applying Computer Vision to Art History

John Resig - http://ejohn.org/research/ Visiting Researcher, Ritsumeikan University

What “Works” TodayReading license plates, zip codes, checks

Optical Character Recognition

• Tesseract

• https://code.google.com/p/tesseract-ocr/

What “Works” TodayFace recognition

Face Matching

• OpenBR

• http://openbiometrics.org/

• Age Estimation

What “Works” TodayRecognition of flat, textured, objects

Computer Vision

• Unsupervised (requires no labeling):

• Comparing an entire image

• Categorizing an image

• Supervised (requires labeling):

• Finding parts of an image

• Finding and categorizing parts of an image

Unsupervised Training

• Requires little-to-no prepping of data

• Can just give the tool a set of images and have it produce results

• Extremely easy to get started, results aren’t always as interesting.

Supervised Training

• Need lots of training data

• Needs to be pre-selected/categorized

• Think: Thousands of images.

• If your collection is smaller than this, perhaps it may not benefit.

• Or you may need crowd sourcing.

• Results can be more interesting:

• “Find all the people in this image”

Image Similarity

• imgSeek (Open Source)

• http://www.imgseek.net/

• TinEye’s MatchEngine

• http://services.tineye.com/MatchEngine

• Both are completely unsupervised. No training data is required.

imgSeek

• Compares entire image.

• Finds similar images, not exact.

• Does not find parts of an image.

• Color sensitive.

Ukiyo-e.org (Using MatchEngine)

• Compares portions of images.

• Finds exact matches.

• Finds images inside other images.

• Color insensitive.

Anonymous Italian Art (Frick PhotoArchive) Using MatchEngine

Conservation

Copies

Partial Image vs. Much Larger Image

Image Portion

Frick 420

420

Zeri 1583642090

Frick 417

417

?

8132a 8132

57129

57134

57130

57138

8131a 8131

?

Image Categorization

• Deep neural networks

• Requires minimal categorization

• Very little user-input required.

• Ersatz

• http://www.ersatzlabs.com/

Requires a lot of training data (thousands of images)

Takes a lot of computers(Not cheap)

The less categories you have, the better.

General Computer Vision

• Ideal for some supervised training problems

• CCV

• http://libccv.org/

• https://github.com/liuliu/ccv

• OpenCV

• http://opencv.org/

Object Detection

Training Caveats

• Requires thousands (if not 10s of thousands) of images

• Will take at least a week to run on a very powerful computer

• Does not work with 3D objects

Learn More about Computer Vision

• Learn more:

• http://cs.brown.edu/courses/csci1430/

• Paper on Frick Computer Vision work:

• http://ejohn.org/research/