when automated analysis goes wrong
TRANSCRIPT
When automated
analysis goes wrong
Tristan Roddis
Cogapp
MCN conference 2017
Tristan Roddis. Cogapp
In this session
• Successes
• FAILURES
Tristan Roddis. Cogapp
Failure 1: Interesting
images
Tristan Roddis. Cogapp
The problem
Adrian Hindle, Tristan Roddis. Cogapp
"Nourishment for the Ailing" and "Nourishment for the Healthy"
http://www.qdl.qa/en/archive/qnlhc/9549.20
Adrian Hindle, Tristan Roddis. Cogapp
"Nourishment for the Ailing" and "Nourishment for the Healthy"
http://www.qdl.qa/en/archive/qnlhc/9549.19
Adrian Hindle, Tristan Roddis. Cogapp
Tristan Roddis
Tristan Roddis
Tristan Roddis. Cogapp
Success 1: Interesting
images
Adrian Hindle, Tristan Roddis. Cogapp
• Automatically tag, organize, and search visual
content with machine learning.
• Create concepts
Adrian Hindle, Tristan Roddis. Cogapp
• Existing concepts:
• Illustration
• Drawing
Adrian Hindle, Tristan Roddis. Cogapp
• Negative concept:
• Text
• Manuscripts
Adrian Hindle, Tristan Roddis. Cogapp
• Combination of two custom concepts
• arabic_manuscript
• arabic_ manuscript_with_image
Adrian Hindle, Tristan Roddis. Cogapp
45 images
45 images
Adrian Hindle, Tristan Roddis. Cogapp
26 images
26 images
Adrian Hindle, Tristan Roddis. Cogapp
• Python script to create sets and train
• Test and train
Adrian Hindle
Tristan Roddis
Adrian Hindle, Tristan Roddis. Cogapp
Adrian Hindle
Tristan Roddis
Adrian Hindle
Tristan Roddis
Tristan Roddis
Tristan Roddis. Cogapp
Other uses
• Blank sheets vs. writing
• Handwritten vs. typewritten
• [your collection-specific question here]
Tristan Roddis. Cogapp
Failure 2: term
extraction
Tristan Roddis. Cogapp
Failure 2: term
extraction
Tristan Roddis. Cogapp
Failure 2: term
extraction
Tristan Roddis. Cogapp
Failure 2: term
extraction
Tristan Roddis. Cogapp
Success 2: add a
human
Tristan Roddis. Cogapp
Failure 3: Finding similar
images
Tristan Roddis. Cogapp
Scikit-image
Tristan Roddis. Cogapp
Scikit-image
Tristan Roddis. Cogapp
Scikit-image
Tristan Roddis. Cogapp
Scikit-image
Tristan Roddis. Cogapp
Scikit-learn
• Machine learning is not easy
• Especially for this problem
Tristan Roddis. Cogapp
Success 3: Finding similar
images
Tristan Roddis. Cogapp
Term extraction
• Clarifai
• Google Vision API
• Microsoft Computer Vision
Tristan Roddis. Cogapp
Clarifai
Tristan Roddis. Cogapp
Google Vision API
Tristan Roddis. Cogapp
MS Computer vision
Tristan Roddis. Cogapp
Images
• Sourced from Nationalmuseum Sweden
• Using Europeana API for discovery
• 2000 images
• http://labs.cogapp.com/iiif-ml/
Tristan Roddis. Cogapp
Success 4:
Optical music recognition
Tristan Roddis. Cogapp
Failure 4: Automated
captions
A vase of flowers on a display
Emanuel Swedenborg
Emanuel Swedenborg sitting in front of
a laptop
Person using a phone
Person sitting in a chair talking on a cell
phone
A man and a woman taking a selfie
A cat with its mouth open
A pizza sitting on top of a window
A man riding a bear in the water
A teddy bear
A group of sheep standing on top of a
horse
A group of sheep standing on top of a
book
Tristan Roddis. Cogapp
Conclusions
• Embrace failure
• Find workarounds
• Add humans to the mix
Thank you.
Questions?
http://labs.cogapp.com/iiif-ml
http://labs.cogapp.com/nls-omr
Tristan Roddis