Callisto – A Content-based Tag Recommendation Tool

M. Lux, A. Pitman, and O. Marques

What does Callisto do?

• Given an image and one or more start tags

• Callisto finds ranked tag recommendations

1. Based on our model (NCP)

2. Based on statistical analysis (Stat)

What are the benefits of NCP & Callisto?

• Different tags are suggested.

• Tags are re-ranked based on visual content.

– Consequently:

• With the NCP model, it is common to see tags that are highly related to visual features being suggested if such features are there, and not suggested if those features are missing.

– E.g.: sunset is not suggested if typical colors of sunsets are missing in the image.

The Application

Image to be tagged

Start tag(s)

Low-level features used

Suggestions

Ranked suggested tags

Suggestions by our content-based model (NCP)

Suggestions based on tag co-occurrence (baseline)

Use Case: Beach

Start tag: beach

• Suggestions by both models are almost the same• Both feature good quality suggestions

Use Case: Beach

Same start tag, different photo

• Suggestions differ• NCP has different tags to offer

Use Case: Ocean

Start tag: ocean

• NCP suggests clouds

Use Case: Juggling

Start tag: juggling

• NCP ranks fire first• NCP doesn‘t include balls in the list, which is good, since there are no balls involved

Use Case: Juggling

Start tag: juggling

• NCP suggests portraitand people• NCP doesn‘t suggest fire

Use Case: Juggling girl

Start tags: juggling girl

• NCP suggests woman• NCP ranks people higher

Performance issues

• Callisto has to download images and tags forsuggestions, which is slow.

• Callisto caches downloads, so next time (with the same start tag) it is much faster.

• The number of downloaded photos is critical.

– 28 works fine and is not too slow

– 100 is much better, but downloading takes forever

Live demo

• Keep your fingers crossed…