ieee 2017 conference on discover and learn new objects...
TRANSCRIPT
IEEE 2017 Conference on
Computer Vision and Pattern
Recognition Discover and Learn New Objects from Documentaries
Kai Chen, Hang Song, Chen Change Loy, Dahua Lin
The Chinese University of Hong Kong
• We develop a novel approach to learning object detectors from
documentary videos and subtitles in an weakly supervised way.
• We propose a framework that can effectively integrate visual and
linguistic cues.
Welcome to visit our project homepage or scan the qrcode.
http://www.chenkai.site/projects/documentary-learning/index.html
Introduction
WildLife Documentary(WLD) Dataset
• Video frames + subtitles
• 15 documentary videos
• >700k frames (7.4h)
• >50 categories
• >4000 annotated tracklets
Framework Results
By the time they leave
her, they will weigh eight
times as much.
Most animals fear sloth
bears as but not
apparently wild boar, at
least not in this food
around.
The sambar deers are feeling
the heat too.
417-542
Background
Timeline
Iter 0
Iter 1
Iter 2
4149-4218
Boar6105-6232
Background
6232-6270
Background6268-6291
Background
4149-4218
Bear
417-542
Cub
6105-
6270(merged)
Deer
6268-6291
Background
417-542
Cub
4149-4218
Bear6105-6291
(merged)
Deer
Ground
truthTiger/Cub Bear Deer
The elephant are about to march throughthem. The spiders themselves have a span aswide as a human hand.
Tigers are one of the few cats that actuallyenjoy swimming.
Unlike mechanics, langurs are the friends ofspotted deer.
But the love serenade is over once a dogarrives.
Male koalas play no role in parenting.
There's a turf war going on and the koalasare losing.
Australian camels appear sick and emaciated.
About 50 animals have died in just threemonths, including this adult orangutan onthe day we arrived.
The mayor has declined offers of assistanceand expert advice from animal welfaregroups.
Examples
She needs to be good. Her cubs havealready got huge appetites
Most animals fear sloth bears as butnot apparently wild boar, at least not inthis food around.
Langurs are the friends of spotted deer.
merge tracklets
and categories
c3
for some babies drinking isn‘t themost comfortable oftime
These are among the rarest sloth bearsThe cubs also like a
cooling dip
c1
c1
c2
cn
…
update
keywords tracklets
screen
samples
…
classifier
classify
tracklets and
ground
keywords
1. Bootstrap 2. Joint Analysis
𝑝 𝑧, 𝑎, 𝑟 𝑜; Θ =1
𝑍 Θexp Ψap z o; 𝜃 + Φ𝑘𝑡 𝑧, 𝑎 𝑜; 𝜂 + Φ𝑠𝑡(𝑟, 𝑧|𝑜)
CRF formulation
Appearance
potentialKeyword-tracklet
potentialGeometric
potential
Ψap z o; 𝜃 = 𝑖=1
𝑛
𝜓𝑎𝑝(𝑧𝑖|𝑣𝑖; 𝜃)
Φkt z, a o; 𝜂 = 𝑖,𝑗 ∈𝐺
𝜙𝑘𝑡(𝑧𝑖 , 𝑎𝑖𝑗|𝜂)
Φst r, z o = 𝑖,𝑖′ ∈𝑅
𝜙𝑠𝑡(𝑟𝑖𝑖′ , 𝑧𝑖 , 𝑧𝑖′|𝑢𝑖 , 𝑢𝑖′)
object category𝑧𝑖whether tracklet 𝜏𝑖assiciates withkeyword 𝑤𝑗
𝑎𝑖𝑗
whether two trackletsshould be merged
𝑟𝑖𝑖′