adaptive segmentation based on a learned quality metric i. frosio 1, e. ratner 2 1 nvidia, usa, 2...
TRANSCRIPT
![Page 1: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/1.jpg)
Adaptive Segmentation Based on a Learned Quality
MetricI. Frosio1, E. Ratner2
1 NVIDIA, USA, 2 Lyrical Labs, USA
![Page 2: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/2.jpg)
2
Motivation: good / bad segmentation
SLIC (Achanta, 2012)
![Page 3: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/3.jpg)
3
Motivation: good / bad segmentation
GRAPH-CUT (Felzenszwalb, 2004)
![Page 4: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/4.jpg)
4
Motivation: good / bad segmentation
ADAPTIVE GRAPH-CUT (our)
![Page 5: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/5.jpg)
5
Motivation: good / bad segmentation
> >
? ? ?SLIC (Achanta, 2012) GRAPH-CUT (Felzenszwalb, 2004) ADAPTIVE GRAPH-CUT (our)
![Page 6: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/6.jpg)
6
Motivation: good / bad segmentation
Achanta, 2012 (SLIC); Kaufhold, 2004: segmentation algorithms aggregate sets of perceptually similar pixels in an image.
Felzenszwalb, 2004 (graph-cut): a segmentation algorithm should capture perceptually important groupings or regions, which often reflect global aspects of the image.
![Page 7: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/7.jpg)
7
Motivation: segmentation & video compression
Segment motion estimationFrame segmentation
Encoding True block and sub-block motion vectors
![Page 8: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/8.jpg)
8
Aim #1: use the human factor(aka segmentation quality metric)
![Page 9: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/9.jpg)
9
Aim #2: automatic parameter tuning
![Page 10: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/10.jpg)
10
Road map
1) Pick a segmentation algorithm…
2) … Learn a quality metric including the human factor (application needs) …
3) … And put them together (autotuning).
![Page 11: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/11.jpg)
11
Graph:
Nodes:
Edges:
Weights:
vi
vj
w(vi, vj)=0
w(vi, vj)>0
Graph-cut
w(vi, vj)>>0
Vvi
Evv ji ,
0, ji vvw
EVG ,
![Page 12: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/12.jpg)
12
Internal difference:
Graph-cut
Cm
ijCvvm wCIntmji ,max
![Page 13: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/13.jpg)
13
Difference between components:
Graph-cut
Cm
ijCvCvnm wCCDifnjmi ,min,
Cn
![Page 14: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/14.jpg)
14
Boundary predicate:
Graph-cut
Ck
Cn
nn
mmnm C
kCInt
C
kCIntCCDif ,min,
10 15 12
![Page 15: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/15.jpg)
15
Graph-cut
C1
C2
Boundary predicate:
nn
mmnm C
kCInt
C
kCIntCCDif ,min,
15 8 11
![Page 16: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/16.jpg)
16
Graph-cut
C1
C2
Boundary predicate:
Observation scale ~ k
nn
mmnm C
kCInt
C
kCIntCCDif ,min,
![Page 17: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/17.jpg)
17
Graph-cutK
= 3
K =
100
K =
10,
000
![Page 18: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/18.jpg)
18
Road map
1) Pick a segmentation algorithm…
2) … Learn a quality metric including the human factor…
3) … And put them together (autotuning).
![Page 19: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/19.jpg)
19
(Weighted) symmetric uncertainty
seg
BGRimg
BGR
BGRBGRBGR SS
segimgIU
,,,,
,,,,,,
,2
segB
segG
segR
segBB
segGG
segRR
w SSS
SUSUSUU
4 bits------------------ = 33%7 bits + 5 bits
Entropy based average
![Page 20: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/20.jpg)
20
k vs. Uw vs. quality
160 x 120 image block
![Page 21: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/21.jpg)
21
k vs. Uw vs. quality
Training
160 x 120 blocks
320x240 rgb images
K = [1, …, 10,000]
visual inspection & classification
![Page 22: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/22.jpg)
22
k vs. Uw vs. quality
Training
160 x 120 blocks
640x480 rgb images
K = [1, …, 10,000]
visual inspection & classification
![Page 23: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/23.jpg)
23
Learning the metric
WEUS N
iiWE
iWiN
iiUS
iWi
m
bUkm
m
bUkmbmE
1,2
,
1,2
,
1
log
1
log,
Uw = m log(k) + b 1, iWE
![Page 24: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/24.jpg)
24
Road map
1) Pick a segmentation algorithm…
2) … Learn a quality metric including the human factor…
3) … And put them together (autotuning).
![Page 25: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/25.jpg)
25
Automatic k selection
![Page 26: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/26.jpg)
26
Automatic k selection
![Page 27: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/27.jpg)
27
Automatic k selection
![Page 28: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/28.jpg)
28
Automatic k selection
![Page 29: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/29.jpg)
29
Automatic k selection
![Page 30: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/30.jpg)
30
… and adaptivity
k = k(x,y)
![Page 31: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/31.jpg)
31
Road map
![Page 32: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/32.jpg)
32
Results - Quality
Adaptive graph-cut (ours)
Graph-cut (Felzensswalb, 2004) *
SLIC (Achanta, 2012) *
* Same number of segments forced for each algorithm
![Page 33: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/33.jpg)
33
Results
![Page 34: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/34.jpg)
34
Results
SLIC
Graph-cut
Adaptive graph-cut
![Page 35: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/35.jpg)
35
Results
![Page 36: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/36.jpg)
36
Results
SLIC Graph-cut Adaptive graph-cut
![Page 37: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/37.jpg)
37
Results: inter-class contrast(the higher the better)
Sum of the contrasts among segments weighted by their areas (Chabrier, 2004)
Ad
ap
tive
gra
ph
-cu
t
Gra
ph
-cu
t
SL
IC
Inter class contrast
0
0.04
0.08
0.12
0.16
0.2
320x240
averagemedian
Ad
ap
tive
gra
ph
-cu
t
Gra
ph
-cu
t
SL
IC
Inter class contrast
00.020.040.060.080.10.120.140.160.18
640x480
averagemedian
![Page 38: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/38.jpg)
38
Results: intra-class uniformity(the lower the better)
Sum of the normalized standard deviation for each region (Chabrier, 2004)
Ad
ap
tive
gra
ph
-cu
t
Gra
ph
-cu
t
SL
IC
Intra class uniformity
0
2
4
6
8
10
12
14
320x240
averagemedian
Ad
ap
tive
gra
ph
-cu
t
Gra
ph
-cu
t
SL
IC
Intra class uniformity
051015202530354045
640x480
averagemedian
![Page 39: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/39.jpg)
39
Results: contrast - uniformity ratio(the higher the better)
Ad
ap
tive
gra
ph
-cu
t
Gra
ph
-cu
t
SL
IC
1000 * Inter / Intra
0
5
10
15
20
25
30
35
320x240
averagemedian
Ad
ap
tive
gra
ph
-cu
t
Gra
ph
-cu
t
SL
IC
1000 * Inter / Intra
0
2
4
6
8
10
12
14
640x480
averagemedian
![Page 40: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/40.jpg)
40
Discussion
LEARNED segmentation quality metric including the HUMAN FACTOR
Iterative method to AUTOMATICALLY and ADAPTIVELY compute the optimal scale parameter
![Page 41: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/41.jpg)
41
A more general approach(edge thresholding segmentation in YUV)
![Page 42: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/42.jpg)
42
A more general approach(edge thresholding segmentation in YUV)
Openboradcast encoding (x264)
Lyricallabs encoding (adaptive
segmentation)
Show
![Page 43: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/43.jpg)
43
A more general approach(edge thresholding segmentation in YUV)
Openboradcast encoding (x264) Lyricallabs encoding (adaptive segmentation)
Show
![Page 44: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/44.jpg)
44
Open issues & improvements
Resolution dependency (160x120 blocks)
Learning: the Berkeley Segmentation Dataset
Avoid iterations (see I. Frosio, SPIE EI 2015)
![Page 45: Adaptive Segmentation Based on a Learned Quality Metric I. Frosio 1, E. Ratner 2 1 NVIDIA, USA, 2 Lyrical Labs, USA](https://reader036.vdocument.in/reader036/viewer/2022062712/56649c745503460f949272a0/html5/thumbnails/45.jpg)
45
Questions
? ? ?