graph signal analysis: imaging, learning, samplinggenec/2019/uofwaterloo_gsp...•image denoising...
TRANSCRIPT
![Page 1: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/1.jpg)
Graph Signal Analysis: Imaging, Learning, Sampling
Gene Cheung
Associate Professor, York University
17th June, 2019
1
.
Univ. of Waterloo 6/17/2019
![Page 2: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/2.jpg)
AcknowledgementCollaborators:
• X. Liu (HIT, China)
• W. Hu, W. Gao (Peking U., China)
• L. Fang (Tsinghua, China)
• Y. Zhao (BJTU, China)
• C.-W. Lin (National Tsing Hua University, Taiwan)
• A. Ortega (USC, USA)
• D. Florencio (MSR, USA)
• J. Liang, I. Bajic (SFU, Canada)
• X. Wu (McMaster U, Canada)
• P. Frossard (EPFL, Switzerland)
• V. Stankovic (U of Strathclyde, UK)
• Y. Nakatsukasa (Oxford, UK)
• P. Le Callet (U of Nantes, France)
2Univ. of Waterloo 6/17/2019
![Page 3: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/3.jpg)
Introducing math tools
Students in EECS4452: “This is math, not engineering!”
3Univ. of Waterloo 6/17/2019
engineering math
![Page 4: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/4.jpg)
Introducing math tools
Students in EECS4452: “This is math, not engineering!”
3Univ. of Waterloo 6/17/2019
math
engineering
Me: “Math is the heart of engineering!”
![Page 5: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/5.jpg)
Outline
• Defining Graph frequencies
• Inverse Imaging• Image denoising
• Image contrast enhancement
• 3D point cloud denoising / super-resolution
• Deep GLR
• Semi-Supervised Learning
• Graph Sampling• Matrix completion
4Univ. of Waterloo 6/17/2019
![Page 6: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/6.jpg)
Outline
• Defining Graph frequencies
• Inverse Imaging• Image denoising
• Image contrast enhancement
• 3D point cloud denoising / super-resolution
• Deep GLR
• Semi-Supervised Learning
• Graph Sampling• Matrix completion
5Univ. of Waterloo 6/17/2019
![Page 7: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/7.jpg)
Signal Decomposition
• Decompose signal into basic components:
• Newton decomposed white light into color components (1730).
6
=k
kkX x
Univ. of Waterloo 6/17/2019
![Page 8: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/8.jpg)
Signal Decomposition
• Decompose signal into basic components:
• Newton decomposed white light into color components (1730).
6
=k
kkX x
( ) dtetxX
eXx
ktj
k
k
ktj
k
2
2
−
=
=
• “Basic” components can be complex exponentials:
Univ. of Waterloo 6/17/2019
![Page 9: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/9.jpg)
Signal Decomposition
• Decompose signal into basic components:
• Newton decomposed white light into color components (1730).
6
=k
kkX x
( ) dtetxX
eXx
ktj
k
k
ktj
k
2
2
−
=
=
• “Basic” components can be complex exponentials:
• Complex exponentials are eigenfunctions of 2nd derivative operator.Univ. of Waterloo 6/17/2019
![Page 10: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/10.jpg)
Digital Signal Processing
• Discrete signals on regular data kernels.• Ex.1: audio on regularly sampled timeline.
• Ex.2: image on 2D grid.
• Harmonic analysis tools (transforms, wavelets):
• Compression, restoration, segmentation, etc.
7Univ. of Waterloo 6/17/2019
xa =desired signal
transformsparse
transform coeff.
2D DCT basis
![Page 11: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/11.jpg)
Graph Signal Processing
• Signals on irregular data kernels described by graphs.• Graph: nodes and edges.
• Edges reveals node-to-node relationships.
1. Data domain is naturally a graph.
2. Embed similarity structure into graph.• Eigenvectors have global info aggregated from local info.
8
Graph Signal Processing (GSP) provides spectral
analysis tools for signals residing on graphs.
[1] A. Ortega, P. Frossard, J. Kovacevic, J. M. F. Moura, and P. Vandergheynst, “Graph signal processing: Overview, challenges,
and applications,” Proceedings of the IEEE, vol. 106, no. 5, pp. 808–828, 2018.
signal on graph kernel
[2] G. Cheung, E. Magli, Y. Tanaka, and M. K. Ng, “Graph spectral image processing,” Proceedings of the IEEE, vol. 106, no.
5, pp. 907–930, 2018.
signal on regular kernel
![Page 12: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/12.jpg)
Graph Signal Processing
• Signals on irregular data kernels described by graphs.• Graph: nodes and edges.
• Edges reveals node-to-node relationships.
1. Data domain is naturally a graph.
2. Embed similarity structure into graph.• Eigenvectors have global info aggregated from local info.
8
Graph Signal Processing (GSP) provides spectral
analysis tools for signals residing on graphs.
[1] A. Ortega, P. Frossard, J. Kovacevic, J. M. F. Moura, and P. Vandergheynst, “Graph signal processing: Overview, challenges,
and applications,” Proceedings of the IEEE, vol. 106, no. 5, pp. 808–828, 2018.
signal on graph kernel
[2] G. Cheung, E. Magli, Y. Tanaka, and M. K. Ng, “Graph spectral image processing,” Proceedings of the IEEE, vol. 106, no.
5, pp. 907–930, 2018.
signal on graph kernel
![Page 13: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/13.jpg)
Graph Fourier Transform (GFT)
Graph Laplacian:
• Adjacency Matrix A: entry Ai,j has non-negativeedge weight wi,j connecting nodes i and j.
• Degree Matrix D: diagonal matrix w/ entry Di,i
being sum of column entries in row i of A.
• Combinatorial Graph Laplacian L: L = D-A• L is related to 2nd derivative.
• L is a differential operator on graph.
9
=j
jiii AD ,,
1 2 3 42,1w
=
0100
1010
010
000
A2,1
2,1
w
w
+=
1000
0200
0010
000
D2,1
2,1
w
w
1 1
−
−−
−+−
−
=
1100
1210
011
00
L2,12,1
2,12,1
ww
ww
*https://en.wikipedia.org/wiki/Second_derivative
( )( ) ( ) ( )
20
2lim
h
hxfxfhxfxf
h
−+−+=
→
undirected graph
432:,3 2x xxxL −+−=
Univ. of Waterloo 6/17/2019
![Page 14: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/14.jpg)
Graph Spectrum from GFT
10
1st AC eigenvector
1 2 3 4 8…2,1w 1 1
TVVL =
eigenvalues along diagonal
eigenvectors in columns
GFT xVx~ T=
• Graph Fourier Transform (GFT) is eigen-matrix of graph Laplacian L.
1. Eigenvectors aggregates info from weights.
• Constant eigenvector is DC.
• # zero-crossings increases as λ increases.
2. Eigenvalues (≥ 0) as graph frequencies.
• GFT defaults to DCT for un-weighted connected line.
• GFT defaults to DFT for un-weighted connected circle.
GFT coefficients
Univ. of Waterloo 6/17/2019
![Page 15: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/15.jpg)
Graph Spectrum from GFT
10
1st AC eigenvector
1 2 3 4 8…2,1w 1 1
TVVL =
eigenvalues along diagonal
eigenvectors in columns
GFT xVx~ T=
• Graph Fourier Transform (GFT) is eigen-matrix of graph Laplacian L.
1. Eigenvectors aggregates info from weights.
• Constant eigenvector is DC.
• # zero-crossings increases as λ increases.
2. Eigenvalues (≥ 0) as graph frequencies.
• GFT defaults to DCT for un-weighted connected line.
• GFT defaults to DFT for un-weighted connected circle.
GFT coefficients
Univ. of Waterloo 6/17/2019
![Page 16: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/16.jpg)
V1: DC component
Graph Frequency Examples (US Temperature)
11
location diff.
−−=
2
2
2, exp
ji
ji
llw
Edge weights
*https://en.wikipedia.org/wiki/Delaunay triangulation
• Weather stations from 100 most populated cities.
• Graph connections from Delaunay Triangulation*.
• Edge weights inverse proportion to distance.
Univ. of Waterloo 6/17/2019
![Page 17: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/17.jpg)
Graph Frequency Examples (US Temperature)
11
location diff.
−−=
2
2
2, exp
ji
ji
llw
Edge weights
*https://en.wikipedia.org/wiki/Delaunay triangulation
• Weather stations from 100 most populated cities.
• Graph connections from Delaunay Triangulation*.
• Edge weights inverse proportion to distance.
V2: 1st AC component
Univ. of Waterloo 6/17/2019
![Page 18: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/18.jpg)
Graph Frequency Examples (US Temperature)
11
location diff.
−−=
2
2
2, exp
ji
ji
llw
Edge weights
*https://en.wikipedia.org/wiki/Delaunay triangulation
• Weather stations from 100 most populated cities.
• Graph connections from Delaunay Triangulation*.
• Edge weights inverse proportion to distance.
V3: 2nd AC component
Univ. of Waterloo 6/17/2019
![Page 19: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/19.jpg)
Graph Frequency Examples (US Temperature)
11
location diff.
−−=
2
2
2, exp
ji
ji
llw
Edge weights
*https://en.wikipedia.org/wiki/Delaunay triangulation
• Weather stations from 100 most populated cities.
• Graph connections from Delaunay Triangulation*.
• Edge weights inverse proportion to distance.
V4: 9th AC component
Univ. of Waterloo 6/17/2019
![Page 20: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/20.jpg)
Outline
• Defining Graph frequencies
• Inverse Imaging• Image denoising
• Image contrast enhancement
• 3D point cloud denoising / super-resolution
• Deep GLR
• Semi-Supervised Learning
• Graph Sampling• Matrix completion
12Univ. of Waterloo 6/17/2019
![Page 21: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/21.jpg)
Graph Laplacian Regularizer
• (graph Laplacian regularizer) [1]) is one smoothness measure.
• Signal Denoising:
• MAP Formulation:
13
( ) =−=k
kk
ji
jiji
T xxxw 2
,
2
,~
2
1Lxx
xLxT
noise
desired signalobservation
vxy +=
xLxxymin2
2
T
x+−
smoothness prior
fidelity term
signal smooth in
nodal domain
signal contains
mostly low graph freq.
[1] P. Milanfar, “A Tour of Modern Image Filtering: New Insights and Methods, Both Practical and Theoretical,” IEEE Signal
Processing Magazine, vol.30, no.1, pp.106-128, January 2013.
( ) yxLI * =+ linear system of eqn’s w/ sparse, symmetric PD matrix
update edge
weights
![Page 22: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/22.jpg)
Graph Laplacian Regularizer
• (graph Laplacian regularizer) [1]) is one smoothness measure.
• Signal Denoising:
• MAP Formulation:
13
( ) =−=k
kk
ji
jiji
T xxxw 2
,
2
,~
2
1Lxx
xLxT
vxy +=
xLxxymin2
2
T
x+−
signal smooth in
nodal domain
signal contains
mostly low graph freq.
[1] P. Milanfar, “A Tour of Modern Image Filtering: New Insights and Methods, Both Practical and Theoretical,” IEEE Signal
Processing Magazine, vol.30, no.1, pp.106-128, January 2013.
( ) yxLI * =+ linear system of eqn’s w/ sparse, symmetric PD matrix
update edge
weights
pixel intensity diff. pixel location diff.
−−
−−=
2
2
2
2
2
1
2
2, expexp
jiji
ji
llxxw
Bilateral filter weights
![Page 23: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/23.jpg)
14
Results: natural image denoising
• Subjective comparisons ( )40 =I
Original Noisy, 16.48 dB K-SVD, 26.84 dB
BM3D, 27.99 dB PLOW, 28.11 dB OGLR, 28.35 dB
[1] J. Pang, G. Cheung, "Graph Laplacian Regularization for Image Denoising: Analysis in the Continuous Domain," IEEE
Transactions on Image Processing, vol. 26, no.4, pp.1770-1785, April 2017.
![Page 24: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/24.jpg)
15
• Subjective comparisons ( )30 =I
Original Noisy, 18.66 dB BM3D, 33.26 dB NLGBT, 33.41dB OGLR, 34.32 dB
Results: depth image denoising
[1] W. Hu et al., "Depth Map Denoising using Graph-based Transform and Group Sparsity," IEEE International Workshop on
Multimedia Signal Processing, Pula (Sardinia), Italy, October, 2013.
![Page 25: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/25.jpg)
GLR for Joint Dequantization / Contrast Enhancement
• Retinex decomposition model:
• Objective: general smoothness for luminance, smoothness w/ negative edges for reflectance.
• Constraints: quantization bin constraints
• Solution: Alternating accelerated proximal gradient alg [1].
16[1] X. Liu, G. Cheung, X. Ji, D. Zhao, W. Gao, "Graph-based Joint Dequantization and Contrast Enhancement of
Poorly Lit JPEG Images," IEEE Transactions on Image Processing, vol. 28, no.3, pp.1205-1219, March 2019.
illumination
reflectance
scalar
noise
generalized smoothpiecewise smooth
![Page 26: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/26.jpg)
Results: Contrast Enhancement
17
![Page 27: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/27.jpg)
Results: Contrast Enhancement
18
![Page 28: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/28.jpg)
Results: Contrast Enhancement
19Univ. of Waterloo 6/17/2019
![Page 29: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/29.jpg)
GTV for Point Cloud Denoising
• Acquisition of point cloud introduces noise.
• Point cloud is irregularly sampled 2D manifold in 3D space.
• Not appropriate to apply GTV directly on 3D coordinates [1]. • only a singular 3D point has zero GTV value.
• Proposal: Apply GTV is to the surface normals of 3D point cloud—a generalization of TV to 3D geometry.
20[1] Y. Schoenenberger, J. Paratte, and P. Vandergheynst, “Graph-based denoising for time-varying point clouds,” in
IEEE 3DTV-Conference, 2015, pp. 1–4
x
f(x)
x
y
functional
smoothness
−−i
ii xfxf )()( 1
geometry
smoothness
−−i
ii yy 1
![Page 30: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/30.jpg)
GTV for Point Cloud Denoising
• Acquisition of point cloud introduces noise.
• Point cloud is irregularly sampled 2D manifold in 3D space.
• Not appropriate to apply GTV directly on 3D coordinates [1]. • only a singular 3D point has zero GTV value.
• Proposal: Apply GTV is to the surface normals of 3D point cloud—a generalization of TV to 3D geometry.
20[1] Y. Schoenenberger, J. Paratte, and P. Vandergheynst, “Graph-based denoising for time-varying point clouds,” in
IEEE 3DTV-Conference, 2015, pp. 1–4
x
f(x)
x
y
functional
smoothness
−−i
ii xfxf )()( 1
geometry
smoothness
−−i
ii yy 1
![Page 31: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/31.jpg)
GTV for Point Cloud Denoising
• Acquisition of point cloud introduces noise.
• Point cloud is irregularly sampled 2D manifold in 3D space.
• Not appropriate to apply GTV directly on 3D coordinates [1]. • only a singular 3D point has zero GTV value.
• Proposal: Apply GTV is to the surface normals of 3D point cloud—a generalization of TV to 3D geometry.
20[1] Y. Schoenenberger, J. Paratte, and P. Vandergheynst, “Graph-based denoising for time-varying point clouds,” in
IEEE 3DTV-Conference, 2015, pp. 1–4
x
f(x)
x
y
functional
smoothness
−−i
ii xfxf )()( 1
geometry
smoothness
−−i
ii yy 1 −−i
ii 11nn
![Page 32: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/32.jpg)
PC Denoising Algorithm
• Use GTV of surface normals over the K-NN graph:
• Denoising problem as l2-norm fidelity plus GTV of surface normals:
• Surface normal estimation of n𝒊 is a nonlinear function of p𝒊 and neighbors.
Proposal:
1. Partition point cloud into two independent classes (say red and blue).
2. When computing surface normal for a red node, use only neighboring blue points.
3. Solve convex optimization for red (blue) nodes alternately.
𝑖 𝑗
n𝒊 n𝒋
21[1] C. Dinesh, G. Cheung, I. V. Bajic, C. Yang, “Fast 3D Point Cloud Denoising via Bipartite Graph Approximation
& Total Variation,” IEEE 20th International Workshop on Multimedia Signal Processing, Vancouver, Canada, August 2018.
−+−Eji
jijiw,
1,
2
2n,pnnpqmin
smoothness on surface normals
![Page 33: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/33.jpg)
Results: Point Cloud DenoisingAnchor model (𝜎=0.3)
22
![Page 34: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/34.jpg)
Results: Point Cloud Denoising
Daratech model (𝜎=0.3)
23
![Page 35: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/35.jpg)
PC Super-Res Algorithm
• Add new interior points to low-res point cloud. 1. Construct triangular mesh using Delaunay triangulation using known points q.
2. Insert new points at the centroids of triangles.
• Partition point cloud into two independent classes (say red and blue).
• When computing normal for a red node, use only neighboring blue points.
• Use graph total variation (GTV) of surface normals over the K-NN graph:
• Solved via augmented Lagrangian + ADMM.
24[1] C. Dinesh, G. Cheung, I. V. Bajic, C. Yang, “3D Point Cloud Super-Resolution via Graph Total Variation on
Surface Normals,” accepted to IEEE International Conference on Image Processing, Taiwan, October 2019.
−Eji
jijiw,
1,n,p
nnmin
smoothness on surface normals
=
−
q
v
p
m
C0
BI
jiji nnm , −=
![Page 36: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/36.jpg)
Results: Point Cloud Super-Resolution
25
• APSS and RIMLS schemes generate overly smooth models.
• Existing methods result in distorted surfaces with some details lost.
[1] C. Dinesh, G. Cheung, I. V. Bajic, “3D Point Cloud Super-Resolution via Graph Total Variation on Surface
Normals,” accepted to IEEE International Conference on Image Processing, October 2019.
![Page 37: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/37.jpg)
26
Results: Point Cloud Super-Resolution
![Page 38: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/38.jpg)
Outline
• Defining Graph frequencies
• Inverse Imaging• Image denoising
• Image contrast enhancement
• 3D point cloud denoising / super-resolution
• Deep GLR
• Semi-Supervised Learning
• Graph Sampling• Matrix completion
27Univ. of Waterloo 6/17/2019
![Page 39: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/39.jpg)
Unrolling Graph Laplacian Regularizer
• Recall MAP formulation of denoising problem with quadratic graph Laplacian regularizer:
• Solution is system of linear equations:
28
xLxxymin2
2
T
x+−
smoothness priorfidelity term
( ) yxLI * =+
linear system of eqn’s w/ sparse, symmetric PD matrix
[1] J. Pang, G. Cheung, "Graph Laplacian Regularization for Inverse Imaging: Analysis in the Continuous Domain," IEEE
Transactions on Image Processing, vol. 26, no.4, pp.1770-1785, April 2017.
![Page 40: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/40.jpg)
Unrolling Graph Laplacian Regularizer
• Recall MAP formulation of denoising problem with quadratic graph Laplacian regularizer:
• Solution is system of linear equations:
28
xLxxymin2
2
T
x+−
smoothness priorfidelity term
( ) yxLI * =+
linear system of eqn’s w/ sparse, symmetric PD matrix
[1] J. Pang, G. Cheung, "Graph Laplacian Regularization for Inverse Imaging: Analysis in the Continuous Domain," IEEE
Transactions on Image Processing, vol. 26, no.4, pp.1770-1785, April 2017.
Q: what is the “most appropriate” graph?
![Page 41: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/41.jpg)
Unrolling Graph Laplacian Regularizer
• Recall MAP formulation of denoising problem with quadratic graph Laplacian regularizer:
• Solution is system of linear equations:
28
xLxxymin2
2
T
x+−
smoothness priorfidelity term
( ) yxLI * =+
linear system of eqn’s w/ sparse, symmetric PD matrix
[1] J. Pang, G. Cheung, "Graph Laplacian Regularization for Inverse Imaging: Analysis in the Continuous Domain," IEEE
Transactions on Image Processing, vol. 26, no.4, pp.1770-1785, April 2017.
Q: what is the “most appropriate” graph?
−−
−−=
2
2
2
2
2
1
2
2, expexp
jiji
ji
llxxw
Bilateral weights:
![Page 42: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/42.jpg)
Unrolling Graph Laplacian Regularizer
• Deep Graph Laplacian Regularization:1. Learn features f’s using CNN.
2. Compute distance from features.
3. Compute edge weights using Gaussian kernel.
4. Construct graph, solve QP.
29
[2] K. Gregor and Y. LeCun, “Learning fast approximations of sparse coding,” in Proc. 27th Int. Conf. Machine Learning, 2010..
features
pre-filter
weight para
[1] M. McCann et al., “Convolutional Neural Networks for Inverse Problems in Imaging,“ IEEE SPM, Nov. 2017.
![Page 43: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/43.jpg)
Unrolling Graph Laplacian Regularizer
30
[1] J. Zeng et al., “Deep Graph Laplacian Regularization for Robust Denoising of Images,” accepted to NTIRE Workshop, CVPR 2019.
Univ. of Waterloo 6/17/2019
![Page 44: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/44.jpg)
Unrolling Graph Laplacian Regularizer
• Graph Model guarantees numerical stability of solution:
• Thm 1: condition number κ of matrix satisfies [1]:
• Observation: By restricting search space of CNN to degree-bounded graphs, we achieve robust learning.
31
( ) yxLI * =+
maximum node degree
[1] J. Zeng et al., “Deep Graph Laplacian Regularization for Robust Denoising of Images,” accepted to NTIRE Workshop, CVPR 2019.
Univ. of Waterloo 6/17/2019
![Page 45: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/45.jpg)
Experimental Results – Numerical Comparison
32
• Trained on AWGN on 5 images, patches of size 26-by-26.
• Batch size is 4, model is trained for 200 epochs.
• Trained for both known and blind noise variance.
[1] Kai Zhang et al, “Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising,” TIP 2017.
[2] Marc Lebrun et al, “The noise clinic: a blind image denoising algorithm,” IPOL 2015.
![Page 46: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/46.jpg)
Experimental Results – Numerical Comparison
33
• Cross-domain generalization.
• trained on Gaussian noise, tested on low-light images in (RENOIR).
• Competing methods: DnCNN [1], noise clinic [2].
• outperformed DnCNN by 5.74 dB, and noise clinic by 1.87 dB.
[1] Kai Zhang et al, “Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising,” TIP 2017.
[2] Marc Lebrun et al, “The noise clinic: a blind image denoising algorithm,” IPOL 2015.
![Page 47: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/47.jpg)
Experimental Results – Visual Comparison
34
• trained on Gaussian noise, tested on low-light images in (RENOIR).
• Competing methods: DnCNN [1], noise clinic [2].
• outperformed DnCNN by 5.74 dB, and noise clinic by 1.87 dB.
[1] Kai Zhang et al, “Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising,” TIP 2017.
[2] Marc Lebrun et al, “The noise clinic: a blind image denoising algorithm,” IPOL 2015.
CDnCNNNoise Clinic DeepGLR
![Page 48: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/48.jpg)
Experimental Results – Visual Comparison
35
• trained on Gaussian noise, tested on low-light images in (RENOIR).
• Competing methods: DnCNN [1], noise clinic [2].
• outperformed DnCNN by 5.74 dB, and noise clinic by 1.87 dB.
[1] Kai Zhang et al, “Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising,” TIP 2017.
[2] Marc Lebrun et al, “The noise clinic: a blind image denoising algorithm,” IPOL 2015.
CDnCNNNoise Clinic DeepGLR
![Page 49: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/49.jpg)
Outline
• Defining Graph frequencies
• Inverse Imaging• Image denoising• Image contrast enhancement• 3D point cloud denoising / super-resolution
• Deep GLR
• Semi-Supervised Learning
• Graph Sampling• Matrix completion
36Univ. of Waterloo 6/17/2019
![Page 50: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/50.jpg)
Semi-Supervised Graph Classifier Learning
• Binary Classifier: given feature vector xi of dimension K, compute f(xi) ∊ {0,1}.
• Classifier Learning: given partial, noisy labels (xi, yi), train classifier f(xi).
• GSP Approach [1]:
1. Construct signed similarity graph with +/- edges.
2. Pose MAP graph-signal restoration problem.
3. Perturb graph Laplacian to ensure PSD.
4. Solve num. stable MAP as sparse lin. system.
37
[1] Yu Mao, Gene Cheung, Chia-Wen Lin, Yusheng Ji, “Image Classifier Learning from Noisy Labels via Generalized Graph
Smoothness Priors,” IEEE IVMSP Workshop, Bordeaux, France, July 2016. (Best student paper award)
example graph-based classifier
[2] G. Cheung, W.-T. Su, Y. Mao, C.-W. Lin, "Robust Semi-Supervised Graph Classifier Learning with Negative Edge Weights," IEEE
Transactions on Signal and Information Processing over Networks, vol. 4, no.4, pp.712-726, December 2018.
![Page 51: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/51.jpg)
Semi-Supervised Graph Classifier Learning
• Binary Classifier: given feature vector xi of dimension K, compute f(xi) ∊ {0,1}.
• Classifier Learning: given partial, noisy labels (xi, yi), train classifier f(xi).
• GSP Approach [1]:
1. Construct signed similarity graph with +/- edges.
2. Pose MAP graph-signal restoration problem.
3. Perturb graph Laplacian to ensure PSD.
4. Solve num. stable MAP as sparse lin. system.
37
[1] Yu Mao, Gene Cheung, Chia-Wen Lin, Yusheng Ji, “Image Classifier Learning from Noisy Labels via Generalized Graph
Smoothness Priors,” IEEE IVMSP Workshop, Bordeaux, France, July 2016. (Best student paper award)
example graph-based classifier
[2] G. Cheung, W.-T. Su, Y. Mao, C.-W. Lin, "Robust Semi-Supervised Graph Classifier Learning with Negative Edge Weights," IEEE
Transactions on Signal and Information Processing over Networks, vol. 4, no.4, pp.712-726, December 2018.
-1
![Page 52: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/52.jpg)
Graph-Signal Smoothness Priorfor signed graphs
• Graph Laplacian Regularizer [1]:
• Promote large / small inter-node differences depending on edge signs.
• Sensible, but numerically unstable.
38
[1] J. Pang and G. Cheung, “Graph Laplacian regularization for image denoising: Analysis inn the continuous
domain,” in IEEE Transactions on Image Processing, vol. 26, no.4, April 2017, pp. 1770–1785.
w=1
eigenvalues / graph freqs
GFT coeff
Promote large differencePromote small difference
![Page 53: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/53.jpg)
Semi-Supervised Learning Formulation
• MAP formulation:
• One sol’n is △=λmin I, i.e. shift all eigenvalues up by η=λmin.
• Intuition: signal variations + signal energies
39
l0 fidelity term smoothness prior for
signed graph
perturbation matrix
to ensure PSD!
[1] G. Cheung, W.-T. Su, Y. Mao, C.-W. Lin, "Robust Semi-Supervised Graph Classifier Learning with Negative Edge Weights," IEEE
Transactions on Signal and Information Processing over Networks, vol. 4, no.4, pp.712-726, December 2018.
![Page 54: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/54.jpg)
Results: Semi-Supervised Learning
• Comparisons w/ other classifiers:
Univ. of Waterloo 6/17/2019 40
![Page 55: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/55.jpg)
Results: Semi-Supervised Learning
• Comparisons w/ other classifiers:
Univ. of Waterloo 6/17/2019 40
![Page 56: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/56.jpg)
Results: Semi-Supervised Learning
• Comparisons w/ other classifiers:
Univ. of Waterloo 6/17/2019 41
![Page 57: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/57.jpg)
Results: Semi-Supervised Learning
• Comparisons w/ other classifiers:
Univ. of Waterloo 6/17/2019 41
![Page 58: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/58.jpg)
Outline
• Defining Graph frequencies
• Inverse Imaging• Image denoising• Image contrast enhancement• 3D point cloud denoising / super-resolution
• Deep GLR
• Semi-Supervised Learning
• Graph Sampling• Matrix completion
42Univ. of Waterloo 6/17/2019
![Page 59: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/59.jpg)
Graph Sampling (with and without noise)
• Q: How to choose best samples for graph-based reconstruction?
43[1] A. Anis, A. Gadde, and A. Ortega, “Efficient sampling set selection for bandlimited graph signals using graph spectral proxies,”
IEEE Transactions on Signal Processing, vol. 64, no. 14, pp. 3775–3789, 2016.
• Existing graph sampling strategies extend
Nyquist sampling to graph data kernels:
• Assume bandlimited signal.
• Greedily select most “informative” samples by
computing extreme eigenvectors of sub-matrix.
• Computation-expensive.
![Page 60: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/60.jpg)
Related Works
44
Graph sampling
Selection sampling
Random selection [3]
Deterministic
Local measurement [2]
Aggressive sampling [1]
Bandlimitedness
Smoothness:
A-optimal(M. Tsitsvero.TSP2016; F. Wang. SPL2018)
E-optimal (S. Chen. TSP2015)
Spectral proxies(A. Anis. TSP2016; A. Anis. TIT2018)
Localized coverage(A. Sakiyama. TSP2019)
GMRF model(P. Chen. ICASSP2018; Y. Bai. ICASSP 2019)
Eigen-decomposition Free
[1] A. G. Marques, S. Segarra, G. Leus, and A. Ribeiro, “Sampling of graph signals with successive local aggregations.” IEEE
Transactions on Signal Processing, vol. 64, no. 7, pp. 1832–1843, 2016.
[2] X. Wang, J. Chen, and Y. Gu, “Local measurement and reconstruction for noisy bandlimited graph signals,” Signal Processing, vol.
129, pp. 119–129, 2016.
[3] G. Puy, N. Tremblay, R. Gribonval, and P. Vandergheynst, “Random sampling of bandlimited signals on graphs,” Applied and
Computational Harmonic Analysis, vol. 44, no. 2, pp. 446–475, 2018.
![Page 61: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/61.jpg)
Signal Reconstruction using GLR
• Signal Model:
• Signal prior is graph Laplacian regularizer (GLR) [1]:
• MAP Formulation:
45
𝐱𝑇Lx =1
2
𝑖,𝑗
𝑤𝑖,𝑗 𝑥𝑖 − 𝑥𝑗2=
𝑘
𝜆𝑘 𝑥𝑘2
noise
desired signalobservation
𝐲 =𝐇𝐱+𝐯
min𝐱
𝐲 − 𝐇𝐱 22 + 𝜇 𝐱𝑇𝐋𝐱
signal priorfidelity term
signal smooth w.r.t. graph
signal contains
mostly low graph freq.
𝐇𝑇𝐇 + 𝜇𝐋 𝐱∗ = 𝐲linear system of eqn’s solved using conjugate gradient
sampling matrix
[1] J. Pang, G. Cheung, "Graph Laplacian Regularization for Image Denoising: Analysis in the Continuous Domain," IEEE
Transactions on Image Processing, vol. 26, no.4, pp.1770-1785, April 2017.
1 2 3 41 11
𝐇 =0 1 0 00 0 0 1
Sample set {2, 4}
![Page 62: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/62.jpg)
Stability of Linear System
• Examine system of linear equations :
• Stability depends on the condition number (λmax/ λmin) of coeff. matrix B.
• λmax is upper-bounded by 1+μ2*dmax.
• Goal: select samples to maximize λmin
(without computing eigen-pairs)!
• Also minimizes worst-case MSE:
46
𝐇𝑇𝐇 + 𝜇𝐋 𝐱∗ = 𝐲
coefficient matrix B
1 2 3 41 11
𝐋 =
1 −1 0 0−1 2 −1 00 −1 2 −10 0 −1 1
Sample set {2, 4}
𝐇T𝐇 =
0 0 0 00 1 0 00 0 0 00 0 0 1
ො𝐱 − 𝐱 2 ≤ 𝜇1
𝜆𝑚𝑖𝑛 𝐁2
𝐋 𝐱 + 𝐧 2 + 𝐧 2
![Page 63: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/63.jpg)
Gershgorin Circle Theorem
• Gershgorin Circle Theorem:
• Row i of L maps to a Gershgorin
disc w/ centre Lii and radius Ri
• λmin is lower-bounded by smallest
left-ends of Gershgorin discs:
• Graph Laplacian L has all Gershgorin disc left-ends at 0 → L is psd.
47[1] Y. Bai, G. Cheung, F. Wang, X. Liu, W. Gao, "Reconstruction-Cognizant Graph Sampling Using Gershgorin Disc Alignment,"
IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK, May 2019.
min𝑖
𝐿𝑖,𝑖− 𝑅𝑖 ≤ 𝜆min
1 2 3 41 11
𝐋 =
1 −1 0 0−1 2 −1 00 −1 2 −10 0 −1 1
1 2 3 4
𝑅𝑖 =
𝑗≠𝑖
𝐿𝑖𝑗
![Page 64: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/64.jpg)
Gershgorin Disc Alignment (math)
• Main Idea: Select samples to max smallest disc left-end of coefficient matrix B:
• Sample node → shift disc.
• Consider similar transform of B:
• Scale row → expand disc radius.
→ shrink neighbors’ disc radius.
48[1] Y. Bai, G. Cheung, F. Wang, X. Liu, W. Gao, "Reconstruction-Cognizant Graph Sampling Using Gershgorin Disc Alignment,"
IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK, May 2019.
1 2 3 41 11
coeff. matrix
similarity transform
𝐁 =
1 −1 0 0−1 2 −1 00 −1 2 −10 0 −1 1
1 2 3 4
Sample set { }
Scale factor {1,1,1,1}diagonal matrix w/ scale factors
𝐁 =𝐇𝑇 𝐇+𝜇 𝐋
𝐂 =𝐒𝐁𝐒−1
![Page 65: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/65.jpg)
Gershgorin Disc Alignment (math)
• Main Idea: Select samples to max smallest disc left-end of coefficient matrix B:
• Sample node → shift disc.
• Consider similar transform of B:
• Scale row → expand disc radius.
→ shrink neighbors’ disc radius.
48[1] Y. Bai, G. Cheung, F. Wang, X. Liu, W. Gao, "Reconstruction-Cognizant Graph Sampling Using Gershgorin Disc Alignment,"
IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK, May 2019.
1 2 3 41 11
coeff. matrix
similarity transform
diagonal matrix w/ scale factors
𝐁 =
1 −1 0 0−1 3 −1 00 −1 2 −10 0 −1 1
1 2 3 4
Sample set {2}
Scale factor {1,1,1,1}
5
𝐁 =𝐇𝑇 𝐇+𝜇 𝐋
𝐂 =𝐒𝐁𝐒−1
![Page 66: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/66.jpg)
Gershgorin Disc Alignment (math)
• Main Idea: Select samples to max smallest disc left-end of coefficient matrix B:
• Sample node → shift disc.
• Consider similar transform of B:
• Scale row → expand disc radius.
→ shrink neighbors’ disc radius.
48[1] Y. Bai, G. Cheung, F. Wang, X. Liu, W. Gao, "Reconstruction-Cognizant Graph Sampling Using Gershgorin Disc Alignment,"
IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK, May 2019.
1 2 3 41 11
coeff. matrix
similarity transform
diagonal matrix w/ scale factors
𝐁 =
1 −1 0 0−1 3 −1 00 −1 2 −10 0 −1 1
1 2 3 4
Sample set {2}
Scale factor {1,s2,1,1}
5
𝐁 =𝐇𝑇 𝐇+𝜇 𝐋
𝐂 =𝐒𝐁𝐒−1
![Page 67: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/67.jpg)
Gershgorin Disc Alignment (math)
• Main Idea: Select samples to max smallest disc left-end of coefficient matrix B:
• Sample node → shift disc.
• Consider similar transform of B:
• Scale row → expand disc radius.
→ shrink neighbors’ disc radius.
48[1] Y. Bai, G. Cheung, F. Wang, X. Liu, W. Gao, "Reconstruction-Cognizant Graph Sampling Using Gershgorin Disc Alignment,"
IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK, May 2019.
1 2 3 41 11
coeff. matrix
similarity transform
diagonal matrix w/ scale factors
𝐁 =
1 −1 0 0−1 3 −1 00 −1 2 −10 0 −1 1
1 2 3 4
Sample set {2}
Scale factor {1,s2,1,1}
5
𝐁 =𝐇𝑇 𝐇+𝜇 𝐋
𝐂 =𝐒𝐁𝐒−1
![Page 68: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/68.jpg)
Gershgorin Disc Alignment (math)
• Breadth First Iterative Sampling:• Given initial node set, threshold T.
1. Sample chosen node i
(shift disc)
2. Scale row i
(expand disc radius i to T )
3. If disc left-end of connected node j >T, Scale row j(expand disc radius j to T )
Else,
Add node j to node set.
4. Goto 1 if node set not empty.
5. Output sample set and count K.
49[1] Y. Bai, G. Cheung, F. Wang, X. Liu, W. Gao, "Reconstruction-Cognizant Graph Sampling Using Gershgorin Disc Alignment,"
IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK, May 2019.
1 2 3 41 11
51
![Page 69: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/69.jpg)
Gershgorin Disc Alignment (math)
• Binary Search with BFIS:• Sample count K inverse proportional to threshold T.
• Binary search on T to drive count K to budget.
• Example: line graph with equal edge weight.• Uniform sampling.
50[1] Y. Bai, G. Cheung, F. Wang, X. Liu, W. Gao, "Reconstruction-Cognizant Graph Sampling Using Gershgorin Disc Alignment,"
IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK, May 2019.
![Page 70: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/70.jpg)
Disc-based Sampling (intuition)
• Analogy: throw pebbles into a pond.
• Disc Shifting: throw pebble at sample node i.
• Disc Scaling: ripple to neighbors of node i.
• Goal: Choose samples to maximize the ripple on the least disturbed node.
51[1] Y. Bai, G. Cheung, F. Wang, X. Liu, W. Gao, "Reconstruction-Cognizant Graph Sampling Using Gershgorin Disc Alignment,"
IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK, May 2019.
![Page 71: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/71.jpg)
Results: Graph Sampling
• GDA has larger λmin and smaller condition number.
• GDA has smaller reconstruction MSE.
• GDA has 10% of computation cost of Spectral proxies.
52[1] Y. Bai, G. Cheung, F. Wang, X. Liu, W. Gao, "Reconstruction-Cognizant Graph Sampling Using Gershgorin Disc Alignment,"
IEEE International Conference on Acoustics, Speech and Signal Processing, Brighton, UK, May 2019.
![Page 72: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/72.jpg)
Results: Graph Sampling• Journal version [1]:
1. formalize the max λmin GDA optimization problem, 2. derive linear-time heuristic based on set cover approx. alg.
• Small graphs: GDA has roughly the same reconstruction MSE.• Large graphs: GDA has smallest reconstruction MSE.
53[1] Y. Bai, F. Wang, G. Cheung, F. Wang, Y. Nakatsukasa, W. Gao, "Fast Sampling Set Selection for Graph Signals Using Gershgorin
Disc Alignment,“ in preparation for IEEE Transactions on Signal Processing.
![Page 73: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/73.jpg)
Matrix Completion
• Fill in missing entries in a matrix:(Low-rank matrix recovery problem)
• Examples of applications:
• Recommendation system—making rating prediction.
• Remote sensing—infer full covariance matrix from partial correlations.
• Structure-from-motion in computer vision.
54
1 2
4
3
2
3
( )
SjiMXts jiji
R NM
=
,,..
Xrankmin
,,
X
[1] E. Candes and Y. Plan, “Matrix completion with noise,” Proceedings of the IEEE, vol. 98, no.6, April 2010, pp. 925–936.
![Page 74: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/74.jpg)
Matrix Completion
• Convex relaxation to nuclear norm:
• Proximal Gradient: SVD plus singular value soft-thresholding.
• Use dual graph-signal smoothness prior to promote low rank [1]:
• Unconstrained convex objective solvable via ADMM, conjugate gradient.
55
SjiMXts jiji
R NM
=
,,..
Xmin
,,
*X
( ) ( ) 2
XXSMSXLXtrXLXtrmin
F
T
cr
T
R NM −++
Synthetic Netflix matrix
Graph Fourier transform on rL Graph Fourier transform on cL
[1] V. Kalofolias, X. Bresson, M. Bronstein, and P. Vandergheynst, “Matrix completion on graphs,”arXiv preprint arXiv:1408.1717, 2014.
![Page 75: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/75.jpg)
Noiseless synthetic rating matrix
Noisy synthetic rating matrixComparison methods: PG [1]; GWC-random [2]; LOC [3]
• [1] Guillermo Ortiz-Jiménez, Mario Coutino, Sundeep Prabhakar Chepuri, and Geert Leus. “Sampling and
reconstruction of signals on product graphs”. arXiv preprint arXiv:1807.00145, 2018.
• [2] G. Puy, N. Tremblay, R. Gribonval, and P. Vandergheynst, “Random sampling of bandlimited signals on graphs,”
Applied and Computational Harmonic Analysis, vol. 44, no. 2, pp. 446–475, 2018.
• [3] A. Sakiyama, Y. Tanaka, T. Tanaka, and A. Ortega, “Eigendecomposition-free sampling set selection for graph
signals,”IEEE Transactions on Signal Processing, 2019.
Figure: Reconstruction MSE of different sampling methods on synthetic dataset. The reconstruction method for matrix completion is dual graph smoothness based method.
(1) Noiseless (2) Noisy
Results: Sampling for matrix completion
56[1] F. Wang, C. Yang, G. Cheung, Y. Wang, “Graph Sampling for Matrix Completion Using Recurrent First Eigenvector
Computation,” arXiv, preprint arXiv:1906.01087, 2019.
![Page 76: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/76.jpg)
SVT [1] GRALS [2] GMC [3] NMC [4]
G1 1.021 | 1.031 0.947 | 0.931 1.036 | 1.037 0.890 | 0.888
G2 1.021 | 0.983 0.945 | 0.893 1.118 | 1.054 0.890 | 0.858
Table: RMSE of different matrix completion methods on Mocielens_100k dataset withdifferent sampling strategies on Feature-based graph (G1) and Content-based graph(G2). In each grid, the value on left side belongs to random sampling; the right side valueis of our proposed IGCS sampling. The best performance in each row is marked in boldand red. In our experiments, the sampling budget is 80k out of 100k available ratings; Wefirst use random 60k samples as given, and then proceed to sample the next 20k samplesbase on random sampling or the proposed IGCS sampling.
• [1] J. Cai, E. J. Candes, and Z. Shen. “A singular value thresholding algorithm for matrix completion”. preprint, 2008.
• [2] N. Rao, H.-F. Yu, P. K. Ravikumar, and I. S. Dhillon. “Collaborative filtering with graph information: Consistency
and scalable methods”. In Proc. NIPS, 2015.
• [3] V. Kalofolias, X. Bresson, M. M. Bronstein, and P. Vandergheynst.” Matrix completion on graphs. ”2014.
• [4] D. M. Nguyen, E. Tsiligianni, and N. Deligiannis, “Extendable neural matrix completion,” in Proc. IEEE Int. Conf.
Acoust., Speech Signal Process., 2018, pp. 1–5.
Results: Sampling for matrix completion
57[1] F. Wang, C. Yang, G. Cheung, Y. Wang, “Graph Sampling for Matrix Completion Using Recurrent First Eigenvector
Computation,” arXiv preprint arXiv:1906.01087, 2019.
![Page 77: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/77.jpg)
Summary
• Graph Spectral Analyis Tools• Similarity graph, graph frequencies.
58Univ. of Waterloo 6/17/2019
GSP
Image ProcessingDenoising, contrast enhancement, PC
denoising / SR
Maching LearningSemi-supervised graph
classifier learning
Data MiningFast sampling for matrix
completion
![Page 78: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/78.jpg)
Q&A
• Email: [email protected]• Homepage: https://www.eecs.yorku.ca/~genec/index.html
59Univ. of Waterloo 6/17/2019
![Page 79: Graph Signal Analysis: Imaging, Learning, Samplinggenec/2019/UofWaterloo_gsp...•Image denoising •Image contrast enhancement •3D point cloud denoising / super-resolution •Deep](https://reader033.vdocument.in/reader033/viewer/2022042213/5eb67b4c51f1520264627efa/html5/thumbnails/79.jpg)
GSP and Graph-related Research
GSP: SP framework that unifies concepts from multiple fields.
60
Graph Signal
Processing* (GSP)
Combinatorial
Graph Theory
Spectral
Graph Theory
Computer
Vision
Computer Graphics
Machine
Learning
spectral
clustering
eigen-analysis of
graph Laplacian,
adjacency matrices
graphical model,
manifold learning,
classifier learning
Laplace-
Beltrami
operator
Laplace
equationPartial Differential
Eq’ns
Max cut, graph
transformation
DSP
Univ. of Waterloo 6/17/2019