semi-supervised deep learning techniques for spectrum
TRANSCRIPT
R&D Center Europe – Stuttgart Lab 1 Copyright 2020 Sony Corporation1
Adriano Simonetto, Pietro Zanuttigh
University of Padova, Dept. of Information Engineering
[email protected], [email protected]
Vincent Parret, Piergiorgio Sartor, Alexander Gatto
Stuttgart Lab 1 of R&D Center, Sony Europe B.V.
{Vincent.Parret, Piergiorgio.Sartor, Alexander.Gatto}@sony.com
Semi-supervised Deep Learning Techniquesfor Spectrum Reconstruction
ICPR 2020
University of Padova
R&D Center Europe – Stuttgart Lab 1 Copyright 2020 Sony Corporation2
Hyperspectral Images and ApplicationsHyperspectral imaging and its applications
▪ Hyperspectral (HS) images provide much more info than RGB images
▪ Many applications benefit from the additional information
Food inspection Health care Recycling
Hyperspectral imagers
Illustration: HSI
?
University of Padova
R&D Center Europe – Stuttgart Lab 1 Copyright 2020 Sony Corporation3
Hyperspectral Images and ApplicationsSpectrum reconstruction from single RGB image
Spectrum Reconstruction
DNN
Wavelength [nm]
400 500 600 700
Norm
aliz
ed
reflecta
nce
0
0.1
0.2
0.3
0.4
0.5
Sky
Man-made
Blue man-made container
Blue sky
Illustration: ICVL
The network implicitly make use of the semantic of the scene.
However, this problem is very ill-posed.
University of Padova
R&D Center Europe – Stuttgart Lab 1 Copyright 2020 Sony Corporation4
Hyperspectral Images and ApplicationsSpectrum reconstruction from single RGB image
Spectrum Reconstruction
DNN
Residual network architecture and training pipeline of the HSCNN-R [1], leading the NTIRE 2018 challenge [2].
[1] Z. Shi, C. Chen, Z. Xiong, D. Liu, and F. Wu, “Hscnn+: Advancedcnn-based hyperspectral recovery from rgb images”in Conference on Computer Vision and Pattern Recognition Workshops, 2018
[2] Arad, O. Ben-Shahar, and R. Timofte, “Ntire 2018 challenge onspectral reconstruction from rgb images” in Conference on Computer Vision and Pattern Recognition Workshops, 2018
Fully supervised HS images required. Not usable in practice.
Illustration: ICVLUniversity of Padova
R&D Center Europe – Stuttgart Lab 1 Copyright 2020 Sony Corporation5
Hyperspectral Images and ApplicationsProposed semi-supervised approaches
Physical model for the inverse transformation Semi-supervised
Unsupervised Training (𝑻𝒖)
Semi-supervised Training with Pixels (𝑻𝒑)
Semi-supervised Training with Images (𝑻𝒊)
Illustration: ICVLUniversity of Padova
R&D Center Europe – Stuttgart Lab 1 Copyright 2020 Sony Corporation6
Unsupervised Training (𝑻𝒖)
DNN
HSI prediction
𝐼
𝐻
LHSI
Ti
Training pipeline
Semi-supervised Training with Images (𝑻𝒊)
Semi-supervised Training with Pixels (𝑻𝒑)
𝐻𝐻𝑙
Compare RGBand assign HSI
Tp
HSI to RGB
መ𝐼
LRGB
Tu LTVM+
University of Padova
R&D Center Europe – Stuttgart Lab 1 Copyright 2020 Sony Corporation7
Unsupervised Training (𝑻𝒖)
DNN
HSI prediction
𝐼
HSI to RGB
መ𝐼
LRGB
Tu LTVM+
LHSI
Training pipeline
Semi-supervised Training with Images (𝑻𝒊)
Semi-supervised Training with Pixels (𝑻𝒑)
University of Padova
R&D Center Europe – Stuttgart Lab 1 Copyright 2020 Sony Corporation8
Results and benchmark with the literature
HS camera (100 images)
Literature: fully supervised
Pixels from HS camera (3 ⋅ 106 pixels)
Literature: fully supervised
RGB camera (100 images)
Spectrometer (100-10k points)
+
Our: semi-supervised (𝐓𝒑)
HS camera (10 images) RGB camera (100 images)
+
Our: semi-supervised (𝑻𝒊)
The approaches were tested on the ICVL database [3].
[3] Arad and O. Ben-Shahar, “Sparse recovery of
hyperspectral signal from natural rgb images” in
European Conference on Computer Vision, 2016. University of Padova
R&D Center Europe – Stuttgart Lab 1 Copyright 2020 Sony Corporation9
Examples of spectra
University of Padova
R&D Center Europe – Stuttgart Lab 1 Copyright 2020 Sony Corporation10
Examples of images
Trained on 100 HD HS images
University of Padova
R&D Center Europe – Stuttgart Lab 1 Copyright 2020 Sony Corporation11
Conclusion
▪ More results and investigations in the paper and the supplementary material➢ Approaches selected from literature➢ More visual examples➢ Fine-tuning ➢ DNN vs non-DL
▪ Our approaches allow to train with very limited supervision, making it usable in practice
▪ Our physical model is the key component; it allows to use information from the RGB domain
▪ Potential future work➢ Include adversarial training➢ Test in real environment
▪ They outperform or reach comparable accuracy to the fully supervised approaches
University of Padova
R&D Center Europe – Stuttgart Lab 1 Copyright 2020 Sony Corporation12
Hyperspectral Images and Applications
Thank you for watching !
We are available for questions or suggestions throughout the conference, or per email.
Adriano Simonetto, Pietro Zanuttigh
University of Padova, Dept. of Information Engineering
[email protected], [email protected]
Vincent Parret, Piergiorgio Sartor, Alexander Gatto
Stuttgart Lab 1 of R&D Center, Sony Europe B.V.
{Vincent.Parret, Piergiorgio.Sartor, Alexander.Gatto}@sony.com
University of Padova