analyses of objects in images by computer vision ...tavares/downloads/... · analyses of objects in...
TRANSCRIPT
Analyses of objects in Images by Computer Vision: Techniques & Applications
João Manuel R. S. Tavares
[email protected] www.fe.up.pt/~tavares
Johann Bernoulli Institute for Mathematics and Computer Science
September 17, 2014
Outline
1. Introduction
2. Segmentation
3. Motion Tracking
4. Analysis of Objects: Matching, Morphing and Registration
5. 3D Reconstruction
6. Conclusions
7. Research Team
8. Publications & Events
2 Analyses of objects in Images by Computer Vision: Techniques & Applications João Manuel R. S. Tavares
Presentation
Presentation
• Associate Professor at the Faculty of Engineering of the University of Porto (FEUP) / Department of Mechanical Engineering
• Senior Research and Projects Coordinator of the Optics and Experimental Mechanics Lab (LOME) of the Institute of Mechanical Engineering and Industrial Management (INEGI)
• PhD and MSc degrees in Electrical and Computer Engineering from FEUP in 2001 and 1995, respectively
• BSc degree in Mechanical Engineering from FEUP in 1992 • Research Areas: Image Processing and Analysis, Medical
Imaging, Biomechanics, Human Posture and Control, Product Development
4 Analyses of objects in Images by Computer Vision: Techniques & Applications João Manuel R. S. Tavares
FEUP: Identity • With more than 80 years of history, FEUP is the largest school of the
University of Porto; since 2000, FEUP has moved to a brand new campus, just outside the city centre, in eastern Porto, Portugal
5 Analyses of objects in Images by Computer Vision: Techniques & Applications João Manuel R. S. Tavares
FEUP: PORTO World heritage
6 Analyses of objects in Images by Computer Vision: Techniques & Applications João Manuel R. S. Tavares
FEUP: In Figures (2012/2013)
7 Analyses of objects in Images by Computer Vision: Techniques & Applications João Manuel R. S. Tavares
FEUP: In Figures (2012/2013)
8 Analyses of objects in Images by Computer Vision: Techniques & Applications João Manuel R. S. Tavares
Introduction
• The researchers of Computational Vision aim the development of algorithms to perform in fully or semi-automatically manner operations and tasks carry out by the (quite complex) human’s vision system
10 Analyses of objects in Images by Computer Vision: Techniques & Applications
Original images Computational 3D models built voxelized and poligonized
Introduction
João Manuel R. S. Tavares
Azevedo et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(3):359-369
• Image processing and analysis are topics of the most importance for our Society
• Algorithms of image processing and analysis are frequently used, for example, in:
– Medicine – Biology – Industry – Natural Sciences – Engineering
• Examples of common tasks involving algorithms of image processing and analysis are:
– noise removal – geometric correction – segmentation, recognition (2D-4D) – motion tracking and analysis, including matching, registration and morphing (2D-4D) – 3D reconstruction
11 Analyses of objects in Images by Computer Vision: Techniques & Applications
Introduction
João Manuel R. S. Tavares
Introduction: Usual Computational Pipeline for Image Processing and Analysis
Analyses of objects in Images by Computer Vision: Techniques & Applications 12
Image(s) enhancement
Image(s) segmentation / features extraction
tracking
matching
morphing
Image(s)
motion analysis registration
image (pre)processing
image analysis / computational
vision João Manuel R. S. Tavares
3D vision
computer vision
Introduction • (Pre)processing of noisy images using an intelligent
worm
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 13
Original images, noisy corrupted images and smoothed using different smoothing methods
Araujo et al. (2014) Expert Systems with Applications 41(13):5892–5906
Segmentation
Segmentation • It is intended to identify in a full or semi- automatic manner
objects (2D/3D) presented in static images or in image sequences
• The most usual methodologies are based on template matching, statistical, geometric or physical modeling, or neuronal networks
• It is one of the most usual operations involved in the computational analysis of objects from images, and very often it is the first “important” step of image processing and analysis
• Frequent problems: noise, low resolution, reduce contrast, shapes not previously known, occlusion, multiple objects, etc.
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 15
Segmentation • Image segmentation by threshold (binarization)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 16
Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246
Segmentation • Example: segmentation of contours in dynamic
pedobarography (Otsu method, morphologic operators)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 17
Original images After segmentation
Bastos & Tavares (2004) Lecture Notes in Computer Science 3179:39-50
camera mirror
contact layer + glass
reflected light glass
pressure opaque layer
lamp
lamp transparent layer
Segmentation • Example: analysis of damage due to drill machining in
composite materials (binarization and region analysis)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 18
Original image After segmentation
Damaged area Measures obtained
Marques et al. (2009) Composites Science and Technology 69(14):2376-2382 Albuquerque et al. (2010) Journal of Composite Materials 44(9):1139-1159
Segmentation • Image segmentation by region growing
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 19
Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246
Segmentation • Example: segmentation of ear structures (region growing)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 20
Region Growing, x=215; y=254
Segmentation obtained (bony labyrinth)
Barroso et al. (2011) CNME 2011 Ferreira et al. (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(8):888-904
X: 254 Y: 214Index: 116.7RGB: 0.459, 0.459, 0.459
Original Image
Segmentation • Example: hardness evaluation from indentation images
(Johannsen & Bille threshold, region growing)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 21
Vickers hardness Brinell hadness
Filho et al. (2010) Journal of Testing and Evaluation 38(1):88-94
Segmentation • Segmentation of images using neuronal networks
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 22
Original images After segmentation (material microstructures )
Albuquerque et al. (2008) Nondestructive Testing and Evaluation 23(4):273-283 Albuquerque et al. (2009) NDT & E International 42(7):644-651
Segmentation • Example: evaluation of the nickel alloy secondary
phases from SEM images (neuronal network)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 23
Original image Image segmented
Albuquerque et al. (2011) Microscopy Research and Technique 74(1):36-46
Segmentation • Example: assessment of material porosity from optical
microscopic images (neuronal network)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 24
Original images with training pixels
Image segmented
Albuquerque et al. (2010) Journal of Microscopy 240(1):50-59
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 25
Segmentation • Segmentation of objects in images using image templates
Carvalho & Tavares (2005) CMNI 2005
×fft fft
ift
( )3ift D CC( )2ift D CC
max CC
Original image Template image
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 26
Segmentation • Segmentation of objects in images using deformable
templates
Carvalho & Tavares (2006) CompIMAGE 2006, 129-134 Carvalho & Tavares (2007) VipIMAGE 2007, 209-215
Example of a deformable template
Segmentation • Example: segmentation of facial features
(deformable geometric template)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 27
Original image and associated energy fields
Segmentation of the iris using a deformable template (a circle)
Segmentation of an eye using an
deformable template Carvalho & Tavares (2006) CompIMAGE 2006, 129-134 Carvalho & Tavares (2007) VipIMAGE 2007, 209-215
• Statistical modeling of objects in images (point distribution models)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 28
Vasconcelos & Tavares (2008) Computer Modeling in Engineering & Sciences 36(3):213-241
Segmentation
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 29
Segmentation • Segmentation of objects in images using active shape
models (point distribution model, optimization)
Vasconcelos & Tavares (2008) Computer Modeling in Engineering & Sciences 36(3):213-241
Segmentation • Example: segmentation of faces and hands in images
(active Shape Model)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 30
Segmentations achieved (initial, intermediate and final steps)
Vasconcelos & Tavares (2008) Computer Modeling in Engineering & Sciences 36(3):213-241
Segmentation • Example: analysis of the vocal tract shape during speech
production from MR images (active shape model)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 31
Intermediate segmentation II
Original image
Final segmentation
Intermediate segmentation I
Vasconcelos et al. (2011) Journal of Voice 25(6):732-742
• Segmentation of objects in images using active appearance models (statistical models, optimization)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 32
Vasconcelos & Tavares (2008) Computer Modeling in Engineering & Sciences 36(3):213-241
Segmentation
Segmentation • Example: segmentation of faces in images (active appearance
model)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 33
Segmentations achieved (initial, intermediate and final steps)
Vasconcelos & Tavares (2008) Computer Modeling in Engineering & Sciences 36(3):213-241
Segmentation • Example: analysis of the vocal tract shape during speech
production from MR images (active appearance model)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 34
Intermediate segmentations
Initial segmentation
Final segmentation
Intermediate segmentations
Vasconcelos et al. (2011) Journal of Engineering in Medicine 225(1):68-76 Vasconcelos et al. (2012) Journal of Engineering in Medicine 226(3):185-196
• Segmentation of objects in images using active contours (i.e. snakes – parametric models)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 35
Tavares et al. (2009) International Journal for Computational Vision and Biomechanics 2(2):209-220
Segmentation
Segmentation • Example: segmentation of a medical image (active contours -
snakes)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 36
Original image and initial contour
Final contour
Tavares et al. (2009) International Journal for Computational Vision and Biomechanics 2(2):209-220
Segmentation • Example: segmentation of objects in images (deformable
contour, FEM, Lagrange equation)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 37
Original images and initial contours Final contours
rubber k = 200N/m 14s
Gonçalves et al. (2008) Computer Modeling in Engineering & Sciences 32(1):45-55
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 38
Segmentation • Segmentation of objects in images using level-set
method (geometrical models)
Ma et al. (2010) Medical Engineering & Physics 32(7):766-774 Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 39
Segmentation • Example: segmentation of the carotid bifurcation in
Doppler images (active contour and level-set)
Segmentation using the contour active method (Yessi’s model)
Segmentation using the level-set method (Chan-Vese’s model)
Silva et al. (2011) VipIMAGE 2011, 117-122 Santos et al. (2013) Expert Systems with Applications 40(16):6570-6579
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 40
Ma & Tavares (2014) ComIMAGE’14
Segmentation examples under different imaging conditions and different types of skin lesions
Segmentation • Example: segmentation of skin lesions in dermoscopic
images (level-set method, color spaces)
An illustration of the segmentation process
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 41
Segmentation • Segmentation of objects in images using level set method
+ prior knowledge
Ma et al. (2010) Medical Engineering & Physics 32(7):766-774 Ma et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(2):235-246
Segmentation • Example: segmentation of the pelvic floor in MR images
(level-set model, prior knowledge, shape influence field)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 42
Pelvic floor segmented Ma et al. (2010) Medical Engineering & Physics 32(7):766-774
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 43
Segmentation
Ma et al. (2013) Computers in Biology and Medicine 43(4):248-258 Ma et al. (2012) The Int. Journal for Numerical Methods in Biomedical Engineering 28(6-7):714-726
• Example: segmentation of organs of the female pelvic cavity in MRI images (level-set method, prior knowledge)
Segmentation of the bladder, vagina and anus from pelvic cavity images (3 examples)
• Example: segmentation of the bladder walls in MRI images (level-set method, prior knowledge)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 44
Segmentation
Ma et al. (2011) Annals of Biomedical Engineering 39(8):2287-2297
Segmentation of the interior and external walls of the bladder (3 examples)
Motion Tracking
Motion Tracking • It is intended to track the motion (and/or the deformation) of
objects along image sequences • In this area, the methodologies based on optical flow, block
matching and stochastic methods are widespread • Usually, it concerns the estimation of the motion involved,
the management of the features being tracked, and the analysis of the motion tracked as well as its quantification
• Usual problems: non-rigid motions, geometric distortions, non-constant illumination, occlusions, noise, multiple objects, etc.
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 46
Motion Tracking • Computational framework to
track features in image sequences (Kalman Filter or Unscented Kalman Filter, optimization, Mahalanobis distance, management model)
Analyses of objects in Images by Computer Vision: Techniques & Applications 47 João Manuel R. S. Tavares
Pinho et al. (2007) Int. Journal of Simulation Modelling 6(2):84-92 Pinho & Tavares (2009) VipIMAGE 2009, 299-304 Pinho & Tavares (2009) Computer Modeling in Engineering & Sciences 46(1):51-75
Motion Tracking • Example: tracking marks in gait analysis (Kalman filter,
Mahalanobis distance, optimization, management model)
Analyses of objects in Images by Computer Vision: Techniques & Applications 48
Prediction Uncertainty Area Measurement Correspondence Result
(5 frames)
João Manuel R. S. Tavares
Pinho et al. (2005) ICCB 2005, 915-926 Pinho & Tavares (2009) Computer Modeling in Engineering & Sciences 46(1):51-75
Motion Tracking • Example: tracking marks to detect gait
events (Kalman filter, Mahalanobis distance, optimization)
Analyses of objects in Images by Computer Vision: Techniques & Applications 49 João Manuel R. S. Tavares
Sousa et al. (2007) ISHF2007, 331-340 Sousa et al. (2007) ICCB2007, 291-296
50 Analyses of objects in Images by Computer Vision: Techniques & Applications
(547 frames)
Motion Tracking • Example: tracking mice in long image sequences (Kalman
filter, Mahalanobis distance, optimization, management model)
João Manuel R. S. Tavares
Pinho et al. (2005) LSCCS, Vol. 4A:463-466 Pinho et al. (2007) International Journal of Simulation Modelling 6(2):84-92
Analysis of Objects: Matching
Analysis of Objects • Matching
– It is regularly used in the computational analysis of objects from images, for example, to register (i.e. align) objects, recognize objects, attain 3D information, analyze the motion tracked, and so forth
– Generally, it is achieved by considering invariant objects’ characteristics, as curvature, or displacements in a global space (like in modal space)
– Common problems: occlusion, non-rigid deformations, high shape variations, etc.
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 52
Matching • Using physical or geometrical modeling and modal
matching
Analyses of objects in Images by Computer Vision: Techniques & Applications 53
Modeling (physical or geometrical)
Eigenvalues / eigenvectors computation
Matching matrix
assembly
Contour 1
Contour 2
Matches achievement (optimization)
Modeling (physical or geometrical)
Eigenvalues / eigenvectors computation
João Manuel R. S. Tavares
Bastos & Tavares (2006) Inverse Problems in Science and Engineering 14(5):529-541 Tavares & Bastos (2010) Progress in Computer Vision and Image Analysis 339-368
• Example: matching contours in dynamic pedobarography (FEM modeling, modal matching, optimization)
Matching
Analyses of objects in Images by Computer Vision: Techniques & Applications 54
Original images Matched contours
camera mirror
contact layer + glass
reflected light glass
pressure opaque layer
lamp
lamp transparent layer
João Manuel R. S. Tavares
Bastos & Tavares (2004) LNCS 3179:39-50 Tavares & Bastos (2010) Progress in Computer Vision and Image Analysis, 339-368
Matching • Example: matching contours and surfaces in dynamic
pedobarography (FEM modeling, modal analysis, optimization)
Analyses of objects in Images by Computer Vision: Techniques & Applications 55
Image of dynamic pedobarography
Tavares & Bastos (2005) Electronic Letters on Computer Vision and Image Analysis 5(3):1-20
Matching found between two contours
Matching found between two intensity (pressure) surfaces (2 views)
Matching found between iso-contours (2 views)
João Manuel R. S. Tavares
Analysis of Objects: Morphing
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 57
Analysis of Objects • Morphing (i.e. simulation)
– It is an especially used in Computer Graphics, but also very useful in the analysis of objects from images, for example, to estimate the deformation involved between two objects or between two configurations of an object, to simulate the transitions between two shapes acquired under a high temporal gap, etc.
– Normally, it is attained by considering simple geometric transformations
– However, when it must be considered the real behavior of the objects, physical methodologies and modeling as, for example, FEM, should be considered
• Common difficulties are related to the estimation of the involved forces and with the properties of the adopted (virtual) material
• The adequate matching of the objects is crucial
Morphing
Analyses of objects in Images by Computer Vision: Techniques & Applications 58
• Physical morphing/simulation of contours in images (FEM modeling, modal analysis, optimization, Lagrange equation)
João Manuel R. S. Tavares
• Example: morphing contours in images (FEM modeling, modal analysis, optimization, Lagrange equation)
Matching found
Deformations estimated
Morphing
Analyses of objects in Images by Computer Vision: Techniques & Applications 59
Gonçalves et al. (2008) Computer Modeling in Engineering & Sciences 32(1):45-55
Original images
João Manuel R. S. Tavares
Analysis of Objects: Registration
Analysis of Objects • Registration
– It is commonly required in order to compare objects in images acquired at different time instants or according to distinct conditions
– It is essential, for example, in Medicine to follow up the evaluation of patients’ diseases from images
– Usually, it is achieved by considering objects’ characteristic features, as points of maximum curvature, and their matching followed by the estimation of the involved transformation
– Common problems: key and invariant features not easily identified, occlusion, non-rigid deformations, severe shape variations, etc.
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 61
Registration • Registration of contours by contours matching,
optimization and dynamic programming
Analyses of objects in Images by Computer Vision: Techniques & Applications 62 João Manuel R. S. Tavares
The cost matrix is built based on geometric or physical principles
The matching is found based on the minimization of the sum of the costs associated to the possible correspondences
To search for the best matching is used an optimization assignment algorithm
Bastos & Tavares (2006) Inverse Problems in Science and Engineering 14(5):529-541 Oliveira & Tavares (2009) Computer Modeling in Engineering & Sciences 43(1):91-110 Oliveira, Tavares, Pataky (2009) Journal of Biomechanics 42(15):2620-2623
Registration • Example: registration of contours in images (geometrical
modeling, matching, optimization, dynamic programming)
Analyses of objects in Images by Computer Vision: Techniques & Applications 63
Original images and contours
Matched contours before registration
Matched contours after registration
Oliveira & Tavares (2009), Computer Modeling in Engineering & Sciences 43(1):91-110 João Manuel R. S. Tavares
Registration • Example: registration of images in pedobarography
(geometrical modeling, matching, optimization, dynamic programming)
Analyses of objects in Images by Computer Vision: Techniques & Applications 64
Original images and contours Contours and images before and after registration
Oliveira et al. (2009) Journal of Biomechanics 42(15):2620-2623
João Manuel R. S. Tavares
Registration: 2D, monomodal, intrasubject
Processing time: 0.5 s (AMD Turion64, 2.0 GHz, 1.0 GB of RAM)
Images dimension: 217x140 pixels
Fixed image and contour (MRI)
Moving image and contour (MRI)
Overlapped images before the registration
Overlapped images after the registration
Difference between the images after the registration
Correspondences found between the Corpus Callosum contours
Oliveira & Tavares (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(2):73-93 João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 65
Registration • Example: registration of brain images (geometrical modeling,
matching, optimization, dynamic programming)
Registration • Registration of images based on Fourier transform
Analyses of objects in Images by Computer Vision: Techniques & Applications 66
Original images Images before and after registration
João Manuel R. S. Tavares
Oliveira, Pataky, Tavares (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(6):731-740
Registration: 2D, monomodal, intrasubject
Processing time: 2.1 s (AMD Turion64, 2.0 GHz, 1.0 GB of RAM)
Images dimension: 221x257 pixels
Analyses of objects in Images by Computer Vision: Techniques & Applications 67 João Manuel R. S. Tavares
Registration • Example: registration of brain images (Fourier transform)
Registration • Registration based on the iterative search for the
parameters of the transformation that optimizes a similarity measure between the input images
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 68
Moving image Fixed image
Pre-registration transformation
(optional) Interpolator Similarity measure
Optimizer Geometric transformation
The optimization algorithm stops when a similarity criterion is achieved
Oliveira & Tavares (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(2):73-93
Registration
Analyses of objects in Images by Computer Vision: Techniques & Applications 69 João Manuel R. S. Tavares
• Example: registration of images in pedobarography (Hybrid method: Fourier transform based registration + optimization of a similarity measure)
Original images Images before and after registration
Oliveira & Tavares (2012) Medical & Biological Engineering & Computing 49,(3):313-323
Registration: 2D, multimodal, intrasubject (without pre-registration)
Similarity measure: MI
Processing time: 4.6 s (AMD Turion64, 2.0 GHz, 1.0 GB of RAM)
Images dimension: 246x234 pixels
Oliveira & Tavares (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(2):73-93 João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 70
Registration • Example: registration/fusion of head images (optimization of a
similarity measure)
A computational platform has been developed to assist biomechanical studies based on the registration of plantar pressure images, which can be used in:
– Foot segmentation – Foot classification: left/right,
high arched, flat, normal, … – Foot axis computation – Footprint indices computation – Posterior statistical analysis
Oliveira, Sousa, Santos, Tavares (2012) Computer Methods in Biomechanics and Biomedical Engineering 15(11):1181-1188
Registration • Example: applications in plantar pressure image studies
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 71
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 72
Registration • Registration using iterative optimization and a curved
transformation (based on B-splines)
Fixed image Moving image
Registered moving image
Pre-registration using a rigid transformation
New pre-registration using an affine transformation
Coarse registration based on B-splines
Fine registration based on B-splines
The registration based on B-splines is of the free-form
deformation type Oliveira & Tavares (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(2):73-93
“Checkerboard” of the slices before the registration (CT/MRI-PD, brain)
F F
F F F
F F F
M M M
M M M
M
M
(The “checkerboard” slice is built by interchanging square patches of both slices and preserving their original spatial position in the fixed (F) and moving (M) slices)
Registration • Example: registration/fusion using iterative optimization
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 73
Registration: 3D, multimodal, intrasubject; Similarity measure: MI
Checkerboard of the slices after the registration (CT/MRI-PD, brain)
Registration • Example: registration/fusion using iterative optimization (cont.)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 74
Checkerboard of the slices (CT, thorax, Δt: 8.5 months) before the registration
Oliveira & Tavares (2014) Computer Methods in Biomechanics and Biomedical Engineering 17(2):73-93
Registration • Example: registration using iterative optimization
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 75
Registration: 3D, monomodal, intrasubject; Similatity measure: MI
Checkerboard of the slices (CT, thorax, Δt: 8.5 months) after a cubic B-spline based reg.
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 76
Registration • Example: registration using iterative optimization (cont.)
Brain DaTSCAN SPECT images are used to assist the diagnosis of the Parkinson’s disease and to distinguish it from other degenerative diseases. The solution developed is able to:
– Segment the relevant areas and perform dimensional analysis – Quantify the binding potential of the basal ganglia – Computation of statistical data relatively to a reference population – Image classification for diagnosis purposes
Normal Alzheimer Idiopathic Parkinsonism
Essential tremor
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 77
Registration • Example: application in brain DaTSCAN SPECT images
Mean slice from the population used as
reference
Corresponding slice of a patient
Difference of intensities
Z-scores mapping over the slice (red – high Z-scores)
3D volume images are automatically registered and statistical analysis relatively to a reference population can be attained
(The blue rectangles represent the 3D ROIs used to compute the binding potentials)
Oliveira et al. (2014) The Quarterly Journal of Nuclear Medicine and Molecular Imaging 58(1):74-84 João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 78
Registration • Example: application in brain DaTSCAN SPECT images
Basal ganglia from a mean
image of a normal population
Basal ganglia from a patient with idiopathic
Parkinson’s disease
Basal ganglia from a patient with vascular
Parkinson’s disease
Basal ganglia 3D shape reconstruction and quantification
Oliveira et al. (2014) The Quarterly Journal of Nuclear Medicine and Molecular Imaging 58(1):74-84
Registration • Example: application in brain DaTSCAN SPECT images
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 79
Three slices (coronal, sagittal and axial) after registration and identification of the potential lesion
3D visualization after CT/SPECT fusion (the lesion identified in the SPECT
slices is indicated)
Registration • Example: application in brain DaTSCAN SPECT/CT fusion
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 80
Fully automated segmentation and classification of the images based on image registration and an artificial classifier
Template image (top), segmented image (bottom-left) and artery mapping (bottom-right)
Registration • Example: application in gated myocardial perfusion
SPECT images
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 81
Oliveira, Faria, Tavares (2014) Proceedings of the Institution of Mechanical Engineers, Part H: Journal of Engineering in Medicine 228(8):810-818
3D surface of the incus and malleus surface built TC slices with the incus and malleus ossicles (inside the red
ellipse) to be segmented
Registration • Example: application in the fully automated segmentation
of the incus and malleus ear ossicles in conventional CT images
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 82
Registration • Registration of image sequences: spatial and temporal
registration
Analyses of objects in Images by Computer Vision: Techniques & Applications 83 João Manuel R. S. Tavares
Moving sequence
Fixed sequence
Apply the spatio & temporal
transformation
Compute the similarity measure
Optimizer Build the spatio &
temporal transformation
Oliveira, Sousa, Santos, Tavares (2011) Medical & Biological Engineering & Computing 49(7):843-850 Oliveira & Tavares (2013) Medical & Biological Engineering & Computing 51(3):267-276
Build the temporal representative
images
Search for the transformation that register the temporal
representative images
Estimate the linear temporal
registration
Pre-registration Registration optimization
Registration • Example: registration of image sequences in dynamic
pedobarography (spatial and temporal registration)
Analyses of objects in Images by Computer Vision: Techniques & Applications João Manuel R. S. Tavares 84
Device: Light reflection (25 fps, resolution 30 pixels/cm2)
Image similarity measure: MSD
Sequences dimension: 160x288x22, 160x288x25
Processing time: 1 min (using an AMD Turion64, 2.0 GHz, 1.0 GB of RAM)
Template sequence
Source sequence
Overlapped sequences
Before the registration
After the registration
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 85
Device: EMED (25 fps, resolution: 2 pixels/cm2, images dimension: 32x55x13; 32x55x18)
Registration: rigid (spatial), polynomial (temporal); similarity measure: MSD
Processing time: 4 s - AMD Turion64, 2.0 GHz, 1.0 GB of RAM
Fixed sequence
Moving sequence
Overlapped sequences
Before the registration
After the registration
Registration • Example: registration of image sequences in dynamic
pedobarography (spatial and temporal registration)
3D Reconstruction
3D Reconstruction • It is intended to accomplish the 3D reconstruction of objects or
scenes from images • In this area, the following methodologies are common:
external shapes – active techniques (with energy projection or relative motion), passive techniques (without energy projection nor relative motion) and of space carving; inner shapes – 2D segmentation (contours, for example) and data interpolation
• Usually, it involves tasks of camera calibration, data segmentation, matching, triangulation, interpolation and fusion
• Common problems: geometric distortions, bad or unstable illumination, occlusion, noise, multiple objects, complex shapes and topologies, etc.
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 87
3D Reconstruction • 3D reconstruction of organs from medical images based
on 2D segmentation, loft, smooth and Delaunay
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 88
Segmentation done in a 2D slice
Pelvic floor reconstructed
Reconstructed organs from a pelvic
cavity
Pimenta et al. (2006) CompIMAGE 2006, 343-348 Alexandre et al. (2007) VipIMAGE 2007, 359-362
slices
3D Reconstruction • 3D reconstruction of scenes using techniques of active
vision (dense stereo vision)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 89
Disparity map obtained
Original image pair
Azevedo et al. (2006) VISAPP 2006, 383-388
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 90
Azevedo et al. (2008) Advances in Computational Vision and Medical Image Processing: Methods and Applications, 117-136
3D Reconstruction • 3D reconstruction of objects by space carving
Pattern and object turntable image sequence
Pattern image sequence
Background/object segmentation
Camera calibration
Volumetric 3D reconstruction
3D model polygonization
3D Reconstruction • Example: 3D reconstruction of objects by space carving
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 91
Azevedo et al. (2008) Advances in Computational Vision and Medical Image Processing: Methods and Applications, 117-136 Azevedo et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(3):359-369
Original images Computational 3D models built voxelized and poligonized
3D Reconstruction • Example: 3D reconstruction of objects by space carving
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 92
Original images Computational 3D models built voxelized and poligonized
Azevedo et al. (2010) Computer Methods in Biomechanics and Biomedical Engineering 13(3):359-369
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 93
Axial and sagittal T2-weighted MR images
3D Reconstruction of the bladder by fusion data from the axial and sagittal images (2 views)
Ma et al. (2013) Medical Engineering & Physics 35(12):1819-1824
3D Reconstruction • Example: 3D reconstruction from multiple views
(registration/fusion)
• Example: 3D reconstruction of the spine from two orthogonal X-ray images using a deformable model (atlas)
João Manuel R. S. Tavares Analyses of objects in Images by Computer Vision: Techniques & Applications 94
Moura et al. (2010) Computer Modeling in Engineering & Sciences 60(2):115-138 Moura et al. (2011) Medical Engineering & Physics 33(8):924-933
Interface developed Adjusted model (2 views) and reconstruction obtained
3D Reconstruction
Conclusions
Conclusions
• The area of image processing and analysis is very complex and demand, but of raised importance in many domains
• Numerous hard challenges exist, as for example, adverse conditions in the image acquisition process, occlusion, objects with complicate shapes, with topological variations or undergoing complex motions
• Considerable work has already been developed, but important and complex goals still to be reached
• Methods and methodologies of other research areas, as of Mathematics, Computational Mechanics, Medicine and Biology, can contribute significantly for their reaching
• For that, collaborations are welcome
96 Analyses of objects in Images by Computer Vision: Techniques & Applications João Manuel R. S. Tavares
Acknowledgments
The work presented has been done with the support of Fundação para a Ciência e a Tecnologia, in Portugal, mainly trough the funding of the research projects:
– PTDC/BBB-BMD/3088/2012 – PTDC/SAU-BEB/102547/2008 – PTDC/SAU-BEB/104992/2008 – PTDC/EEA-CRO/103320/2008 – UTAustin/CA/0047/2008 – UTAustin/MAT/0009/2008 – PDTC/EME-PME/81229/2006 – PDTC/SAU-BEB/71459/2006 – POSC/EEA-SRI/55386/2004
97 Analyses of objects in Images by Computer Vision: Techniques & Applications João Manuel R. S. Tavares
Research Team (Computational Vision)
Research Team (Computational Vision)
• Post-Doc students (3): – Finished: Alexandre Carvalho – In course: Zhen Ma, Simone Prado
• PhD students (14): – Finished: Zhen Ma, Francisco Oliveira, Teresa Azevedo, Daniel Moura, Sandra
Rua – In course: Maria Vasconcelos, João Nunes, Alex Araujo, Carlos Gulo, Roberta
Oliveira, Danilo Jodas, Pedro Morais, Andre Pilastri, Nuno Sousa • MSc students (27):
– Finished: Carolina Tabuas, Jorge Pereira, Luis Ribeiro, Luis Ferro, Rita Teixeira, Liliana Azevedo, Diana Cidre, Célia Cruz, Priscila Alves, Pedro Gomes, Nuno Sousa, Diogo Faria, Elisa Barroso, Ana Jesus, Frederico Jacobs, Gabriela Queirós, Daniela Sousa, Francisco Oliveira, Teresa Azevedo, Maria Vasconcelos, Raquel Pinho, Luísa Bastos, Cândida Coelho, Jorge Gonçalves
– In course: Raquel Alves, André Silva, Silva Bessa • BSc students (2)
– Finished: Ricardo Ferreira, Soraia Pimenta
99 Analyses of objects in Images by Computer Vision: Techniques & Applications João Manuel R. S. Tavares
Publications & Events
Taylor & Francis journal “Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization”
www.tandfonline.com/tciv
Lecture Notes in Computational Vision and Biomechanics (LNCV&B) Series Editors: João Manuel R. S. Tavares, Renato Natal Jorge ISSN: 2212-9391, Publisher: SPRINGER
www.springer.com/series/8910
Webpage (www.fe.up.pt/~tavares)
Analyses of objects in Images by Computer Vision: Techniques & Applications
João Manuel R. S. Tavares
[email protected] www.fe.up.pt/~tavares