Download - ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces
![Page 1: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/1.jpg)
Digital Days 29/6/2001
ISTORAMA: A Content-Based Image Search Engine andHierarchical Triangulation of 3D Surfaces.
Dr. Ioannis Kompatsiaris
Centre for Research and Technology Hellas
Informatics and Telematics Institute
Thermi-Thessaloniki, Greece
![Page 2: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/2.jpg)
Digital Days 29/6/2001
Outline• Introduction• Istorama architecture• K-Means with Connectivity Constraint Algorithm (KMCC)• Demo• Object/model based coding• Adaptive Triangulation and Progressive transmission• Reduced pyramid - quincunx sampling• Experimental results• Conclusions
![Page 3: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/3.jpg)
Digital Days 29/6/2001
Need for efficient image search
• Huge number of images or databases of images
• Highly visual and graphical nature of the Web
• Text descriptors are not always efficient
• Greater flexibility with “content-based” access
• Queries which are more natural to humans
![Page 4: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/4.jpg)
Digital Days 29/6/2001
Proposed approach
• Usually a description, a “signature” or a set indexes is created for the whole image
• Images usually contain different objects• Proposed approach: the image is first separated
into objects (segmentation)• Descriptors are created for each object• The user can search for a specific object
contained in images
![Page 5: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/5.jpg)
Digital Days 29/6/2001
ISTORAMA architecture
Server
World Wide Web
Data BaseJDBC
Java Data Base Connection
User
PHP
Crawler - Spider
Indexing - Retrieval Algorithms
![Page 6: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/6.jpg)
Digital Days 29/6/2001
The K-Means with Connectivity Constraint Algorithm (KMCC) I
• Based on K-Means algorithm• K-Means does not take into account spatial
information• In KMCC, the spatial proximity of each region is
also taken into account by defining a new spatial center and by integrating the K-Means with a component labeling procedure
• Automatic correction of the number of regions KK
![Page 7: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/7.jpg)
Digital Days 29/6/2001
The K-Means with Connectivity Constraint Algorithm (KMCC) II
• Step1 K-Means is performed • Step2 Spatial centers are calculated
• Step3 Generalised distance
• Step 4 Component labeling LL connected regions
kCI
k
k
Sk
I AAkD
CSpCIpIp
22
1)(),(
kCS
![Page 8: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/8.jpg)
Digital Days 29/6/2001
The K-Means with Connectivity Constraint Algorithm (KMCC) III
![Page 9: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/9.jpg)
Digital Days 29/6/2001
Object descriptors
• Color, texture and spatial characteristics
• Color: histogram, 8 bins
• Spatial: (centroid),
• Shape: area, eccentricity
where λ1, λ2 are the two first eigenvalues
kCS
2
11
![Page 10: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/10.jpg)
Digital Days 29/6/2001
Experimental Results (Synthetic)
![Page 11: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/11.jpg)
Digital Days 29/6/2001
Experimental Results (Synthetic)
![Page 12: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/12.jpg)
Digital Days 29/6/2001
Experimental Results
![Page 13: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/13.jpg)
Digital Days 29/6/2001
Experimental Results (Claire)
Facial region
Moving object
Original sequenceFrames 1-
10
Segmentation
![Page 14: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/14.jpg)
Digital Days 29/6/2001
Experimental results (Claire)
![Page 15: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/15.jpg)
Digital Days 29/6/2001
Experimental results (table-tennis)
Original sequenceFrames 1-10
![Page 16: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/16.jpg)
Digital Days 29/6/2001
Experimental results (table-tennis)
Segmentation
Moving objects
![Page 17: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/17.jpg)
Digital Days 29/6/2001
Experimental results (Akiyo+Foreman)
Facial region
Facial region
Original sequenceFrames 1-
10
Original sequenceFrames 1-
10
![Page 18: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/18.jpg)
Digital Days 29/6/2001
Conclusions
• K-means with spatial proximity algorithm• Multiple features segmentation• Higher order segmentation• Correspondence of objects between consequent
frames• Max-min criterion for automatic regularisation
parameters
![Page 19: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/19.jpg)
Digital Days 29/6/2001
Future work
• Use of texture
• Indexing of video
• Integration with text descriptors
![Page 20: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/20.jpg)
Digital Days 29/6/2001
• Triangular meshes of high quality are used in:
• Computer Aided Design • 3D representation of objects
(e.g. archaeological artifacts)• Animation and visual simulation• Entertainment (computer games)• Digital Terrain Modelling
Introduction
![Page 21: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/21.jpg)
Digital Days 29/6/2001
Object/model-based coding
![Page 22: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/22.jpg)
Digital Days 29/6/2001
Object/model-based coding
![Page 23: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/23.jpg)
Digital Days 29/6/2001
Object/model-based coding
![Page 24: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/24.jpg)
Digital Days 29/6/2001
Compression of finely detailed surfaces is necessary for:
• computation
• storage
• transmission
• display efficiency
Adaptive triangulation
![Page 25: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/25.jpg)
Digital Days 29/6/2001
• Early, coarse approximations are refined though additional bits
Progressive transmission
![Page 26: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/26.jpg)
Digital Days 29/6/2001
• Vertices removal and retriangulation [Schroeder] [Cohen]
• General mesh optimization process/function [Hoppe]
• Multiresolution analysis (MRA) [Lounsbery]
• Wavelets [Schroeder] [Gross]
• Progressive transmission [Schroeder] [Hoppe]
• Generalized triangle mesh representation [Deering]
Background
![Page 27: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/27.jpg)
Digital Days 29/6/2001
Properties of the algorithm
• Efficient compression of the wireframe information• Simplification of the wireframe by adaptive
triangulation• Progressive transmission of the wireframe
information• Prioritised transmission of the wireframe• Straightforward correspondence between
successive scales
![Page 28: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/28.jpg)
Digital Days 29/6/2001
Input surfaces
• Surface represented as a parametric function
in the parametric space
• determined by the position of a set of control points or nodes
• It allows for arbitrary, possibly closed wire-frame surfaces to be defined.
TvuzvuyvuxvuP ),(),,(),,(),(
2R
Tklklkl zyxlkr ,,),(
![Page 29: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/29.jpg)
Digital Days 29/6/2001
Input surfaces
• The filters are applied to the 2D parametric representation of the surface as though it were a 2D image with intensity equal to
• Such surfaces include also:• depth images estimated from stereo pairs and• every surface that is homomorphic to a plane,
cylinder or torus
),( vuP
),( vuz
![Page 30: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/30.jpg)
Digital Days 29/6/2001
Block diagram of the proposed procedure
![Page 31: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/31.jpg)
Digital Days 29/6/2001
Reduced pyramid with quincunx sampling matrix
![Page 32: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/32.jpg)
Digital Days 29/6/2001
Corresponding triangulation
![Page 33: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/33.jpg)
Digital Days 29/6/2001
Optimal filters
• Optimal filters are determined by their Fourier transform:
• where is the power spectral density.
• Alternatively may be determined by the equation:
1,,1,0,1
021
Nr
eeeG M
iqj
rM
jrj
ri
T Mw
ww
Mk)(ig
wjr e
tktMMkpMtk
,)()(r
irir RgR
![Page 34: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/34.jpg)
Digital Days 29/6/2001
Optimum bit allocation
• bits/vertex is assumed to be transmitted
• bits/vertex are allocated to each level using
• is the sum of error variances
BB rr 2
2
B
rB
2
r
ir
0
22
![Page 35: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/35.jpg)
Digital Days 29/6/2001
Error prioritization
• The prediction errors corresponding to all predicted vertices are calculated and sorted with the vertices corresponding to higher errors being put first on the list
Higher Errors
Lower Errors
![Page 36: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/36.jpg)
Digital Days 29/6/2001
Entropy estimation
• Entropy coding is used• The number of bits needed for error transmission
is the entropy of the errors • Using the quincunx sampling geometry at the
receiver, there is no need to transmit the exact co-ordinates of the position of each transmitted vertex
• The final cost of the transmission is the sum of the error entropy and the position entropy
![Page 37: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/37.jpg)
Digital Days 29/6/2001
Adaptive Triangulation Procedure
• Synthesis stage of the QMVINT pyramid
• The vertex along with the vertices used to predict it are added to the mesh
• Handling of cracks
• Triangulation of the next vertex
)()(ˆ)( )()()( rk
rk
rk PerrPIPI
)(rkP
![Page 38: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/38.jpg)
Digital Days 29/6/2001
Adaptive triangulation procedure
![Page 39: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/39.jpg)
Digital Days 29/6/2001
Adaptive triangulation procedure
![Page 40: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/40.jpg)
Digital Days 29/6/2001
Experimental results
• Original dense depth map and surface of the “Venus” data
![Page 41: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/41.jpg)
Digital Days 29/6/2001
Experimental results
• 2569 vertices and 4006 triangles at level 2 MSE = 1.30
![Page 42: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/42.jpg)
Digital Days 29/6/2001
Experimental results
• 7661 vertices and 11135 triangles at level 1 MSE = 1.30
![Page 43: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/43.jpg)
Digital Days 29/6/2001
Experimental results
• 11416 vertices and 15827 triangles at level 0 MSE = 0.12
![Page 44: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/44.jpg)
Digital Days 29/6/2001
Experimental results
![Page 45: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/45.jpg)
Digital Days 29/6/2001
Conclusions
• Hierarchical representation of 3D surfaces using 3D adaptive triangular wireframes
• The variance of the error transmitted is minimised and therefore results to optimal compression of the wireframe information
• It produces a hierarchy where coarse meshes are as similar to their finer versions as is possible
![Page 46: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/46.jpg)
Digital Days 29/6/2001
Conclusions
• The triangulation algorithm is integrated with a bit allocation procedure
• The number of nodes and triangles of the wireframe as well as the information needed for the transmission or storage of the wireframe are reduced simultaneously using a unified approach (QMVINT filtering)
• Precise correspondence between triangles at each level is achieved
![Page 47: ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces](https://reader035.vdocument.in/reader035/viewer/2022062521/56814c7d550346895db999cc/html5/thumbnails/47.jpg)
Digital Days 29/6/2001
Future work
• Expansion and application directly to 3D surfaces
• Estimation of filters