retrieval of multimedia objects using color segmentation and dimension reduction of features
DESCRIPTION
RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES. Mingming Lu, Qiyu Zhang, Wei-Hung Cheng, Cheng-Chang Lu Department of Computer Science Kent State University. Data Mining and Knowledge Management. Processing multimedia objects - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/1.jpg)
RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR
SEGMENTATION AND DIMENSION REDUCTION OF FEATURES
Mingming Lu, Qiyu Zhang, Wei-Hung Cheng, Cheng-Chang Lu
Department of Computer Science
Kent State University
![Page 2: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/2.jpg)
September, 2009 Kent State University 2
Data Mining and Knowledge Management
• Processing multimedia objects
• Defining and extracting features
• Feature dimension reduction
• Multimedia data retrieval
• Knowledge representation and management
![Page 3: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/3.jpg)
September, 2009 Kent State University 3
Current Tasks
• Off-line data training– Segment images – batch mode– Find region of interest (ROI)– Interface with feature extraction and analysis– Feature domain processing
![Page 4: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/4.jpg)
September, 2009 Kent State University 4
Current Tasks (cont.)
• Users Interfaces– Reading user-input images– Segmentation– Find ROI– Feature extraction of ROI– Compare with trained data in repository– Return data (images) satisfying certain criteria
![Page 5: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/5.jpg)
September, 2009 Kent State University 5
Data Training
Image &Feature DataRepository
Segmentation Finding ROI Interface Feature ExtractDimension Reduction
Sending Images for Processing Store Feature Data back
Image Domain Feature Domain
![Page 6: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/6.jpg)
September, 2009 Kent State University 6
Image Domain Procesisng
• Segmentation – Color VQ, Texture based image segmentation
• Find ROI – ROI occupies large area– ROI locates near the image center– ROI contains homogenous texture
![Page 7: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/7.jpg)
September, 2009 Kent State University 7
Color-Texture SegmentationApplications
• Identify Regions of Interest (ROI) in a scene
• Image classification
• Image annotation
• Object based image and video coding
![Page 8: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/8.jpg)
September, 2009 Kent State University 8
Color-Texture SegmentationCurrent Limitations
• Many existing techniques work well on homogeneous color regions, while natural scenes are rich in color and texture.
• Many texture segmentation algorithms require the estimation of texture model parameters, which is a difficult problem and often requires a good homogeneous region for robust estimation.
![Page 9: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/9.jpg)
September, 2009 Kent State University 9
Color-Texture SegmentationAdvantage of Color VQ and Texture
based segmentation • Does not attempt to estimate a specific
model for a texture region.
• Tests for the homogeneity of a given color-texture pattern, which is computationally more feasible than estimation of model parameters.
![Page 10: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/10.jpg)
September, 2009 Kent State University 10
Color-Texture SegmentationTwo-Step Process
• Color Quantization– Performed in the color space without
consideration of spatial distribution of colors.– Label each pixel with a quantized color to form
a class-map.
• Spatial Segmentation– Performed on the class-map
![Page 11: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/11.jpg)
September, 2009 Kent State University 11
Color-Texture SegmentationColor Quantization
• Use Peer Group Filtering
• As a result, coarse quantization can be obtained while preserving the color information in the original images.
• Usually 10-20 colors are needed in the images of natural scenes.
![Page 12: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/12.jpg)
September, 2009 Kent State University 12
Color-Texture SegmentationCriteria for Good Segmentation
WWT
W
C
i
C
i ZziiW
ZzT
SSSJ
S
mzSS
mzS
i
/)(
Define
class. same the tobelonging points of variance total theis
and
Let
1 1
2
2
![Page 13: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/13.jpg)
September, 2009 Kent State University 13
Color-Texture Segmentation-A Criterion for Good Segmentation
• When the color classes are more separated from each other, J is getting larger.
• If all color classes are uniformly distributed over the entire image, J tends to be small.
![Page 14: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/14.jpg)
September, 2009 Kent State University 14
Color-Texture SegmentationA Criterion for Good Segmentation
• Now let us recalculate J over each segmented region instead of the entire class-map and define the average by
• A segmentation which can minimize J is considered a good segmentation.
k
Kk JMN
J1
![Page 15: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/15.jpg)
September, 2009 Kent State University 15
Color-Texture Segmentation-Spatial Segmentation
• Seed Determination
• Seed Growing
• Region Merge
![Page 16: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/16.jpg)
September, 2009 Kent State University 16
Color-Texture Segmentation-Spatial Segmentation
![Page 17: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/17.jpg)
September, 2009 Kent State University 17
ROI Determination
• Find ROI – Mechanism– Pixel closer to the center contributes more
weight to the region it belongs to.
– Region with more pixels tends to get higher weight
dw
1
![Page 18: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/18.jpg)
September, 2009 Kent State University 18
Results of Image Domain Processing
• Results of Color Quantization
• Results of Finding ROI
![Page 19: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/19.jpg)
Results of Image Domain Processing
March, 2004 Kent State University 19
V1 = 500, V2 = 1, V3 = 0.5 AutoV1 = 500, V2 = 1, V3 = 0.5
![Page 20: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/20.jpg)
Results of Image Domain Processing
March, 2004 Kent State University 20
V1 = 500, V2 = 1, V3 = 0.5 Auto
![Page 21: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/21.jpg)
March, 2004 Kent State University 21
![Page 22: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/22.jpg)
September, 2009 Kent State University 22
Interface with Feature Domain
• Find the rectangle circumscribing the ROI
• Store its coordinate information into to a temporary file for feature domain’s use.
![Page 23: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/23.jpg)
September, 2009 Kent State University 23
Feature Domain(Overview)• Two Stages:
– Feature Extraction
– Dimension Reduction (DR)
Image &Feature DataRepository
Interface
Store Feature Data back
Image Domain Feature ExtractDimension Reduction
Feature Domain
![Page 24: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/24.jpg)
September, 2009 Kent State University 24
Implementations
• Acquire ROI information from the image domain
• Extract features based on Gabor Filter and color histogram on HSV space
• Integrate two feature spaces
• Reduce the high feature dimensions to a very low number
![Page 25: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/25.jpg)
September, 2009 Kent State University 25
Implementations (cont.)
• Calculate the similarity measurement between the query object and the objects in the image repository
• Search the similar images in the repository based on similarity index
• Output the corresponding retrieval images
• Knowledge extraction
![Page 26: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/26.jpg)
September, 2009 Kent State University 26
Feature Extraction Algorithm
• Gabor Filter Feature– One of the most important wavelets with multi-
scale and multi-resolution – Mainly reflect texture information
• Color histogram on HSV space– Provide color features
![Page 27: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/27.jpg)
September, 2009 Kent State University 27
Gabor Filter Concept
• A complete but non-orthogonal basis wavelet set
• A significant aspect: localized frequency description – composed of space information
![Page 28: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/28.jpg)
September, 2009 Kent State University 28
Gabor(cont.)
• A two dimensional Gabor function g(x, y) and its Fourier transform G(u, v) can be written as:
jWx
yxyxg
yxyx
22
1exp
2
1),(
2
2
2
2
2
2
2
2)(
2
1exp),(
vu
vWuvuG
yvxu andwhere 2/1,2/1
![Page 29: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/29.jpg)
September, 2009 Kent State University 29
Gabor(cont.)
• Let g(x, y) be the mother Gabor wavelet, then this self-similar filter dictionary can be obtained by appropriate dilations and rotations of g(x, y) through the generating function
number. scale themeans
ns.orientatio ofnumber total theis and , where
)cossin(),sincos(
int,,1),,(),(
m
Knππ/θ
yxayandyxax
nmayxGayxgmm
mmn
![Page 30: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/30.jpg)
September, 2009 Kent State University 30
Color Histogram in HSV Space
• HSV color space includes – Hue (H)– Saturation (S)– Value (V or Lightness)
• Only consider Hue and saturation information, since the lightness of pictures is very sensitive to the surrounding conditions.
![Page 31: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/31.jpg)
September, 2009 Kent State University 31
HSV space Figure
![Page 32: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/32.jpg)
September, 2009 Kent State University 32
HSV space bands
• Design bands in the HSV space– 8 hue bands – 4 saturation bands, – Total 32 sub-spaces
• Compute color histogram feature in each sub-space to form 32 feature dimensions eventually
![Page 33: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/33.jpg)
September, 2009 Kent State University 33
Feature Integration
• Normalize both Gabor filter and HSV color histogram features
• Set a weight factor to balance two feature spaces. Usually Gabor filter features will have the bigger weight value.
![Page 34: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/34.jpg)
September, 2009 Kent State University 34
DR Algorithm
• Disadvantages in the high dimension space– The computational complexity arise sharply– The database indexing becomes difficult
• Principal Component Analysis (PCA) – PCA seeks to reduce the dimension of the data
by finding a few orthogonal linear combinations (Principal Component “PC”)
![Page 35: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/35.jpg)
September, 2009 Kent State University 35
DR implementation
• Original feature dimensions– Gabor filter features: 6*5*2 = 60– HSV color histogram features: 4*8 = 32– Total dimensions: 92
• Feature dimensions after DR– 10 ~15 dimensions
![Page 36: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/36.jpg)
September, 2009 Kent State University 36
Simulation Results in the Feature Domain
• We randomly select 11 query pictures as the test samples in this report.
• At each query time, at most 14 retrieval pictures are retrieved.
• The minimum square error method is served as the similarity measurement.
• The value in the tables as below means the positive pictures out of the 14 retrieval pictures.
![Page 37: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/37.jpg)
September, 2009 Kent State University 37
Performance between different feature extraction techniques
• the integration of Gabor Filter and HSV color Histogram
gains the better performance. • See pictures in detail. Click here
Query pic#
1 2 3 4 5 6 7 8 9 10 11
Gabor 6 7 7 4 12 1 1 2 4 3 2
HSV 8 2 9 1 2 3 1 1 4 2 3
Integrated 10 5 11 4 12 3 3 2 5 2 4
![Page 38: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/38.jpg)
September, 2009 Kent State University 38
Performance between with and without DR applied
• The performance after DR applied slightly degrades on average in comparison to the results before DR takes on
stage • See pictures in detail. Click here
Query pic#
1 2 3 4 5 6 7 8 9 10
11
Integrated
10 5 11 4 12 3 3 2 5 2 4
DR 9 6 5 5 12 2 1 1 4 3 2
![Page 39: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/39.jpg)
September, 2009 Kent State University 39
More Simulations
• Performance between different weight used
• Performance between different dimensions retained after DR
![Page 40: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/40.jpg)
September, 2009 Kent State University 40
Final Integration Results
• Simulation results when both the image domain and the feature domain are used
• See the detail pictures, Click here
![Page 41: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/41.jpg)
September, 2009 Kent State University 41
Integration
• UAV media capture and analysis
• WWW based media analysis
• Vehicle based media capture and analysis
![Page 42: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/42.jpg)
September, 2009 Kent State University 42
Future ResearchExtended to video objects
• Object based video coding
• Non-object based video coding
• Video indexing
• Knowledge extraction and management
![Page 43: RETRIEVAL OF MULTIMEDIA OBJECTS USING COLOR SEGMENTATION AND DIMENSION REDUCTION OF FEATURES](https://reader035.vdocument.in/reader035/viewer/2022070405/56813f52550346895daa1087/html5/thumbnails/43.jpg)
September, 2009 Kent State University 43
Future ResearchData Fusion
• Multimodality medical imaging
• CT – Structural information
• PET – Functional information
• Fusion
• Knowledge management