ee645: independent component analysis elliot taniguchi advisor: dr. kuh may 16, 2003
TRANSCRIPT
EE645: Independent Component Analysis
Elliot Taniguchi
Advisor: Dr. Kuh
May 16, 2003
Presentation Overview
ICA Motivation Mathematical Formulation Fast ICA Algorithm Applications
Noise Separation and Feature Extraction Digital Watermarking
Motivation for ICA
Cocktail Party Problem Suppose you are in a crowded room with
many people. How do you understand what any one person is saying?
Separation of Independent Signals Similar to Blind Source Separation Little knowledge of the signals Access to mixed signals only
Cocktail Party Problem??
Cocktail Party Problem
ICA Separation Algorithm Separation of Speech
Signals Humans can separate
multiple signals with only two ears/sensors
ICA needs as many ears/sensors as message signals
Here we assume he has four ears!
Recovered Messages
Mathematical Formulation Overview
ICA Definition ICA Assumptions
Independent Signals Non-Gaussian
ICA Limitations Scaling Permutations No. of Sensors
ICA Definition
Mixed Signals in Matrix Notation
n
i
is1
iasAx
Variable Definitions
Signalt Independen i ~
Matrix ngMultiplexi ~
Signal dMultiplexe~
this
A
xj
ICA Solution
Signal Separation
Find using the ICA Algorithm
ssIsA)(W
s)(AWxWs
ˆ
W
ICA Block Diagram (2 Signals)
Signal #1 Signal #2
MultiplexedSignal #2
MultiplexedSignal #1
a11
a12 a21
a22
ICA Assumption #1: Independence Probability Density Definition
Expected Value Definition
functions are ,
)}({)}({)}()({
21
22112211
hh
yhEyhEyhyhE
spdf' are ,
)()()(
21
22112,1
pp
ypypyyp
ICA Assumption #2: Non-Gaussian Property of Gaussian signals
Addition of two independent Gaussian random variables is another single Gaussian random variable.
Information Lost! Kurtosis Function
Special Case: kurt(N) = 0
224 }){(3}{)( yEyEykurt
Limitation #1: Scaling
ICA maximizes independence between signals.
)}ˆ({*)}ˆ({
)}ˆ(*)ˆ({)}ˆ(*)ˆ({
222111
221121222111
XhEXhE
XhXhEXhXhE
Assuming theSeparated Signalsare Independent
Scaling Factorfor Signal #1
Scaling Factorfor Signal #2
Limitation #2: Signal Permutations
The mixing matrix and independent components are unknown.
sAsPPA
sPPAsIAsAx1
1
*)*(*)*(
*)*(***
sPssIsPP
sPIPsPAWP
sPAWPsAWx
1
11
1
***)*(
*))*(*(*)*)*(*(
*)*(*)*(**ˆ
Limitation #3: No. of Sensors
Sensor Requirement The number of separated signals cannot be
larger than the number of inputs. Current research is being done to reduce this
constraint.
ICA Separation Technique
Central Limit Theorem If two random (non-Gaussian) signals added,
the resulting signal will be more Gaussian than the original two random signals
ICA Separation Concept Central Limit Theorem (in Reverse) Maximizing Non-Gaussianity
Results in separating the two signals
Fast ICA Algorithm Overview
“Fixed-Point” Algorithm Implementation
Fast ICA algorithm Extensions
Algorithm Speed & Performance Currently the fastest Most Commonly Used
Fast ICA Algorithm
1. Choose a random initial weight vector.
2. Let,
3. Let,
4. Repeat until converges.
www /wxwxwxw TT )}(g{)}g({ EE
w
Fast ICA Extensions
Preprocessing Normalize mean to zero Pre-Whitening
Activation Functions g(u)=u^3 g(u)=u^2 g(u)=tanh(a1*u) g(u)=u*exp(-a2*u^2/2)
Noise Separation Example
Separation of Noise Impulsive Noise Additive White Gaussian Noise
Implementation Two Sensor Setup Fast ICA Algorithm
Noise Generation
AWGN Gaussian R.V.
Impulsive Noise Poisson R.V. Gaussian R.V.
)(*),( PmNI
),( mN
Physical Setup
Receiver #1Receiver #2
ImpulsiveNoise
GaussianNoise
Noise Separation Example
0 100 200 300 400 500 600 700 800 900 1000-30
-20
-10
0
10
20
30Noise Separation Simulation
P(
= 1
)*N
(m =
0,
= ,
5)
0 100 200 300 400 500 600 700 800 900 1000-3
-2
-1
0
1
2
3
N(m
= 0
, =
1)
Noise Separation Example
0 100 200 300 400 500 600 700 800 900 1000-30
-20
-10
0
10
20
30Noise Separation Simulation
Mul
tiple
xed
Sig
nal #
1
0 100 200 300 400 500 600 700 800 900 1000-30
-20
-10
0
10
20
30
Mul
tiple
xed
Sig
nal #
2
Noise Separation Example
0 100 200 300 400 500 600 700 800 900 1000-20
-10
0
10
20Noise Separation Simulation
Sep
arat
ed S
igna
l #1
0 100 200 300 400 500 600 700 800 900 1000-3
-2
-1
0
1
2
3
Sep
arat
ed S
igna
l #2
Noise Separation Performance
0 100 200 300 400 500 600 700 800 900 1000-40
-20
0
20
40
Orig
inal
Sig
nal #
1
Noise Separation Simulation
0 100 200 300 400 500 600 700 800 900 1000-40
-20
0
20
40
Nor
mal
ized
IC
A S
igna
l #1
0 100 200 300 400 500 600 700 800 900 1000-0.1
-0.05
0
0.05
0.1
Err
or :
Sig
nal #
1 MSE = 0.00051558
Noise Separation Performance
0 100 200 300 400 500 600 700 800 900 1000-4
-2
0
2
4
Orig
inal
Sig
nal #
2
Noise Separation Simulation
0 100 200 300 400 500 600 700 800 900 1000-4
-2
0
2
4
Nor
mal
ized
IC
A S
igna
l #2
0 100 200 300 400 500 600 700 800 900 1000-0.2
-0.1
0
0.1
0.2
Err
or :
Sig
nal #
2 MSE = 7.3829e-005
Noise Separation & Feature Extraction
ICA performs well in Blind Source Separation ICA for Feature Extraction
Reduce Complexity of the Neural Network Train only on the appropriate signal
Detection and Estimation of Hidden Signals
Noise Detection and Estimation: Method #1
FastICASensor #3
Sensor #4
Sensor #2
Sensor #1
Feature #3
Feature #4
Feature #2
Feature #1
N.N.(BP,RBF,SVM)
Noise Detection and Estimation: Method #2
FastICASensor #3
Sensor #4
Sensor #2
Sensor #1
Feature #3
Feature #4
Feature #2
Feature #1
N.N.(BP,RBF,SVM)
DecisionDevice
Noise Detection and Estimation:Conclusion Additional Preprocessing
Segmentation of Impulsive Noise (Time-Limited) Possible Inputs to the Neural Network
Statistical Moments Signal Samples
Possible Neural Networks Back Propagation SVM Radial Basis Functions
Problem Need to train Neural Network in Matlab
Digital Watermarking of Music
Motivation Popularity of Digital Storage Devices
Reliable, Fast, Ease of duplication, etc. How to protect copyrighted information?
Leaving digital signatures of its artist
Essential Properties for Watermarking Undetectable Irremovable Resilient
Detection & Estimation of Watermarks Detection of Watermark
Authenticate copyrighted music Estimation of Watermark
Authenticate copyrighted music Information on artist, producer, etc.
Watermarking Model
Process Mix the original musical data with watermark Keep watermark Power relatively low
Ensure high quality of the watermarked music Watermark is better hidden
hidden is watermark that theensure to :Note
WatermarkEmbedded
Data MusicalCopyright
Key Encryption
Data Musical dWatermarke
12 11
2221
1211
aa
aa
aa
Popular Digital Formats
Wave Format Mp3 Format
Bit Rate 1411 [kbps] 128 [kbps]
Channels 2 2
Audio Sample Rate
44 [kHz] 44 [kHz]
Detection of Watermark
Watermarking Detection Algorithm1. Use the ICA model to randomly mix the watermark
and music file.
2. Save the watermarked music in the popular *.wav format
3. Read the saved *.wave file. Separate the watermark and the music file.
4. Identify the watermark using statistical methods (mean, std, etc.)
Performance Statistic Correlation Coefficient (Absolute Value)
Detection Performance
3.85 3.852 3.854 3.856 3.858 3.86 3.862 3.864 3.866 3.868 3.87
x 10-6
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Signal-to-Noise Ratio
Cor
rela
tion
Coe
ffic
ient
Performance of ICA Watermarking: 16 Bit, 44100 Hz
3.853 3.854 3.855 3.856 3.857 3.858 3.859 3.86 3.861
x 10-6
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
Signal-to-Noise RatioC
orre
latio
n C
oeff
icie
nt
Performance of ICA Watermarking: 8 Bit, 44100 Hz
Estimation of Watermark
Watermarking Estimation Algorithm1. Use the ICA model to randomly mix the watermark
and music file.2. Save the watermarked music in the popular *.wav
format3. Read the saved *.wave file. Separate the watermark
and the music file.4. Identify the watermark using statistical methods
(mean, std, etc.)5. Digitize the watermark signal.
Performance Statistic Bit Error Rate
Estimation Performance
3.85 3.855 3.86 3.865
x 10-6
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1Performance of ICA Watermarking: 16 Bit, 44100 Hz
Signal-to-Noise Ratio
Bit
Err
or R
ate
3.853 3.854 3.855 3.856 3.857 3.858 3.859 3.86 3.861 3.862 3.863
x 10-6
-1
-0.8
-0.6
-0.4
-0.2
0
0.2
0.4
0.6
0.8
1
Signal-to-Noise RatioB
it E
rror
Rat
e
Performance of ICA Watermarking: 8 Bit, 44100 Hz
Is the Music Content Preserved?
“Hero” – Mariah CareyOriginal Wave file from CD
Hero.wav29 [s]
“Hero” – Mariah CareyWatermarked Wave fileHeroWatermarked.wav
29 [s]
Resilience of Proposed Watermark
Resiliency Previous Simulations show that wav format is
resilient to 8-bit and 16-bit quantization. Can the watermark be detected after Mp3
compression and decompression?
Mp3 Compression/Decompression
Actual Mp3 compression program used CDex Version 1.40 Release
Mp3 (lossy) Compression Down Sampling Filter banks
Mp3 Decompression Up Sampling Reconstruction Filter
Detection and Estimation Performance
Bad News Correlation Coefficients 0 MSE 0.5
Possible Problems with Mp3 Compression Down Sampling
Watermark information is lost Quantization Noise
Watermark information absorbs into the quantization noise
Lossy Compression 11:1 Compression Rate
How to Improve its Resilience?
Alternative Approaches Synchronization of the music data
Time shift in Mp3 compression? Storing watermark in certain frequencies
(where less quantization occurs) Error Coding
Hamming Reed-Solomon
Conclusion
Wave to Wave Format Very good performance (even for 8-bit wave
files) SNR is very low. Music Integrity is excellent
Mp3 Compression Very bad performance Alternative methods need to be found! Need a greater understanding of current Mp3
Compression Algorithms
References
[1] Araki and others. Suband Based Blind Source Separation with Appropriate Processing for Each Frequency Band. 4th International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.
[2] Hoyer and Hyvarinen. Independent Component Analysis Applied to Feature Extraction from Colour and Stereo Images. August 2000.
[3] Hyvarinen, Aapo. The Fixed-Point Algorithm and Maximum Likelihood Estimation for Independent Component Analysis. http://www.cis.hut.fi/~aapo/.
[4] Hyvarinen, Aapo. Fast and Robust Fixed-Point Algorithms for Independent Component Analysis. http://www.cis.hut.fi/~aapo/. April 1999.
[5] Hyvarinen and Oja. Independent Component Analysis: A Tutorial. http://www.cis.hut.fi/projects/ica/. April 1999.
[6] Introduction to Blind Source Separation. http://www.cnl.salk.edu/~tewon/Blind/. [7] Liu and others. A Digital Watermarking Scheme based on ICA Detection. 4th
International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.
[8] Mitra, Sanjit K. Digital Signal Processing: Second Addition. McGraw Hill, 1998.[9] Shen and others. A Method for Digital Image Watermarking Using ICA. 4th
International Symposium on Independent Component Analysis and Blind Source Separation (ICA 2003). April 2003.