rethinking clutter filtering and improving signal statistics...regression clutter filtering...
TRANSCRIPT
![Page 1: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/1.jpg)
John Hubbert, Greg Meymaris, Mike Dixon, Scott Ellis
NATIONAL CENTER FOR ATMOSPHERIC RESEARCHBoulder, Colorado
NEXRAD TAC MeetingNorman OK
29 April 2019
Rethinking Clutter Filteringand
Improving Signal Statistics
0
.2
.4
.6
.8
1.0
-.2
-.4
-.6
-.8
-1.0
-1.2
-1.4
0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0
Time (sec)
Clutter signal
2 sec.
![Page 2: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/2.jpg)
Spectra-Based Clutter filtering• Since the advent of fast digital receivers this has been the
standard, e.g., GMAP• Replaced time domain filters• Very common in weather radars
![Page 3: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/3.jpg)
Discrete Fourier Transform
Fourier Transform pairFrequency domain
Time domain,i.e., sum of sinusoids
![Page 4: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/4.jpg)
0
.05
.10
.15
.20
.25
.30
.35
.40
.45
.50
-.05
-.101 12
Real Part of a S-Pol Time Series.
Raw Time series
![Page 5: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/5.jpg)
0
.05
.10
.15
.20
.25
.30
.35
.40
.45
.50
-.05
-.101 128 256 384 512
Time (ms)
Real Part of a S-Pol Time Series. DFT creates a periodic signal
Raw Time series
![Page 6: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/6.jpg)
0
.05
.10
.15
.20
.25
.30
.35
.40
.45
.50
-.05
-.101 128 256 384 512
Time (ms)
Raw Time series In order to create a
sum of sinusoids that can replicate this jump discontinuity, many higher frequency sinusoids are required.
Real Part of a S-Pol Time Series. DFT creates a periodic signal
![Page 7: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/7.jpg)
0
.04
.08
.12
.16
.20
.24
.28
.32
1 128 256 384 512
Time (ms)
0
.05
.10
.15
.20
.25
.30
.35
.40
.45
.50
-.05
-.101 128 256 384 512
Time (ms)
Raw Time series
HanningWindowedTime series
In order to create a sum of sinusoids that can replicate this jump discontinuity, many higher frequency sinusoids are required.
HanningWindowedTime series
Real Part of a S-Pol Time Series. DFT creates a periodic signal
![Page 8: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/8.jpg)
0
10
-10
-20
-30
-40
-50
-60
02468
1012141618202224
-2-4-6-8
-10
Raw spectrumHanning windowed
spectrum
m/s m/sVelocity Velocity
Window Effects on SpectraRaw Hanning windowed
The smooth curve spectrum is an artifact of the jump discontinuity
-40dB
-10dB
![Page 9: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/9.jpg)
0
.2
.4
.6
.8
-.2
-.4
-.6
-.8
-1.0
-1.2
-1.4
-1.6
-1.8
0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0
Time (sec)
0
.2
.4
.6
.8
1.0
-.2
-.4
-.6
-.8
-1.0
-1.2
-1.4
0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0
Time (sec)
(a) (b)
Scan of Rocky Mountains by S-Pol at 8 deg/sec.
(16 degrees az.)
Typical Clutter I and Q Signals
I Q
Degs.
![Page 10: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/10.jpg)
What is Regression Filtering?
Clutter + Weather Weather Only
![Page 11: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/11.jpg)
Represent the I and Q Signals as a Sum of These Orthogonal PolynomialsFast, low round off error algorithm: http://jean-pierre.moreau.pagesperso-orange.fr/
![Page 12: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/12.jpg)
Regression Clutter filtering• Regressions filters have a history in biomedical field
•
• And weather radar• Torres, S. and D. Zrnic ́, 1999: Ground clutter canceling with a
regression filter. J. Atmos. Oceanic Technol.
![Page 13: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/13.jpg)
Regression Filter Example on S-Pol Data
Power Spectrum Time series
![Page 14: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/14.jpg)
0
.05
.10
.15
.20
.25
-.05
20 40 60 80 100 120
Time (ms)
RealPart
Raw dataFitted data
0
.05
.10
.15
.20
.25
-.05
20 40 60 80 100 120
Time (ms)
RealPart
Raw dataFitted data
8th order polynomial fit
![Page 15: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/15.jpg)
Subtracting the polynomial fit from the time series…..
![Page 16: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/16.jpg)
Regression Filtered versus Raw Spectrum
-14
-45
![Page 17: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/17.jpg)
Blackman Window and Notch Technique
![Page 18: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/18.jpg)
Regression Window and NotchComparison
![Page 19: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/19.jpg)
Regression versus Window and Notch
• WHY USE A REGRESSION FILTER???
![Page 20: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/20.jpg)
Signal Statistics5.23dB attenuation
• Hanning window: 4.19 dB attenuation
About 50% increase in variance!
About 35% increase in variance!
![Page 21: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/21.jpg)
Regression Frequency Response64 point time series for polynomial orders 3 - 7
![Page 22: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/22.jpg)
Frequency Response 264 point time series for polynomial orders 9, 11, 13, 15, 19
Frequency response depends on sequence length and polynomial order
![Page 23: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/23.jpg)
Modified Regression Filtering
1 2 3 4 ….. …N
Standard technique: Take length N sequence, fit a polynomial to it and subtract to remove the low frequency clutter components.
Modified technique: Break the sequence into blocks, lets say 4 for this example:
1 Nl l l
N/4 N/2 3N/4
1. Do 4 regression fits, thus suppressing the ground clutter in each block2. Concatenate the blocks back into one sequence3. Calculate radar variables
block 1 block 2 block 3 block 4
time series
![Page 24: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/24.jpg)
1 N
m points
m points
Length N sequence broken into overlapping blocks. The overlap is m pointsand these regions are shown in green.After filtering, overlapping pairs of points are averaged so that a N pointfiltered sequences is created. This is done to smooth the transition from oneblock to the next.
The Blocks Can Be Overlapped and WeightedThis is to reduce slight discontinuities at end points.
.1 .3 .7 .9
.9 .7 .3 .1
weightsBlock1
Block2
![Page 25: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/25.jpg)
S-Pol Clutter Environmentat Marshall
22 April 20180.5 elevation
![Page 26: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/26.jpg)
Compare Clutter Rejection of the Regression and GMAP like filters (notch width is constant)
![Page 27: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/27.jpg)
-15
-20
-25
-30
-35
-40
-45
-50
-55
-60
-65
-70
-75-15-20-25-30-35-40-45-50-55-60-65-70-75
-15
-20
-25
-30
-35
-40
-45
-50
-55
-60
-65
-70
-75-15-20-25-30-35-40-45-50-55-60-65-70-75
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
++
+
+
+
+
+
+
+
+
+
+
+
+
+
+
++
+
Scatter Plots Regression vs Window and Notch(4 blocks of 16)
![Page 28: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/28.jpg)
KFTG Data. “Bomb Cyclone” 13 March 2019Short PRT data from VCP 212
25 km50 km
225o
Z Vel
![Page 29: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/29.jpg)
Raw Blackman-Nuttall
9 pt notch
KFTG LPRT Data VCP212, 20 deg/sec, 16 points
6th order
![Page 30: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/30.jpg)
KFTG 227 Az. 0.45 elev13 March 2019
Unfiltered data
![Page 31: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/31.jpg)
5th order regression filter Blackman window 7 point notch
A Comparison of Regression versus Window and Notch Clutter Filters
![Page 32: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/32.jpg)
Smearing Effects of Windowing
Raw data (rectangular window) Blackman-Nuttall Window
S-Pol data
![Page 33: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/33.jpg)
Resulting Implications for the Clutter Filter Bandwidth
5 pt notch insufficient!!
Window function spreads the clutterpower so that wider bandwidth is needed
![Page 34: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/34.jpg)
Vmax-Vmax Vmax
Clutter 0.25 m/s
0
Weather 2 m/s
Modeling Efforts Have Begun
28 m/s-28 m/s
64 points30 dB SNR weather30 dB CNR clutterBlackman Window, 9pt notchRegression 7th order
~ 50% increase in STD~ 30% increase in STD
![Page 35: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/35.jpg)
Application to Data Sets Has Begun
Bomb Cyclone Level 2 Data Times Series Processed with Regression Filter
KFTG
![Page 36: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/36.jpg)
Old frequency response
Compensatedfrequency response
Passband Stopband
Frequency
Frequency Compensation
After regression filtering, FFT the resulting timeseries and compensate the frequencies as desired as guided by the frequency response of the regression filter.
An interesting possibility
![Page 37: Rethinking Clutter Filtering and Improving Signal Statistics...Regression Clutter filtering •Regressions filters have a history in biomedical field • •And weather radar •Torres,](https://reader034.vdocument.in/reader034/viewer/2022052003/60166debbd0029496874d8ed/html5/thumbnails/37.jpg)
Conclusions
much better signal statistics can be achieved.
• Windowing the data, while containing “clutter leakage”, spreads the clutter (causes wider clutter bandwidth). This then requires a wider bandwidth notch as compared to regression filtering.
• Thus underlying weather signal is is more effectively recovered with a regression filter