CLEAN-AP UpdateClutter Environment Analysis using Adaptive Processing

Sebastián Torres and David Warde CIMMS/The University of Oklahoma

and National Severe Storms Laboratory/NOAA

NEXRAD TACNorman, OK

29 August, 2012

CLEAN-AP is a novel real-time, automatic, integrated technique for ground clutter detection and filtering that produces meteorological data with improved quality while meeting NEXRAD technical requirements.

what is CLEAN-AP?

2

While CLEAN-AP can deal with normal- (NP) and anomalous-propagation (AP) clutter, it is not a mitigation technique for moving clutter such as airplanes, cars, or wind turbines.

what is CLEAN-AP not?

3

• Developed concept (Spring ‘08)– Filed invention disclosure with OU’s OTD

• Evaluated performance using simulations (Spring ‘08)• Implemented in real time on the NWRT PAR (Fall ‘08) • Compared with current WSR-88D clutter mitigation (Fall

‘09)– KEMX (Tucson, AZ), KTLX (Oklahoma City, OK), KABX (Albuquerque, NM), KCRI (ROC

testbed)• Presented at November 2009 TAC meeting

– initially presented CLEAN-AP• “… the technique shows potential utility…”, “… encouraged by early results…”• “… [have not] shown the scientific details behind the algorithm…”• “… [need] case comparisons with CMD…”

• Extended to dual-polarization (Spring ‘10)– KOUN (S-band) and OU’ (C-band)

• Developed clutter model (Spring ‘10)– Form/Function/Fit for RVP8

milestones (I)

4

• Presented at fall 2010 ROC/NSSL/NCAR TIM– presented more technical details and data cases– received positive feedback from ROC eng and agency reps

• Delivered CLEAN-AP algorithm description (Fall ’10)– NSSL Report 14 (confidential attachment)

• Extended to staggered PRT (Fall ‘10)• Presented at March 2011 TAC meeting

– decision briefing• “The TAC members agreed to move forward with conducting

an engineering evaluation on CLEAN AP.”

• Documented CLEAN-AP performance (Winter ’11)– NSSL Report 15

• Delivered SPRT CLEAN-AP algorithmdescription (Spring ’12)

– NSSL Report 16 (confidential)

milestones (II)

5

Ground Clutter Filtering

Ground Clutter Detection

Unfiltered data

Filtered data

clutter mitigation at the RDA

Control

CMD

GMAP

CLEAN-AP

6

CLEAN-AP’s core: the ASDlag-1 autocorrelation spectral density

• For periodic signals: F1(k) = F0(k) ej2k/M

– S1(k) = F0*(k) F0(k) ej2k/M =S (k) ej2k/M

– the ASD is the PSD with linear phase

sample time

10 M2 M1

V0

V1

window & DFT F1

window & DFT F0

S1(k) = F0*(k)F1(k)

MeasuredTrue

v

v

ASD

v

v

ASD

v

PSD

Ground Clutter

v

PSD

Weatherv

v

ASD

v

v

ASD

Leakage

Leakage

nothing is ideal…

Spectral leakage in the measured ASD is the basis for clutter detection in CLEAN-AP

how does CLEAN-AP work? Integrated Detection and Filtering

• Estimate CNR• Select appropriate data

window• Compute ASD• Identify components

with clutter contamination

– Phase of ASD is near zero due to leakage effect

• Remove clutter• Reconstruct weather

• ASD is used to estimate meteorological variables

v

v

|ASD|

zero phase

DC

Arg(ASD)

PSD

9

Performance Summary

data window selectionCLEAN-AP vs. GMAP

More aggressive data windows contain spectral leakage but increase variance of estimates

better suppression

lower variance

GMAP

%

11

notch width selectionCLEAN-AP vs. GMAP

CLEAN-AP uses magnitude and phase for improved notch width determination

better suppression

lower bias

NW determinationmag phase

GMAP CLEAN-AP

12

13

CLEAN-AP has more clutter suppression and less variance of estimates

Clutter SuppressionDoppler Mode

CLEAN-AP GMAP

14

CLEAN-AP has a more robust notch width determination that results in smaller biases

Reflectivity (Power) BiasDoppler Mode

CLEAN-AP GMAP

Case Examples

• Ground-clutter suppression– Due to the mountainous terrain close to the

radar, KEMX data exhibited “hot spots” (clutter residue) after the initial release of CMD (B11). This led to a few software changes (B11 fixed)

KEMX Tucson, AZCan you see the mountains?

16

KEMX Reflectivity

Unfiltered CMD+GMAP (B11 fixed)CLEAN-AP

CLEAN-AP mitigates both weak and strong clutter contamination

17

KEMX Tucson, AZCLEAN-AP Performance

CLEAN-AP adapts to clutter environment

18

CLEAN-APPower Removed

CLEAN-APWindow Used

• Zero-isodop loss– Weather with narrow spectrum width and near

zero velocity has nearly the same spectrum as clutter

KCRI Norman, OKWhat happened to the zero-isodop?

if it looks like a duck… 19

20

KCRIReflectivity

Unfiltered UnfilteredCLEAN-APCMD+GMAP (B11

fixed)

CLEAN-AP better preserves the zero isodop

21

KCRIVelocity

CLEAN-AP better preserves the zero isodop

Unfiltered UnfilteredCLEAN-AP CMD+GMAP (B11)

• Support implementation on the ORDA• Support engineering evaluation • Investigate using dual-pol information to

improve detection performance• Investigate using H and V channels to

determine the filter’s notch width– This also applies to

the current filter

Future Work

CLEAN-AP is a novel real-time, automatic, integrated technique for ground clutter mitigation that produces meteorological data with improved quality while meeting NEXRAD technical requirements.

In March 2011, the TAC endorsedits engineering evaluation on the WSR-88D.

Summary

23

back up slides

24

• CLEAN-AP is automatic– no need for user intervention– real-time ground-clutter detection– no need for clutter maps

• CLEAN-AP produces data withimproved quality

– adaptive data windowing finds a good compromise between clutter suppression and data quality

• CLEAN-AP meets NEXRAD requirements– improved clutter suppression– requirements (Z) met with as few as 8 samples

• CLEAN-AP is integrated– one algorithm for ground clutter detection and filtering– gate-by-gate operation

the CLEAN-AP filter (I)

25

• CLEAN-AP “sets the stage” for further spectral processing

– phase provides useful information– immune to biases from circular convolution

• CLEAN-AP is operational on the NWRT PAR– running in real-time since Sep 2008– performance informally evaluated by meteorologists

and forecasters (PARISE experiments)• CLEAN-AP is compatible with current and

future upgrades– SZ-2, super resolution, dual pol., hybrid SW,– SPRT, range oversampling, etc.

the CLEAN-AP filter (II)

26

KEMX

Clutter Map (Pwr Rmvd > 10 dB)Clutter Map (Pwr Rmvd > 20 dB)

27

• Snow event with embedded storms

KTLX Oklahoma City, OKWhere is the zero?

28

29

KTLXReflectivity

Unfiltered CLEAN-AP

Where is the zero isodop?

30

KTLXVelocity

Unfiltered CLEAN-AP

No detectable zero isodop loss

KOUN What about dual polarization?

ReflectivityDifferential ReflectivityCorrelation CoefficientDifferential Phase

CLEAN-AP is compatible with dual polarization

31