l2os threshold optimisation 20 june 2014, pm26
DESCRIPTION
L2OS threshold optimisation 20 June 2014, PM26. JL Vergely, J. Boutin, P. Spurgeon ACRI-ST,LOCEAN, ARGANS. RFI/outlier detection. Aim : To improve the thresholds of the L2OS processor to be applied on TB measurements in order to remove outliers. About thresholds : - PowerPoint PPT PresentationTRANSCRIPT
L2OS threshold optimisation 20 June 2014, PM26
JL Vergely, J. Boutin, P. Spurgeon
ACRI-ST,LOCEAN, ARGANS
RFI/outlier detection
Aim :
• To improve the thresholds of the L2OS processor to be applied on TB measurements in order to remove outliers.
About thresholds :
• should be independent on L1c quality products
Thresholds to be testedtest for outlier detection (dwell test)nsig
test for out of range TB detection (FOV test)Tm_out_of_range_affov Tm_out_of_range_eaffovTm_out_of_range_stokes3_affovTm_out_of_range_stokes3_eaffovTm_out_of_range_stokes4_affovTm_out_of_range_stokes4_eaffov
test for oscillation TB detection (FOV test)Ts_stdTs_std_stokes3Ts_std_stokes4
Other tests : max of iteration
Tests conditions
|X_swath| < 400 km
Coast : 1000 km
PCT_var < 80
-40° < lat < 40°
SSS ref: Coriolis global NRTOA (MyOcean)
Day : 1,2,3,4,5/5/2013, L1C v550
L2OS proc : v600 (CATDS processing chain)
Indicators / SSS quality filter
- chi2P : good TB fit if chi2P high. Chi2P > 0.05 in current processor. Warning : Dg_chi2P in L2OS processor = 1-Chi2P
- SSS error < 1.4 psu- mean(SSS SMOS – SSS Coriolis) and std(SSS
SMOS – SSS Coriolis) - X = (SSS SMOS – SSS Coriolis)/SSS_error. X
should be close to a Gaussian law with mean(X)=0 and std(X)=1. Does not depend on SSS accuracy (close to the ratio between empirical error and theoretical error).
Chi2P and RFI
Chi2P, 5/5/2013, asc
Percentage of RFI contamination : january 2012, asc
nsig : full ocean
Outlier detection.
Dwell test.
TB removed if :
|TBsmos – TBmodel – DA| > nsig.rad_noise
DA = mean dwell correction
Current value : 5
Tested values : 2, 3, 4, 5
nsig full ocean
nsig full ocean
Expected distribution
Centred reduced variable
4 sigmas test
Queue distribution : outliers
nsig full ocean
No specific filter
Chi2p > 0.05 & sigSSS < 1.35
Mean and std(X)
nsig = 2
nsig full ocean
No specific filter
Chi2p > 0.05 & sigSSS < 1.35
Mean and std(X)
nsig = 3
nsig full ocean
No specific filter
Chi2p > 0.05 & sigSSS < 1.35
Mean and std(X)
nsig = 4
nsig full ocean
No specific filter
Chi2p > 0.05 & sigSSS < 1.35
Mean and std(X)
nsig = 5
nsig full ocean
nsig=2: Many outliers at 4 sigmas
nsig : coast (1000km)
nsig coast
Expected distribution
nsig coast
No specific filter
Chi2p > 0.05 & sigSSS < 1.35
Mean and std(X)
nsig = 2
nsig coast
No specific filter
Chi2p > 0.05 & sigSSS < 1.35
Mean and std(X)
nsig = 3
nsig coast
No specific filter
Chi2p > 0.05 & sigSSS < 1.35
Mean and std(X)
nsig = 4
nsig coast
No specific filter
Chi2p > 0.05 & sigSSS < 1.35
Mean and std(X)
nsig = 5
nsig coast
nsig=2 : very biased !!
nsig=2: Many outliers at 4 sigmas
Tm_out_of_range_affov or eaffov (polar X,Y,3,4)
Snapshot removed if at least one TB is an outlier : |TB smos – TB model| > Tm_out_of_range
Problem because the test is applied directly on the TBs and not on the TBs normalised by the radiometric noise
X and Y from short and long integration timeCurrent value : 50 K for affov and 100 K for eaffovTested value : 10, 20, 30, 40 K
Tm_out_of_range_affov full ocean
Tm_out_of_range_affov full ocean
No specific filter
Chi2p > 0.05 & sigSSS < 1.35
Tm = 10K
Tm_out_of_range_affov full ocean
No specific filter
Chi2p > 0.05 & sigSSS < 1.35
Tm = 40K
Tm_out_of_range_affov full ocean
Tm_out_of_range_eaffov full ocean
Tm=10K :
Little bit better but lost of accuracy
No significative change
(with Tm_out_of_range_affov = 12)
Tm_out_of_range_stokes3_affov full ocean
Tm=6K :
Little bit better
-> try to work in dual pol mode ?
No significative change
(with Tm_out_of_range_affov/eaffov = 12/18)
No significative change
(with Tm_out_of_range_stokes3_affov = 8)
Tm_out_of_range_stokes3_eaffov full ocean
Tm_out_of_range_stokes4_affov full ocean
No significative change
(with Tm_out_of_range_stokes3_affov/eaffov = 8/16)
Tm_out_of_range_stokes4_eaffov full ocean
No significative change
(with Tm_out_of_range_stokes3_affov/eaffov = 8/16 & Tm_out_of_range_stokes4_affov = 10)
Ts_std thresholds
Snapshot is removed if :rms((TB smos –TB model)/ra) > Ts_std
rms((TB smos –TB model)/ra) is expected to be close to 1 (if OTT well applied)
Current value = 2.5
Ts_std full ocean
Ts_std=1.5 :
Little bit better
Ts_std_stokes3 full ocean
Ts_std=0.5 or 1 :
Too low
Too biased
No significative improvement
Ts_std_stokes4 full ocean
No significative improvement
Ts_std=1 :
Too low
Too inaccurate
Comparison current configuration and configuration without thresholds
Comp current/without thres. full ocean
current conf
Chi2p > 0.05 & sigSSS < 1.35
Comp current/without thres. full ocean
without filterNo signicative change
Chi2p > 0.05 & sigSSS < 1.35
Comp current/without thres. full ocean
A little bit better without thresholds
Comp current/without thres. coast
Without thresholds : better for 4 sigmas SSS
5 day processing
1,2,3,4,5/05/2013; nsig = 2, 3, 4, 5
Dwell test.
TB removed if :
|TBsmos – TBmodel – DA| > nsig.noise
DA = mean dwell correction
Current config : nsig = 5
Without LS mask. chi2P > 0.05nsig=2
nsig=3
nsig=4
nsig=5
Iteration number
2 modes !!
Iteration numberSignature TEC ?
RFI or island ?
Hot spot
Global improvement using iterMax
Small global effect.
What about specific area ?
4 zones with RFI/coast contamination
Pacific
Pacific + coast
Atlantic
Indian ocean
4 zones with RFI/coast contamination
Pacific
Pacific + coast
Atlantic
Indian ocean
iterMax = 5
Itermax=5 is too low : lost of accuracy
Low bias
Gulf of Bengal
iterMax = 10
good accuracy
iterMax = 15
No significative change comparatively to 10
iterMax = 20
iterMax=20 (current)
iterMax=15
iterMax=10
iterMax=5
Stripes
conclusions• Chi2P (or chi2) is improved with TB filtering but SSS biases
increase. • No TB filtering is requiered ? It helps for coast (only from some
threshods) but not for full ocean.• The current configuration has very high thresholds which filter
almost nothing. A configuration with 0 TB filtering gives almost the same results than the current configuration.
• Filtering has to be done at SSS level from Dg_chi2P and sigSSS.
• Use of Tm_out_of_range ? Algorithm to be improved and tested again ?
• Some improvements from St3 filtering : try to work in dual pol ?• Add a specific mask for faint RFI ?• confirmed with more data : tests from 5 day data (for nsig). Use
of iterMax. iterMax = 10 to remove some RFIs/outliers. • Dg_chi2P in L2OS processor = 1-chi2P