kernel-based_retrieval_of_atmospheric_profiles_from_iasi_data.pdf
Post on 26-May-2015
145 Views
Preview:
TRANSCRIPT
Intro Methodology Results Conclusions
Kernel-Based Retrieval of Atmospheric Profiles from IASI Data
Gustavo Camps-Valls, Valero Laparra, Jordi Munoz-Marı,Luis Gomez-Chova, Xavier Calbet∗
Image Processing Laboratory (IPL), Universitat de Valencia. Spaingustavo.camps@uv.es – http://isp.uv.es∗EUMETSAT, Darmstadt, Germany
IGARSS 2011, 25-29th July, Vancouver, Canada
1 / 23
Intro Methodology Results Conclusions
Motivation
Retrieval of atmospheric profiles
Temperature and humidity are basic meteorological parameters for weatherforecasting and atmospheric chemistry studies
High spectral resolution infrared sounding instruments provide highaccuracy and vertical resolution of temperature and water vapour profiles
Fast, non-linear, multi-output regression methods are needed
2 / 23
Intro Methodology Results Conclusions
MetOp-IASI
MetOp satellite series managed by EUMETSAT
Band 1 Band 2 Band 3
Complex signal processing problem:High input (radiances) dimensionalityHigh output (state vectors) dimensionalityHigh levels of noise in particular channelsHigh temporal and spatial redundancy: high data volumeNonlinear input-output relations
3 / 23
Intro Methodology Results Conclusions
Objectives
Main objective
Nonlinear retrieval of atmospheric states from IASI radiance spectra
Specific objectives
Develop advanced nonlinear multi-output regression for IASI data
The retrieval method must be scalable, fast and accurate
Robust to noisy scenarios, clouds, both over ocean and land
Provide confidence intervals for estimationsNonlinear anomaly detection (quality flags) are developed
X Y
n x 8461 n x 270
T Td O3
4 / 23
Intro Methodology Results Conclusions
Approaches
Current approach in EUMETSAT
L2 PPF: PCA + Linear Regression (LR) retrievals are fed to OptimalEstimation (OE) procedure
LR is fast, but too simple and inaccurate
OE is accurate, but extremely slow
Neural networks nonlinear retrieval (Aires, 02; Blackwell 05; Camps-Valls, 10)
Neural nets have been successfully used for IASI and AIRS
Fast (on test) and accurate
Slow and difficult to train (many parameters to adjust)
Kernel Ridge Regression (KRR) retrieval
Can tackle efficiently with multioutput problems
Training is easier and faster (only two intuitive parameters must be tuned)
It provides a ranked list of most important IFOV used in training
Confidence intervals for the predictions can be obtained
5 / 23
Intro Methodology Results Conclusions
Approaches
Current approach in EUMETSAT
L2 PPF: PCA + Linear Regression (LR) retrievals are fed to OptimalEstimation (OE) procedure
LR is fast, but too simple and inaccurate
OE is accurate, but extremely slow
Neural networks nonlinear retrieval (Aires, 02; Blackwell 05; Camps-Valls, 10)
Neural nets have been successfully used for IASI and AIRS
Fast (on test) and accurate
Slow and difficult to train (many parameters to adjust)
Kernel Ridge Regression (KRR) retrieval
Can tackle efficiently with multioutput problems
Training is easier and faster (only two intuitive parameters must be tuned)
It provides a ranked list of most important IFOV used in training
Confidence intervals for the predictions can be obtained
6 / 23
Intro Methodology Results Conclusions
Approaches
Current approach in EUMETSAT
L2 PPF: PCA + Linear Regression (LR) retrievals are fed to OptimalEstimation (OE) procedure
LR is fast, but too simple and inaccurate
OE is accurate, but extremely slow
Neural networks nonlinear retrieval (Aires, 02; Blackwell 05; Camps-Valls, 10)
Neural nets have been successfully used for IASI and AIRS
Fast (on test) and accurate
Slow and difficult to train (many parameters to adjust)
Kernel Ridge Regression (KRR) retrieval
Can tackle efficiently with multioutput problems
Training is easier and faster (only two intuitive parameters must be tuned)
It provides a ranked list of most important IFOV used in training
Confidence intervals for the predictions can be obtained
7 / 23
Intro Methodology Results Conclusions
Learning scheme
Developed NLR processor
Feature Selection(Calbet, 2008)
Feature Extraction(PCA)
Nonlinear Regression(KRR)
Feature selection (Calbet, 2008):Avoid channels with negative radiancesNoise bias-variance criteria: bias > 4K and std dev. > 3K for IASI
Feature extraction:
PCA feature extraction
Multioutput nonlinear regression:
Kernel ridge regression
8 / 23
Intro Methodology Results Conclusions
Nonlinear regression
Kernel ridge regression (KRR), aka Least Squares SVM
Regression model: Y = ΦW + E
Assume a squared loss function in H:
minW
n‖Y −ΦW‖2 + λ‖W‖2
oRepresenter’s theorem: W = Φ>α
Solve:α = (λI + ΦΦ>| {z }
K
)−1Y
The prediction function:
Y = f (x∗) = Φ(x∗)W = Φ(x∗)Φ>α = K(X, x∗)α
We use the RBF kernel function: K(xi ,xj) = exp(-‖xi − xj‖2/(2σ2))
Confidence on the prediction:
V[f (x∗)] = K(x∗, x∗)− K(x∗,X)(K + λI)−1K(X, x∗)
9 / 23
Intro Methodology Results Conclusions
Nonlinear regression
Key features
KRR generalizes LR
Tune two parameters: σ and λ
Fast for training (few hours) and test (25 ms/FOV)
Fast implementation
Standard code: >> alpha = inv(lambda*eye(n) + K) * Y;
Cholesky decomposition is ∼ 4 times faster:>> R = chol(gamma*eye(n) + K);
>> alpha = R\(R’\Y);
10 / 23
Intro Methodology Results Conclusions
Nonlinear regression
Key features
KRR generalizes LR
Tune two parameters: σ and λ
Fast for training (few hours) and test (25 ms/FOV)
Fast implementation
Standard code: >> alpha = inv(lambda*eye(n) + K) * Y;
Cholesky decomposition is ∼ 4 times faster:>> R = chol(gamma*eye(n) + K);
>> alpha = R\(R’\Y);
11 / 23
Intro Methodology Results Conclusions
Experiments
Datasets
Training done with ECMWFa Chevallier’s database:IASI cloud free, emissivity ‘sea’, ‘noise-free’FOVs: 13456
Methodology for training:Feature selection (Calbet, 2008): XX′ = [X, surface pressure, scan angle, latitude]Feature extraction: PCA with X′, and retain a number of features pLR: use all training data to estimate model weightsKRR:
cross-validation ( 23 , 1
3 ) to estimate model parametersuse all data to estimate model weights
aEuropean Centre for Medium-Range Weather Forecasts
Results
Real datasets, IASI orbits with 91800 FOVs
Predicted error profiles of temperature and water vapour
Confidence maps and detection of anomalies
12 / 23
Intro Methodology Results Conclusions
KRR testing at EUMETSAT over ocean ...
KRR clearly outperforms LRVery good results in water vapour
13 / 23
Intro Methodology Results Conclusions
KRR testing at EUMETSAT over land ...
KRR outperforms LR, which dramatically failsErrors are similar to estimations over the oceanTemperature errors are reasonable, while water vapour is really good 14 / 23
Intro Methodology Results Conclusions
KRR results
Results with and without cloud masking ...
0 5 10 15
102
103
RMSE [K]
p [h
Pa]
T
LR (all)KRR (all)LR (masked)KRR (masked)
0 5 10 15
102
103
RMSE [K]p
[hP
a]
Td
Clouds and anomalies are an important error source
Cloud screening is mandatory
An anomaly detector can be developed
15 / 23
Intro Methodology Results Conclusions
Predictions, discrepancies and confidence maps: Madagascar
AVHRR KRR confidence map
IASI cloud flag ∆T : |TECMWF − TKRR| Cloud flag ×∆T
16 / 23
Intro Methodology Results Conclusions
Predictions, discrepancies and confidence maps: Mexico coast
AVHRR KRR confidence map
IASI cloud flag ∆T : |TECMWF − TKRR| Cloud flag ×∆T
17 / 23
Intro Methodology Results Conclusions
Anomaly detection scheme
PCA SVM
Radiances
T predictionsT errors > 5K
Inputs: radiances and/or predictions
Output: nonlinear prediction of KRR big discrepancies with ECMWF
18 / 23
Intro Methodology Results Conclusions
Anomaly detection results
500 1000 1500 200078
80
82
84
86
88
90
92
Training samples
Tes
t OA
Overall accuracy
500 1000 1500 20000.5
0.55
0.6
0.65
0.7
0.75
0.8
0.85
0.9
Training samplesT
est κ
Kappa statistic, κ
LDA
QDA
MAHAL
TREE
SVM
Linear and nonlinear classifiers developed
Anomalies can be detected accurately (OA ∼ 92%, κ = 0.81)
SVM outperforms all other classifiers
19 / 23
Intro Methodology Results Conclusions
Conclusions
Developed and implemented KRR nonlinear regression
KRR outperforms LR
KRR provides confidence maps
Developed nonlinear anomaly detection methods
1 Thresholding the discrepancies to ECMWF, ∆ = |TECMWF − TKRR |2 KRR confidence on predictions, σT ∈ [0, 1]
3 SVM prediction of anomalies: ∼90%
Future work
Include better spatial information
Channel emissivity prediction
20 / 23
Intro Methodology Results Conclusions
Conclusions
Developed and implemented KRR nonlinear regression
KRR outperforms LR
KRR provides confidence maps
Developed nonlinear anomaly detection methods
1 Thresholding the discrepancies to ECMWF, ∆ = |TECMWF − TKRR |2 KRR confidence on predictions, σT ∈ [0, 1]
3 SVM prediction of anomalies: ∼90%
Future work
Include better spatial information
Channel emissivity prediction
21 / 23
Intro Methodology Results Conclusions
Conclusions
Developed and implemented KRR nonlinear regression
KRR outperforms LR
KRR provides confidence maps
Developed nonlinear anomaly detection methods
1 Thresholding the discrepancies to ECMWF, ∆ = |TECMWF − TKRR |2 KRR confidence on predictions, σT ∈ [0, 1]
3 SVM prediction of anomalies: ∼90%
Future work
Include better spatial information
Channel emissivity prediction
22 / 23
Intro Methodology Results Conclusions
Conclusions
Kernel-Based Retrieval of Atmospheric Profiles from IASI Data
Gustavo Camps-Valls, Valero Laparra, Jordi Munoz-Marı,Luis Gomez-Chova, Xavier Calbet∗
Image Processing Laboratory (IPL), Universitat de Valencia. Spaingustavo.camps@uv.es – http://isp.uv.es∗EUMETSAT, Darmstadt, Germany
IGARSS 2011, 25-29th July, Vancouver, Canada
23 / 23
top related