Intro Methodology Results Conclusions

Kernel-Based Retrieval of Atmospheric Profiles from IASI Data

Gustavo Camps-Valls, Valero Laparra, Jordi Munoz-Marı,Luis Gomez-Chova, Xavier Calbet∗

Image Processing Laboratory (IPL), Universitat de Valencia. Spaingustavo.camps@uv.es – http://isp.uv.es∗EUMETSAT, Darmstadt, Germany

IGARSS 2011, 25-29th July, Vancouver, Canada

1 / 23

Intro Methodology Results Conclusions

Motivation

Retrieval of atmospheric profiles

Temperature and humidity are basic meteorological parameters for weatherforecasting and atmospheric chemistry studies

High spectral resolution infrared sounding instruments provide highaccuracy and vertical resolution of temperature and water vapour profiles

Fast, non-linear, multi-output regression methods are needed

2 / 23

Intro Methodology Results Conclusions

MetOp-IASI

MetOp satellite series managed by EUMETSAT

Band 1 Band 2 Band 3

Complex signal processing problem:High input (radiances) dimensionalityHigh output (state vectors) dimensionalityHigh levels of noise in particular channelsHigh temporal and spatial redundancy: high data volumeNonlinear input-output relations

3 / 23

Intro Methodology Results Conclusions

Objectives

Main objective

Nonlinear retrieval of atmospheric states from IASI radiance spectra

Specific objectives

Develop advanced nonlinear multi-output regression for IASI data

The retrieval method must be scalable, fast and accurate

Robust to noisy scenarios, clouds, both over ocean and land

Provide confidence intervals for estimationsNonlinear anomaly detection (quality flags) are developed

X Y

n x 8461 n x 270

T Td O3

4 / 23

Intro Methodology Results Conclusions

Approaches

Current approach in EUMETSAT

L2 PPF: PCA + Linear Regression (LR) retrievals are fed to OptimalEstimation (OE) procedure

LR is fast, but too simple and inaccurate

OE is accurate, but extremely slow

Neural networks nonlinear retrieval (Aires, 02; Blackwell 05; Camps-Valls, 10)

Neural nets have been successfully used for IASI and AIRS

Fast (on test) and accurate

Slow and difficult to train (many parameters to adjust)

Kernel Ridge Regression (KRR) retrieval

Can tackle efficiently with multioutput problems

Training is easier and faster (only two intuitive parameters must be tuned)

It provides a ranked list of most important IFOV used in training

Confidence intervals for the predictions can be obtained

5 / 23

Intro Methodology Results Conclusions

Approaches

Current approach in EUMETSAT

L2 PPF: PCA + Linear Regression (LR) retrievals are fed to OptimalEstimation (OE) procedure

LR is fast, but too simple and inaccurate

OE is accurate, but extremely slow

Neural networks nonlinear retrieval (Aires, 02; Blackwell 05; Camps-Valls, 10)

Neural nets have been successfully used for IASI and AIRS

Fast (on test) and accurate

Slow and difficult to train (many parameters to adjust)

Kernel Ridge Regression (KRR) retrieval

Can tackle efficiently with multioutput problems

Training is easier and faster (only two intuitive parameters must be tuned)

It provides a ranked list of most important IFOV used in training

Confidence intervals for the predictions can be obtained

6 / 23

Intro Methodology Results Conclusions

Approaches

Current approach in EUMETSAT

L2 PPF: PCA + Linear Regression (LR) retrievals are fed to OptimalEstimation (OE) procedure

LR is fast, but too simple and inaccurate

OE is accurate, but extremely slow

Neural networks nonlinear retrieval (Aires, 02; Blackwell 05; Camps-Valls, 10)

Neural nets have been successfully used for IASI and AIRS

Fast (on test) and accurate

Slow and difficult to train (many parameters to adjust)

Kernel Ridge Regression (KRR) retrieval

Can tackle efficiently with multioutput problems

Training is easier and faster (only two intuitive parameters must be tuned)

It provides a ranked list of most important IFOV used in training

Confidence intervals for the predictions can be obtained

7 / 23

Intro Methodology Results Conclusions

Learning scheme

Developed NLR processor

Feature Selection(Calbet, 2008)

Feature Extraction(PCA)

Nonlinear Regression(KRR)

Feature selection (Calbet, 2008):Avoid channels with negative radiancesNoise bias-variance criteria: bias > 4K and std dev. > 3K for IASI

Feature extraction:

PCA feature extraction

Multioutput nonlinear regression:

Kernel ridge regression

8 / 23

Intro Methodology Results Conclusions

Nonlinear regression

Kernel ridge regression (KRR), aka Least Squares SVM

Regression model: Y = ΦW + E

Assume a squared loss function in H:

minW

n‖Y −ΦW‖2 + λ‖W‖2

oRepresenter’s theorem: W = Φ>α

Solve:α = (λI + ΦΦ>| {z }

K

)−1Y

The prediction function:

Y = f (x∗) = Φ(x∗)W = Φ(x∗)Φ>α = K(X, x∗)α

We use the RBF kernel function: K(xi ,xj) = exp(-‖xi − xj‖2/(2σ2))

Confidence on the prediction:

V[f (x∗)] = K(x∗, x∗)− K(x∗,X)(K + λI)−1K(X, x∗)

9 / 23

Intro Methodology Results Conclusions

Nonlinear regression

Key features

KRR generalizes LR

Tune two parameters: σ and λ

Fast for training (few hours) and test (25 ms/FOV)

Fast implementation

Standard code: >> alpha = inv(lambda*eye(n) + K) * Y;

Cholesky decomposition is ∼ 4 times faster:>> R = chol(gamma*eye(n) + K);

>> alpha = R\(R’\Y);

10 / 23

Intro Methodology Results Conclusions

Nonlinear regression

Key features

KRR generalizes LR

Tune two parameters: σ and λ

Fast for training (few hours) and test (25 ms/FOV)

Fast implementation

Standard code: >> alpha = inv(lambda*eye(n) + K) * Y;

Cholesky decomposition is ∼ 4 times faster:>> R = chol(gamma*eye(n) + K);

>> alpha = R\(R’\Y);

11 / 23

Intro Methodology Results Conclusions

Experiments

Datasets

Training done with ECMWFa Chevallier’s database:IASI cloud free, emissivity ‘sea’, ‘noise-free’FOVs: 13456

Methodology for training:Feature selection (Calbet, 2008): XX′ = [X, surface pressure, scan angle, latitude]Feature extraction: PCA with X′, and retain a number of features pLR: use all training data to estimate model weightsKRR:

cross-validation ( 23 , 1

3 ) to estimate model parametersuse all data to estimate model weights

aEuropean Centre for Medium-Range Weather Forecasts

Results

Real datasets, IASI orbits with 91800 FOVs

Predicted error profiles of temperature and water vapour

Confidence maps and detection of anomalies

12 / 23

Intro Methodology Results Conclusions

KRR testing at EUMETSAT over ocean ...

KRR clearly outperforms LRVery good results in water vapour

13 / 23

Intro Methodology Results Conclusions

KRR testing at EUMETSAT over land ...

KRR outperforms LR, which dramatically failsErrors are similar to estimations over the oceanTemperature errors are reasonable, while water vapour is really good 14 / 23

Intro Methodology Results Conclusions

KRR results

Results with and without cloud masking ...

0 5 10 15

102

103

RMSE [K]

p [h

Pa]

T

LR (all)KRR (all)LR (masked)KRR (masked)

0 5 10 15

102

103

RMSE [K]p

[hP

a]

Td

Clouds and anomalies are an important error source

Cloud screening is mandatory

An anomaly detector can be developed

15 / 23

Intro Methodology Results Conclusions

Predictions, discrepancies and confidence maps: Madagascar

AVHRR KRR confidence map

IASI cloud flag ∆T : |TECMWF − TKRR| Cloud flag ×∆T

16 / 23

Intro Methodology Results Conclusions

Predictions, discrepancies and confidence maps: Mexico coast

AVHRR KRR confidence map

IASI cloud flag ∆T : |TECMWF − TKRR| Cloud flag ×∆T

17 / 23

Intro Methodology Results Conclusions

Anomaly detection scheme

PCA SVM

Radiances

T predictionsT errors > 5K

Inputs: radiances and/or predictions

Output: nonlinear prediction of KRR big discrepancies with ECMWF

18 / 23

Intro Methodology Results Conclusions

Anomaly detection results

500 1000 1500 200078

80

82

84

86

88

90

92

Training samples

Tes

t OA

Overall accuracy

500 1000 1500 20000.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

Training samplesT

est κ

Kappa statistic, κ

LDA

QDA

MAHAL

TREE

SVM

Linear and nonlinear classifiers developed

Anomalies can be detected accurately (OA ∼ 92%, κ = 0.81)

SVM outperforms all other classifiers

19 / 23

Intro Methodology Results Conclusions

Conclusions

Developed and implemented KRR nonlinear regression

KRR outperforms LR

KRR provides confidence maps

Developed nonlinear anomaly detection methods

1 Thresholding the discrepancies to ECMWF, ∆ = |TECMWF − TKRR |2 KRR confidence on predictions, σT ∈ [0, 1]

3 SVM prediction of anomalies: ∼90%

Future work

Include better spatial information

Channel emissivity prediction

20 / 23

Intro Methodology Results Conclusions

Conclusions

Developed and implemented KRR nonlinear regression

KRR outperforms LR

KRR provides confidence maps

Developed nonlinear anomaly detection methods

1 Thresholding the discrepancies to ECMWF, ∆ = |TECMWF − TKRR |2 KRR confidence on predictions, σT ∈ [0, 1]

3 SVM prediction of anomalies: ∼90%

Future work

Include better spatial information

Channel emissivity prediction

21 / 23

Intro Methodology Results Conclusions

Conclusions

Developed and implemented KRR nonlinear regression

KRR outperforms LR

KRR provides confidence maps

Developed nonlinear anomaly detection methods

1 Thresholding the discrepancies to ECMWF, ∆ = |TECMWF − TKRR |2 KRR confidence on predictions, σT ∈ [0, 1]

3 SVM prediction of anomalies: ∼90%

Future work

Include better spatial information

Channel emissivity prediction

22 / 23

Intro Methodology Results Conclusions

Conclusions

Kernel-Based Retrieval of Atmospheric Profiles from IASI Data

Gustavo Camps-Valls, Valero Laparra, Jordi Munoz-Marı,Luis Gomez-Chova, Xavier Calbet∗

Image Processing Laboratory (IPL), Universitat de Valencia. Spaingustavo.camps@uv.es – http://isp.uv.es∗EUMETSAT, Darmstadt, Germany

IGARSS 2011, 25-29th July, Vancouver, Canada

23 / 23