tim smyth and jamie shutler assessment of analysis and forecast skill assessment using satellite...
TRANSCRIPT
Tim Smyth and Jamie Shutler
Assessment of analysis and forecast skill
Assessment using satellite data
• Rationale – what are we trying to achieve ?• Original approaches and web portal.• New metrics and approaches
– Annual SST comparisons
– Annual chl-a comparisons
– Weekly chl-a comparisons ( 2 examples)
• Conclusions.
Overview
Rationale
• Sat in too many conferences, squinting from the back, where models and satellite data were said to be “in good agreement”:– Highly dissatisfactory as is subjective;– Authors frequently play with scales to suit;– Not necessarily comparing like with like– SOLUTION: come up with comparison web site …
Advantages• Immediately shows up where discrepancy between satellite and model;• Gives potential for models to be improved over shorter time-cycles (need to
discuss this at the workshop);• The shame factor!
Disadvantages• Not necessarily comparing like with like;
• Only surface measurements
• Problems with cloud coverage and averaging;
• “What is truth?”
Some immediate results
• On short time-scales: poor agreement;– Spring bloom timing is crucial: time of dramatic changes.
• On seasonal time-scales: better agreement; (see Allen et al. (2008));– Not surprising as our models are based on trying to predict
things like seasonal succession.• There is some model skill in predicting the likelihood of
blooms.• However still need to produce more useful metrics to
judge different aspects of the model …
New metrics
Towards determining reasons for differences:• Principal component analysis (or EOF) of time series.• Kappa coefficient
– Method traditionally employed to look at land change.
– Measure of ‘difference’ between two sets of data.
• Receiver operator curves (ROC).– Used extensively in pattern recognition.
– Employed in precipitation analyses (Met Office).
– Performance graphing technique.
• Wavelet decomposition method.– Employed in precipitation analyses (Met Office).
– Spatial decomposition.
– Provides a measure of skill and mean squared error.
New metrics
General approach: • Metrics are automatically generated every week.• Comparing weekly composite data.• All data mapped to same scale and domain.• Common cloud masking (of composite).• Satellite data spatially averaged to approximate spatial scale of model data.
Two sets of results:• Annual analyses (February to October) • Weekly analyses
Annual : SST metric resultsVisual observations: spatially and temporally similar
Satellite PC1 Satellite PC2
model PC1 model PC2
Satellite temporal weighting
Model temporal weighting
Annual : SST metric resultsVisual observations: spatially and temporally similar
Satellite PC1 Satellite PC2
model PC1 model PC2
Satellite temporal weighting
Model temporal weighting
Annual : chlorophyll metric resultsVisual observations: spatially and temporally different
Satellite PC1 Satellite PC2
model PC1 model PC2
Satellite temporal weighting
Model temporal weighting
Weekly : 5 July chlorophyll metric results
Satellite MRCS model %age difference
5th July 2008
Visual observations:• model data appear to contain a large bias• some structures appear similar (changes in gradient)
Same geophysical scale
Weekly : 5 July chlorophyll metric results
Satellite MRCS model %age difference
5th July 2008
Visual observations:• model data appear to contain a large bias• some structures appear similar (changes in gradient).
Same geophysical scale Similar gradients (with bias)
Weekly : 5 July chlorophyll metric results
Satellite MRCS model %age difference
5th July 2008
Visual observations:• model data appear to contain a large bias.• some structures appear similar (changes in gradient).
Same geophysical scale Similar gradients (with bias)
Weekly : 5 July chlorophyll metric results
Kappa
MSE Skill score
ROC
Weekly : 5 July chlorophyll metric results
Kappa ROC
MSE Skill score
Data divergeAt ~0.5mg m-3
Errors at all scalesacross range of chlorophyll
-ve skill score for high chlorophyll, particularly at lower spatial scales
Better than randomperformance
Weekly : 11 October chlorophyll metric results
Satellite MRCS model %age difference
11th October 2008
Visual observations:• mixture of similar and dissimilar structures• model data inverse of satellite data in some regions.
Weekly : 11 October chlorophyll metric results
Satellite MRCS model %age difference
11th October 2008
Visual observations:• mixture of similar and dissimilar structures• model data inverse of satellite data in some regions.
opposite responses
Weekly : 11 October chlorophyll metric results
Satellite MRCS model %age difference
11th October 2008
Visual observations:• mixture of similar and dissimilar structures• model data inverse of satellite data in some regions.
Feature in model data not apparent in satellite
Weekly : 11 October chlorophyll metric results
Kappa ROC
MSE Skill score
Weekly : 11 October chlorophyll metric results
Data divergeAt ~0.1mg m-3
Performance closer to random
Errors at all scalesAcross range of chlorophyll
-ve skill score for high chlorophyll, at low spatial scales
ROCKappa
MSE Skill score
Conclusions
• Presented operational framework for evaluating MRCS vs satellite data• Apparent need to use multiple metric approach.• Seasonal signal captured well by model SST.• Chlorophyll comparisons show the biggest differences.• Performance of model chlorophyll varies (when compared with satellite
data).
• However, satellite chlorophyll signal in winter months is likely to be incorrect or biased (algorithm performance reduces during this time).
• Both datasets have uncertainties.
• Still not comparing like with like (e.g. temporal differences in input data, possible that binning the satellite data to match model data will improve results).