1

Joint OSSE Progress

Simulation of observation and initial results.

http://www.emc.ncep.noaa.gov/research/JointOSSEs

August 2010

Michiko Masutani

What is OSSE

Observing System Simulation Experiments (OSSEs) are typically designed to use data assimilation ideas to investigate the potential impacts of prospective observing systems.

In an OSSE, simulated rather than real observations are the input to a data assimilation system (DAS for short). Simulated observational values are drawn from the Nature Run, proxi truth atmosphere for OSSE.

The Nature Run is a long, uninterrupted forecast by a NWP model whose statistical behaviour matches that of the real atmosphere

Introductory presentation about data assimilation.

Jones, Andrew  S. 2008:What is Data Assimilation? A Tutorial, Presentation at AMS data assimilation forum.  AMS annual meeting, New Orleans, LA, January 20-24,2008http://www.emc.ncep.noaa.gov/research/JointOSSEs/references/Jones_Data_Assimilation_Ed_Forum.ppt

3

Real dataOSE

Observing System Experiment– Typically aimed at assessing the

impact of a given existing data type on a system

– Using existing observational data and operational analyses, the candidate data are either added to withheld from the forecast system, and the impact is assessed

– Control run (all operationally used observations)

– Perturbation run (control plus candidate data)

– Compare!

Simulated data (OSSE)Observing System Simulation

Experiment• Typically aimed at assessing the

impact of a hypothetical data type on a forecast system

• Simulated atmosphere (“Nature Run”)

• Simulated reference observations (corresponding to existing observations)

• Simulate perturbation observations (object of study)

• Verify simulated observation• Simulate observational error• Control run (all operationally used

observations)• Perturbation run (control plus

candidate data)• Compare!Costly in terms of computing and

manpower

Data Denial Experiments

DATA DUMP

OSSE Quality Control

(Simulated conventional data)

OSSEDATA DUMP

Quality Control (Real conventional data)

OSSE DA (Include new obs)

Real

Radiance data

DA

NWP forecast

Simulation of Existing data + Proposed data

(DWL, CrIS, ATMS, UAS, etc)

Verification of data

Adding Observational error

Nature RunCurrent observing system

NWP forecastGDAS OSSE

NCEP system

Simulated

Radiance data

DAS,OSE,OSSE Diagram

Diagram by

Lars Peter Riishojgaard JCSDA

7

Data impact on analysis and forecast will be evaluated.

A Full OSSE can provide detailed quantitative evaluations of the configuration of observing systems.

A Full OSSE can use an existing operational system and help the development of an operational system

.Existing Data assimilation system and velification method are used for Full OSSEs. This will help development of DAS and verification tools.

Full OSSEs

Masutani, M., T. W. Schlatter, R. M. Errico, A. Stoffelen, E. Andersson, W. Lahoz, J. S.. Woollen, G. D. Emmitt,L.-P. Riishøjgaard, S. J. Lord : Observing System Simulation Experiments. Chapter 24 of Data Assimilation: Making sense of observations, published from Springer in 2010. The final draft of the chapter which describe OSSE is posted athttp://www.emc.ncep.noaa.gov/research/JointOSSEs/references/Springer_Data_Assimilation_Ch24_OSSE_Draft.pdf

8

While “Rapid Response OSSEs” or “QuickOSSEs” may provide less expensive yet important and reliable insights to potential data impacts from proposed new observing systems, full OSSEs offer a more realistic representation of those data impacts on analyses and forecasts.

A Full OSSE can use the full extent of an existing operational forecast system and provide input to the preparation of that system for the ingestion of the new data set(s) in an operational setting (e.g. ADM).

Advantages of Full OSSEs

9

Why an International Joint OSSE capability

• Full OSSEs are expensive

– Nature Run, entire reference observing system, additional observations must be simulated. Sharing one Nature Run saves $$.

– Calibration experiments, perturbation experiments must be assessed according to standard operational practices and using operational metrics and tools

• OSSE-based decisions have international stakeholders

– Decisions on major space systems have important scientific, technical, financial and political ramifications

– Community ownership and oversight of OSSE capability is important for maintaining credibility

• Independent but related data assimilation systems allow us to test robustness of answers

OSSE Calibration

● In order to conduct calibration all major existing observations must be simulated.

● The calibration includes adjusting observational error.

● If the difference can be explained, we will be able to interpret the OSSE results as to real data impact.

● The results from calibration experiments provide guidelines for interpreting OSSE results on data impact in the real world.

● Without calibration, quantitative evaluation data impact using OSSE could mislead the meteorological community. In this OSSE, calibration is performed and presented.

Simulation of control data for calibration

● Simulation of control data require significant funding for specific OSSEs● GMAO and NCEP are simulating control data

Experience from the past

In 1993, ECMWF produced Nature Run in T213.Observations are simulated while the NR was producedThe data were distributed with significant cost

ProblemsRadiance data was not simulated correctly and cannot be used.SST was kept constant and that affect the OSSE results.Took long time to find fund to conduct OSSEs

OSSE using ECMWF T213 Nature Run data Stoffelen, A., G. J. Marseille, F. Bouttier, D. Vasiljevic, S. De Haan And C. Cardinali  2006:ADM-Aeolus Doppler wind lidar Observing System Simulation Experiment,  Quar.J.Roy. Metorol. Soc. , 619, 1927-1948 Focused on NH due to the lack of radiance data and showed DWL impact over ocean.

NOAA simulated observation including radiance data and conduct OSSEs to evaluation DWL.Masutani, M., J. S. Woollen, S. J. Lord, G. D. Emmitt, T. J. Kleespies, S. A. Wood, S. Greco, H. Sun, J. Terry, V. Kapoor, R. Treadon, and K. A. Campana, 2009. Observing System Simulation Experiments at the National Centers for Environmental Prediction. J. Geophys. Res., 114, doi:10.1029/2009JD012528.Results were mainly presented over NH, which has less affected by constant SST. (To be published from JGR)

Note: This data must not be used for commercial purposes and re-distribution rights are not given. User lists are maintained by Michiko Masutani and ECMWF.

New Nature Run by ECMWFProduced by Erik Andersson(ECMWF)

Based on discussion with JCSDA, NCEP, GMAO, GLA, SIVO, SWA, NESDIS, ESRL, and ECMWF

Low Resolution Nature Run Spectral resolution : T511 , Vertical levels: L91, 3 hourly dumpInitial conditions: 12Z May 1st, 2005 , Ends at: 0Z Jun 1,2006

Daily SST and ICE: provided by NCEPModel: Version cy31r1

Two High Resolution Nature Runs35 days long

Hurricane season: Starting at 12z September 27,2005, Convective precipitation over US: starting at 12Z April 10, 2006

T799 resolution, 91 levels, one hourly dumpGet initial conditions from T511 NR

13

To be archived in the MARS system at ECMWF To access T511 NR, set expver = etwu

Copies are available to designated users for research purposes & users known to ECMWF

Saved at NCEP, ESRL, and NASA/GSFC

Complete data available from portal at NASA/NCCS portal  http://portal.nccs.nasa.gov/osse

Contacts: Michiko Masutani (michiko.masutani@noaa.gov), and Password protected . Accounts arei arranged by Ellen Salmon (Ellen.M.Salmon@NASA.gov)

Gradsdods access is available for T511 NR. The data can be downloaded  in grib1, NetCDF, or binary. The data can be retrieved globally or for selected regions.

Provide IP number to: Arlindo da Silva (Arlindo.Dasilva@nasa.gov)

Archive and Distribution

14

Pressure level data: 31 levels, Potential temperature level data: 315,330,350,370,530K

Selected surface data for T511 NR: Convective precip, Large scale precip,

MSLP,T2m,TD2m, U10,V10, HCC, LCC, MCC, TCC, Sfc Skin

Temp

T511 verification data is posted from NCAR CISL Research Data Archive. Data set ID ds621.0. Currently an NCAR account is required for access.

(Also available from NCEP HPSS, ESRL, NCAR, NRL/MRY, Univ. of Utah, JMA, Mississippi State Univ., JPL and Environment of Canada as well as NASA/NCCS portal)

Supplemental low resolution regular lat lon data 1deg x 1deg for T511 NR

[Usage and credit] This data must not be used for commercial purposes and re-distribution rights are not given. ECMWF and Joint OSSEs must be given credit in any publications in which this data is used. If you are interested in using the data set it is necessary to send E-mail with the statement below.Please send following information to Michiko Masutani (michiko.masutani@noaa.gov).

your name, Email address, Affiliation, Title of the project (or how NR will be used)

Then your name will be added to the user list and sent to ECMWF. The usage permission will be given to individual not to the institute. Therefore, everyone has to submit above information. "I agree not to copy the ECMWF data for the use of other persons, and I agree not to use these data and/or software for commercial purposes. ECMWF will be given credit in any publications in which these data and/or software are used. I understand that if other persons in my organization wish to use these data and/or software, they must also sign a copy of this agreement."

16

Comparison between the ECMWF T511 Nature Run against

climatology20050601-20060531, exp=eskb,

cycle=31r1

Adrian Tompkins, ECMWF

NR

MODIS

NR-MODIS

Utilize Goddard’s cyclone tracking software. - By J. Terry(NASA/GSFC)

Evaluation of the Nature run

Vertical structure of a HL vortex shows, even at the degraded resolution of 1 deg, a distinct eye-like feature and a very prominent warm core. Structure even more impressive than the system observed in August. Low-level wind speed exceeds 55 m/s.

Tropics by Oreste Reale (NASA/GSFC/GLA)

Seasonal mean zonal mean zonal wind jet maximum strength and latitude of the jet maxima for the ECMWF reanalysis (1989-2001, blue circles) and the Nature Run (), northern hemisphere. (N. Prive.)

Time series showing the night intensification of the LLJ at the lee of the Andes in the simulation.Gridpoint at 18 S / 63 W

THE SOUTH AMERICAN LOW LEVEL JET

Juan Carlos Jusem (NASA/GSFC)

M.Masutani (NOAA/NCEP)

-90 -60 -30 0 30 60 90L a titu d e

0

10

20

30

40

50

60

70

80

90

100

Tota

l Clo

ud C

ove

r (%

)

Tota l C loud C over (Land and O cean)

- N R- ISC C P- W W M C A-- H IR S

Evaluation of T511(1°) clouds

by SWA

17

T511 Nature Run is found to be representative of the real atmosphere and suitable for conducting reliable OSSEs for midlatitude systems and tropical cyclones. (Note: MJO in T511 Nature Run is still weak.)

There are significant developments in high resolution forecast models at ECMWF since 2006 and a more realistic tropics for T799 Nature Run is expected with a newer version of the ECMWF model.

ECMWF agreed to generate a new T799 NR, when the Joint OSSE team has gained enough experience in OSSEs with T511NR and is ready to make the best use of the high resolution Nature Run.

For the time being, the Joint OSSE team will concentrate on OSSEs using the T511 Nature Run.

Andersson, Erik and Michiko Masutani 2010: Collaboration on Observing System Simulation Experiments (Joint OSSE), ECMWF News Letter No. 123, Spring 2010, 14-16. http://www.ecmwf.int/publications/newsletters/pdf/123.pdf

Reale O., J. Terry, M. Masutani, E. Andersson, L. P. Riishojgaard, J. C. Jusem (2007), Preliminary evaluation of the European Centre for Medium-Range Weather Forecasts' (ECMWF) Nature Run over the tropical Atlantic and African monsoon region, Geophys. Res. Lett., 34, L22810, doi:10.1029/2007GL031640.http://www.agu.org/pubs/crossref/2007.../2007GL031640.shtml

Simulated of radiance Michiko Masutani and Jack Woollen

NOAA/NWS/NCEP/EMCTong Zhu, Haibing Sun, Tom Kleespies, Yong Han, Fuzhong Weng

NOAA/NESDISLars Peter Riishojgaard

JCSDA

For full OSSE, existing observation has to be simulated. This is an initial investment.Flexile Radiance data Simulation strategies at NCEP

20

Flexible Radiance data Simulation strategies at NCEP

Content of DBL91 Nature Run data at foot print

91 level 3-D data (12 Variables)

2-D data (71 Variables)

Climatological data

All information to simulate Radiances

The DBL91 also used for development of RTM.DBL91 can be processed for other sampling such as GMAO samplingDBL91 can be processed for new observation

It is an option whether DBL91 to be saved and exchange among various project, or DBL91 to be treated as temporary file produced in simulation process. This depends on size of DBL91 compare to the Nature Run.

DBL91

Experts for data handling and experts of RTM are different people.

21

Nature Run(grib1 reduced Gaussian)91 level 3-D data (12 Variables)2-D data (71 Variables)Climatological data

Observation templateGeometryLocationMask

Decoding grib1Horizontal Interpolation

DBL91

Running Simulation program (RTM)

Post Processing (Add mask for channel, Packing to BUFR)

Simulated Radiance Data

Need large cpuNeed Radiation experts

Need complete NR (3.5TB)Random access to grib1 data

Need Data Experts

Need Data Experts but this will be small program

Compare Tb(NR) with Tb(OBS)AMSU-A 20050502 00Z

Old NCEP.v2 simulation Observation

NOAA-17 HIRS3 Ch-07

CRTM SimulationObservation

IASI simulation Evolution at Windows channel

IASI simulation for 00Z May 2nd 2005

Using template based on usage on 00Z may 2nd 2009

Compared with observation at 00Z May 2nd 2005

IASI simulation over ocean( Clear atmosphere)

Dataset Originating Institute Contact

NCEP Obs NOAA/NCEP Michiko Masutani (Michiko.Masutani@noaa.gov)

NCEP-NESDIS NOAA/NCEP Michiko Masutani (Michiko.Masutani@noaa.gov)

NOAA/NESDIS

NCEP-NESDIS data posted in December 2009 http://portal.nccs.nasa.gov/josse

NCEP-NESDIS

osbuvb.n_t511.v0906prepbufr.n_t511.v0903thinsats.n_t511.dbl91.v0909

NCEP_Obs

thinsats_rad.gdas.mask.v0909thinsats_rad.gdas.v0905

Subdirectories under each data set

 [Interpolation program by Jack Woollen] Horizontal interpolation code by Jack Woollen is posted athttp://www.emc.ncep.noaa.gov/research/JointOSSEs/Manuals/NCEP_SimObs/JW-mkthinrad/ The directory contains software by Jack to do sampling and  horizontalinterpolation.  He is using random access to grib data and that made the program verysimple.   Please contact Jack Woollen Jack.Woollen@noaa.gov for the detail. [Libraries used in the programs]grib decorder from ECMWF can be downloaded fromhttp://www.ecmwf.int/products/data/software/download/gribex.htmlManual of gribx is posted fromhttp://www.ecmwf.int/publications/manuals/libraries/gribex/subroutineGribex.htmlIf you are interested in more recent version of ECMWF gribdecordergrib API is available fromhttp://www.ecmwf.int/publications/manuals/grib_api/ w3lib and bacio lib are available fromhttp://www.nco.ncep.noaa.gov/pmb/codes/GRIB2/ 

The simulated radiance using CRTM1.2.2, at foot print based on usage by 2005 GDAS is reasonable. The simulated is continued to complete entire period of T511 Nature run.

Observational error will be added based on method developed by T. J. Kleespies.

GMAO also provided softwarre to add random error.

Calibration will be performed for observational errors

At NOAA currently OSSE experiments are conducted for July 2005 using data provided by GMAO. Preliminary calibrated data are produced for this period.

Plans

Preliminary simulated radiances are being updated  atftp://ftp.emc.ncep.noaa.gov/exper/mmasutani/SIMOBS/NOAA_RAD/ The data need to be evaluated

Simple description is provided but these are posted to help people who are interested in simulating observation and not design to be potable. Please contact Jack.Woollen@noaa.gov and Michiko.masutani@noaa.govfor the detail.We appreciate any comments and improvement to the code.

Simulation of radiance is done using CRTM REL-1.2.2 but CRTM REL-2.0.2 is available from ftp://ftp.emc.ncep.noaa.gov/jcsda/CRTM/ We appreciate if anyone can upgrade these code to REL-2.0.2 and share with Joint OSSE.

NCEP will post simulated observation as progress. First we post from NCEP ftp site to be evaluated. After the data is evaluated it will be transferred to NASA NCCS portal. We appreciate any help in evaluation of the simulated observations.

GMAO is also working on simulation of radiance. Lars Peter Riishojgaard is the contact person for GMAO OSSE. Please contact him if you have any questions regarding GMAO OSSEs. We hope all efforts may be merged in some stage.

Remarks

Simulated TC vital

TC-vital for 13 monthhttp://www.emc.ncep.noaa.gov/research/JointOSSEs/Manuals/NCEP_SimObs/TC/

The software are posted from http://www.emc.ncep.noaa.gov/research/JointOSSEs/Manuals/NCEP_SimObs/TC/Software/

The simulated data is also posted from ftp://ftp.emc.ncep.noaa.gov/exper/mmasutani/SIMOBS/TC-vital/T511-v3

[Simulation of TC vital]TC vital was simulated using software originally written by Tim Marchock and currently developed by Guan Ping Lou of NCEP. The simulated observation has not been evaluated.

Testing Line-Of Sight (LOS) analysis code FY10 GDAS implemented on December 15, 2009

Jack Woollen, Michiko Masutani, Lars Peter Riishojgaard, Zaizhong Ma

AcknowledgementDave Emmitt, Sid Wood, Steve Greco, Daryl Kleist, John Derber

Back ground Single Obs test using (U,V) and two LOS are suggested by Dave Emmitt to test

Early test was conducted in 2006 by Jack Woollen, Yuanfu Xie, Joe Terry, Yucheng Song, Genia Brin and M. Masutani but the results was not conclusive and the summary was not produced.

Assimilation code for DWL was originally written by John Derber in 1997. Further development added by Weiyu Yang and other people at NCEP.

Observation exactly same as the Nature Run

Date: July 3rd 6z ,2005 Position: 96W 29S 200mb height 11991mWind: U=46.00 V=19.18

Single obs test

Input as two LOSDATA(ExactNR) (96W 29S) Pressure level=200 HGT=11991 Representativeness Error =3.0LOS 34.51 BEARAZ=90 ELEV= 41.38deg, and LOS 14.39 BEARAZ=0 ELEV=41.38deg

U and V observation as pibalPosition: 96W 29S 200mb height 11991m, Wind:U=46.00 V=19.18Representativeness Error =3.0

U200

U200

V200

V200

Shading: anal-ges

34

Global Wind Observing Sounder (GWOS)

Observing Systems Simulation ExperimentsFor Global Wind Observing Sounder

Dual Technology Sampling

• The coherent subsystem provides very accurate (< 1.5m/s) observations when sufficient aerosols (and clouds) exist.

• The direct detection (molecular) subsystem provides observations meeting the threshold requirements above 2km, clouds permitting.

• When both sample the same volume, the most accurate observation is chosen for assimilation.

• The combination of direct and coherent detection yields higher data utility than either system alone. Note that in the background aerosol mode, the combination of the coherent and direct provide ~ 20 % more coverage near 3 -5 km than could either technology by itself.

36

Hybrid Doppler Wind LidarHybrid Doppler Wind LidarMeasurement Geometry: 400 kmMeasurement Geometry: 400 km

350 km/217 mi53 secAlong-Track Repeat“Horiz. Resolution”

586 km/363 mi

GWOS Sampling

Reduction of RMSE from NR compared to CTLAverage between 00 ZJuly 3 -18Z July 3

Red: Positive impactBlue: negative impact

First look of Impact of GWOS with Hybrid DWLVery Initial results

Preliminary initial condition and control data

V200U200

U850

V500U500

V850

Reduction of RMSE from the Nature run by GWOSAveraged between July 3rd 00Z and July 3rd 18z

Area averaged between 60S to 60N

UV

First look for three day experiments for three days.

Hybrid GWOS showed positive strong impact in analysis

The positive analysis impact is centered over tropics and spread to mid latitude. Negative impact over both poles.

Further investigation to improve LOS data assimilation code may increase the impact of DWL data.

Simulated observation require more realistic observational error. Further calibration is required to gain the confidence in results.

These results do not guarantee positive impact GWOS DWL in forecast. Longer time experiments are required for forecast impact.

Further work

The experiments are done using preliminary initial condition and simulated observation. Need to be repeated with clean initial condition and control data.

Complete longer time period (minimum 6 week) to study forecast verification.

The experiments for different season particularly NH winter

More metric reduction of failed forecast (See slides 41-43) Cyclone track forecasts.

Investigate the cause of the negative impact.

Test using flow dependent error covariance being developed at NCEP.

Evolution of GFS Forecast Skill

S. Lord and Fanglin Yang

5/20/2010

500 hPa Anomaly Correlation

• “But these are just ‘width of the line’ improvements…”• Why are these changes important to users?• How can you justify $M for new computing power based

on these anticipated improvements?

Does It Make a Difference toHow Forecasters Use Product?

Percent Good Forecasts

NCEP

Percent Poor Forecasts

NCEP

Useful References

Basic guide lines for Full OSSE

Masutani, M., T. W. Schlatter, R. M. Errico, A. Stoffelen, E. Andersson, W. Lahoz, J. S. Woollen, G. D. Emmitt,L.-P. Riishøjgaard, S. J. Lord : Observing System Simulation Experiments. The final draft of the chapter which describe OSSE is posted athttp://www.emc.ncep.noaa.gov/research/JointOSSEs/references/Springer_Data_Assimilation_Ch24_OSSE_Draft.pdf

This is Chapter 24 of Data assimilation:Making Sense of ObservationLahoz, William, Khattatov, Boris, Menard, Richard (Eds.) 2010: Data Assimilation, Making Sense of Observation.. Springer, 732 p., Hardcover ISBN: 978-3-540-74702-4http://www.springer.com/earth+sciences+and+geography/computer+%26+mathematical+applications/book/978-3-540-74702-4

Introductory presentation about data assimilation.

Jones, Andrew  S. 2008:What is Data Assimilation? A Tutorial, Presentation at AMS data assimilation forum.  AMS annual meeting, New Orleans, LA, January 20-24,2008http://www.emc.ncep.noaa.gov/research/JointOSSEs/references/Jones_Data_Assimilation_Ed_Forum.ppt

Publication about Joint OSSE Nature Run

Andersson, Erik and Michiko Masutani 2010: Collaboration on Observing System Simulation Experiments (Joint OSSE), ECMWF News Letter No. 123, Spring 2010, 14-16. http://www.ecmwf.int/publications/newsletters/pdf/123.pdf

Reale O., J. Terry, M. Masutani, E. Andersson, L. P. Riishojgaard, J. C. Jusem (2007), Preliminary evaluation of the European Centre for Medium-Range Weather Forecasts' (ECMWF) Nature Run over the tropical Atlantic and African monsoon region, Geophys. Res. Lett., 34, L22810, doi:10.1029/2007GL031640.

http://www.agu.org/pubs/crossref/2007.../2007GL031640.shtml

Publication for OSSEs with T213 Nature Run

Masutani, M, John S. Woollen, Stephen J. Lord, G. David Emmitt, Thomas J. Kleespies, Sidney A. Wood, Steven Greco, Haibing Sun, Joseph Terry, Vaishali Kapoor, Russ Treadon, Kenneth A. Campana (2010), Observing system simulation experiments at the National Centers for Environmental Prediction, J. Geophys. Res., 115, D07101, doi:10.1029/2009JD012528. http://www.agu.org/pubs/crossref/2010/2009JD012528.shtml

Masutani, M. K. Campana, S. Lord, and S.-K. Yang 1999: Note on Cloud Cover of the ECMWF nature run used for OSSE/NPOESS project. NCEP Office Note No.427http://wwwt.ncep.noaa.gov/officenotes/NOAA-NPM-NCEPON-0006/01408B76.pdf

Becker, B. D., H. Roquet, and A. Stoffelen 1996: A simulated future atmospheric observation database including ATOVS, ASCAT, and DWL. BAMS, 77, 2279-2294.http://ams.allenpress.com/archive/1520-0477/77/10/pdf/i1520-0477-77-10-2279.pdf Errico, R.M., R. Yang, M. Masutani, M., and J. Woollen, 2007: Estimation of some characteristics of analysis error inferred from an observation system simulation experiment. Meteorologische Zeitschrift, 16, 695-708.http://www.emc.ncep.noaa.gov/research/JointOSSEs/references/Errico_GB_071005.ps

Stoffelen, A., G. J. Marseille, F. Bouttier, D. Vasiljevic, S. De Haan And C. Cardinali 2006:ADM-Aeolus Doppler wind lidar Observing System Simulation Experiment, Quar.J.Roy. Metorol. Soc. , 619, 1927-1948

Other references related to OSSE and simulation experiments for DWL

Arnold, C. P., Jr. and C. H. Dey, 1986: Observing-systems simulation experiments: Past, present, and future. Bull. Amer., Meteor. Soc., 67, 687-695.

Atlas, R. 1997: Atmospheric observations and experiments to assess their usefulness in data assimilation.  Journal of the Meteorological Society of Japan, 75, 111-130.

http://www.emc.ncep.noaa.gov/research/JointOSSEs/references/Beesley_ECMWF_SHEBA_Arctic_2000JD900079.pdfHalem, H. and R. Dlouhy, 1984: Observing system simulation experiments related to space-borne lidar

wind profiling. Part 1: Forecast impacts of highly idealized observing systems. Preprints, Conference on Satellite Meteorology/ remote Sensing and Applications, Clearwater, Fla., Americal Meteorological Society, 272-279.

Kalnay, E, Jusem, J C and J Pfaendtner, 1985. The relative importance of mass and wind data in the FGGE observing system. Proceedings of the NASA Symposium on Global Wind Measurements, Clumbia, MD, NASA, 1-5.

Lorenc,A.C., Graham,R.J., Dharssi,I., MacPherson,B., Ingleby,N.B., Lunnon,R.W.,1992, Preparation for the use of a Doppler wind lidar information in meteorological assimilation systems, ESA-CR(P)-3454 http://www.emc.ncep.noaa.gov/research/osse/NR/references/Lorenc.1992.TIDCCR4129.pdf

Lord, S.J., E. Kalnay, R. Daley, G.D. Emmitt, R. Atlas, 1997: Using OSSEs in thedesign of future generation integrated observing systems.  Preprints, 1st Symposium on Integrated Observing Systems, Long Beach, CA, AMS, 45-47.

http://www.emc.ncep.noaa.gov/research/osse/NR/references/Lord_ams97.htmlMarseille, G.J., Stoffelen, A., Barkmeijer J. , 2008a, Sensitivity Observing System Experiment (SOSE) - A New Effective

NWP-based Tool in Designing the Global Observing System, Tellus, 60A, 216–233. Marseille, G.J., Stoffelen, A., Barkmeijer J. , 2008b, Impact Assessment of Prospective Space-Borne Doppler Wind Lidar

Observation Scenarios, Tellus, 60A, 234-248.Marseille, G.J., Stoffelen, A., Barkmeijer J. , 2008c, A Cycled Sensitivity Observing System Experiment on Simulated

Doppler Wind Lidar Data during the 1999 Christmas Storm "Martin" , Tellus, 60A, 249-260.Tan, D.G.H., E. Andersson, M. Fisher and L. Isaksen 2007: Observing system impact assessment using a data

assimilation ensemble technique: Application to the ADM-Aeolus wind profiling mission .  Q.J.Roy.Met.Soc, 133, 381-390.