mosac-17 - met office · pdf filemosac-17 14-16 november 2012 ... document sets out proposed...

15
© Crown Copyright 2012 1 MOSAC-17 14-16 November 2012 ANNEX V Met Office ‘Scientific Excellence’ Corporate Objective Performance Measures draft proposal

Upload: dangnguyet

Post on 18-Mar-2018

221 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 1

MOSAC-17

14-16 November 2012

ANNEX V

Met Office ‘Scientific Excellence’ Corporate Objective Performance Measures – draft proposal

Page 2: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 2

The Science Strategy Implementation Plan (Annex II) sets out the top level plan to enable the Met Office to deliver the Science Strategy 2010-2015. This is reflected within the Met Office Corporate Plan under the ‘Scientific Excellence’ corporate objective. Work is ongoing during FY1213 to develop a set of appropriate performance measures with which to assess progress under the ‘Scientific Excellence’ corporate objective. This document sets out proposed metrics for discussion, and some information on the current baseline where available. The following table summarises the metrics selected and to be defined.

Scientific Excellence Corporate Objective – performance measures: DRAFT PROPOSAL FOR DISCUSSION

Measure Metrics

A.

Cap

ab

ilit

y

A1. Model performance across timescales

a) Global model capability

b) Global and UK data assimilation capability

c) Ocean data assimilation capability

d) UK model capability

A2. Critical activities a) in-year theme plan ‘underpinning’ deliverables b) longer-term implementation plan deliverables

A3. R&D pull-through

A4. Shared funding Total non-government income originated from EUMETSAT, ESA, EU, NERC during a financial year Monitoring indicator only

A5. Collaboration benefit i) Estimated gearing benefit from collaborations (revenue or effort)

ii) UM Partnership net benefit iii) MOAP directed effort iv) MONSooN utilisation

A6. External review Independent assessments of Met Office science quality within year (e.g. MOSAC, SRG, government, customers, etc)

B.

Re

pu

tati

on

Publication record Basket of measures, based on Web of Science (e.g. number of papers, citations, h-index, impact, % intl.)

Committee Service Number of external committee memberships; proportion of staff

Studentship grants Number of Met Office-sponsored CASE studentships

External Appointments Number of part-time staff appointments with collaborators

Workshops and conferences Number of workshops and conference attendances; proportion of staff

Awards Number of awards/honours * Note Reputation metrics are mostly for monitoring purposes only

Page 3: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 3

A. Capability Indicators

A1. Model performance across timescales

The existing set of metrics that are monitored as part of the Science programme have largely been specified from the point of view of customer requirements for forecast accuracy, rather than as a means of tracking the improving capability of the forecasting systems to reproduce the observed structures of the atmospheric circulation. As the pressure on forecast accuracy monitoring is to become even more user focused, with metrics limited to specific sites and impacts-related variables and thresholds, it is important to ensure that suitable metrics are used to track and justify improvements in underlying model capability. This paper represents a first attempt to define such a set of metrics. It is currently a draft for discussion, and will no doubt be altered considerably before being implemented. The metrics are currently classified into six categories, measuring model capability and data assimilation (initial state) capability at three model scales: UK, global atmosphere and global coupled. In the event, the global atmosphere and global coupled model capabilities have been combined in one set of metrics, separate from the UK model capability, while data assimilation capability currently deals with global and UK atmosphere models together but the global ocean separately. It will be noted that several aspects of capability are not yet measured in this proposal, emphasizing the needs for further work before implementation.

a) Global Model Capability (atmosphere-only and coupled)

The following set of metrics aims to summarise global model performance across weather and climate timescales and measure improving capability through delivery of science strategy and programme. Both atmosphere-only and coupled models are included, though some metrics are more relevant to coupled model performance. Since these are a subset of the INTEGRATE scores, it is proposed that the metrics should be extracted for presentation during the annual INTEGRATE release review.

Rationale: 1. Develop a subset of metrics that reflect main model systematic errors as identified in Process Evaluation Groups (PEGS1) under INTEGRATE project and calculated in Auto Assess and TRUI/VER packages 2. Measures of both mean errors (bias) and errors in modes of variability. 3. Choose a subset of metrics that can be measured relative to other models as a benchmark of improving capability for both operational NWP (CBS scores) and climate prediction (CMIP3/CMIP5 - see Gleckler et 1998). This requires some coordination and the metrics chosen here do all fulfill this objective. 4. Metrics should eventually cover all aspects of physical coupled Atmosphere-Ocean-Land-SeaIce-Aerosol system. Also involve process based diagnostics. 5. Presentation/visualisations of the metrics is also important in order to see pattern of change. See Figures 1 and 2. 6. How often should this verification be measured? Our proposal is once a year for new model versions as part of the GA/GO/GL/GSI assessment process, drawing on NWP case studies and trials, MOGREPS testing, seasonal hindcasts, and multi-decade climate runs. 7. Review – the metrics should be reviewed on a yearly basis to ensure they are fit for purpose and reflect changing customer requirements and evolving model performance. Work is ongoing to develop a summary metric for each of the PEGS and the likelihood is these could eventually replace the single measures suggested here.

1 Current PEGS and top model issues: i) Processes over Africa; ii) Global Monsoons; iii) MJO and

Teleleconnections; iv) North Atlantic Ocean biases; v) Southern Ocean Biases; vi) Clouds and radiation; vii) ENSO and Teleconnections; viii) Continental surface biases; ix) Storms and blocking; x) Tropical cyclones; xi) Polar processes; xii) Conservation.

Page 4: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 4

Metrics – First Draft Metrics chosen here are a subset of the hundreds of metrics actually used to evaluate new atmosphere and coupled model versions as part of the GA/GO/GL/GSI assessment process. They have been chosen to reflect the current top model biases as identified at the UM user workshop in June 2012. 1. Extra-tropical variability (storms and blocking PEG)

1.1. NWP index vs. other NWP centres - as measure of synoptic scales. 1.2. Tibaldi-Molteni/Pelly-Hoskins Blocking index. - applied to NWP and MOGREPS timescales

(biases and skill). Similar index can be applied to seasonal and climate predictions. 1.3. Extratropical cyclones

1.3.1. NWP & MOGREPS - intensity and track measures as for tropical cyclones (see below). These need to be developed (TRACK software)

1.3.2. Seasonal and climate? Stormtracks and NAO variability 1.4. Jets - intensity and position - requires feature tracking software (Marion?).

2. Tropical cyclones 2.1. NWP & MOGREPS- skill measures for track prediction in various basins (J.Hemings

current measures). Add intensity/depth measures. 2.2. Seasonal and climate – Tropical cyclone numbers?

3. MJO 3.1. NWP & MOGREPS - use Wheeler and Hendon index (skill scores). Currently being carried

out as part of (John Gotschalch). 3.2. Seasonal and climate – Choose suitable metric from current set (consult with Prince

Xavier).

4. Clouds and radiation 4.1. NWP – 4.2. Seasonal and Climate –

5. SST biases 6. ENSO

6.1. Niño 3 & 4 SST from seasonal and multi-decadal climate runs 6.2. Wind stress in east Pacific (as above).

7. Africa 7.1. African easterly waves (Do we have a summary metric for this – Caroline?). 7.2. West African Monsoon Onset date.

8. Asian Monsoon 8.1. Intra-seasonal variability - Std.dev. of precipitation in 30–50 day JJAS – sub-regions

9. Water Cycle

9.1. Global P, E and P-E (all timescales) vs GPCP/TRMM 9.2. Regional P,E and P-E - NH, SH, Tropics, land, ocean, monsoon regions etc. (all

timescales) vs GPCP TRMM

10. Near Surface Weather (Continental surface biases PEG) 10.1. 1.5m Temperature and humidity vs SYNOPS – selected regions 10.2. 1.5m Temperatures vs CRUTEM climatology – selected regions

Page 5: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 5

Figure 1: Example of current metric set for the Global Troposphere (A1) assessment (main panel). The aim is to eventually develop a summary metric for each of the assessment areas (A1,A2) as in right hand panels.

Figure 2: Metric of model performance for a range of parameters across CMIP3 models from Gleckler et. al. (2008). The metric itself is a measure of relative error using error variances, where a figure 0f +0.2 implies a given model is 20% better than the ensemble mean.

Reference Gleckler, P. J., K. E. Taylor,and C. Doutriaux (2008): Performance metrics for climate models. JOURNAL OF GEOPHYSICAL RESEARCH, VOL. 113, D06104,doi:10.1029/2007JD008972

A1 A2A1 metrics

Gleckler et al. [2008]

Page 6: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 6

Figure 3: Examples of MJO metrics from Auto Assess.

Page 7: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 7

b) Global and UK Data Assimilation Capability

We wish to define an objective methodology to track improvements in the quality of the output of data assimilation, i.e. the analysis, used as the initial condition for global and convective-scale NWP. Clearly, the quality of the analysis feeds directly into the skill of the subsequent NWP forecast, which will have its own metrics. However, the motivation here is to provide additional diagnostics from the data assimilation system itself that can be used to complement, and inform an assessment of the skill and monitor improvements to the overall NWP system. 1. Observation Minus Background Statistics For both global and convective-scale data assimilation, the fit to observations of the background (short, 1-6hr NWP forecast: O-B) is a fundamental data assimilation diagnostic. Statistical are available for all observations assimilated (and also those being monitored in preparation for potential assimilation) – a far wider range than is available for standard observation-space verification (O-F) software. Two examples are given in Fig. 1:

Figure 1: Observation minus background (short-range forecast) differences for surface pressure (a., left) and selected radiance channel (MetOP AMSUA5 – b. right) for each month between 2009 and 2012.

The reduction in mean daily surface pressure O-B between 2009 and 2012 is clearly shown in Fig.1a, indicating a continually improving background field (at this short range, it is assumed that it is the improvements in DA that dominate those of the very-short range forecast). Fig 1b illustrates one of the challenges of measuring improvements to data assimilation from O-B statistics: in AMSU Channel 5 radiance space, there is a much less clear signal due to the more complex (radiative transfer model) observation operator and temporally varying observation (radiance) bias corrections. So, although data assimilation experts will use O-B statistics from the full range of observation types in their development work, it is prudent to choose only a subset of observations as an external diagnostic of improving data assimilation performance. Proposal 1: Produce monthly summary O-B statistics (mean, variance) geographic distribution) for operational NWP for the following observations types:

a) Global: surface pressure (land and sea), radiosonde/aircraft (temperature, wind), GNSS (temperature profiles). Global (+ monitor NH/Tr/SH) averages. b) UK convective-scale NWP: surface temperature/winds, aircraft/sonde humidities, visibility. Cloud amount and reflectivities should be considered in future once direct assimilation makes use of O-B statistics appropriate (currently assimilated indirectly via humidity retrievals and latent heat nudging respectively) and confidence in the

Page 8: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 8

accuracy/representiveness of the observation is gained (currently observation errors are poorly understood and represented).

Specific deficiencies: Impact of data assimilation on the PBLs’ humidity profile near inversions (case study comparisons of O-B, O-A during Sc events). 2. Analysis Fields and Increments In addition to the observation space O-B diagnostic, the size of the gridded analysis increments (A-B) is informative in measuring both the quality of the assimilation as well as model bias; a better model requires a smaller correction through the data assimilation, and hence mean A-B values should reduce as models and data assimilation algorithms improve. Caveats here are that a) It is difficult to separate out reductions in model bias from data assimilation improvements, and b) The introduction of new observation types in previously data-sparse areas may increase A-B values (in this situation, the assimilation previously had no observations to show the local forecast was biased). Proposal 2: Produce monthly summary A-B statistics (mean and variance) and maps for operational analysis increments to inform assessment of assimilation/model improvements. Variables selected could be those that appear in the relative global and UK indices (note: in research mode, a wider range of fields will be considered e.g. as part of the INTEGRATE project’s process evaluation groups). Proposal 3 (in collaboration with verification effort): Perform routine verification of operational global NWP against observations, analysis, and ECMWF, NCEP and JMA analyses. Reductions in the differences between observation/MetO analysis based verification indicate improvements in the representation of truth by the analysis (not necessarily the same thing as the analysis improving forecast skill). Specific deficiency: Analysis/observation-based verification of wind forecasts differs greatly in the tropics. Cross-validation against ECMWF analyses indicates our tropical analysis is the outlier. 3. Ensemble Metrics The spread of ensemble solutions is used as a primary indicator of confidence in probabilistic NWP products. The detailed structure of ensemble perturbations is also used to define complex, flow-dependent forecast error covariances within ensemble data assimilation (EnDA). Ensemble improvements via the EnDA are measured by the primary data assimilation diagnostics proposed above, and resulting NWP forecast metrics. Hence, the focus here is in assessing the quality of the raw (i.e. unprocessed) ensemble itself, specifically the difference between ensemble spread and forecast error (derived from observation minus ensemble mean forecast differences). An example is shown in Fig. 2 below. Proposal 4: Continue to monitor:

a. Global (MOGREPS-G) ensemble: Difference in raw ensemble spread/skill as a function of forecast lead time for the parameters of the global NWP index.

Specific deficiency: Rate of growth of spread with forecast lead time for above variables. b. Convective-scale (MOGREPS-UK) ensemble: Difference in raw ensemble spread/skill as a function of forecast lead time for the parameters of the UK NWP index, including precipitation (caveat: precipitation is a priority, but precipitation skill is notoriously difficult to pin down e.g. observation errors, representivity).

Specific deficiency: Improving representation of small-scale uncertainty via initial and model ensemble perturbations tailored for the convective-scale.

Page 9: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 9

Figure 2: Spread (asterisks) and skill of ensemble mean (crosses) as a function on forecast range for NH MSLP (left) and surfacew wind (right). The difference between the two is a fundamental measure of improvements to the raw ensemble.

4. Secondary (internal) Data Assimilation and Ensemble Diagnostics There a wide range of secondary data assimilation diagnostics used in day-to-day studies by DAE staff. Examples include: spin-up/down (rapid systematic changes in cloud, precipitation, vertical velocity, etc in the first few hours of the forecast), speed of minimization convergence, weak/strong computational scalability, ‘Desroziers’ diagnostics of the accuracy of observation/forecast error estimates, comparison of observation/analysis based verification, Brier skill scores, rank histograms, etc. It is proposed that these are excluded from external monitoring of improvements to the data assimilation and ensemble prediction systems as their interpretation can be difficult for non-experts.

Page 10: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 10

c) Ocean Data Assimilation Capability

Eventually it will be important to measure aspects of the consistency of initialization of ocean and atmosphere. However, for the present we limit this assessment to measures of aspects of the ocean state that will be important contributors to the evolution of both the ocean state and the atmosphere state. It is proposed to track the bias and RMS errors in short range forecasts of the following quantities, measured against observations:

sea surface temperature

sea surface height

surface velocities

sea-ice concentration

temperature profiles

salinity profiles

d) UK Model Capability

Routine metrics of sensible weather elements measured for the UK Index. These measure the accuracy of individual variables. For cloud, visibility and precipitation, thresholds are chosen so as to give priority to evaluating the prediction of potentially hazardous weather.

The list of variables is as follows:

(a) screen temperature (b) 10m wind speed and direction (c) Visibility (≤200m, ≤1000m, ≤4000m) (d) cloud cover (≥2.5oktas, ≥4.5oktas, ≥6.5oktas) (e) cloud base height (≤100m, ≤300m, ≤1000m) (f) hourly precipitation (≥0.5mm, ≥1mm, ≥4mm)

Current scores are site specific RMSVE for wind, RMS for temperature and ETS for other variables, vs point observations, each normalized to produce a skill score. However, Mittermaier (2012) has proposed using Continuous Ranked Probability Skill Score (CRPSS) for temperature, the Ranked Probability Skill Score (RPSS) for wind speed (in Beafort forces) and the Brier Skill Score (BSS) for the thresholded variables, with neighbourhood processing to avoid the double penalty and to mitigate against representativeness errors of the observations. It is proposed that these recommendations are adopted at the earliest opportunity. UK Index metrics are unable to track improvements in the model’s capability to predict specific phenomena, such as radiation fog, thunderstorms or boundary layer cloud. Weaknesses in these capabilities are, however, identified by forecasters and fed into the Operations and NWP Problem Database, maintained by the NWP Problem Group. Research is carried out to improve performance in these respects and case studies are used to track improvements from changes prior to their implementation in parallel suites. It is proposed that the problems classified as critical or major, that relate to the UK models, should be tracked using results from case study and parallel suite comparisons. The current set of such model weaknesses is as follows:

(a) Poor prediction of boundary layer structure, including inversion strength and presence of cloud.

(b) Winter minimum temperature warm bias under clear skies, especially over snow (c) Over prediction of sea fog (d) Over prediction of wind gusts in convective situations

Page 11: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 11

A2. Critical activities The Science programme theme plan (see Annex III Introduction) contains a number of activities and milestones. These cover items to progress in order to deliver the Public Weather Service, Met Office Hadley Centre Climate Programme and a number of other customer-specific deliverables. Progress against these ‘deliverables’ is routinely assessed and monitored. A subset (about 100) of these deliverables have been identified as ‘underpinning’ (and thereby contribute to the Scientific Excellence corporate objective). In addition, a smaller number of longer-term objectives have been identified as being a priority in order to deliver the Science Strategy. Progress against these deliverables should be routinely monitored to assess performance, even though key milestones may not be due within the reporting year. The list below gives an indicative draft of potential measures for FY1213 (for comment and agreement).

Integrate the global medium range and seasonal forecasting systems as part of the development of a seamless ensemble prediction system from days to decades

Develop and operationally implement hourly hi-res NWP forecast for the UK to improve on current nowcasting approaches beyond one hour.

Implement higher resolution air quality model for the UK (AQUM) operationally

Upgrade monthly to decadal system with key earth system feedbacks ( river flows, vegetation, dust and aerosols); improved resolution (N320 or better), and coupled atmosphere-ocean assimilation

Develop next generation modelling system for efficiently using new computer architectures from ~2020.

Delivery of 2015 HPC upgrade and associated infrastructure

A3. R&D pull-through

A proposal is put forward to instigate an annual report of impact statements against the ‘underpinning’ deliverables (see A2). This approach could be extended to all Science deliverables to provide an assessment of the impact of the programme. This approach could replace the existing pull through measure as an indicator of the effectiveness of the Met Office Science Programme. Proposal for impact based pull-though assessment: Foundation, Weather and Climate Science all have key deliverables identified each year. These deliverables are already monitored monthly and status provided as to delivered or not delivered. It is proposed that an additional report is provided when a deliverable is met which summarises which category of impact the deliverable meets and its impact. Recommended Categories of deliverable:

Underpinning Model improvement,

System development

System optimization

Communication of science,

Inform national policy,

Customer revenue,

(others??)

Categories of impact

Page 12: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 12

In addition to a verbal summary of the impact it may be desirable to try to categorize the level of impact into the following categories:

Major breakthrough,

Large step,

Incremental progress,

Enable strategy for progress,

Maintaining status quo,

Enable informed decision to terminate this work.

A trial has recently been undertaken in the Science programme to assess the feasibility of this approach. Currently a report is produced each month of the status of all key deliverables. The proposal is that in future when a deliverable is met additional information on the type, level and impact if the deliverable be recorded. Typical examples would be:

Deliverable Status Deliverable type Delivery level

Impact statement

High resolution wind model implemented to support Weymouth Bay Olympic forecasting

Met New products/capability

Major breakthrough

Great feedback from customers and brilliant showcase capability.

Development version of UKPP to provide gridded products from the NWP Nowcasting system for Southern England(PS30)

Met New products/capability

Enable strategy for progress

Associated with DAE key deliverable from 2011-12. Informs shape of UKV 2015 data assimilation system& consequently whether NWP can improve on traditional methods of very short range precipitation

Software support in place for 2012 MASS data classification exercise

Met System development

Enable strategy for progress

Easier access to climate model data archive to improve Climate consultancy

By using these categories for monitoring it will be possible to consider if there are year on year trends in the proportion of deliverables for different categories or levels of impact. A recent trial of the method on the Deliverables due April-August 2012 showed that it is feasible. The only key concern raised was that this is a qualitative rather than quantitative assessment. Previously the Pull through index linked man years of effort to improvement in NWP scores however it was a cumbersome score to calculate and did not take account of improvements that did not link through to NWP score such as moving to probabilistic forecasting etc, relied on subjective assessment of managers of how much project contributed to the observed improvement and is best suited to discreet projects rather than gradual evolution through long term research goals.

Page 13: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 13

A4. Shared Funding

Monitoring the amount of shared research funding that the Met Office attracts is another indicator of our scientific capability and excellence. By doing, and being seen to do, excellent and relevant science, we are able to attract additional research funding. The shared funding metric is defined as the total income originated from EUMETSAT, ESA, EU and NERC during a financial year. For example, in FY1112 the Met Office attracted £2.3M from these sources, which funds PWS and/or HCCP relevant science using non-core funding.

It is expected that the shared funding trend should be upwards, however strict revenue targets should not be part within the Scientific Excellence objective. The shared funding metric is therefore to be used in this context as a monitoring indicator only.

A5. Collaboration benefit

To influence and pull-through the best science taking place in the UK and internationally, the Met Office needs to be actively engaged with a number of collaborative research and development projects.

FY1011 BASELINE:

165 collaborative projects (73 classed as ‘Category 1’), total gearing of £15.035M

FY1112 BASELINE:

185 collaborative projects (79 classed as ‘Category 1’), total gearing of £15.885M.

FY1213 TARGET:

>150 collaborative projects logged; >40% classed as Category 1; >£16M gearing.

Note it has also been proposed that ‘impact statement’ analogous to A3 should also be compiled.

A5. External review

Each year the quality and relevance of the Met Office’s scientific research is independently assessed by MOSAC. In addition, the content and progress of the MOHC Climate Programme is reviewed for DECC and Defra customers by the MOHC Science Review Group.

FY1112 BASELINE: “We [the House of Commons Science and Technology Committee] were pleased to find that science is very much at the heart of the services provided by the Met Office. Its science strategy...has been very well received across the meteorological community.”2 “The [MOSAC] Committee has usually been impressed with the material presented to it at the MOSAC meeting, but this year it was particularly so. The new organisational structure had bedded down well and a remarkable amount of good work had been achieved.”3 “The Science Review Group considers that the Met Office Hadley Centre continues to produce world class science and maintains a world leading centre for climate research...The SRG believes the MOHC continues to provide their customers the best possible work programme for their funding.” 4 FY1213 TARGET: Evidence of independent positive appraisal of science programme from MOSAC and SRG review committees, in addition to other sources. Sign-off by Chief Executive.

2 http://www.publications.parliament.uk/pa/cm201012/cmselect/cmsctech/1538/153802.htm

3 MOSAC Chair’s Report 2011

4 Met Office Hadley Centre Science Review Group meeting 2011 – Key Conclusions and Recommendations Report

Page 14: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 14

B. Reputation Indicators

The lists below sets out a baseline measure of the current Met Office science reputation, as assessed by a number of complementary metrics. A proposed set of targets are also provided where relevant. It is anticipated that reputation metrics would be collated annually (or bi-annually). See http://nar.ucar.edu/2011/lar/page/metrics for an example from NCAR.

1) Publication record5 The Met Office’s publication record is an incomplete proxy for our contribution to scientific development. However, it is one of the main quantifiable, globally collated and commonly used source of data on the production and consumption of science. FY1011 BASELINE: 236 papers; 46 different countries; 438 different institutions; h-index 132 FY1112 BASELINE: 252 papers; 49 different countries; 440 different institutions; h-index 143 FY1213 TARGET: >255 papers published; h-index>150

2) External committee service6 Met Office staff are called upon to participate in and often lead external scientific, technical, policy and educational committees. These committees are instrumental in advancing and promoting the work of the scientific and technical community.

FY1011 BASELINE: 82 Met Office science staff served on 183 national and international science committees FY1112 BASELINE: 90 Met Office science staff served on 211 national and international science committees FY1213 – MONITORING INDICATOR ONLY

3) Studentship grants7

The Met Office sponsors post-graduate studentships to contribute to the education, training and career development of young researchers, both in an external institution and in the Met Office. This facilitates stronger research links with academia and improved levels of knowledge exchange.

FY1011 BASELINE: 59 industrial studentships ongoing across 16 different universities FY1112 BASELINE: 67 industrial studentships ongoing across 15 different universities FY1213 – MONITORING INDICATOR ONLY

4) External appointments8 Met Office staff make increasingly important contributions through part-time and honorary appointments within university departments, typically in professorial positions. These appointments recognise individuals’ status and achievements in the wider scientific community. FY1011 BASELINE:

5 Assessed as part of annual publication record, based on Web of Science records and search criteria developed by

Library (conducted by PS to CSc or Science Admin). 6 Assessed as part of annual collaboration gearing activity (see footnote 4)

7 Records maintained by Science Admin, and assessed as part of annual collaboration gearing activity (see footnote 4)

8 TO BE ASSESSED as part of annual collaboration gearing activity (see footnote 4)

Page 15: MOSAC-17 - Met Office · PDF fileMOSAC-17 14-16 November 2012 ... document sets out proposed metrics for discussion, ... MOGREPS testing, seasonal hindcasts,

© Crown Copyright 2012 15

X part-time appointments at Y universities; A honorary positions at B universities FY1112 BASELINE: X part-time appointments at Y universities; A honorary positions at B universities FY1213 – MONITORING INDICATOR ONLY

5) Workshops and conferences9

Met Office staff give presentations about their work to audiences around the world to audiences ranging from scientists and engineers to the general public. Engagement in the scientific development through conferences and workshops is a critical part of our education, visibility, influence and reputation. FY1011 BASELINE: 247 different members of science staff attended 320 different conferences and workshops FY1112 BASELINE: 326 different members of science staff attended 385 different conferences and workshops FY1213 – MONITORING INDICATOR ONLY

6) Awards10 Each year a number of Met Office staff are honoured by independent bodies for their work and contributions to weather and climate, and related, sciences.

FY1112 BASELINE: X staff received special recognition for their work in FY1212. Further details are provided.

Julia Slingo was shortlisted for ‘Public Servant of the Year’ in the Women in Public Life Awards 2011, celebrating women leaders in society.

A groundbreaking study led by Adam Scaife won the Lloyd’s of London annual prize for the Science of Risk in Climate Change.

The Met Office Atmospheric Dispersion group won the Science, Engineering & Technology Award at the prestigious Civil Service Awards, in recognition of developing and exploiting dispersion modelling capabilities to provide expert advice to key partners in response to both the Fukushima incident in Japan and the eruption of the Grimsvotn volcano in Iceland.

Richard Graham was awarded an OBE in the 2011 New Year’s Honours for contributions to seasonal weather forecasting in the developing world, especially in Africa.

Environmental Research Letters vote the paper “How difficult is it to recover from dangerous levels of global warming” by Jason Lowe et al. as one of its ‘papers of the year’ as part of its 5th anniversary.

Maggie Hendry received the QJ Editor’s Award for 2011 for “providing and outstanding and exemplary series of reviews for one particular manuscript”.

Jim Haywood received the RMetS Buchan Prize 2011 for a paper “adjudged to contain the most important original contribution or contributions to meteorology”.

Helen Wells received the RMetS LF Richardson Prize 2011 for “an outstanding paper” published by an early-career researcher.

John Eyre was awarded a ‘lifetime contribution’ prize at the 18th International TOVS Conference. Also recognised were Stuart Newman (best presentation) and Anna Booton (best poster)

Julia Slingo received an honorary degree of Doctor of Science at the University of Reading FY1213 – MONITORING INDICATOR ONLY

9 Assessment based on Science Travel Spreadsheet. Records maintained by Science Admin

10 MORE FORMAL MONITORING REQUIRED (e.g. as part of annual assessment). Current measures based on

review of MetNet briefings/announcements.