quality audit of draft nera water bills model -...

Quality audit of draft Nera water bills model

Report prepared for Defra

Final report

December 2014

2 Quality audit of draft Nera water bills model

Contents

1 Review process ...................................................................................... 4

2 Comments by area ................................................................................ 7


List of tables

Table 1. Record of model audit review activities and inputs .............................. 4


1 Review process The model has passed the quality audit and only a few minor

checks and revisions are recommended

1.1 Audit procedure

To answer the questions set out by the model review process, the audit was carried out at two levels.

The first level is to look at the aims, objectives, design and structure of the model. The second level is to dig

deep into the data, assumptions and formulae used in the model. The overall structure was evaluated on both

the coding of formulae and the logic behind the main calculations. The basic inputs and outputs of the model

were checked for clarity, consistency and replicability (usability of the model).

A detailed evaluation of the spreadsheet examined all the assumptions, scenarios, data and data

processing sheets in considerable detail, evaluating each formula or data field for consistency. Other

aspects of the model such as colour coding, error code handling and assumptions and data management were

also checked. Each formula was transcribed in a word form alongside the variables and arguments it uses and

each input\argument was given a name. Formulae were checked based on first principles and on whether

they made economic and \or accounting sense. Also where a formula was used in more than one place (for

different companies and\or elements of the value chain), its inputs were verified for consistency.

Nera responded to the comments made in this report and as a result some figures changed between Nera’s

draft final and final work. Figures taken from Nera’s draft final work are discussed here and have not been

updated with its final figures in this document, otherwise the audit comments attached to them would no

longer make sense. Hence there will be differences between the figures quoted in this report (particularly in

section 2.6) and the final Nera report.

Table 1. Record of model audit review activities and inputs

Task Percentage of

model covered

by review

Time

allocated to

task

Finding

Overall

4.5 h

Are the overall aims of the model clearly

specified

100 per cent 2 h Yes

Does the model solve the problem it was

intended to? 100 per cent 2 h Yes

Is there a colour key outlined? 100 per cent 10 min Yes, but it could be extended


Is there a table setting out the purpose of

each of the sheets in the model? 100 per cent 10 min Yes

Assumptions 2.5 h

Are all the assumptions specified in

assumptions sheets? If not, is it clear

where else they are?

100 per cent 20 min Most, but not all are specified in

the assumptions sheets

Are the units for all assumptions

identified?

100 per cent 30 min Some units are missing

Do all assumptions have dependents? 100 per cent 30 min Yes

Is it easy for the user to restore default

assumption values?

100 per cent 30 min There are a few hard-coded

numbers

Does the model store scenario

assumptions alongside stored outputs?

100 per cent 30 min Yes

Structure 20 h

Is the structure consistent and coherent

throughout the model?

80 per cent 2 h Yes

Are formulae used consistently? 65 per cent 2 h Yes

Is it flexible enough for input variables to

be changed easily without causing errors?

100 per cent 2 h Yes

Does the flow of calculations and the

assumptions used make logical sense?

65 per cent 6 h Yes

Is the model working correctly – i.e. does

each set of calculations (i) calculate the

right thing and (ii) calculate it correctly?

a. Data forecasting b. Data compiling c. Main calculations

65 per cent 8 h

1. There are a some issues with data forecasting

2. Data compiling works correctly

3. Main calculations work correctly, with small number of issues identified

Calculations 18 h

Is the purpose of each calculation clear

and accurately labelled?

65 per cent Formulae and tables are not

labelled. There is no detailed

calculation architecture or list of

word equations

Has the colour key been followed?

65 per cent Yes

Do the ranges in any formulae cover the

appropriate cells? Pay particular attention

to the data in the last row and last column

of any table which has repeating

formulae.

65 per cent Yes

Are there any ‘#ERR’, ‘#REF’ etc. cells?

65 per cent A few instances of #Value and

#N/A


Are there labelled ‘checks’ built in for

checking important calculations i.e. do

shares sum to 100 per cent?

65 per cent Yes

Are the arguments of functions such as

VLOOKUP, IF etc. correctly articulated?

65 per cent Yes

Do any calculations use numbers rather

than referring to other cells?

65 per cent A few instances of hardcoded

numbers

Are unique formulas

arithmetically/economically correct?

65 per cent Yes

Output

Are there separate sheets summarising

the outputs from the model?

100 per cent Yes

Are all units correctly identified? 100 per cent Yes

Does the model contain a convenient way

of labelling and storing output data?

100 per cent Yes

Source: Vivid Economics


2 Comments by area The model performs well but would benefit from narrowing of

assumption ranges

2.1 Documentation

The draft report is clear and well structured. The draft main report is clear, well written and well laid out.

There is evidence that issues have been recorded and addressed. Nera kept a log of comments received

during the model development process and these were shared with and read by Vivid.

The report introduces the reader to some of the main calculations and all the user interface sheets. The

report explains how to operate the model, sets out its functionality and displays example outputs. It is written

clearly and is easy to understand. The model explains data sources and states which principal calculations

take place.

There are a number of areas not covered by the report, which are candidates for appendices. The

report could go further in appraising the evidence for input assumptions. It could explain off-model

calculations and in some places, could more fully elaborate explanations of key steps in the calculations. The

report would greatly benefit from a table of abbreviations and this table should be included in the spreadsheet

as well.

The model provides particular value through its ability to highlight the relative importance of drivers

of bills. The model allows the user to explore the sensitivity of bills to a wide range of drivers and

assumptions. This is very useful, however, the user will find it very taxing to explain why bills have

particular sensitivity to drivers. Two additional pieces of model documentation could help: an influence

diagram and word equations. Nera responded by adding this material. An influence diagram shows how

important input drivers and assumptions feed through intermediate variables and calculations to output

variables. It is complemented by word equations which show how variables are combined at key stages of

the calculation. It is common but not universal practice in planning model construction to build influence

diagrams and to write out equations in words before encoding them. If such information is available it would

be very worthwhile including them in the report as appendices, or if too large, as e-documents.

While the report is generally very clean, there are some minor tidying edits needed. There are some

minor editing revisions to attend to in the finalisation of the report. Some tables are presented without units

and some figures are quoted without time periods, and there is ambiguity in some cases whether long term

growth rates are arithmetic or geometric averages. There are a few typographical errors. Vivid has marked

these up on the draft report except for the typographical errors which could be picked up with a spelling

check.


2.2 Structure

The model has a clear and organised structure, the logic of the model is sensible. The spreadsheet has

information on the structure of the model, the purpose of each sheet is documented and consistent; however

it can be improved by including a glossary of the abbreviations used. The model was split into raw data, data

forecasting, data processing and main calculations. The details in each block are comprehensible and the data

is constantly being summarised and consolidated for ease of use. Policy and driver shocks are treated as

deviations from a baseline scenario and, where there are many scenarios, ample care was taken in defining

and coding each scenario separately. Drivers and policy scenarios are laid out explicitly with a clear purpose

and organisation. The formulae used in the model can be characterised into:

– forecasting formulae;

– accounting identities;

– accounting for cost of growth and quality enhancement;

– accounting for own and cross price elasticity of demand;

– cost appropriation along the value chain; and

– data retrieval and consolidation.

The structure of data handling is coherent throughout the model. The main calculations repeat across

the companies and the elements of the water and sewerage value chain. Vivid Economics checked that

repeated calculations adopt the same formulae across companies and value chain elements and found that the

calculations are carried out in a consistent and coherent fashion across all these elements. The two main

panels for controlling the scenarios and data input make the model flexible enough to introduce assumptions

without altering any of the core calculations. Vivid Economics checked almost two thirds of the model

formulae meticulously by transcribing the formulae into words and naming each variable. The word

representation of entire sheets of calculations ensures a clear understanding of the logic of the formulae. For

each of these formula, Vivid verified that:

– it was coded correctly;

– it used the right inputs;

– it makes economic\accounting sense; and

– it was subsequently used properly.

There are a small number of instances where formulae might be considered for revision, as described in the

issues log.

2.3 Calculations

The formulae used are correct and there are very few coding and logical errors. There are no major

instances where formulae were deemed inappropriate, however there were few instances where the inputs

were deemed inappropriate. Nera adopted all suggestions from the log including minor suggestions such as

formatting.

The data in each row is carefully described, however the spreadsheet would be more accessible if it

included descriptions of some of the key calculations. Nera added responded by adding descriptions.


Given the complicated coding involved in retrieving data by constant reference to scenario and variable

codes, the logic of the calculation can be hard to follow. There is a basic colour coding that is followed in the

spreadsheet for data inputs. A remedy could be to introduce colour coding for the calculation sheets,

mapping colours to companies and value chain elements, to improve the navigation and usability of the

model. We found only three other minor issues with the calculations:

– there is a minor concern with the handling of error codes because #N/A and #Value appear frequently.

This can be addressed by explicitly indicating which data is not necessary and can be omitted from

subsequent calculations. Nera added this to the list of potential future upgrades;

– there are a few checks built in to verify the consistency of a string of calculations but the model could

benefit from the inclusion of more checks. Nera added this to the list of potential future upgrades; and

– there is a small amount of hard coding of values in the model (see issues log). Some calculations refer to

empty cells which may cause a problem if the empty cells are altered by a user. This problem can be

resolved by locking the empty cells for editing. Nera responded to this issue.

2.4 Assumptions and inputs sheets technical review

All the assumptions are specified in the assumptions sheet, with similar sheets laid out for economic

drivers and policy shocks. All the variables defined in the assumption sheets have dependents. There are

instances where assumption variables are not recorded in the assumption sheets. On occasion, assumption

variables are introduced directly into formulae and there are other instances where assumed values were

explicitly inserted in separate rows in the calculation sheet, rather than in the assumptions sheets. The

introduction of variables directly into calculations is best avoided altogether. It is preferable to gather all

assumptions in one place, rather than to have them spread out among the calculations. Nera responded by

creating an assumptions worksheet. There were some occasions where assumptions figures were adjusted for

inflation by hard coding the inflation number into the calculation. This may cause a problem of inconsistency

with the user’s choice of economic drivers. Nera addressed this comment.

Some units of measurement were missing from data rows, making the sheets harder to read.

Generally the sources of assumed variables are stated. The assumption variables are stored appropriately

alongside the outputs.

2.5 Assumption ranges

We have examined the ranges of assumptions and in most cases find them to be reasonable. We have

looked at the ranges of assumptions and based on our experience, or on the quality of evidence offered in

support of the ranges, identified those which merit further discussion. Those meriting further discussion are:

– RPI; and

– efficiency improvement rate.

Nera revisited and updated the assumptions.


The RPI range spans very low and very high figures and may not be consistent with other

assumptions. In Table 3.8, the central and high RPI rates are well above the Bank of England target rate and

the long term low figure is historically extremely low. Table 3.9 even suggests large negative RPI figures

compounded for 28 years, which is an extreme scenario. There is a question whether it is possible to choose

assumptions so that the RPI, CPI or other price indices and GDP projections are inconsistent, which could

introduce unintended relative price effects. It appears that inconsistent assumptions can be introduced. These

two issues combined could lead to some odd projections, perhaps including very low bills. We strongly

recommend the review of additional evidence on RPI rates and the adoption of a narrower range of estimates

for the long term. This review could usefully encompass the GDP and CPI rates which also appear to span

negative figures which are improbable in the long run.

The compound efficiency assumption is important but unsupported. The model uses a compound

efficiency assumption of 1 per cent a year in the baseline and offers a range of 0.5 per cent to 2 per cent. The

outputs are sensitive to this assumption. No evidence is given for the range of assumptions. The higher figure

would generate a reduction of 50 per cent over a 34 year period from 2015 to 2049 if it is applied to the draft

determination figures and beyond. This is a large real-terms decrease and, although significant efficiency

improvements are likely to be found in the sector, a 50 per cent reduction may be too large as a realistic

planning assumption. Nera added a sensitivity section to its report.

There is a sudden fall in enhancement capex in the baseline scenario in the mid-2020s. This is

presumably in part because the demand for quality enhancements becomes much more uncertain beyond that

date. It is possible that a combination of aging assets, population growth, climate change, stricter

environmental legislation, and demand growth will lead to higher levels of capex and that the sudden drop in

capex will not occur. The user should be made aware that alternative, higher capex scenarios are possible,

and the change in the mid-2020s is worth highlighting. It may not be possible to introduce simple

functionality to allow users to raise the level of capex in the second half of the projection period, but it is

worth considering a more ambitious Water Framework Directive programme from 2025 as an option in the

policy drivers settings. There is a lot of detail in the model on water resources but much less detail on waste

water and water quality, which may become a spending priority in the mid-2020s, and this action would help

to address that balance.

The basis of the maintenance projections could be made clearer. Maintenance is a key building block of

bills. The use of totex and pay as you go calculations make it harder to understand how the capital

maintenance projections have been prepared. It would be helpful to the user to describe in more detail how

maintenance is calculated and projected forward.

GDP is a driver of demand. Population and GDP are both drivers of demand and they are treated

exogenously. However, population growth is a factor in GDP growth so there may be the potential for double

counting of population effects on demand or inconsistency between GDP scenarios and the hard-coded

population forecast.

One of the effects of upstream and retail competition is on cost of finance. It is not clear whether the

transition to a contestable market is assumed to change the cost of capital for the whole service regulatory


capital value (RCV) or whether the increase is focused on the value chain elements which are contested. It

would seem most appropriate to focus it on the contested activities. Nera confirmed that this is the approach

they took and that they had assumed zero effect on the WACC from retail competition.

2.6 Outputs

The graphical and tabular outputs cover a wide range of useful and intuitive reports. They are easy to

interpret but it requires some effort of investigation to find out what is driving trends. Some of the figures

quoted below, for example, the projected average bill, capex and opex, will differ from those published in the

final Nera water bills report.

There are a number of small changes in presentation which could be considered:

– it would be helpful to include a breakdown of the capital enhancement programme by value chain

element because capital expenditure is an important driver and it will help to explain what is driving

capital enhancement to report it across the value chain. Nera added this to the list of potential future

upgrades;

– the charts of bills expressed as a proportion of household income are useful and should be included in the

report;

– Figure 4.45 shows absolute totex by value chain element. Complementary charts could be included

showing absolute and percentages changes in totex by value chain element over time, allowing the user to

check whether there are large changes in some value chain elements and whether these are plausible; and

– it would be of policy interest to report bill impacts along the value chain, perhaps using waterfall charts,

but it is not clear whether the model can accommodate this; this is something for Nera to consider. Nera

added this to the list of potential future upgrades.

The baseline model runs suggest a long term downward movement in bills. There are two principal

drivers of the trend:

– enhancement CAPEX falls by half between 2015 and 2049, shaving 25 per cent off totex, leaving the

RCV fairly constant over the period; and

– operating expenditure falls by 18 per cent, due to assumed efficiency improvements.

These are offset by a 24 per cent increase in capital maintenance, despite efficiency improvements, reflecting

a larger physical capital base. We discuss these in turn.

The declining enhancement expenditure is plausible. The arguments supporting a declining enhancement

capital expenditure over time are that the proportion of assets remaining to be upgraded to address

intermittent sewage discharges, to improve effluent treatment, to increase drinking water quality or to

enhance resilience of resources all decline over time. It may be that the standards which the sector has to

attain are raised over time, but as has been seen in other utility sectors, it is possible that an historic peak of

capital enhancement in the sector has now passed.

The increasing level of capital maintenance is justified. Although additions to the capital base may be

slowing and the RCV, which is an accounting record, may be stable, the physical asset base of the water


industry is projected to grow. The argument for this is that the population is growing, quality of service is

improving and the technology employed is fairly constant. This growing physical asset base demands

increases in expenditure to maintain it. The projection of capital maintenance requirements has always been

a difficult area of water industry science and there is quite a lot of uncertainty around maintenance

projections. Figure 4.41 shows baseline flat capital maintenance until 2035 followed by a gradual increase.

Nera could provide a fuller explanation of its projection of capital maintenance expenditure including why

capital maintenance remains flat before increasing when capital enhancement is on a declining trend. If the

capital maintenance is pegged to movements in the RCV or totex, then consideration should be given as to

whether these relationships will be stable over time.

The model projects a very substantial shift in the balance between operating expenditure and capital

maintenance. The figures in Table 4.29 can be used to derive input shares. In 2015 the ratio of operating

expenditure to capital maintenance is 1.9:1.0, which falls to 1.27:1.0 by 2049. This would be consistent with

an industry which has become more capital intensive. It seems reasonable to expect that the sector will

become more capital intensive and that capital maintenance costs will grow relative to operating expenditure,

as technology changes and as quality of service rises. However, this is quite a large change in input shares

and it merits some discussion and explanation of why it is reasonable.

One of the factors driving the 18 per cent fall in operating expenditure is the efficiency assumption.

The efficiency assumption, which is 1 per cent per annum in the baseline, applies to aggregate operating

expenditure, not to unit costs of activities. If underlying operating activities are increasing, it implies a

greater than 1 per cent per annum efficiency increase, and if they are shrinking, a figure lower than 1 per

cent. Given forecast population increases and further capital enhancement, it seems likely that the former

case persists. The assumption has a large effect on OPEX over the period to 2049 because it is compounded

for 34 years. A factor of 0.99 raised to the power 34 is 0.71. The baseline assumption on efficiency is driving

an underlying OPEX reduction of 1 – 0.71, or 29 per cent, which is offset to the extent of around 11 per cent

by drivers such as population and volume supplied, to give a net change in OPEX of 18 per cent. The model

outputs are sensitive to this assumption and Defra should consider whether 1 per cent is the best estimate to

use in the baseline, a it may be too high.

The model predicted a very wide range of possible future bills from £165 to £478/hh/yr. Nera

subsequently revised its assumptions, producing a much narrower range. Not all of this range appears

plausible. Although bills have moved dramatically downwards over time in some other utility sectors, such

as communications, the technology and market structure changes which have been the cause there will not be

replicated in the water sector. Very large real reductions in bills seem implausible though some real

reductions in bills over time are possible, especially for customers in regions where the capital expenditure

drivers are weaker or absent.

The variation in future bills is mostly on the upside (increasing bills). There is little difference between

the bill estimates under the low settings and the baseline settings, but there is a large difference on the

upside. There are reasons why reductions in bills may be truncated from the baseline position, for example,

because capital enhancement expenditure cannot be reduced below zero, while variation on the upside is

much less constrained, but the closeness of the baseline and low scenario estimates suggests that the baseline


has been set too low. A careful review of the baseline assumptions relative to the low and high scenarios is

worthwhile and may lead to some adjustments of the baseline.

Within the body of water customers, average measured and unmeasured bills may change relative to

each other as metering penetration increases. These changes are not presented in the report but they are

visible in the model output sheets. There is no discussion of the calculation of the measured-unmeasured bill

differential or these results within the report, but this is a policy-relevant issue and should be covered, if the

model is intended to cover it within its scope, or alternatively the outputs could be removed from the model.

The model currently predicts very large increases in average unmeasured bills over time. Nera added this to

its list of potential future upgrades.

The long run projections in the model rely on constant compound annual growth rates and highly

uncertain projections of capital expenditure. The model gives a much more accurate picture to 2025 than

it does thereafter. There is no solution to this problem, since the uncertainty is inherent in the problem, but it

does suggest that users should select ranges of assumptions of compounding drivers which are narrower than

the range used for short term projections. That is, users should assume some long term mean reversion of

drivers, and Defra may wish to add this advice to the report.

It would be helpful to present botex and enhancement expenditure as separate figures. Table 4.28

shows that the baseline projection states a 26 per cent reduction in totex in the sewerage network compared

with 10 per cent reduction in treated water distribution. In order to understand these relative trends, it would

be helpful to decompose the total figures into botex (totex less enhancement capex) and enhancement capex

and include these data in the table. The same argument applies to the absolute change in expenditure to 2049.

OPEX and capital maintenance are slowly declining and slowly increasing respectively over time and

the large decrease in totex is due to reduced capex enhancement. The botex and enhancement capex is

stated for the combined services value chain in Table 4.29 and shows that enhancement capex falls by 50 per

cent between 2015 and 2049. This seems plausible. The share of enhancement in totex halves from 53 per

cent to 27 per cent between 2015 and 2049 in the baseline scenario, which again seems reasonable.

The model shows decreases in regulatory gearing, debt divided by RCV, over time. The estimated

regulatory gearing falls from 60 per cent in 2015 to 25 per cent in 2049. The level of debt falls from £40 to

23 billion over the same period. The company financial gearing is not reported. It seems likely that the model

is underestimating the level of debt carried by companies in the late 2020s and beyond and will thus slightly

underestimate the value of the tax shield and therefore slightly overestimate the required return on capital.

The model only reports figures from 2015 onwards and there is no comparison with historic data or

trends. The model starts projections in 2015. The report states that trend analysis has been performed off

model but no details are given. It is not possible to see whether the model’s projections continue smoothly

from the historic data. Greater transparency on the trend analysis and consistency between past and future

would enhance the credibility of the trend assumptions and projections. Nera responded by adding a chart

with historic average household bills.

Contact us:

Vivid Economics Limited T: +44 (0)844 8000 254

The Media Village E: [email protected]

131-151 Great Titchfield Street

London W1W 5BB

United Kingdom

Company Profile

Vivid Economics is a leading strategic economics consultancy with global reach. We strive

to create lasting value for our clients, both in government and the private sector, and for

society at large.

We are a premier consultant in the policy-commerce interface and resource- and

environment-intensive sectors, where we advise on the most critical and complex policy

and commercial questions facing clients around the world. The success we bring to our

clients reflects a strong partnership culture, solid foundation of skills and analytical assets,

and close cooperation with a large network of contacts across key organisations.

quality audit of draft nera water bills model -...

Documents