beginners guide to weather and climate data

A Beginners guide to weather and climate data

Bristol Data Scientists Meetup | 24 January 2017

Margriet Groenendijk | Developer Advocate | IBM Watson Data Platform

@MargrietGr

https://medium.com/ibm-watson-data-lab

https://github.com/MargrietGroenendijk/Bristol

https://github.com/MargrietGroenendijk/Bristol

https://github.com/MargrietGroenendijk/Bristol

https://github.com/MargrietGroenendijk/Bristol

https://github.com/MargrietGroenendijk/Bristol

https://github.com/MargrietGroenendijk/Bristol

Observations + ModelsForecast =

Temperature

Humidity

Windspeed and direction

Air pressure

Rainfall

Radiation

http://www.metoffice.gov.uk/public/weather/climate-network/#?tab=climateNetwork

Historic weather

Mean daily maximum temperature

Mean daily minimum temperature

Days of air frost

Total rainfall

Total sunshine duration

http://www.metoffice.gov.uk/datapoint/

https://business.weather.com/products/the-weather-company-data-packages

Note: data not cleaned

Historic weather

Example: London City airport 1997-2017

https://climexp.knmi.nl

http://www.cru.uea.ac.uk/data/

https://developer.ibm.com/clouddataservices/2016/04/18/predict-temperatures-using-dashdb-python-and-r/

pseudo-code test before using!

Models

3D grid

Atmosphere

Energy

Water

Differential equations

Boundary conditions

Land

Ocean

Lots of parameters

http://www.climateprediction.net/

Important to separate the different PFTs and large cover fraction, because these are all influenced20the f0 parameters and by the evaporation from the soil.21

Where can the differences between models be related to?22

What are the uncertainties?23

Any constraints of the models with observations?24

2 Theory25

At the leaf-scale CO2 (A) and water vapour (E) fluxes across stomata are given by (Katul et al., 2009):26

A = gs(ca ci) (1)27

E = 1.6gs(ei ea) (2)

where gs is the stomatal conductance of CO2, ca and ci the ambient and intercellular CO2 concentra-28tions, ea and ei the ambient and intercellular water vapour concentration, and 1.6 the relative diffu-29sivity of water with respect to carbon. The ratio of these fluxes is defined as the leaf-scale water use30efficient (WUEleaf ):31

WUEleaf =A

E(3)

Additionally, WUEleaf can be derived from the gradients of the CO2 concentration and water vapour32between the ambient air and air within the leaf. When combining Eqs. 1 to 3 it follows that:33

WUEleaf =ca ci

1.6(es ea)(4)

where ca and ci are the ambient and internal partial pressures of CO2 [Pa] and es and ea the sat-34urated and ambient water vapour pressure [Pa] The assumption is used that the intercellular water35vapour concentration is saturated and therefore ei in Eq. 1 is replaced with es. [See Katul papers for36conversions between units and why vpd is a good estimate in this equation]37

To simulate the ecosystem-scale water and carbon fluxes the model developed for the leaf-scale is38used (e.g., Farquhar et al., 1980). This implies that in theory WUE at the ecosystem-scale can be39estimated with both eqs. 3 and 4 and that they should be equal:40

WUEe =GPP

ET(5)

2GPP ET

http://www.earth-syst-dynam.net/7/525/2016/esd-7-525-2016.pdf

0 50 100 1500

100

200

300

400

500

600

vcm,25 (mol m2 s1)

j m,2

5 (

mol

m2

s1 )

1 2 3 4 5 6 7

0 50 100 1500

0.2

0.4

0.6

0.8

1

vcm,25 (mol m2 s1)

(m

ol m

ol1

)

0 50 100 1500

200

400

600

800

1000

vcm,25 (mol m2 s1)

(m

ol m

ol1

)

0 200 400 6000

0.2

0.4

0.6

0.8

1

jm,25 (mol m2 s1)

(m

ol m

ol1

)

0 200 400 6000

200

400

600

800

1000

jm,25 (mol m2 s1)

(m

ol m

ol1

)

0 0.5 10

200

400

600

800

1000

(mol mol1)

(m

ol m

ol1

)

k-means clustering

0 50 100 1500

100

200

300

400

500

600

vcm,25 (mol m2 s1)

j m,2

5 (

mol

m2

s1 )

1 2 3 4 5 6 7

0 50 100 1500

0.2

0.4

0.6

0.8

1

vcm,25 (mol m2 s1)

(m

ol m

ol1

)

0 50 100 1500

200

400

600

800

1000

vcm,25 (mol m2 s1)

(m

ol m

ol1

)

0 200 400 6000

0.2

0.4

0.6

0.8

1

jm,25 (mol m2 s1)

(m

ol m

ol1

)

0 200 400 6000

200

400

600

800

1000

jm,25 (mol m2 s1)

(m

ol m

ol1

)0 0.5 1

0

200

400

600

800

1000

(mol mol1)

(mol

mol1

)

1 2 3 4 5 6 70

0.05

0.1

0.15

0.2

0.25

0.3

0.35

Group

Rela

tive

vege

tatio

n di

strib

utio

n (

)

CROSAVDBFEBFENFGRAMFO

1 2 3 4 5 6 70

0.05

0.1

0.15

0.2

0.25

0.3

0.35

GroupRe

lativ

e cl

imat

e di

strib

utio

n (

)

ARBOTMTETCTR

k-means clusters vs. vegetation and climate classifications

0.25

0.15

0.20

A CB

http://www.earth-syst-dynam.net/7/525/2016/esd-7-525-2016.pdf

Climate models

=

Algorithms + Observations + Parameters

=

Machine Learning

invehiclehaildamageclaimseveryyear

increaseintemperaturemeans$24Mmorein

electricityspendingperday

drop insalesforareaswithmorethana10%drop

intemperature

Insurance

Energy & Utility

Retail

Some applications

Next find the data!

Open Data

Open data in Bristol

https://opendata.bristol.gov.uk

Air quality River water quality

Water levels Flood alerts

M32 traffic flow Road works

Crime stats Census data

Energy consumption by ward Hospital admissions

NHS

Weather + Driving Difficulty Index API

+

M32 traffic flow + Road works

+

air quality observations

=

Predict air quality

Weather and Climate Data

Historical time series

Historical maps

Climate model data

Real-time APIs

Forecast APIs

Thanks!https://github.com/MargrietGroenendijk/Bristol

http://www.slideshare.net/MargrietGroenendijk/presentations

@MargrietGr

https://medium.com/ibm-watson-data-lab