fmi open data interface and usage

Open Meteorological DataUsage and Best Practices

Finnish Meteorological InstituteRoope Tervo

05/02/2023 2

Finnish Meteorological Institute opened its data in 2013.

Basically everything FMI has rights was opened.

Data is provided freely in machine readable format.

FMI Open Meteorological Data Services | Roope Tervo

FMI Open Data

https://en.ilmatieteenlaitos.fi/open-data



05/02/2023 3

FMI Open Data Portal follows INSPIRE requirements.

FMI Open Data Portal

Meta data

Data Models

Services

The very same data portal works as Open Data and INSPIRE portal.


05/02/2023 4

Data set Description Time Interval

Estimated publish date

Weather Observations

Temperature, Wind, Humidity, Ground Temperature…

10 min Open,older data to be added

Sun Radiation UV, Short and Long Term Radiation…

1 min Open

Marine Observations

Waves, Sea Temperature, Sea Level…

1 h Open

Weather Radars Precipitation Rate, Precipitation Amount…

5 min Open, older and more data to be added

Lightning Thunder Strikes in Finland

5 min Open

Example of Data Sets


05/02/2023 5

Example of Data SetsData set Description Time Interval Estimated

publish date

Real Time Observations

Real Time Observations from specific location(s)

AWS 2010 – Soundings 1959 –Flashes 1998 – Sea Level 1971 – Waves 2005 –

Openolder data will be added

Climatological Observations

Daily and monthly temperature mean and extreme values from weather stations

1959 - Open

Climatological Observations

Monthly temperature and precipitation rate mean values interpolated to grid

1961 - Open

Climatological Reference

Climatological Reference. Temperature, humidity, pressure, precipitation amount and snow depth.

Reference seasons:1971-2000 1981-2010

Open


05/02/2023 6


publish dateWeather forecast model HIRLAM RCR

Point forecasts and grid data

Latest model run(4 times a day) 0…54 h

Open

Sea forecast models Sea level point forecasts, Wave (WAM) and current (HBM) as grid data

Latest model run(4 times a day) 0...54 h

Open

Environmental Monitoring Facilities

Weather observation stations, radars…

- 2017

Aviation Observations

METAR/IWXXM 30 min Open

Mast observations Special observations from masts

open


05/02/2023 7


publish dateAir Quality Observations

Air Quality Observations 1h open

Silam Model Dispersion Model for Air Quality, Forest Fire and Pollen

Latest model run (once a day)0…96h

2017

HELMI Ice Model Ice forecast model Latest model run(4 times a day) 0...54 h

open

Soundings Temperature, Humidity, Pressure, Wind from ground to 25 km height

2 times a day open

Quality information

Quality information for observations

open


05/02/2023 8

Registrationo Registration is required to use View and Download

Services o Working email address is the only mandatory

informationo After registration the user gets an API key which have to

be added into all requestso GET parameter fmi-apikey=…&o Header fmi-apikey; …o Part of url http://wms.fmi.fi/fmi-apikey/…/wms?

o One can create several API keys with one email


05/02/2023 9

Usage LimitsWith one API key it’s allowed to

o do at most 20 000 requests per day to Download Serviceo do at most 10 000 requests per day to View Serviceo do at most 600 requests per 5 minutes to both services

o If all observations from one time step is calculated to as one, little over 17 000 new data sets are published daily

o So, with one API key it’s allowed load everything onceo View service can be used for testing but can not be used as a

back end for popular clients


05/02/2023 10

Catalog Service (CSW)

o Based on GeoNetworko High-level metadatao Provides UIo Maybe more useful sources:

o http://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-services (based on WFS DescribeStoredQueries)

o http://www.paikkatietoikkuna.fi/web/fi/metatietohaku (based on GeoNetwork)


http://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-services



http://www.paikkatietoikkuna.fi/web/fi/metatietohaku



05/02/2023 11

View Service (WMS)o Based on GeoServero Visualize observation and

forecast data and provides radar images

o Only the most common layers are published

o Can be used for prototyping - not for production use


05/02/2023 12

Download Service (WFS 2.0)o Web Feature Service (WFS) 2.0

Simple Profileo Based on stored queries

o Predefined data sets with possibility for additional parameters (i.e. time and area)

o In-house production


05/02/2023 FMI Open Meteorological Data Services | Roope Tervo

13

GetCapabilities:• Describes capabilities of the service• Normally start point for client applications• http

://data.fmi.fi/fmi-apikey/0017a4fc-b1e9-4851-850c-0f9ff66f8cae/wfs?request=GetCapabilities

WFS Basicsobseravtions and point forecasts

http://data.fmi.fi/fmi-apikey/0017a4fc-b1e9-4851-850c-0f9ff66f8cae/wfs?request=GetCapabilities





14

XSLT

DescribeStoredQueries• List and describes available stored queries (pre-defined

data sets)• Stored query name and description

• Possible additional parameters for the data query

• http://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-services






15

GetFeature• Actual data query• In FMI service, stored query id is compulsory

• &storedquery_id=..

• Other possible parameters depend on the stored query• Possible values can be fetched with DescribeStoredQueries

request.

• Location has to be often defined some how.

http://data.fmi.fi/fmi-apikey/.../wfs?request=GetFeature&storedquery_id=fmi::observations::weather::cities::timevaluepair&bbox=21,62,22,64


http://data.fmi.fi/fmi-apikey/0017a4fc-b1e9-4851-850c-0f9ff66f8cae/wfs?request=GetFeature&storedquery_id=fmi::observations::weather::cities::timevaluepair&bbox=21,62,22,64

http://data.fmi.fi/fmi-apikey/0017a4fc-b1e9-4851-850c-0f9ff66f8cae/wfs?request=GetFeature&storedquery_id=fmi::observations::weather::cities::timevaluepair&bbox=21,62,22,64


16

Location parameters in stored queries& bbox area

• By default, in same srs with the (requested) srs. Specify as fifth element if needed (i.e. &bbox=132249,6433579,631999,6933329,epsg:3067)

& fmisid FMI id for observation station& wmo WMO id for observation station



17

Location parameters in stored queries& geoid http://geonames.org id for location& place human readable name

• use region to accurate your query (i.e. &place=kumpula,tampere)

• can be used several times in one query (i.e. &place=kumpula,helsinki&place=heinola)

& latlon coordinates (used only in marine data sets)


http://geonames.org/

05/02/2023 18

GetPropertyValue• Responses only requested part of the GetFeature response• Use ValueReference=xpath to define part of interest• Same additional parameters for stored query with GetFeature

request are still valid • For example: http://data.fmi.fi/fmi-apikey/…/wfs

?request=GetPropertyValue&storedquery_id=fmi::observations::weather::multipointcoverage&place=kumpula,tampere&ValueReference=wfs:FeatureCollection/wfs:member/omso:GridSeriesObservation/om:result



http://data.fmi.fi/fmi-apikey/0017a4fc-b1e9-4851-850c-0f9ff66f8cae/wfs?request=GetPropertyValue&storedquery_id=fmi::observations::weather::multipointcoverage&place=kumpula,tampere&ValueReference=wfs:FeatureCollection/wfs:member/omso:GridSeriesObservation/om:result






05/02/2023 19

Data Modelso Observations and point

forecasts as GMLo The same data is published in

two different formats.o Gridded data is provided in

appropriate binary format (grib, NetCDF, HDF…)o WFS members contains the

metadata ‘envelope’ with a link to a actual data


o Every XML document as self-explanatory as possivble


20

om:phenomenonTime• Tells start and end time of returned data

Data Modelsmost important elements


21

om:procedure• Describes data producing procedureom:parameter• Changing parameters in the process

• i.e. analysis time of the numerical model



22

om:featureOfInterest• Describes location(s) or area of returned data• Several different kind of identifiers

• target:SurfaceWeatherTargetArea area (grid data)

• fmisid FMI id for observation station

• wmo WMO id for observation station

• geoid http://geonames.org id for location

• name Human readable name

• region county of the station


http://geonames.org/


23

om:observedProperty• Describes meteorological parameters in returned data• Defined as xlink• Describes label, basePhenomenon and units of measure of

every parameter in the reponse• Tip: use the same service without parameter definitions to

query all possible parameters• http://data.fmi.fi/fmi-apikey/.../meta?observableProperty=forecast

• Note that all parameters are not available in all data sets


http://data.fmi.fi/fmi-apikey/.../meta?observableProperty=forecast


24

om:result• Data in required format in observations and point forecasts and link to a

external binary data in grid formatted data


05/02/2023 25

Data Modelsgmlcov:MultiPointCoverage

gml:rangeSetgml:doubleOrNilReasonTupleList

The data is listed for every point defined in domain set.

gml:domainSetgmlcov:simpleMultiPoint

The coverage is defined as a list of

points in 4 dimensional grid (lat,

lon, height, time).

gmlcov:rangeType

The parameters listed in range set

are defined in separate element.


05/02/2023 26

Cons- Not intuitive- No natural

structure of XML XSLT and Xpath don’t work

Pros+ Compact+ Efficient+ Small file size+ Works for many

data types

Data Modelsgmlcov:MultiPointCoverage


05/02/2023 27

Data Modelswml2:MeasurementTimeseries

wml2:MeasurementTimeseries

One member contains time series of one parameter and

one location


05/02/2023 28

Cons- Lots of repetition- Large file size- Heavy for DOM-

based parsers- Don’t work i.e. for

thunder strikes

Pros+ Intuitive+ Easy to use+ XSLT & XPath

works

Data Modelswml2:MeasurementTimeseries


05/02/2023 29

Data ModelsSimpleFeature

SimpleFeature

One member contains one time step of one parameter

and one location


05/02/2023 30

Cons- Huge file size- Heavy for DOM-

based parsers

Pros+ Intuitive+ Easy to use+ Wide application

support

Data ModelsSimpleFeature


05/02/2023 31

Data Models file size comparison

Simple

Featur

e

Measu

remen

tTim

eseri

es

MultiPoin

tCov

erage

0

20

40

60

80

10081.7

52.9

1.81.3 1.2 0.2

Document Size [MB]

Compressed DocumentSize [MB]

o 138 o

bser

vatio

n stat

ions

o 11 pa

ramete

rs

o 72 tim

e step

s (12

hour

s)


Data Type Data Format

Observations wml2:MeasurementTimeseries

gmlcov:MultiPointCoverage

SimpleFeature

Point Forecasts wml2:MeasurementTimeseries

gmlcov:MultiPointCoverage

SimpleFeature

Lightning Observations gmlcov:MultiPointCoverage

SimpleFeature

Grid Forecasts XML Envelope + Grib2/NetCDF

Radar Images GeoTiff / PNG images

METAR IWXXM

05/02/2023 32FMI Open Meteorological Data Services | Roope Tervo


33

• Aviation weather reposts are delivered as IWXXM• New data model coming into use in aviation

• Consists of the same elements than other messages• om:phenomenonTime, om:procedure, om:featureOfInterest,

om:result

• Content of the METAR is in om:result part as• extracted into XML elements

• original, “old fashion”, METAR string

Data Modelsaviation observations IWXXM / METARS

http://en.wikipedia.org/wiki/IWXXM


34

• Once published, content and structure of IWXXM messages will not change

• Messages will contain a digital signature

• Use GetPropertyValue to download only original METAR string

• http://data.fmi.fi/fmi-apikey/…/wfs?request=GetPropertyValue&storedquery_id=fmi::avi::observations::finland::latest::iwxxm&valuereference=wfs:FeatureCollection/wfs:member/avi:VerifiableMessage/avi:metadata/avi:MessageMetadata/avi:source/avi:Process/avi:input


http://data.fmi.fi/fmi-apikey/0017a4fc-b1e9-4851-850c-0f9ff66f8cae/wfs?request=GetPropertyValue&storedquery_id=fmi::avi::observations::finland::latest::iwxxm&valuereference=wfs:FeatureCollection/wfs:member/avi:VerifiableMessage/avi:metadata/avi:MessageMetadata/avi:source/avi:Process/avi:input





















35

• Messages have been automatically created based on the original METAR/SPECI code messages.

• Messages are NOT an authoritative aviation weather report. They SHOULD NOT be used as a weather report to be used in flight planning or other aviation related use.



36

• WFS response contains the same meta data information than in observations and point forecasts

• om:result contains gmlcov:RectifiedGridCoverage• Basically the same with multipointcoverage

• domainSet defines the grid (now as regular grid)

• rangeSet contains data as an external link to the binary content• rangeType defines the parameters

Data Modelsbinary data


37

• Note that fileReference points often to an other service• Can be used directly, but

• It is always good practice to consult WFS for available times and parameters


Client

WFS

WMSFMI Data Server

Available radar images?Available weather model outputs?


38

• Area, time and weather parameters may be defined in the request

• For weather models, intersection of requested area and available data area is returned

• For radar images, all images which intersects requested spatial and temporal space are returned


Requested area

Radar images

Radar images Requested area

Model coverage

Weather models


39

Radar images• The reference points to a “original” gray scale GeoTiff image• Images can also be downloaded as color images

• Remove &styles=raster from the data request

• But then information is lost

• Used SLD-files can be downloaded from https://github.com/fmidev/opendata-resources/tree/master/sld

• Consult om:parameter element for single radar measurement parameters

• Scanning angle, bin count and bin length


https://github.com/fmidev/opendata-resources/tree/master/sld

https://github.com/fmidev/opendata-resources/tree/master/sld


40

Numerical models• The reference points to (often a subset) the model output in

appropriate format• GRIB2 for weather models

• NetCDF for marine models

• Note that whole model output can be large• Up to 17 GB

• Download always only area, parameters and time range of interest



41

For binary data• PanoPly

• http://www.giss.nasa.gov/tools/panoply/

• Simple

• Handle only very basic data

• IDV• http://www.unidata.ucar.edu/software/idv/

• Handle most of the data

Helpful Software

http://www.giss.nasa.gov/tools/panoply/



http://www.unidata.ucar.edu/software/idv/

http://www.unidata.ucar.edu/software/idv/


42

For exploring the data• Google Earth & QGIS• Start from GetCapabilities, UI guides thereafterhttp

://wms.fmi.fi/fmi-apikey/.../geoserver/wms?service=WMS&version=1.3.0&request=GetCapabilities

Helpful Software

http://wms.fmi.fi/fmi-apikey/0017a4fc-b1e9-4851-850c-0f9ff66f8cae/geoserver/wms?service=WMS&version=1.3.0&request=GetCapabilities




43

Libraries• MetoLib (JS) https://github.com/fmidev/metolib• Weather Downloader (Python)https

://github.com/Tumetsu/FMI-weather-downloader• API R client (R) https://github.com/rOpenGov/fmi

Helpful Software

https://github.com/fmidev/metolib



https://github.com/Tumetsu/FMI-weather-downloader



https://github.com/rOpenGov/fmi

https://github.com/rOpenGov/fmi

05/02/2023 44

Please contribute!

https://github.com/fmidev/

MetoLibo Open source JavaScript library produced by Finnish

Meteorological Instituteo Helps users to load and use the data

o Supports multi point coverage data formato Python version is on the list

Easy requests Cache

Parse the data to as JSON


www.fmi.fi

fmi open data interface and usage

Internet