bde esd tool - big data met norway rasmus benestad

Utskifting av bakgrunnsbilde:

- Høyreklikk på lysbildet og velg «Slide» -> «Set Background Picture for Slide...»

- Velg ønsket bilde og klikk «Open»

Brussels 2015-06-15

Climate analysis & Big data

Rasmus E. Benestad

Abdelkader Mezghani, & Kajsa M. Parding

esd for retrieving, processing, dissecting, analysing, and visualisation

How do I find useful climate informationin large volumes of data?

Information & Data & Knowledge

● Information "...in its most restricted technical sense, is a sequence of symbols that can be interpreted as a message".(wikipedia)

● Data are measurements, observations, calculations

● Knowledge is expectation about causalities, dependencies, and why?

Norwegian Meteorological Institute

The ultimate objective is to find the answers.

The data is the source and the means finding answers.

Analytical tools designed for data analysis/statistics.

What is the question?

Big data - simple algorithmsFast “distillation” of information

Open source tool ‘esd’http://github.com/metno/esd

Norwegian Meteorological Institute

esd: open-source and free

Data accessSource of information

Temperature

ECA&D:~1.1 Gb-temperature-precipitation

Temperature

ECA&D +GHCN +MET Norway-temperature

Precipitation

Reanalyses and satellite data

Climate model results

Norwegian Meteorological Institute

Make use of information hidden in vast archives of climate model results and observations

ExampleExtracting information embedded in global climate models and observations

Climate model results

Norwegian Meteorological Institute

Norwegian Meteorological Institute

Norwegian Meteorological Institute

What is hidden behind the results?

Norwegian Meteorological Institute

Building block

Empirical-statistical downscaling

Dependencies & connections

Redundancy

Norwegian Meteorological Institute

Extension of the results to regions

Norwegian Meteorological Institute

High one-in-twenty mean summer temperature in 2100

InfographicsExposing different aspects

Different sides to information

Changes in rainfall statistics?

Quick look at temperatures

Maps

Correlation

AnalysisPatterns of behaviour

Anomaly with respect to latitude

How similar are the reanalyses? Correlation maps

How good are the models?

Using mathematics to make sense of the data

Is there a change in the precipitation?

Norwegian Meteorological Institute

How much does it rain?

Norwegian Meteorological Institute

Global rain gauge data

= huge volume

35,000 rain gauges with daily data

Best way to mine hidden information?

Norwegian Meteorological Institute

Understand the data

Exponential distribution?

Vast number of data points on top of each other

Norwegian Meteorological Institute

Extract the essence (cleverly)

Principal component analysis: two main characteristics

Norwegian Meteorological Institute

E.g. The wet-day 95-percentile for 24-hr precipitation

Calculated Observed

Norwegian Meteorological Institute

Test results

Norwegian Meteorological Institute

Changing rainfall patterns

Norwegian Meteorological Institute

Changing rainfall patterns

Norwegian Meteorological Institute

Aggregate

Principal Component Analysis (PCA)

Regression analysis

Quick search

Combination of sources

Mapping/gridding

Statistical distributions

Main instruments

Norwegian Meteorological Institute

Tricks for efficiency

Structured and standardised metadataCommon information model (CIM)Data reference syntax (DRS)classes and ‘S3’ methodsFast algorithms making use knowns

Norwegian Meteorological Institute

• Facilitate intercomparisons

• Sharing of generic methods

• Traceability and replicability

• Promotes community building & discussions

Benefits of common standards & structures

Norwegian Meteorological Institute

Summary

●esd - “easy and simple data” or empirical-statistical downscaling

●Statistics - information from the data

●Aim to address specific questions

Meteorologisk institutt

Thanks for your attention!