4D Data Visualisation and Quality Control

Peter Löwe, Jens Klump

GeoForschungsZentrum Potsdamploewe@gfz-potsdam.de, jklump@gfz-potsdam.de

Intention of the talk● Give a better understanding of the relevance of

„preview formats“ for quality control of complex enironmental data.

● Demonstrate how preview formats can be generated with FOSS geoinformatics applications.

● Real world example:– A soil erosion topic, relying on the processing of

„complex data“– which uses FOSS GIS for this task.

•Topic, theory •Hypothesis and approach•Technical frameworkTechnical framework•Modelling•Results / Lessons learnedResults / Lessons learned

Overview: Soil Erosion Project

Soil erosion is the process of soil destruction, a natural process, which can be initiated or amplified by human land management. ... Soil erosion can deminish the agricultural yield significantly.

Soil Erosion: The Problem

In addition to human use of the Earth's surface, climate is a key factor. It provides a means of transportation for soil material to be carried away.

How can these How can these processes be processes be modelled ?modelled ?

ClimateTerrain

Soils

Vegetation

Humans

SOIL EROSIONWind WaterTransportation

Control Enhance

Start

Water, as in ... rainfall

● We need a map of the rainfall distribution (simple task)– Regular updates – Sufficient resolution – Reliable

● The changes of this map can be used to calculate the „potential erosivity“ of the rainfall for a given area:– Total amount: how much water comes down in total ?– Temporal Pattern: Once only, repeated soaking ?– Small droplets, big droplets ?

Idea and approachHypothesis: There are small, temporally fluctuating peaks of There are small, temporally fluctuating peaks of erosiviy due to the convective weather situation. How can these erosiviy due to the convective weather situation. How can these erositiy peaks be charted?erositiy peaks be charted?A sufficiently high temporal (When ?) and spatial data coverage (Where ?), is needed, and also a measure of confidence for the data values (How much ?).

To answer „When - How much - Where“ the radar reflectivity products must be accessed and processed.

ModellingTechnical Framework

Results

Analysis and encoding of the Erosivity.

Geoinformatics,

Information-logistics,

Remote sensing,

Radarmeteorology

•Verification•Validation

•Results

Practical Approach

We use ground-based weather radar for a test site:– 5 Minutes scan rate (200 km radius, 18 km vertical)– Pixel/Voxel resolution 1 km – continous Reflectivity Data (not rain)

400 km

18 km Lower Atmosphere

X

XX

XX

ab

cd

Erosivitymaps

Reflectivity GIS-layers for defined altitudes

Rainfall maps, Pluviogram

XXXX

X X XX

1

42

Erosivity-Model

What type of weather occurs when, where?

Where does erosivity potential occur ?

When does how much rain fall where ?

-

Data flow

3D -> 2Dtransformation

18km altitude

1km altitude

Precipitation-data stream

of „radar rain“

InitState 1

Hibernate

State 2Store

State 3Pause

P: Precipitation

D: Dry

DD

D

PP

P

A „virtual rain gauge“ [state machine] is simulated for each spatial cell of the radar coverage. Erosivity values are derived according to the cell's individual input data stream.

For this reason, agent technology is used, as each „gauge-agent“ must keep its own record of previous precipitation events.

Erosivity-Modelling

Erosivity-data stream Maps

Index-value

Cell-Agent

Tools of the trade● GRASS GIS

– Raster and volume data processing– 2D Animations (flip-books)

● NVIZ (part of GRASS)– 3D visualisation and animation

● Paraview– 3D visualisation and animation

GRASS GIS

● GRASS GIS is a Geographic Information System (GIS) used for geospatial data management and analysis, image processing, graphics/maps production, spatial modeling, and visualization.

● Oldest and largest FOSS GIS project● GRASS is official project of the Open Source

Geospatial Foundation.● Scriptable● „Backend use“ in QuantumGIS, PyWPS,

JGRASS

Calculating Totals

24h total

erosive

16:18:50 Hours

16:43:30 Hours

16:59:56 Hours

Erosivity

Reflectivity

Σ

Σ

Left: ReflectivityCentre: RainfallRight: Erosivity

Erosivity-Totals display of local erosivity pulses

Elevation: Rainfall totalColor: Erosivity total

Conclusion: The model implementation works !

Animated Time SeriesAnimated displays of reflectivity, derived “radar rainfall” and the

corresponding erosivity peak pattern were created with GRASS GIS.

Reflectivity “Radar-Rain” Erosivity

The erosivity ribbons (right) follow the rainfall fields (center): The model works

The Challenge● 2D animations and 2.5D images show that the

erosivity modelling „works well“– [the erosivitiy peaks follow the rainfall fields]

● However, the model depends on input data:● Can we trust the data ?

– Metadata appears correct.– [are the rainfall fields correct ?]

● Weather Radar provides 3D data. – [3D->2D transformation: Correctly done?]

3D: Straightforward Approach

NVIZ-Animation (GRASS): „Real Clouds“

Volume [Full Information]= Reflectivity Values (30/40/50 dBz) 2.5D Surface: MaxReflecivity(Color), Radar horizon (Elevation)

Flattening

NVIZ● „GRASS in-house“ visualisation tool● Pro: Works directly on the internal database● Pro: Scriptable● Con: small user base, bugs, missing documentation● http://grass.itc.it/nviz/

● Parallel Visualisation Toolkit● Frontend to VTK + QT● Large userbase.● GRASS-related import issues:

– Loosely coupled via file system– Ascii-VTK-Format – currently not effective for use. Improvements

are hoped for later this year● http://www.paraview.org

Garbage in, Garbage out

● Can we trust the rainfall information of the weather radar ?

● Model results are based on rainfall data.● Errors and Biases in the rainfall data will affect

all derived products.● What about transient biases which might

vary in time or space?

Flattening TrustTrust

3D data

2D information

Boredom in, Boredom out● Large data archives exist and more data are

added every day.● How can we easily identify time intervals when

„some interesting weather“ has occurred?● We could watch it all in 4D (3D over time):

– Takes too much time, is incredibly boring– Problem to watch the right things at the right time.

Introducing Preview Formats● A visual preview format provides a condensed

view with the relevant information for the current question („determinant“).

● Humans are visualy oriented: preview formats (shapes/volumes) are easier to comprehend than numbers.

Transient Phenomena Preview● What was the weather like

for a 24 h period ?● Try this: 2D Animation

(Flip-Book)

● Alternative: Stacking of the flip-books pages (just the ink, really) and look at all pages at once.

● Howto: Creation of a volume in GRASS GIS, visualisation by Paraview.

From single drawings

1 2 3

„Radar Rain“

Erosivity

Flip-book Volume

time

1

2D Space: Rainfall field

2

3

Yellow: RainfallRed: Erosivity

Data Errors(ground targets)

Rainfall field

Rainfall field

Erosivity Peaks

Not realclouds !

1

2

3

Ce n'est pas un nuage!

● Detailed top-view of the track of a precipitation field (yellow) and the derived erosivity pulses (red). Note the highly localized distribution of the erosivity pulses. This information can be used to calibrate the interaction between point-sampling rain gauge networks, weather radar calibration and soil erosion plots.

Painting of a pipe

Rainfall tracks of clouds

(+ „erosivity tracks“)

Quality Control 1.0

A precipitation field and its resulting erosivity pulses shown in side-view.

The image does not show real world clouds but precipitation- and erosivity tracks.

„Soaking“ The height of a rainfall track tells us how long it did rain at a certain location

Rods of eternal

soaking: Data errors

Dimensional Collapse

The 2D (xy) rainfall field was „squeezed“ out of the 3D (xyz) weather radar data, implicitly „collapsing“ the vertical dimension.

The stacking of the time frame „flip-book“ pages substituted the altitude (z) dimension by the time dimension.

Collapse 2.0

This approach can be followed further:● In the previous example we collapsed the z-

dimension● Now we collapse the horizontal (xy)

dimension.● The resulting diagram is a preview format

commonly used in meteorology: the „Contoured Altitude by Frequency Diagram“ (CFAD).

Contoured Frequency by Altitude Diagrams (CFAD)

● CFAD can be created from 3D radar reflectivity data (original airspace radar scan). The 3D data set is sliced vertically.

● A histogram of the reflectivities (1D) is generated for each slice/layer.

● Stacking the histograms gives us a 2D synopsis of the current situation in the scanned airspace.

● This tells us a lot about the weather and potential measurement errors.

● In our case, this task is done in GRASS.

CFAD – An Example

● Contoured Frequency by Altitude Diagram (CFAD). Numbers on contour lines give the number of voxels in the observation area with a given radar reflectivity. The CFAD gives a snapshot of weather intensity at different altitudes in the lower atmosphere.

2D Animation

Leafing through the flip-book:Weather CFAD

CFATD: One step beyond● Contoured Frequency Altitude by Time Diagram

adds the time dimension, resulting in a volume body.

● The shape of the CFATD makes it easy to identify:

● periods of high radar reflectivity, i.e. intense weather, and

● Errors in the radar or processing chain.● Done in GRASS, displayed in Paraview

Flip-book volume● Once again we can create a volume from the

flipbook.

Time

Altitude

„Droplet Size“

Iso Surfaces resemble levels of droplet counts (a few, many, lots)

Critical threshold: If the inner layer (many droplets) of the „loaf“ exceeds it, then there is heavy downpour or even hail.

Visual Quality Control● CFATD gives a convenient and reliable quality measure

for observations not to use

● If the CFATD structure appears blocky, or „non-organic“: discard the data

Faulty data

Faulty data

Better data, better models● 4D previews for „Live Quality Control“ in sensor

systems:– Weather Radar does „now-casting“

● It looks into the distance (right now) ● but not into the future

– Real-time generation of CFATD „loaves“ could be used for radar system calibration and maintenance.

What level of quality do we get RIGHT NOW ?

Applications in Grid/eScience● Dimensional collapse and 3D animation of data

can be used as a preview format for very large/complex datasets.

● The computing power needed for the generation of these preview formats can be sourced from the Grid.

Conclusion

● Complex (4D) data are not easy to interpret.● Preview formats enable identification of biases

drifting in time and space in complex data.● Preview formats save time in the selection of

useful data.● You can do it, too:

– Ukrainian Radarsystem (MRL-5 + Linux-based Operating Software (Titan))

– GRASS + Clips + Database + Paraview

Thank you for your attention !