multimedia on the mountaintop: presentation at acm mm2016

Multimedia on the Mountaintop: Using Public Snow Images to Improve Water

Systems Operation

A. Castelletti, R. Fedorov, P. Fraternali, M. Giuliani Politecnico di Milano, Italy

ACM MM 2016, AmsterdamBNI session

The (hopefully brave new) idea

• There is a lot of multimedia content out there, produced by– People– Ground sensors

• There are many environmental problems that lack affordable and accessible input data

• Question: is public web visual content good enough to help in such environmental problems?

Andrea Francesco Castelletti

insisterei sul concetto di validation data

Observing the earth

• Not everything can be done from above• There is not a single satellite product good for all• (Useful) satellite products are costly• Clouds may be a problem


for all ?? data

The grand challenge: water scarcity• Climate change, urban concentration and agriculture

put water resources under stress• Predicting future availability is key• When you have mountains, water is stored as snow

UK_WATER SUPPLY UTILITY15 million customers2.6 Gl/day drinking water3 billion $ revenue (2013-14)


ti ho messo una figura alternativa

The contentInput• User generated

– 700.000 Flickr images crawled so far within 300x160 km

• Sensor generated– 2000 webcams queried every

minute (10 – to 1500 images per web cam per day)

– More than 10M images crawled so far

Output• Virtual Snow Indexes:

numerical time series that are a proxy of the quantity of water stored in the snow pack (Snow Water Equivalent – SWE)

The multimedia pipelines

• Differences– Web cam images have high temporal density, UG images

have broader spatial coverage– UG photos searched by keywords may be irrelevant,

webcam images always portrait mountains– UG photo mountain classifier already discards bad

weather images

UG Image relevance

• 7000 images randomly sampled and used for a crowdsourcing experiment: “Do you see a mountain in this picture?”

• Classifier trained (94% precision, 96.3% recall)

Webcam image enhancement

Remove/attenuate:• Variability of illumination• Shadows• People & irrelevant objects

Daily median image

Mountain peak identification

orginal image edge maps

skyline estimationDEM generated virtual panoram

VCC best matching

Snow mask extractionSnow classification at the pixel level

Snow mask extraction

Snow Virtual Indexes

The case study• Regulation of mountain inflow dependent lakes

Lake Como Catchment area Lake Como 4500 km2

Reservoirs Lake Como 247 Mm3 Alpine HP 545 Mm3

StakeholdersFarmers:

irrigated area 1400 km2

Floods:lake and downstream

….

Local folklore

Formalization: 2 objectives optimization

• Decide the daily lake outflow ( lake level)

• So to– Maximize water for

downstream irrigation– Minimize # of flood days

• Respecting– Minimum outflow

requirement for ecological preservation of effluents

• Based on– Policy input (X)

• Regulator's policies– Baseline: regulator only considers lake

level and day of year– Upper bound: regulator knows the

water that will be available (lake inflow) in the future

– P_x: regulator knows partial information (x) on the water that will be available (lake inflow) in the future

• What is X?– P1: Official snow water equivalent

data estimated from Region Lombardy– P2: virtual snow indexes from nearby

mountain images– P3: official SWE data + virtual snow

indexes

PS: Upper bound policy can be calculated retrospectively for the past, where you know how much water you actually got day by day


to be correct: conosce un proxy/surrogate dell'acqua che sarà disponibile, ma non ha una previsione. Non so se la distinzione è rilevante per l'audience probabilmente no

Assessment method

Select information based on its

expected value(Iterative

Input Selection)

Design control policy based on selected input

information

Quantify performance of policy + selected

information

Quantify value of perfect

information Expected Value of Perfect Information (EVPI)

Inflow data series Outflow data series

Baseline policy Upper

boundpolicy

Input data

series(exogenous

variables)

Most Valuable Information

(X)

X_informed control policy(P_x)

J(P_x)Performance of

P_x

Performance metricsHyper Volume Indicator

(HV)

Performance improvement over baseline(ΔHV)

Assessment results

Thank you & … see you soon in the PlayStore

Content processing pipeline• Photo contains/does not contain mountain landscape

binary classifier– SVM with Dense SIFT, Spatial Histograms. 7k annotated

images (majority of 3 votes). 95.1% Accuracy on balanced dataset.

• Peak identification / Photo orientation estimation– Ad-hoc algorithm with edge extraction and vector cross-

correlation. 160 images manually aligned w.r.t. Digital Elevation Model. 75-81% of images correctly aligned (depending on weather conditions).

• Pixel-wise snow/non snow classifierRandom Forest, trained/evaluated on 60 manually segmented images (single annotator) for a total of 7M of labeled pixels. 91% accuracy.

Iterative input selection



Input Selection)


information


information





boundpolicy

Input data

series(exogenous

variables)


(X)



P_x


(HV)


D=distance metric

Policy search



Input Selection)


information


information





boundpolicy

Input data

series(exogenous

variables)


(X)



P_x


(HV)


Good decisions matter

WATER DEFICIT

FLOOD THRESHOLD

EFFECT OF REGULATION

For more info• A. Castelletti, R. Fedorov, P. Fraternali, M. Giuliani:

[email protected]• http://snowwatch.polimi.it/

mailto:[email protected]

multimedia on the mountaintop: presentation at acm mm2016

Environment