Public policy implications of accuracy assessment for fire and

deforestation monitoring: Don’t forget the error bars

A story of sin and redemption

Foster Brown

Woods Hole Research Center/

Universidade Federal do Acrefbrown@uol.com.br

III LBA Scientific Conference

Brasilia, DF, 28 July 2004

Premises for this presentation

1. Indicators are not the reality, but approximations to a reality, such are deforestation estimates and hot pixel data

2. All data have error, or less pejoratively, have uncertainty.

3. Binary logic can be overused, particularly for generalities, such as: reliable/unreliable, correct/wrong, Needs to be coupled the word ‘depends.’

Premises applied

1. Approximations – Classified satellite pixels or fractions of pixels to

estimate area deforested.

– Satellite pixels classified hot to estimate fires.

2. Uncertainty (error)– Is the uncertainty small enough so to be able the

answer the research question?

3. Are the data of X, Y or Z reliable/correct? Answer: Depends on the question asked.

Explicit versus Implicit Uncertainty

• Explicit (error bars): – Reports a range (absolute or probabilistic) around a value:

35 ha ± 3.0 ha.

– Reports a range: 1-3 Pg C/year, without central value.

• Implicit– Uncertainty is in the last digit

– 12.4 means a value between 12.35 and 12.45 or 12.4 ± 0.05

– Number of significant figures determines implicit uncertainty.

Examples of explicit uncertainty I: opinion polls

• Candidate A was preferred by 50 percent of those polled with a margin of error of plus or minus 3 percent.

• Uncertainty based on binomial distribution and is likely to be higher due to sampling factors.

Examples of explicit uncertainty II: weather forests

• Question: Will it rain today?• Responses

– 1. There is a 10% chance– 2. There is a 30% chance– 3. There is a 60% chance– 4. There is a 90% chance

• When to carry an umbrella becomes a user’s decision.

The danger of too many significant figures: World Bank estimate in 1988 for deforestation in Brazilian Amazonia

(Mahar 1988)

• 598,921.7 km2, 7 significant figures, implicit uncertainty of ±0.00005%

• Student in class: “The gringos know all about us.” • If it is published, especially by the World Bank, then in must

be true.• Subsequent analysis: 415,000 km2

• Difference of approximately 50% of most likely value, a million-fold higher uncertainty than the implicit uncertainty.

• Right or wrong? Wrong in implying that the number is so accurate.

• Implications for public policy: World Bank is fallible.

If the World Bank can make mistakes, so can researchers,

local institutions, governments.

The World Bank would have been right if it had reported 600,000 km2 ±35%

Excessive use of significant figures, 598,921.7 km2, means loss of credibility

Examples for public policy for Acre The data are good/bad, depending on the question.

• Zoning for Acre State, Brazil (Sassagawa and Brown ZEE, 2000)

• Data sources: INPE, Funtac, Ibama, Michigan State University.

• Question: Is the future of Acre in the forestry sector? – If more than two-thirds of the state has forest cover, it

would be a good sign.

9.0

7.4

5.0

6.7

INPE FUNTAC IBAMA MSU

Percent of Acre State, Brazil deforested by 1996Note: All values < 10% deforested >90% of Acre has forest in 1996.

Good news for public policy.

But what about how much deforestation has occurred in

Acre?

• The simple answer: “By 1996, estimates ranged from 5.0 to 9.0%.”

• Other questions:• Where is it occurring? INPE couldn’t answer, it is

a state-wide estimate. Need to use the others.• Is the rate changing? Only INPE has yearly data

for which yearly deforestation rates can be calculated.

Example of Deforestation Rates for Acre State

Zoning report for Acre – 2000, reconciling two different data

• 9.0% for INPE and it is put in a time perspective with deforestation rates during the 1990s.

• 7.4% from Funtac which shows that most of the deforestation occurs in eastern Acre.

• We needed to explain why we can use 9.0% and 7.4% in the same chapter for area deforested in Acre.

• Different methods and definitions can produce different areas of deforestation.

Implications for Public Policy• Increasing deforestation rates for Brazilian Amazon have

been of great concern (INPE website19jan04):

– 2000-2001 18,166.00 km2/yr– 2001-2002 25,476.00 km2/yr

• This reported increase of 40% has influenced Amazonia Sustentavel and other public policy initiatives.

• Most recent (INPE website 28Jul03)

– 2000-2001 18,165 km2/yr– 2001-2002 23,266 km2/yr

– Increase 28%

454.86 726.1105.62 704.3369.03 677.9365.79 535.91561.71 499.91611.56 464.1232.28 457.09

227-67002-67231-67226-67228-67225-64001-67

CENA 2000-2001 2001-2002 km2/year

Orginal data recently revised. Two image scenes from Acre in red.

Implicit uncertainty <0.05%.

263.48

178.50

2001-2002revised

450 730110 700370 680370 540560 500610 460230 460

227-67002-67231-67226-67228-67225-64001-67

CENA 2000-2001 2001-2002 km2/year

Official data rounded to ± 5 km2 /year (2 significant figures)

Implicit uncertainty <5%.

260

180

2001-2002revised

Official data rounded to ± 50 km2/year (1 significant figure)

Implicit uncertainty < 50%.

CENA (Image) 2000-2001

2001-2002

Revised

2001-2002

002-67 100 700 300

Revision >50%

001-67 200 500 200

Revision >50%

Fires in Acre

• Questions important for public policy:– A) How many fires are occurring in Acre?– B) Where and when are the fires occurring in

Acre?

“Error” Matrix for Hot Pixel Data and public policy implications

Fire observed on ground

No fire observed on ground

Error of commission = B/(A+B)

Fire observed by satellite

A, good B, false fire/ comission error/ false positive

No fire observed by satellite

C, missed fire/ omission error/ false negative

Error of omission = C/(A+C)

BAD FORENFORCE-MENT

BAD FOR FIRE CONTROL

Synthesis of indicators for fires in the State of Acre for 2001 – NOAA 12 – great underestimate

for 2001 (1/10?)

Selhorst and Brown (2003)

Need to choose between low omission and low commission errors

• To minimize commission errors typically means accepting higher omission errors.

• Costs of being wrong depends on the objective.– For enforcement, high cost of commission errors led

to INPE to minimize this error to avoid sending crews on wild goose chases (looking for fires that are not there).

– For fire control, the fire not detected could be very costly: decrease omission error and increase commission error (fewer missed fires and more wild goose chases).

Critical role of hot pixel maps in showing where burning is likely

to be occurring

Concentration of hot pixels showing where burning is

occurring (MODIS data June-September 2003)

Pontos MODIS (jun-set03)Imagem LandSat 1999

Pontos em

Fazendas

P. A. Tocantins(1997)

P. C. Humaitá(1981)

Public Policy Implications

• Absolute number of hot pixels can be misleading, frequently underestimates fires (omission errors).

• Combining hot pixels from various satellites can be useful show relative concentrations.

• Maps of hot pixels showing their spatial location can be very useful to indicate where a closer look is needed.

• Maps need to be timely: available to those who can use the information shortly after observations.

Keynote address for the 1999 Simposium

N.R. Chrisman - Speaking truth to power: an agenda for change.

(Title based on Quaker religious saying)

The sins of unjustified significant figures for deforestation estimates

(I have sinned, too)

• 1988. World Bank. 0.00005% went to 50% uncertainty: factor of 1,000,000.

• 1996. INPE+IBAMA+FUNTAC+MSU. 5% to 20+%. differences: factor of 4.

• 2003. INPE Prodes. 0,05% to 50%: factor of 1,000.

Why Do We Sin? - 1

• The legacy of 19th Century Physics and calculus – the myth that absolute knowledge exists and can be calculated.– Concept of Exact Sciences is alive and well. University

in Israel and Brazil have divisions of exact sciences.– "One aim of the physical sciences has been to give an

exact picture of the material world. One achievement of physics in the twentieth century has been to prove that aim is unattainable.“ - Jacob Bronowski (1973)

– So why do we keep trying to attain it?– Habit and world view

Habit – World View

• “One of the most pervasive forces in technology is the tyranny of the way things have always been done…

• The research community…does not influence the practice of GIS until the research concepts change the routine ways of doing things.” N.R. Chrisman (1999).

Why Do We Sin? - II

• Fear: “Perhaps the worst nightmare of a natural resources manager is to appear ‘uncertain’ to the public, or to admit that there is ‘error’ in the decision process being presented.” H.T. Mowrer (1999) Accuracy (re)assurance: selling uncertainty assessment to the

uncertain.

Why Do We Sin? - III

• The bookkeeping approach is the most common means to present data– Neat, tidy, and always two numbers to the right of the

decimal point– 123.34– 12.22– 1.12– .02– Data are commonly reported in newspapers in a similar

fashion. Spreadsheets are designed with this structure in mind.

How do we seek redemption?

• Develop sensitivity analysis for uncertainty propagation (Monte Carlo approach)

• Compare various sources of data for consistency/inconsistency. Example Aster, Modis, Landsat.

• Make uncertainty explicit with error bars and ranges – Don’t sin like the World Bank.

Questions and recommendations from the workshop on accuracy assessment 26 July 2004

• How good is good enough for public policy? - I

• For deforestation rate estimates, let us try to document ±20% or less uncertainty.– 2000-2001: 18,000 ± 3,600 km2/yr– 2001-2002: 25,000 ± 5,000 km2/yr

• Remember the difference between precision and accuracy.

Questions and recommendations from the workshop on accuracy assessment 26 July 2004

• How good is good enough for public policy? – II• Hot pixel data – tendency for omission errors. • Acre: 40,000 small rural producers who burn most years.

Hot pixels: NOAA-12 ~ 1,000, MODIS ~ 7,000. • Use hot pixels as a qualitative indicator (low, medium,

high) and as an alarm (near real time) accessible to society. • Compare hot pixels from different satellites as a

redundancy check for alarms as a measure of uncertainty.

Questions and recommendations from the workshop on accuracy assessment 26 July 2004

• Involve users in validation exercises –research and government institutions and rural organizations.

• Document explicit uncertainties for deforestation estimates and hot pixels/fire counts