1

GODDARD SPACE FLIGHT CENTER

Reflectance intercomparisons at a RadCalNetsite for training and traceability

K. Thome, C. Ong, C. Barrientos Gajardo, I. Lau, J. Czapla-Myers, B. Wenny

JACIE Workshop

College Park, MD 17-19 September 2018

2

Training course background

Geoscience Spaceborne Imaging Spectroscopy

Technical Committee (GSIS TC) within IEEE

conducted a training activity in July 2017 Follow-on to a successful course at the 2016 IGARSS

meeting in Beijing

Outcome of the original course was that practical

experience with field measurements would be beneficial

Decision made to leverage the 2017 IGARSS locale of Ft.

Worth, Texas to do the practicum at Railroad Valley Playa

in Nevada

3

Goals

Training for personnel working across various

earth observation areas including sensor

engineers to calibration scientists

Visit a CEOS WGCV RadCalNet test site and

the associated calibration laboratory for that

site

Demonstrate measurement protocols for

surface and atmospheric parameters

Concentrate on vicarious calibration of

imaging spectrometers

Minimize uncertainties

Perform a reflectance-based calibration of

an on-orbit satellite sensor

Expose the students to >30 hours of old guy

road trip music

9/12/18, 3)34 PMwestern us map - Bing Maps

Page 1 of 2ht tps://www.bing.com/maps?q=western+us+map&FORM=HDRSC4

1600 Elm St, Fort Worth, TX 76102

US-6, Tonopah, NV 89049

26 hr 53 min, 2522.2 km

Moderate traffic (22 hr 38 min without traffic)

Via I-20 W, I-10 W

A

B

© 2018 HERE© 2018 HERE

250 miles250 miles 250 km250 km

Railroad Valley Playa, USA

• >20 years working experience on site

• 4 radiometers, sun photometer, met

station

• Dry lakebed

4

Course provided an opportunity

Number of people and availability of equipment allowed for

studies of reflectance measurement uncertainties

Surface reflectance inter-comparison similar to past campaigns

including those from Lunar Lake 2000 JACIE efforts

Impacts of spatial and spectral homogeneity of the site being

measured

Approaches to separate instrumental variability from test site variability

Methods to improve site sampling

Wavelength (nm)

400 700 1000 1300 1600 1900 2200

0.25

0.35

0.45

0.55

JAC UA

JPL SDSU

5

Laboratory visit

Exercise included a tour of the optical calibration lab operated

by University of Arizona Remote Sensing Group (RSG)

S. Biggar of RSG described the facilities and led the tour of the group’s

radiometric calibration facility

Tutorial put the laboratory measurements into context of the field data

Also provided a chance to describe the measurement campaign

plans

6

Measurement protocols

Underlying theme of the training course is related to SI-

traceable uncertainties within reflectance-based calibration

Lectures emphasize the development of error budgets and

approaches to limit uncertainties

2017 Training Course examined uncertainties for reflectance-based

vicarious calibration

Leveling reference

Alignment to reference

Reference measurement

and note taking

7

Intercomparison studyTwo areas were used to evaluate the consistency

between retrievals of surface reflectance

Results would indicate how well the training went

Also provide insight into uncertainties of the

vaalidation of atmospherically-corrected, analysis

ready surface reflectance products

Help evaluate error budgets for Railroad Valley

RadCalNet TOA reflectances

Instrumentation and references were from RSG,

thus evaluation of absolute uncertainty was more

difficult

Did collect data to evaluate two different SI-

traceability paths to surface reflectance

On-site characterization of RadCalNet

radiometers

8

Repeatability

80 m linear transect collected multiple times

by each group over multiple times Three teams of two people

Spectroradiometer operator

Note taker/spotter/reference panel operator

Teams consisted of a mixture of levels of field spectroradiometer

experience

Reference followed by 80-m transect and finish with reference

Spectra of site collected while walking

80 m East-West

9

Intercomparison Results

12 runs over 3 hours on the first day by 7 operators using 2

spectroradiometers and 2 panels

Clear skies over sun with some patchy clouds as day progressed and

temperatures >37 oC

~4% variability in the average reflectance of all runs from July 31

Instrument dependant variations were seen between the

spectroradiometers

-8

-6

-4

-2

0

2

4

350 650 950 1250 1550 1850 2150 2450

Perc

ent

Dif

fere

nce

Wavelength (nm)User 1 to User 2 Morning to afternoon Day 1 to Day 2 Continuous to stop/measure-8

-6

-4

-2

0

2

4

350 650 950 1250 1550 1850 2150 2450

Pe

rce

nt

Dif

fere

nce

Wavelength (nm)

10

Measurement method comparisonComparison of the moving collection (continuous)

method versus a stop and measure method Collect data at 8 spots on 80-m transect

Operator collects at while standing stationary

Panel reference measurement at start and end of each

surface measurement

Minimize impacts of atmosphere, instrument variations

Drawback is collection samples less area and can take

longer to cover site

11

Stop-Measure Results

Retrieved reflectance from two different collection approaches

within variability of a single collection approach

Noticeable difference between results of the two sampling strategies

Standard deviations were typically larger for the stop-measure

approach (overall, per sample, averaged per sample)

-4

-2

0

2

4

6

8

350 650 950 1250 1550 1850 2150 2450

Co

nti

nu

ou

s to

sto

p-m

easu

re

Perc

ent

Dif

fere

ce

Wavelength (nm)

-10

-8

-6

-4

-2

0

2

4

6

8

10

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

350 650 950 1250 1550 1850 2150 2450

Perc

ent

Dif

fere

nce

Ref

flec

tan

ce

Wavelength

Continuous Stop-measure Percent Difference Continuous Std Dev. Stop-measure Std Dev

0

2

4

6

8

350 850 1350 1850 2350

Perc

en

t St

and

ard

D

evia

tio

n

Wavelength (nm)

0

1

2

3

4

5

6

350 650 950 1250 1550 1850 2150 2450Perc

ent

Stan

dar

d D

evia

tio

n

Wavelength

Continuous

Stop-measure

12

Reflectance-based test site

Sentinel-2a overpass offered an opportunity to evaluate large-

site retrievals

Used continuous sampling and stop-measure sampling approaches

Transects are 20 m apart over a 240-m x 80-m area

Reference panel measurement, relocated at the southern end, at the

start and every second transect

Stop-measure relied on 12 random transect locations from the 48

possible

13

Reflectance-based site results

Cloudiness prevented all operators from collecting and from

determining the Sentinel-2A comparison

Results match with 80-m transect

collections

Results are within typical results of

reflectance-based collections

-6

-4

-2

0

2

4

6

350 650 950 1250 1550 1850 2150 2450

Pe

rce

nt

Dif

fere

nce

fro

m 1

90

0 U

TC

colle

ctio

n

Wavelength

14

Reflectance- versus radiance-based retrieval

Compared surface reflectance reflectance ratio to absolute radiance measurements

Near simultaneous collection

Field spectrometer data converted to

reflectance using diffuser

Radiance data converted to reflectance

via radiative transfer code calculations

based on measured atmospheric conditions

15

Was the training successful?

Groups demonstrated that they

could retrieve surface

reflectance within the

uncertainties of the approach

Results raised further questions

Variability of the stop and measure approach was larger than

continuous sampling

Cause is unclear whether it was surface, instrumental, or

atmospheric

Points to the need for a better sample for Type A uncertainty

assessment

Difference between the continuous results and stop and measure

results indicate the need for better site sampling techniques

16

2018 Joint campaign sought to answer questions from 2017 training

Refine collection approaches and develop a set of guidelines for

future intercomparisons and surface reflectance validation Four basic collections

Reflectance-based calibration of Terra platform sensors

80-m playa transect

60 cm x 60 cm gray tarp sample

60 cm x 60 cm barium sulfate painted panel

Sky conditions were not the most cooperative

Intercomparison results still being evaluated

17

2018 NASA GSFC results

Ratio of retrieved reflectance from NASA GSFC data sets relative to

clear-sky, optimal collection for that target

0.88

0.9

0.92

0.94

0.96

0.98

1

1.02

1.04

1.06

1.08

350 850 1350 1850 2350

Rat

io t

o M

ay 3

Cle

ar S

ky

Wavelength (nm)

0.9

0.95

1

1.05

1.1

350 650 950 1250 1550 1850 2150 2450

Rat

io t

o M

ay 3

Cle

ar S

ky C

ase

Wavelength (nm)

Barium sulfate 48% tarp

GSFC results indicate reflectance can be retrieved with better than 5% repeatability regardless of sky conditions

Key is representative reference sample

Result of near-lambertian samples

Will impact experiment designs for surface reflectance validation campaigns

0.98

0.99

1

1.01

1.02

1.03

1.04

1.05

1.06

1.07

1.08

1.09

1.1

350 650 950 1250 1550 1850 2150 2450

Rat

io t

o M

ay 3

Cle

ar S

ky C

ase

Wavelength (nm)

80 m by 80 m area

18

Added measurement from 2017 - Gray tarp

Limiting reflectance collection to a small sample area leads to

teams measuring the same area

Tarp sample measurements will isolate differences

caused by the instruments or the reference

Effects from from changing solar irradiance are

limited because of short time between

references and the gray sample

Short time between groups limits the BRDF

effects of the sample

Tarp is being evaluated as a traveling standard

that would allow intercomparisons without the

need for groups to gather in a single location

One round of collections had all groups using the

same reference standard

19

Added measurement from 2017 - Barium sulfate panel

Barium sulfate has a reflectance similar to those of the

reference standard

Using BaSO4 panel reduces instrument effects such as those caused by

non-linearity, lower SNR at lower reflectance, etc.

Both BaSO4 and gray tarp retrievals can be compared to laboratory-

based predictions of reflectance allowing a check on the absolute

uncertainty

20

What we hope to learn from 2017 and 2018

The ensemble of results should help indicate what the limits

could be to absolute and relative uncertainties

Best agreement should be for retrieval of reflectance of BaSO4 panel

Highest SNR

Uniform sample

Rapid data collection

Minimal BRDF effects

Gray sample should give next best results since it is still of a very limited

spatial area

Lower reflectance could point out SNR or linearity effects

Can still be some BRDF and spatial heterogeneity effects

80-m transect gives a realistic measurement while limiting area

Full site measurements evaluated against calibration of an imaging

sensor viewing all of the characterized areas

21

Summary

Training of students during the practicum shows that following

protocols leads to repeatability similar to experienced groups

Previous work shows absolute uncertainties are dominated by

reflectance standard uncertainty as show

Still unclear what dominates the lack of repeatability

Methods to isolate Type A uncertainties need development

Small, stable samples

Stable mounting of spectrometer relative to small sample

Stop and measure approach should limit uncertainties from variability

in spectrometer and atmosphere

Need to implement more rigorous evaluations of temporal and spatial

sampling approaches

22

Acknowledgements Thank you to IEEE Geoscience & Remote Sensing Society and

the Geoscience Spaceborne Imaging Spectroscopy (GSIS)

Technical Committee

Thank you to the University of Arizona for access to their

laboratory and test site

http://www.grss-ieee.org/hyperspectral-calval-workshop-2/

http://www.grss-ieee.org/hyperspectral-calval-field-practicum/