fsa user sensitivity study for quality of naip imagery · fsa user sensitivity study for quality of...
TRANSCRIPT
![Page 1: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/1.jpg)
FSA User Sensitivity Study for Quality of NAIP Imagery
(GS-23F-0284M ORDER #AG-8447-D-06-0026)
December 7, 2006Sharon Lunt (585)269-5089Tracey BjickTom HeatonSeth Weith-Glushko
![Page 2: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/2.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 2
Agenda
– Background and goals of Study
– FSA User Study methodology
– Observations and preliminary conclusions
![Page 3: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/3.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 3
2006 USDA AFPO Product Quality Support
Phase 1 – User Sensitivity Investigation– Defining FSA quality needs
Phase 2 – Processing Chain “Best Practices”– Applying FSA needs to vendor “Best Practices” recommendations
Final Report Due February 2, 2007
![Page 4: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/4.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 4
User Study Motivation
NAIP imagery can be of very high quality, however multiple vendors and image processing procedures can sometimes provide inconsistent imagery to APFO
![Page 5: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/5.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 5
User Sensitivity Investigation Goals
Determine the sensitivity of FSA end users at county and state offices to variations in quality
Determine type of improvements that might be necessary in upstream processing to provide quality consistent with end user needs
![Page 6: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/6.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 6
User Sensitivity Investigation Plan
Determine common
quality issues• Analyze previous NAIP imagery
• Categorize and prioritize quality issues
Create Image Series
• Identify representative images
• Create a series for each quality issue
• Scaled levels
Perform image evaluation
• Feedback from FSA personnel
• Performed at County and State Offices
• Document viewing environment and workstation calibration
Data Analysis and Report
•Analyze field office results
•Summarize findings
•Define input recommendations for Processing Chain “Best Practices”
![Page 7: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/7.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 7
FSA Field Office Image Evaluation
Goals– To obtain FSA personnel input– To understand general and area
specific usage/preference of imagery – Observe and quantify image viewing
environment and tools
Areas of concentration– Color balance– Dynamic range – Tone scale – Sharpness– Misregistration of color records– Noise– Clipping
FSA Field Sites included
Monroe County, NYBox Elder County, UTCache County, UTUtah FSA State OfficePottawatomie County, KSClay County, KSKansas FSA State OfficeWaller County, TXBurleson County, TXTexas FSA State Office
![Page 8: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/8.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 8
User Study Methodology
Test takers were FSA state and county employees that used NAIP imagery for their jobsAll users saw an Oklahoma common image (2 Meter) selected by APFO as being representative of good imageryAll users also saw an image from their home state selected to be close to the common image in contrast and colorBoth qualitative (subjective impression) and quantitative (numeric) information was obtainedInformation regarding user’s current image viewing setup was also collected
![Page 9: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/9.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 9
FSA User Evaluated Images
Box Elder Co., UT Clay Co., Kansas
RegionalHicks Co., TX Mikado Co., MIGrady Co., OK
Control
![Page 10: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/10.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 10
Mosaic Non-Uniformities
Users ratedImage qualityType of non-uniformity Levels of usability in different areas Frequency of non-uniformities in delivered imagery
RESULTS - PreliminaryContrast variation
– Users split whether the low contrast dark, or low contrast light region was less usable
– Users see images like it frequently in some areas(~25% of the time)
Exposure variation– Most users found the overexposed (bright)
quadrant the most difficult to locate boundaries
Lower contrast and
darker
Lower contrast and lighter
+1 stop
exposure
-1 stop
exposure
![Page 11: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/11.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 11
Mosaic Non-Uniformities15%
noise
25%
noise
RESULTS - Preliminary
Noise variation– Users tended to rate the
complete image not usable
Color variation– Users tended to rate the whole
image as usable+10%
Green
-10%
Green
![Page 12: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/12.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 12
Sharpness Threshold Task User Threshold
ReferenceUsers ranked a series of variations in sharpness
RESULTS -Preliminary
Low tolerance to poor sharpness
Impacts CLU definition
Level of worst sample problem image
![Page 13: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/13.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 13
Noise Threshold Task
Level of worst sample problem image
Reference
User Threshold
Users ranked a series of variations in noise
RESULTS -PreliminaryTolerant to noise until level obscures CLU definition
High noise fatigue factor identified by users
![Page 14: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/14.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 14
Misregistration Threshold Task
Users ranked a series of variations in misregistration
RESULTS -PreliminaryA two pixel shift was found to make an image too difficult to use without strain
![Page 15: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/15.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 15
Saturation Task Not Preferred
Users ranked a series of variations in saturation from 1-5 (Best to Worst)
RESULTS -Preliminary
• Users tended to pick the center as preferred.
Very high saturation and very low not preferred
Preferred
![Page 16: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/16.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 16
Color Balance Variations
Paired Comparison of images – Test taker picked preferred image from a pair
Color variations were 6% and 12% changes in cyan, green, blue, and yellow
RESULTS – PreliminaryNeutrally balanced image preferred overallUsers not tolerant to large color shifts
– Least tolerant to large green shifts– Most tolerant to blue/yellow variations
![Page 17: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/17.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 17
Color Balance Variations
Cyan Blue
YellowGreen
![Page 18: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/18.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 18
Quality Rating of Images on a 5-point Subjective Usability Scale
Users were presented with images that varied in apparent contrast due to the introduction of clipping or manipulation of the histogram.
– Images presented one at a time
Users were asked to rate the image based on a on a 5-point Subjective Usability Scale
![Page 19: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/19.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 19
Clipping Variations – High-End (White) Clipping
Original 1% Clipped 3% Clipped
![Page 20: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/20.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 20
Clipping Variations – Low-End (Black) Clipping
Original 2% Clipped 3% Clipped
![Page 21: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/21.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 21
Clipping Variations – Equal-End Clipping
Original 1% Clipped Each Side 2% Clipped Each Side
![Page 22: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/22.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 22
Tone Scale Variations – Histogram Stretch and Compress(Sigmoidal Lookup Table)
b = 2 b = 5
b = 6b = 3
Originalb = 4 b = 7
![Page 23: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/23.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 23
Rating Scale Definitions
1 – Not UsableThis image would be difficult to get information from to do the type of job I do2- Somewhat UsableThis image would provide some information with some effort, but is not what I’d
prefer using.3 – UsableThis image would provide the information for me to do the type of job I do, but
could be improved4- Very UsableIt is easy to find information in this image to do the type of job I do5- Best ImageThis image is optimum for finding information for the type of job I do. I’d like all
images to look like this.
![Page 24: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/24.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 24
Clipping and Contrast Task
Quality Rating of Images on a 5-point Subjective Usability Scale RESULTS - Preliminary
Contrast Variations: – Users selected higher contrast images in preference to the
reference– Users rated low contrast images significantly lower than the
reference and the clipped imagesClipping Variations:
– A small amount of clipping (~1%) was acceptable– Greater than 1% rated consistently lower than reference image– Clipping of dark (shadows) considered worse than light
(highlights)
![Page 25: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/25.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 25
Results (Preliminary) – Top two rated images –both higher contrast images Sigma 5
Sigma 6
Reference
![Page 26: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/26.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 26
Results (Preliminary) – Bottom two rated images3% Clip of dark values
Reference
Low Contrast
(Sigma 2)
![Page 27: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/27.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 27
Observations on Current Viewing SetupsLarge variation in lighting levels and set-up
Monitors set at factory defaults which are not optimum for viewing imagery
– 9300K Color Temperature
– Lower resolution and/or 60 Hz refresh rate
– No calibration software
Monitor responses are not consistent and so the users may see the imagery differently than APFO QA or Vendor QA (or each other)
User Monitor 1Calibrated User Monitor 2
![Page 28: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/28.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 28
Summary (Preliminary Results)– Mosaic: Users very sensitive to non-uniformities
– Sharpness: Strong preference for well-defined boundaries
– Noise: Minimal noise preferred (impacts boundary delineation)
– Misregistration: Less than 2-pixels required to keep images easily usable
– Color balance: Neutral with allowance for small color shifts
– Saturation: Users preferred more natural looking images overall
– Clipping: Only small amount allowable
– Contrast: Dynamic range maximum with minimal clipping
– User Viewing Environment: Monitor optimization aided ability to more easily extract information
– Many positive comments to set-up suggestions and changes
![Page 29: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/29.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 29
Overall Conclusions
FSA users require well defined image detail to delineate and verify CLUs
FSA users want good color balance and definition to aid in determining crop type
Improvements in upstream processing are warranted– Addressed in Phase 2– Processing chain “Best Practices”
– Applying FSA needs to vendor “Best Practices”recommendations
![Page 30: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/30.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 30
Phase 2 – Development of Best Practice Recommendations
Definition of protocols, procedures and metrics, for controlling:
– Clipping
– Contrast and Dynamic Range
– Saturation
– Color balance
– Noise
Collection and processing parameters around exposure, film scanning, and color record registration
![Page 31: FSA User Sensitivity Study for Quality of NAIP Imagery · FSA User Sensitivity Study for Quality of NAIP Imagery (GS-23F-0284M ORDER #AG-8447-D-06-0026) December 7, 2006 Sharon Lunt](https://reader033.vdocument.in/reader033/viewer/2022052719/5f0704497e708231d41ae180/html5/thumbnails/31.jpg)
FSA User Sensitivity Study for Image Quality of NAIP Imagery Dec 2006 31
Variation in NAIP imagery and demonstration of potential “Best Practice” improvementsProblematic NAIP imagery
samples
Original images delivered to APFO
Reprocessed to bring out information content Original ITT corrected
Original ITT corrected