analyzing precision ag data : intermediate workshop on what is needed to move precision agriculture...

Analyzing Precision Ag DataAnalyzing Precision Ag Data: :

Intermediate workshop on what is needed to move Intermediate workshop on what is needed to move Precision Agriculture beyond mappingPrecision Agriculture beyond mapping

Joseph K. BerryJoseph K. BerryW. M. Keck Visiting Scholar in Geosciences, Geography, University of DenverW. M. Keck Visiting Scholar in Geosciences, Geography, University of Denver

Principal, Berry & Associates // Spatial Information SystemsPrincipal, Berry & Associates // Spatial Information Systems

Email Email [email protected]@innovativegis.com — Web — Web www.innovativegis.comwww.innovativegis.com/basis//basis/

Geographic Information and Spatial Technologies WorkshopGeographic Information and Spatial Technologies WorkshopAg Canada — October, 2006 — Winnipeg, Manitoba, CanadaAg Canada — October, 2006 — Winnipeg, Manitoba, Canada

mailto:[email protected]

http://www.innovativegis.com/basis/

http://www.innovativegis.com/basis/

Map Data Visualization and SummaryMap Data Visualization and Summary

((BerryBerry))

Workshop TopicsWorkshop Topics

Overview Overview (keynote)(keynote)

Mapped Data Visualization and SummaryMapped Data Visualization and Summary

Comparing Mapped DataComparing Mapped Data

Spatial InterpolationSpatial Interpolation

Characterizing Data GroupsCharacterizing Data Groups

Developing Predictive ModelsDeveloping Predictive Models

Generating Prescription MapsGenerating Prescription Maps

Overview …keynote presentationOverview …keynote presentation

Map Data Visualization and SummaryMap Data Visualization and Summary

((BerryBerry))









MAP Analysis FrameworkMAP Analysis Framework

(Berry)

Continuous, regular grid cells (objects)Continuous, regular grid cells (objects)

::--, --, --, --,--, --, --, --,--, --, --, --,--, --, --, --,--, --, --, --,--, --, --, --,--, --, 19441944, --,, --,--, --, --, --,--, --, --, --,::

GridGridTableTable

Click on…Click on…

Zoom Pan RotateZoom Pan Rotate DisplayDisplay

ShadingShadingManagerManager

Now for someNow for someMap Analysis…Map Analysis…

GridGridAnalysisAnalysis

……calculate a calculate a slope map and slope map and drape on the drape on the elevation surfaceelevation surface

Grid-based Map Data VisualizationGrid-based Map Data Visualization

((BerryBerry))

Thematic MappingThematic Mapping

Display and Data TypesDisplay and Data Types

Lattice versus GridLattice versus Grid

2D Map versus 3D Surface2D Map versus 3D Surface

Discrete (Qualitative) versus Discrete (Qualitative) versus Continuous (Quantitative)Continuous (Quantitative)

Investigating and Normalizing Mapped DataInvestigating and Normalizing Mapped Data

(Berry)(Berry)

GridMath equationGridMath equationNorm_GOAL = (mapValue / 250) * 100Norm_GOAL = (mapValue / 250) * 100

Data Vales (matrix)Data Vales (matrix)

Drill downDrill down

Comparing Map DataComparing Map Data

((BerryBerry))









Visually Comparing Mapped DataVisually Comparing Mapped Data

((BerryBerry))

What differences do you see?What differences do you see?

““How different are the maps?” “How are they different?” “Where are they different?”How different are the maps?” “How are they different?” “Where are they different?”

MUST have a common

legend

Class 5Class 4Class 3Class 2Class 1

DISCRETE integer

values 1-5

CONTINUOUS ratio values 2.33 – 295.00

240-300 Class 5180-240 Class 4120-180 Class 3

60-120 Class 20-60 Class 1

…you don’t “see” the data values– just the COLORS

Quantitative ComparisonQuantitative Comparison

Align grid maps and count Align grid maps and count

240-300 Class 5180-240 Class 4120-180 Class 3

60-120 Class 20-60 Class 1

Comparing Discrete Maps Comparing Discrete Maps (Joint coincidence)(Joint coincidence)

((BerryBerry))

……a a Coincidence TableCoincidence Table reports the number of cells reports the number of cells

for each joint condition with for each joint condition with diagonal cells identifying diagonal cells identifying agreement (no change)agreement (no change)

40+144+1648+18+0= 1850/3289= 40+144+1648+18+0= 1850/3289= 56.25 56.25 overall agreementoverall agreement

Comparing Continuous SurfacesComparing Continuous Surfaces (Difference map)(Difference map)

1997_Yield_Volume1997_Yield_Volume- 1998_Yield_Volume- 1998_Yield_Volume

Yield_DiffYield_Diff

Map VariablesMap Variables… map values within an analysis grid … map values within an analysis grid can be can be mathematically and statistically analyzed mathematically and statistically analyzed

((BerryBerry))

……greengreen indicates indicates areas of increased areas of increased productionproduction

……yellowyellow indicates indicates minimal changeminimal change

……redred indicates indicates decreased decreased productionproduction


((BerryBerry))









The geo-registered soil The geo-registered soil samples form a pattern of samples form a pattern of “spikes” throughout the field. “spikes” throughout the field.

Spatial InterpolationSpatial Interpolation is is similar to throwing a blanket similar to throwing a blanket over the spikes that conforms over the spikes that conforms to the pattern.to the pattern.

Spatial InterpolationSpatial Interpolation (Mapping spatial variability)(Mapping spatial variability)

((BerryBerry))

All interpolation algorithms assume that— All interpolation algorithms assume that—

1) “1) “nearby things are more alike than nearby things are more alike than distant thingsdistant things” (spatial autocorrelation), ” (spatial autocorrelation), 2) appropriate 2) appropriate sampling intensitysampling intensity, and , and 3) suitable 3) suitable sampling patternsampling pattern..

……the continuous surfaces form a “map” the continuous surfaces form a “map” of the spatial variation in the data samples.of the spatial variation in the data samples.

Spatial InterpolationSpatial Interpolation (Average vs. IDW)(Average vs. IDW)

Comparison of the interpolated surface to the whole field average Comparison of the interpolated surface to the whole field average shows shows large differenceslarge differences in localized estimates in localized estimates

Difference Map

((BerryBerry))

Spatial InterpolationSpatial Interpolation (Compare maps)(Compare maps)

Comparison of the IDW and Krig interpolated surfaces shows Comparison of the IDW and Krig interpolated surfaces shows small small differencesdifferences in in localized estimates in in localized estimates

Difference Map

((BerryBerry))

Spatial Interpolation TechniquesSpatial Interpolation Techniques

((BerryBerry))

Characterizes the spatial distribution by fitting a mathematical equation Characterizes the spatial distribution by fitting a mathematical equation to localized portions of the data (roving window)to localized portions of the data (roving window)

AVG= 23 everywhere

Spatial Interpolation techniques use “roving windows” to summarize sample values within a specified reach of each map location. Window shape/size and summary technique result in different interpolation surfaces for a given set of field data

…no single techniques is best for all data.

Inverse Distance Weighted (IDW) technique weights the samples such that values farther away contribute less to the average

…1/Distance Power

Assessing Spatial Interpolation ResultsAssessing Spatial Interpolation ResultsResidual AnalysisResidual Analysis

……the best map is the the best map is the one that has the bestone that has the best““guesses”guesses”

((BerryBerry))

A Map of Error A Map of Error (Residual Map)(Residual Map)

……shows you where your estimates are likely good/badshows you where your estimates are likely good/bad

((BerryBerry))

Spatial Interpolation TechniquesSpatial Interpolation Techniques

((BerryBerry))

A A variogramvariogram depicts the relationship between the distance between depicts the relationship between the distance between sample points and the difference between the measurement valuessample points and the difference between the measurement values


1) “1) “nearby things are more alike than distant thingsnearby things are more alike than distant things” (spatial autocorrelation), ” (spatial autocorrelation), 2) appropriate sampling intensity, and 2) appropriate sampling intensity, and 3) suitable sampling pattern.3) suitable sampling pattern.

Basic Point Sampling Design ConcernsBasic Point Sampling Design Concerns

StratificationStratification—— appropriate appropriate groupingsgroupings for sampling for samplingSample SizeSample Size—— appropriate appropriate sampling intensitysampling intensity for each stratified group (N) for each stratified group (N)

Sampling GridSampling Grid—— appropriate appropriate analysis gridanalysis grid for locating point samples for locating point samples

(Berry)(Berry)


1) “nearby things are more alike than distant things” (spatial autocorrelation),1) “nearby things are more alike than distant things” (spatial autocorrelation), 2) appropriate 2) appropriate sampling intensitysampling intensity, and , and 3) suitable sampling pattern.3) suitable sampling pattern.

Point Sampling Design ConcernsPoint Sampling Design Concerns (Sampling Pattern)(Sampling Pattern)

Sampling PatternSampling Pattern— — appropriate arrangementappropriate arrangement of samples considering of samples considering both spatial interpolation and statistical inferenceboth spatial interpolation and statistical inference

(Berry)(Berry)


1) “nearby things are more alike than distant things” (spatial autocorrelation),1) “nearby things are more alike than distant things” (spatial autocorrelation), 2) appropriate sampling intensity, 2) appropriate sampling intensity, and and 3) suitable 3) suitable sampling patternsampling pattern..


((BerryBerry))









Visualizing Spatial RelationshipsVisualizing Spatial Relationships

(Berry)(Berry)

What spatial What spatial relationships do you relationships do you see?see?

Interpolated Spatial DistributionInterpolated Spatial Distribution

Phosphorous (P)

……do relatively high levels do relatively high levels of P often occur with high of P often occur with high levels of K and N?levels of K and N?

……how often?how often?

……where?where?

Identifying Unusual AreasIdentifying Unusual Areas

……locations that are more than one standard deviation above the mean locations that are more than one standard deviation above the mean are identified as unusually high are identified as unusually high

(Berry)(Berry)

Calculating Data DistanceCalculating Data Distance……an n-dimensional plot depicts the multivariate distribution; the distance an n-dimensional plot depicts the multivariate distribution; the distance

between points determines the relative similarity in data patterns between points determines the relative similarity in data patterns

(Berry)(Berry)……the closest floating ball is the least similar (largest data distance) from the comparison pointthe closest floating ball is the least similar (largest data distance) from the comparison point

Identifying Map SimilarityIdentifying Map Similarity

(Berry)(Berry)

The green tones indicate field locations with fairly similar P, K and N levels; red tones indicate dissimilar The green tones indicate field locations with fairly similar P, K and N levels; red tones indicate dissimilar areas. areas.

……the relative data distance between the comparison point’s data pattern the relative data distance between the comparison point’s data pattern and those of all other map locations form a and those of all other map locations form a Similarity IndexSimilarity Index

Clustering Maps for Data ZonesClustering Maps for Data Zones

(Berry)(Berry)

(Cyber-Farmer, Circa 1992)(Cyber-Farmer, Circa 1992)

Variable Rate ApplicationVariable Rate Application……fertilization rates vary for the different fertilization rates vary for the different clusters “clusters “on-the-flyon-the-fly””

……groups of “floating balls” in data space groups of “floating balls” in data space identify locations in the field with similar data identify locations in the field with similar data patterns– patterns– data zonesdata zones

…a map stack is a spatially organized set of numbers

Evaluating Clustering ResultsEvaluating Clustering Results

(Berry)(Berry)

……if the boxes do not overlap (much), the data clusters are distinctif the boxes do not overlap (much), the data clusters are distinct


((BerryBerry))









NIR NIR (R)(R)

Red Red (G)(G)

Green Green (B)(B)(Beyond our sight)(Beyond our sight) Color Infrared Color Infrared

PP

KK

phph

etc.etc.

RS Imagery as GIS Data LayersRS Imagery as GIS Data Layers

Remote sensing Remote sensing images are composed images are composed of numbersof numbers, just like , just like any other map in a any other map in a grid-based GIS…grid-based GIS…

““Map-ematical Map-ematical Processing”Processing”

1481485252

2626

4444

4343

257257

7.27.2

4646

3434

5757

312312

7.57.5

A RS image is just a “A RS image is just a “shishkebab shishkebab of numbersof numbers” like any other ” like any other grid map (raster)grid map (raster)

ImageImage

((BerryBerry))

Creating Prediction Models Creating Prediction Models (Scatter Plot)(Scatter Plot)

……aa Scatter Plot Scatter Plot identifies the “joint condition” at each map identifies the “joint condition” at each map location; the trend in the plot forms a prediction equationlocation; the trend in the plot forms a prediction equation

(Berry)(Berry)

Map SetMap Set New Graph New Graph Scatter Plot Scatter Plot

Deriving a Predictive Index Deriving a Predictive Index (NDVI)(NDVI)

……an index combining the Red and NIR maps can be used to generate an index combining the Red and NIR maps can be used to generate a better predictive model a better predictive model

Normalized Difference Vegetation IndexNormalized Difference Vegetation Index NDVI= ((NIR – Red) / (NIR + Red))NDVI= ((NIR – Red) / (NIR + Red))

for the sample grid locationfor the sample grid location NDVI= ((121-14.7) / (121 + 14.7))= 106.3 / 135.7= .783NDVI= ((121-14.7) / (121 + 14.7))= 106.3 / 135.7= .783

(Berry)(Berry)

Evaluating Prediction Maps Evaluating Prediction Maps (Spatial error analysis)(Spatial error analysis)

……the regression equation is evaluated and the predicted map is compared the regression equation is evaluated and the predicted map is compared to the actual measurements to generate an error mapto the actual measurements to generate an error map

Error = Predicted - ActualError = Predicted - Actual

for the sample grid locationfor the sample grid location YYestest = 55 + (180 * .783) = 196 …error is 196 – 218 = 22 bu/ac = 55 + (180 * .783) = 196 …error is 196 – 218 = 22 bu/ac

Note that the average error is 2.62 and 67% of the predictions are within +/- 20 bu/acNote that the average error is 2.62 and 67% of the predictions are within +/- 20 bu/acAlso, most of the error is concentrated along the edge of the fieldAlso, most of the error is concentrated along the edge of the field

(Berry)(Berry)

Error = Predicted - Actual

Stratifying Maps for Better PredictionsStratifying Maps for Better Predictions

(Berry)(Berry)

Stratifying by Error ZonesStratifying by Error Zones

Other ways to stratify mapped data—Other ways to stratify mapped data—1) 1) Geographic ZonesGeographic Zones, such as proximity to the field , such as proximity to the field edge; 2) edge; 2) Dependent Map ZonesDependent Map Zones, such as areas of low, , such as areas of low, medium and high yield; 3) medium and high yield; 3) Data ZonesData Zones, such as areas , such as areas of similar soil nutrient levels; and 4) of similar soil nutrient levels; and 4) Correlated Map Correlated Map ZonesZones, such as micro terrain features identifying , such as micro terrain features identifying small ridges and depressionssmall ridges and depressions. .

The The Error ZonesError Zones map is used map is used as a template to identify the as a template to identify the NDVI and Yield values used to NDVI and Yield values used to calculate three separate calculate three separate prediction equations. prediction equations. A A Composite PredictionComposite Prediction map is map is created by applying the created by applying the equations to the NDVI data equations to the NDVI data respecting the template map respecting the template map zones.zones.

Assessing Prediction ResultsAssessing Prediction Results

(Berry)(Berry)

Whole FieldPrediction

StratifiedPrediction

ActualYield

none

none

Error Map for Stratified Prediction

80%

Error Map

The Precision Ag Process The Precision Ag Process (Fertility example)(Fertility example)

((BerryBerry))

As a combine moves through a field As a combine moves through a field 1)1) it uses GPS to it uses GPS to check its location then check its location then 2)2) checks the yield at that checks the yield at that

location to location to 3)3) create a continuous map of the yield create a continuous map of the yield variation every few feet (variation every few feet (dependent map variabledependent map variable). ).

On-the-Fly On-the-Fly Yield MapYield Map

Steps 1)–3)Steps 1)–3)

Variable Rate ApplicationVariable Rate Application

Step 5)Step 5)

5)5) …that is used to adjust fertilization levels every …that is used to adjust fertilization levels every few feet in the field (few feet in the field (actionaction).).

Intelligent ImplementsIntelligent Implements

Derived Derived Nutrient MapsNutrient Maps

Step 4)Step 4)

Prescription MapPrescription Map

Zone 3

Zone 2

Zone 1

The yield map The yield map 4)4) is analyzed in combination with is analyzed in combination with soil, terrain and other maps (soil, terrain and other maps (independent map independent map variablesvariables) to derive a “Prescription Map” …) to derive a “Prescription Map” …

Analyzing Spatial ContextAnalyzing Spatial Context

((BerryBerry))









Generating Prescription Maps Generating Prescription Maps (Step & Equations)(Step & Equations)

(after Elaine McCallum, Red hen Farming)(after Elaine McCallum, Red hen Farming)

RENUMBER 1996_Fall_P ASSIGNING 50 TO 0 THRU 4 ASSIGNING 30 TO 4 THRU RENUMBER 1996_Fall_P ASSIGNING 50 TO 0 THRU 4 ASSIGNING 30 TO 4 THRU 8 ASSIGNING 15 TO 8 THRU 12 ASSIGNING 0 TO 12 THRU 500 FOR P_application8 ASSIGNING 15 TO 8 THRU 12 ASSIGNING 0 TO 12 THRU 500 FOR P_application

Prescription maps of P and K Prescription maps of P and K are based on decision rulesare based on decision rules

Prescription MappingPrescription MappingIf <condition> then <action>If <condition> then <action>

……this is where science and this is where science and technology meet technology meet

Prescription map for N is Prescription map for N is based on equationbased on equation

Economic MapsEconomic Maps

(after Elaine McCallum, Red hen Farming)(after Elaine McCallum, Red hen Farming)

A total costs map is derived by summing the variable and fixed cost maps.A total costs map is derived by summing the variable and fixed cost maps.

Calculate ( (P_lbs_per_cell * .697) / (1997_Yield_Volume * 2.75) ) * 100 For Revenue_%P Calculate ( (P_lbs_per_cell * .697) / (1997_Yield_Volume * 2.75) ) * 100 For Revenue_%P

On-Farm Testing On-Farm Testing (site-specific studies/research)(site-specific studies/research)

((BerryBerry))

On-farm studies, such as seed hybrid performance, can On-farm studies, such as seed hybrid performance, can be conducted using actual farm conditions.be conducted using actual farm conditions.

Is GIS Technology Ahead of Science?Is GIS Technology Ahead of Science?

((BerryBerry))

1)1) Is the “ Is the “scientific methodscientific method” relevant in the data-rich age ” relevant in the data-rich age of knowledge engineering? of knowledge engineering?

2)2) Is the “ Is the “random thingrandom thing” pertinent in deriving mapped data? ” pertinent in deriving mapped data?

3)3) Are Are geographic distributionsgeographic distributions a natural extension of numerical distributions? a natural extension of numerical distributions?

4)4) Can spatial dependencies be Can spatial dependencies be modeledmodeled??

5)5) How can “site-specific” analysis contribute to the How can “site-specific” analysis contribute to the scientific body of knowledgescientific body of knowledge? ?

……the bottom line is that modern maps are the bottom line is that modern maps are numbers firstnumbers first, , pictures laterpictures later

Five critical questions underlying Precision Agriculture…Five critical questions underlying Precision Agriculture…

Where To Go From Here…Where To Go From Here…

Who’s Minding the FarmWho’s Minding the Farm, GeoWorld, Adams Business Media, Chicago, Illinois, Feb 1998, 11:2 , GeoWorld, Adams Business Media, Chicago, Illinois, Feb 1998, 11:2 46-51. J.K. Berry. 46-51. J.K. Berry. http://www.geoplace.com/gw/1998/0298/GW980200Feature1.asphttp://www.geoplace.com/gw/1998/0298/GW980200Feature1.asp

Applying Spatial Analysis for Precision Conservation Across the LandscapeApplying Spatial Analysis for Precision Conservation Across the Landscape , , J. of Soil and Water Conservation, Nov/Dec 2005, 60:6 22-29. J.K. Berry, J. A. Delgado, R. Khosla and F.J. J. of Soil and Water Conservation, Nov/Dec 2005, 60:6 22-29. J.K. Berry, J. A. Delgado, R. Khosla and F.J. Pierce. Pierce. http://www.swcs.org/documents/DelgadoBerry_Precision_Conservation_040406095108.pdfhttp://www.swcs.org/documents/DelgadoBerry_Precision_Conservation_040406095108.pdf

Precision Conservation for Environmental SustainabilityPrecision Conservation for Environmental Sustainability , J. of Soil and Water , J. of Soil and Water Conservation, Nov/Dec 2003, 58:6 332-339. J.K. Berry, J. A. Delgado, R. Khosla and F.J. Pierce. Conservation, Nov/Dec 2003, 58:6 332-339. J.K. Berry, J. A. Delgado, R. Khosla and F.J. Pierce. http://www.swcs.org/documents/Precision_Conservation_111605114832.pdfhttp://www.swcs.org/documents/Precision_Conservation_111605114832.pdf

Analyzing Precision Ag DataAnalyzing Precision Ag Data: : bookbook published by BASIS Press, published by BASIS Press, Fort Collins, Colorado, 2003, 84 pages, 49 illustrations with exercises and Fort Collins, Colorado, 2003, 84 pages, 49 illustrations with exercises and companion software. J.K. Berry. companion software. J.K. Berry. http://www.innovativegis.com/basis/Books/AnalyzingPAdata/http://www.innovativegis.com/basis/Books/AnalyzingPAdata/

____________________________________________________________________Companion Software Companion Software

Analyzing Precision Ag DataAnalyzing Precision Ag Data: workshop this afternoon : workshop this afternoon 1:30-4:30pm and repeated tomorrow morning 8:45-11:45am1:30-4:30pm and repeated tomorrow morning 8:45-11:45am

____________________________________________________________________

analyzing precision ag data : intermediate workshop on what is needed to move precision agriculture...

Documents

canadamap data visualization

precision ag data

idwberryspatial interpolation

suitable sampling pattern

appropriate sampling

nearby things

krig interpolated surfaces

high levels of p