Yield Cleaning Software and Techniques

OFPE Meeting

10-27-2015

Yield data cleaning conventions

• No standardized, accepted approach, but:

• Commonalities exist among investigated techniques

• Between 10-50% of data typically removed before analysis

• Visual interpretation of applied filters near end of processing is common

• Commercial software automatically applies some filters, but does not allow for

manipulation of all important parameters

Yield Editor Software

• Created by Ken Sudduth and Scott Drummond, USDA-ARS

• Version 1, 2007; Version 2, 2012

• Allows for manual and automatic manipulation of important

parameters not accessible in commercial software

long lat flowgps time

log interval dist width moist. header

status pass #

• Flow Delay

• already incorporated by some extent into the data (typically)

• use this to slightly adjust further and visually inspect

• results in some of the data deleted from end of transects

• Moisture Delay

• not usually same amount of time as flow delay

• Start and End Pass Delay

• trims values off start/end of transect

• used because as combine enters/exits the crop, values will be unreliable for

some distance

• may interact with flow delay (e.g., a flow and start delay of 5 will each

remove the first 5 points of a transect)

• Max Velocity

• sometimes useful to remove partial swath transects that were harvest at a

faster speed but not marked in the display by the operator

• Min Velocity

• one of the most critical to yield cleaning

• as combine slows, but intake remains high for a few seconds, yield values can

become unreasonably high

• Smooth Velocity

• filters out areas of rapid velocity changes

• represents an allowable ratio of velocity change from one point to the next

• default ratio of 0.2 mean that points not within 20% of the speed of

neighbors will be removed

• Minimum Swath

• only useful to us if the operator has recorded partial swaths while harvesting

• have not seen this in the data we have

• Min/Max Yield

• Standard Deviation

• used to remove values that are more/less than a specified number of STDs

from the field-wide mean

• Header Down Requirement

• removes values where the header was up, but still recording

• often this is filtered out by default within commercial software

• have not seen this in the data we have

• Position

• removes any ‘fliers’ which are out of the area of interest

• Adjust for Moisture

• doesn’t remove data

• Moisture Adjustments

• allows for modification of yield values based on moisture status of the grain

• it is a complicated decision that involves several considerations:

“First, if we are going to adjust grain yield back to a market moisture level, will

we use the moisture sensor data to make this correction? Sometimes this is not

a good idea, as the moisture sensor can have significant problems that may

cause very large (and incorrect) adjustments to yield. Even if we do use the

sensor, we will need a manual value to use in problem areas (the end of

transects, etc.). Another approach is to use a constant manual value to make

the correction across the whole field (for example, the moisture value reported

at the point of sale). This approach is less prone to major sensor based errors,

but may miss real variations that exist within a field. A final option that we

need to consider is whether we want to expand the grain yields where the

moisture falls below the market moisture level up to the yields they would

have been at the market moisture level.”

Automatic Yield Cleaning Editor (AYCE)

AYCE – auto min/max filters• AYCE picks ranges for position, velocity, and yield

• determined by analysis of histograms

Sudduth et. al., 2012; Yield Editor 2.0: Software for Automated Removal of Yield Map Errors

AYCE – delay computations

• Based on Phase Correlation Identification method (Lee et al. 2012)

• Determines the spatial consistency of flow times at each interval

• Repeats 10 times – blue line is the ‘normalized’ average

AYCE – overlap filter

• Based on bitmap method developed by Han et al. 1997• Removes observations where the combine travelled over previously harvested

areas• Extremely effective for fields harvested with one combine• Not suitable for fields harvested with multiple combines unless able to sort all

records among all combines chronologically• Limited by poor GPS accuracy

AYCE – Localized STD filter (LSD)

• Effective at catching fliers missed by yield and velocity filters, as well as reducing ramping at border between end rows and interior field transects• Searches an area equal to user-specified neighborhood (# of header widths) and

filtering by a user-specified number of STD

AYCE – Performance compared to manual filtering (Sudduth et al. 2012)

• Yield cleaning methods compared on 50 yield maps

• high variation of field configurations and harvest patterns: multiple combines, operators,

yield monitors, variability in harvesting conditions, and a range of required delay predictions

• Comparisons by ‘expert’ yield cleaners and automatic methods (using default

settings) produced comparable results

• Of yield observations retained by the two methods, 95% were in common

AYCE – Performance compared to manual filtering• Regressions on values (field mean yield, STD, # rem. obs.) were similar for the

two cleaning methods

Advanced Editing

• To be used after AYCE• Masks the effects of previously applied filters in small areas where it

is not realistic to have applied them• For Example:• we have a field where a start delay of 7 seconds and an end delay of 12

seconds was applied to all transects, but there is a ditch in the middle of the field where the header was raised briefly and then lowered. From the previous filters the 7 values after and the 12 values before the header was raised to cross the ditch are thrown out but are actually valid values. We can use Advanced Editing to ‘mask’ these small areas from various filters.

Recommended Procedures

Start with automated cleaning then refine using manual filters and lastly, used advanced editing methods

Image before automated cleaning

After automated cleaning

14% of values removed

Automated cleaning + my filters

15.5% of values removed