methods for cropline following - robotics institute...suni of the square differences (ssdi of the...

Methods for Cropline Following

Michael Happold

CMU-RI-TR-00-14

Submitted in partial fulfillment of the requirements for the degree of Master of Science in Robotics

The Robotics Institute Camegie Mellon University

Pittsburgh, Pennsylvania 15213

May 2000

Copyright 0 2000 by Michael Happold. All rights reserved.

May 4,2000


Michael Happold

1.0 Introduction ~~ ~__ - -

The Demeter Project of the National Robotics Enpineerinp Consortium of Cmnegic Mellon University seeks IO develop a robotic h'wester capable of guiding itself t l i r (~~gt i fields of crop. A variety of techniques for guidance have been experiiiiented with. finni GPS t o stereo vision. This repon details effom at guidatice hy mealis 0 1 procesiiig 2-D image? taken from color cameras positioned 011 each side of a hwvesler. These effrm\ are meant to provide a low-cost and computationally inexpensive solution IO the p s i - tioning-problem.

Finding the dividing line berweeii cut iind uncut CIOP in 2-D iniager. the cropIi[ic. is n w urally formulated as an image se&?mentation problem. Although extensive work hi15 been done in this <area. most segmentation algorithms do not opcrate iii real-time. Real- time in this context means cycling at 4-5 H2.. which is the approximate iniiiitnum cycling time to smoothly guide a harvester Iravelling at 4 m.p.h. Even with the rapid increases in the speed of computing hardware. sophisticated segmentation routines often take several seconds, if not minutes. to complcte. The rehe . a suitable trade-off between the reliability ofrcsults and the speed of the algorithm must he found.

This repon covers four methods for finding croplines. These *are: a modified vcrsinii of the algorithm presented by Ollis and Stentz 1131: a model-based color segmenter: a tcx- rure-based segmenter; and a color edge-detecror. The speed and accuracy of these four algorithms are compared on images of alfalfa and Sudan in flat m d bedded fields.

1 of 23

Prlor Work _____ -

__~ __ 2.0 Prior Work _ _ _ _ _ ~

liiiyushi and Fuji 191 proposed diflcrentiating images in the lafaill direction to litid rdges cnrrcsponding t o the cropline. A llouph tmisfomi i < then applied [ ( I find ii

straipht line throuph these edge points. T h i s technique is inadequate lw i m ; q r s in wli ich cut and uncut crop lime similir intensity lcvels nnd images in which tl i t cropline i\ not a btraipht line.

Billingsley and Schoentisch [ I I introduced il technique for guiding vehicles in r w ~ crop by fittirig lines to [he r o w and monitriring the movenient r i f the vitnishiiig point and ceiiters of three o f the lines. Although fields with beds often present liner similar 10

those found in row crops. these lines disappear if the crcrp is thick. XI t h i s technique would not be applicable to this domain.

Gerrish et al. 1x1 combined edge-detection and template-inatching nicthnds tu find crop edges in images (if fields. They tested their tcchnique o n crop images such 2, allalta with mixed results. ‘Too many spurious edges w r e fouiid. created hy shildo\vh LIS well as the cut crop.

Zhanp, Reid. and Noguchi [ I R ] describe a multi-sensor svsirm fcir guiding iln ;igricul- turd vehcle in row crops. They use a ncnrinfrarcd filter to distinguish between v i i l a n d crop. They provide results for a vchicle guided by vision. GPS. and a fusion 0 1 the two. Vision guidance resulted in tracking t n within 0.15 In of the desired path. They du nrrt. however. describe the field and crop types in which they tested thcii- vision sy\trnr. ;ind so it is unclear from this work how applicable an infrared filter is to distinguihhing crop from soil in general.

Brandon and Searcy 121 use run length encoding t n pick out edges of rows i n MIW crop% in thresholded grayscale images. l h r rows were identified ah the centcrs of the hlobs bounded by the edges. Th is method is designed esclusively fbr row crops. wlierc the assumption that intensity edges correspond to row bounddries htdds.

Ollis and Stentz 112 I describe a color discriminant for finding the boundary between cut and uncut crop. This work is superseded by Ollis and Stenrz I1 3 1 and Piliuksi et al. [ I 31. Pilarski et ill. describe results for both GFS- and vision-guided systems. The vision algorilhms described in these papers allow for detection r)f croplines of.arhitr:~ry tun+ turc. an imptinant improvement over techniques relying on hough trJnsform\.

3.0 Problem Definition ~ - ~ _ _ _ _ _ _ ~ - -_____ ~

Cropline tracking is complicated hy the wide variety of crop and field typcs iound in agricultural settings. Crops can be planted on flat fields or on fields with beds. The lat ter are fields with evenly-spaced. raised mounds. When cutting in fields with heds, the harvester rides with its front wheels on top of the beds. This rcsults in crop of wrying heights beiog left behind between the beds. obscuring the soil from thc tracker in the next pass. A tracker that dspends on discriminating crop from soil would he h;ird pressed to find a cropline in such fields.

~ _ _ _ _ _ _ _ ____________- 2 of 23 Methods for Cropline Following

Problem Delinition

Crrip height and ccilor varies widcly as well. Two types of c r o p were i ondcrcd i n the experinienrs here: nlfdla and sudan. T'he color ofthchr crops 1101 (inly differ f r w i one another. but alw from crop ofthe same type planted in diffcrent soil and ~limiite condi- tions. A fuunher coniplication is the possibility of crop such as sudan drying O U I . D r i e d crop often more ckiscly resembles cut crop or soil than it rescmhle\ fyesh. uncui crop.

The diversity of crop and field types can be seen in Figures 1-4.

- .~

FIGURE 1. Alfalfa on Beds

~ _ _ FIGURE 2. Alfalfa on Flats

_ _ _ _ _ _ ~ Methods lor Cropline Following 3 of 23

_ _ _ _ _ _ _ 4.0 Ollis’ Adaptive Algorithm -______ ___________

The algorithm described by Ollis 1131 represents the first fensihlc inelhod for vision- h&ed cropline following. It is hoth computationally inexpensive. running at 20 liz o n 3 200 Mhr. Pentium. and fairly reliable. Ollis repons that when measured a~a i i ih t an idcal cume fitted to the cropline, the color adaptive algorithm produce& a postulated line with an average rrmr in the position of each p i n t of 7.8 pixels ( i n images < 1 f h 1 0 colun~ns). The alforithiii is not sensitive to intensity similarities hetween uncut ;md cut crop. unlikc thresholding methods descrihed above.

_________ - - _ _ _ _ _ _ _ _ _ ~ - 4 of 23 Methods for Croplioe Following

Ollis' Adaptive Algorithm

The algorithm seeks n best-fir stcp in each scanline o f an iinafe. The best-lit \Icp is dcfined as the position in the scanline that minimizes the squarc rcn)t of the sum 01 Ihc suni of the square differences (SSDI of the pixel values on the left 0 1 ttie position froin tticir mean and the SSD of the pixel values o n the right from thcir i i iei i i i . Thih is rcprc- sented as:

(EQ 1)

where.;, is the cument column position for scanline f..;,,,,,,, is the tofal number O I ~ L Y I I ~ umns, and d(ilj! is the adapted value for the pixel at i j

On each pass of the algirrithm, the adapted value for each pixel is the wrifhted w i r i 01

it.; R. G and B values. The weights ( i ~ ~ , , u v , ~ ' ~ ~ rn initialized to 1.0 f o r the first pass. For each suhsequeiit image to be processed. the weights are adjusted tu rctlect the segmtma- tion obtained inthe previous image. The cropline coniputcd in the previouh im:ige deliii- rates two cla\ses: left-of-the-cnrpline and riphl-of-the-cropline. The Firhel- Linear Discriminant for these two classes i s computed. yielding weights IIiiil project pixel values onto a line that maximizes the ratio of interclass to intraclass scatter. Thus. the weights adjust the pixel values t n achieve the greatest sepuatiun herweeii the I w o classes. The criterion function. Jfu.1, ti) be maximized fnr the Fisher Linear Dixrim- nant is expressed as follows 151:

(EO 2)

where H is the weight vector projecting samples onto the linc. SB is the betwcen-class scatter matrix. and Slli is the within-class scatter matrix. For a two-cliiss prrrblem. S, is

the sum of the scatter matrices of the two classes, with scatter matrix for c.lass i defined as:

(EO 3)

.I E x i

and S R defined as:

Methods for Cropline Following 5 of 23

Ollis' Adaptive Algorithm

Figures 5 - 8 denionstrate [he affect ofnpplyinf the adaptive weigl1w. In figures 5-7. thc raw intmsity \,dues for R. G B ilc~ozs a scmline arc shown. F i p r e X s h w \ r i le weightcd suni of the R . G md B values h r this same scdnlior. A clrar step cmergcs from w h a are othcrwise noisy signair.

------ _-_ -____ - -_ 6 of 23 Methods for Cropiine Following

Modifications t o Ollis' Adaptive Algorithm

~~

FIGURE 8. Graph of Adapted Values across the Scanline

.P'a6a%.Sa "e- lcl, a %&#"a , . ,he * r~_...__

.~ 5.0 Modifications to Ollis' Adaptive Algorithm

~~

The success of the adaptive algorithm depends on whether a key assumption hc~lds for ths images in question: the location of the cnrpline niust crrrespond t o i l bimodal wp. That is to say. the image must be segmentable into two repionr. And these t w o re~ irns must be separable by means of the linear discriminant used. This ndds a11 additional assumption, namely that cut and uncut cmp classes are linearly sep.uilblc.

The first assumption is often violated in images of C N ~ Rows nf cut crop produccd by the harvester, called windrows, appear in the images. Windrows produce [wo ndditioniii steps. one on each side ofthe windrow. Depending on the appenrmce of the CUI scctions between windrows and uncut crop and between windrows. thc best step in the image might be a windrow edge. This ha5 been found to occur frequently in images of .;ud;rn.


Modifications to Ollis’ Adaptive Algorithm

To iivoid r e p e n t i n g images along the windrow edges. two methods for forcing the bimodal asruinption were developed. The first attempts to rcstrici the regirin 01’ the irnnfe that is proccrssed by the adaptive algorithm to ooly those porlirins which iiiclude crrip and the cui region adjacent to the crop, as shown in Figure 9. This methiid riiiiply cuts iiut ii scction of the imapt. and processes the rest. The second method considers f l i t

entir? image. but restricts huw much of the image is included i n thc srcp cr,inputiiri(in lor each column in the image. As the alforithin mrives i)cmss the colunins lookin&? fbr ii step. the mount of the image on the side of the dividing line considered lo c~irrespwid to cut crup that will he used for the step computafion is limited to the nurnher 01 pixels that would he expected to be found in the sectirin between the uncnt crop iiiid the cl i isesi windrobr..

--___ ____ ~ __ -__ FIGURE 9. Example of Restricting Area of Processing. Dark region on the left I S not processed

8 of 23

Each (if these methods requires two user inputs. These u e the desircd v:idth of the cut and rhe expecied width of the windrow. Because the width uf the windrow is cnntrolled by seftings r m the hamester. the user can sasiiy provide this inti~rinorinii. Similiirly. the desired CUI width is known, because the fields are laid o u t in sectims of uiiiforin si7.e separated by raised dirt horders.

Using these two usel inputs and the known camera pa~mieters. Ihe perspective projection 0 1 the valid regions can be computrd. Frrr the first rnethcid. tlic positioii ut t h t ideal cropline in the image given that the harvester is cutting nf the desired width is easily computed. tisinp the expected windrow width and the expectsd cut width. we can coni- pute an estimated width ofthe cut region between the windrow and the uiicut crop. The rcst of the image beyond this cut region is not used.

The srcund methcid also uses the expected size of the cut region adjacent to the uncut crop to restrict how much of the image i s considered. The dityerence is lhat the regions (hat are processed changes from column to column; no single rcgion is excluded. For each scanline. the algorithm passes owr each colunin and computes the step using all the pixels on what is the expected uncut crop hide of the current column. hut usins rinly those pixels rin the expected cut side which would fall in the immediately adjacent cut region as it is pnijectcd on the image with the current column as one of its edgcs. l h e number ( i f pixels on the cut side used for computing the fit of the step is thcrefore il

- --____ - - ______ Methods for Gropline Following

Modifications to Ollis' Adaptive Algorithm

funcritin of column position. the expectcd size o f the adjacent C U I rc@m 21nd the p w spctive projection. Figure 10 shows how procrssing would bc restricted liir 3 rows i n an image. 'Ihc lcftnmost circle of each of rhr lines indicatcs the left bwndxr); ot the region prixessed for that scanline and the column indicated by Ihc iniilille circlc. The algorithm is searching for a cropline point a1 the position ofthe iniddle circlc by considering everything between it and the leftmost circle ax cut crop iuid cverytliiiis between it and the rightmost circle as uncut. As the middle circle is moved iicross the colutiiiis. the positionof the leftmost circle changes. as can thc distmce ktweeti i t and the rniddle circle.

- ~~ ~

FIGURE 10. Example of Scanline processing restrictions lor 3 Rows of an Image

The secorid method has two important advantages over the first. Ti) exclude the ccinect region. the first method needs n third user input. namely the initial expccted cumature of the cropline. In subsequent steps of the algorithm. the crirrect cuwature cii~i bc con- puled for the cruplines found. but this assumes that the curvature dws not change auh- stailtially from image IO image. Furthemiore. the second method works even if the true cropline is no1 close to the ideal cropline in the image. The drawback r i f the r s i u ~ ~ l rnethoc is that it can still find best-fir steps that do not comespoiid to the true cropline. although with less frequency than the original adaptive algurithin.

~- __ ~ _ _ ~ Methods lor Cropline Following 9 01 23

Adaptive Color Edge Detection

6.0 Adaptive Color Edge Detection ------

The work (if Gcnish et al. [SI using edpdetection tecliniques ti) ~ t n d crop e d p fwused on intensity edges in grsyscalc image&. This resullcil i l l the detrclion r j t inu1iier- ous spurious edfes. which severely complicated the plr)ce\s of picking out #he cnqlinc. Cases in which there is no significant intensity difference between cut ilud uncut crop regions ciln albo be expected. which mould cause their nlgoritluu to fail. Con\pa~-ccd t u

searching fur inlensit? edges. detecting color edges is iiiore likely to pick out 1 1 1 ~ cropline.

Rather than conuolvc masks with the image 11) find edges. tlic color edse detsctioti ; i l p

rithm developed here pnsses two adjacent windom across the im;i:e row^ iirid cornpiirc\ the color propenies ofthe windows with one another. The cnnlpasison producer :I singlc value that expresses the strength of the edge for the colunni and row under coiwider- ation. This method adapts the comparative measure to characicrisiics d t h e w i n d t w x under consideration. as compared with using a static edge detection m;isl. 'The \:slur for cornpaison is coinputed as the sum of the inter-class squared Ma\hal;inobis disrnoccs between the RGB pixel values of the two windows. Squared Mahalonohis distilnce i h

given as:

where p is the mean of the class and E-' the inverse of its coviiriilnce niillrix. 'The ~ 1 1 1

of the inter-class squared h'lahalanobis distancc is rcpresented as:

where .I l i and .r2, are samples from windows 1 'and 2. u l and 1: the m m n s o f wiw

dows 1 and 2. and XT and 2;' the inverses of thc covariance matrices i~ twindows I

and 2 respectively.

I

The algwithm proceeds as follows:

* for rurh IOH' i iii inra,qr

* .fer cnch co/umn,j in >-ow i


FIGURE 11. Color Edge Intensities for a scanline

Plot of Edge Intenslues Across a Row 60 r

I 0 100 200 300 4m 500 600 700

Column Posjbon

Often, a cropline cm he extracted from rhe distances image by searching for the niasi- mum distance score for each scanline. An example of a cropline fciund in this iiianner is

~~



FIGURE 12.

12 of 23

___________ ~ -___ _ _ _ _ _ Cropfine found by Color Edge Routrne Searching for Maxrrnum Edge Values

Although the prucedure described ahrrve requires several thresholds. ihcse threshdds are eusily detcmiined md comespond to expected features in an image. Croplinc\ should be coritinuous and five major peaks can br expected at the top of thc image coirscpond- in: to the edges of two windrows and Ihe crupline itself. Increasing Ihih threshuld to account for spuriuus peaks in the crop will nor significantly add to the cni1iput;ition;il

Model-based Color Segmentation

burden. as such pe& will he thrown out when candidate criiplines x c generated because they won’t produce a long enough cropline.

7.0 Model-based Color segmentation

Ollis [ I 1 describes a Bayesian npprcrach to finding croplines. He umstructs pdfs of CUI

and uncut crop by filling out histograms for cdch class. identifying 311 pixels 1111 cmc side of the cropline as one particular class and all pixels on the other side AS the other <Lash

(which side is CUI and which uncut is detemiined by which camera is raking the image. and therefore which side the uncut crop is expected tu he on). .Ihc hi\togr.inih updated on tlie previous cycle are used to determine the cropline fix the currem cycle. This is done by compuring the maximum likelihood estimate. as given by:

(EO 7)

where j,,,,,,.,,, is the current column position and.f(ijJ is the pixel for row i. column j This maximum likelihood estimate relies on the assumption that the cut crop is on the left side of my prospective cropline and the uncut crop is 011 the right. Kevcrsing thir assumption produce!, an estimate foracmpline to be detected on the tippasite side ofthe hares te r .

This method is susceptible to throwing out prospective cmpline points hecause 0 1 the influence of a few pixels with high pmbabiliries of helonging tu the class opposite the side which they fall on relative to the cropline points under consideration. For cx;iniple. if most of fhe pixels on the left side ofa prospective cropline point only weakly bclong to the left class. but those few pixels on the left thal belong to rhc right class do s o strongly. this prospective cropline point is likely to be rejected as tlie maXiiiium likelihood estimate of the cropline.

An alternati\,e method is toclassify each of the pixels individually. and use the im;igc of classified points to s e p e n t the original image. A third class. shadows. is added to fhc classes for cut and uncut crop. The pdfs for these three classes we structured as multivariate nomial densities. The general f h n for the multivariate i i onn~ l density is given as [j]:

(EO 8)

~ _ _ ~ ~ _ _ Memods for Cropline Following 13 of 23

Texture-based Segmentation

where .Y i i a r/-componcnr fcature vector. p, i i the nieilii vector for c l n ~ i . m d Z: ir it>

wvxiancc matrix. The probahility of a panicular class ;riven the pixel in quchtiuo is, according to Bayrs rule. written as:

(EQ 91

JJ( 1 1 , l . V ) 5 11 (.I( l t ' ; ) p ( U ' r 1

ifies pixels in thc initial i n l ay using clash nirdels obtaincd f 1 - w previ- tmsly collected data. Fipurc 13 shows an example image and the clilssificatirrn 0 1 the pixels. Pixels colored ill gray are classified as cut crop. those i n black as u w u t crop. and those in white as shadow. The windrow i s classitied as uncut crop. which is not surpris- ing given the sharper difference between soil and uncut crop than thit hetwten windrim and crop).

FIGURE 13. Image of Sudan and Classlficatlon of Plxels

Once a classified image is prduced. the cropline is crmputed, The classified im;y.s of Figure I3 indicates that using a connected components followed by il method itr exfraci the edge of the main crop cwnpunent and a method to smorrth this edge would yield 11

reasonable cropline. Unformoately. counectcd cnmpments routines arc notoriously slow. Instead. the step-function algoritluri used hy the second modified adaptive a l p rithm is used here. without. of course. the initial step of multiplying Ihe iimge v;~lucs by adaptive weights. Applying this algorithm to the clasified iinagz is similar t o doily template marchinp. with the tempkite consisting of a siep changin, 0 In size a s it is n i ~ ~ c d down the image.

8.0 Texture-based Segmentation ~ -_-----

?he example of sudan in Figure 3 reveals the similarity in cdor between uncut crop and cut regions t h ~ t is often found in such fields. This similarity presents difficulties for any segmentation mutinr that keys off color differences. especially one that relier on the assumption that the regions in RGB-space corresponding to cut and uncut crop are linearly separable. The difference in t e x t m between cut and uncut regions ofsudim. h m -

------ __ - - - 14 Of 23 Methods f o r Cropline Following

Texture-based Segmentation

cver. is striking, Using a texture-based segmcnration routine. eithcr in conjunction with or :is a replacement for color-based methods. is n self-evidently iippropriiite iipproach lor finding croplines in sudan images. There are a number of remons. i1et;iilzd k l u \ v . why this approach might not be appropriate for firding croplines i n ;iIkilfi i i i i i i p .

I t is imponant to keep in mind the constraints placed on cmiput;itional expenw hy this prohlem donrain. Many popular texture sepntentatim routines ciiniiot mum of-4 Hz. on currently available hnrdware unles\ run UII ii superc example. the method reponed by Daily 131 using M:lrkou Random Fields tMRF,) tn segment textures required 60 hours on 8 Lisp machines to teht Ihc parameter, l or thc MKFs to determine thc best set for se&vnenting tlie test image\. Altttouyh MRFh are apt for detecting changes in microtexture. an attribute likely to he useful for regmcoting alfalfa images. the computational complexity is too great for a re:iI-tini

8.1 Texture Segmentation Using Gabor Filters

Because the image plane i s not parallel to Ihe scene plane (the ciintera ir nut looking straight down, but is tilted away from the scene). perspective eflccts can be expected. The result Is that the area. apect-ratio. and density of trxture eleineiir\ chnnpe iis deplh varies. If 3 method based ou texture filters such as LOG operator.: is used. it %ill he nec- e s s q to apply filters w e r a range of scales to capture the change i n texture eleiiicm attributes. This raises an additional consideration: texture attributes indicative of uncut crop at one depth might be indicative of cut crop at another depth. A simple uiethcld for avoiding this problem will he discussed later.

Sudan places an additional constraint on the type of texture operator that can be used. Because the crop grows in excess ilf6 ft. tall and is not rigid. the stalks iiiove consider- ably in evrn moderate winds. The crop. therefore. can he found in any orientation. and. in hct, a mix of orientalious can he found in the same image. Thui. i[ does not make sense to use oriented texture operators.

l h e texture operator used for segmenting sudan imases is t i le gahor filter [ h . l i l . whicli is a complex sinusoid moduleted by a Gaussian envelope. A 2-D gahilr filter is repre- sented by the function 11.51:

(EQ 10)

wherer,, and ?,, locate the center of the Gaushian. o is Ihe frequency o l the sinusoid. is its orientation. and 'p its phase. Gabor [7] demonsrrated that thc I-r) filter hiis optimal joint resolution, that is. i t demonstrates the optimal trade-off between qxiliiil and frequency domain resolutions. Daugman [4l extended this proof to tlir X I filter.

To meet the constraints imposed hy sudan images a id the need for real-time oprratinns. I-D gahorfilters at several scales arc used to filter the input images. Becnuhc orientation is not of extreme importance. the I -D filters iue convolved veiticail? and horimntidly

~-

Methods lor Cropline Following 15 of 23

Texture-based Segmentation ----------

with the iinage at each scale. Threc scales are used for each i)rieiirilti~l~. ydcliii2 SILC

total filters. These filters were had-selected. although ineihixldr d(1 exist fr~rdcren~lining optimal filtcr characteristics [ 15-16].

Once o filtered image is produced. the magnitudc of the rczponse tor wch tilter is dcrer- mined, and the 1 . a ~ Energy [IO1 for the output ofthe magnitude operator 1311 e x h 61- tered iniage is computed. Laws Encrgy fora 7x7 neighborhood 01 in i ? i,, dcfiticd a>:

[EQ 11) i ?

wheref,) is the value of the image filtered hy kernel I I

?he output of the energy computation for each filler is then numialired to liiive uni t mrun. The nonnalizcd. filtered irnoges are then used to compute the cropline. 4 wriet? of techniques can used at this juncture. including the step-function meth(xl 11fOIh’ :ilgorirhni c)r 3 classificnlion scheme such as presented in section 7.0. To acc(iunr f o r pmjcctivc distortion. the images are divided into lhree sections. The tw(7 vertic;il i~nd two horizontal filters with the most appropriate scale for the depth in the wction in question are considered. Experimentation as to which scale of filter provides the stron- gest response to Sudan at the range in question is easily done. This method produccs i 1 4- dimcnbional feature vector for cacti position in the image. rather ttxm the h-diniensim;il vector resulting from considering the output of every filter. These feature vectorr ciw tticn be used as the input to the adiptive algorithm described by Ollib. This iiiethml ci111 he modified so that each third ofthe imape has its own sei of adaptive weights.

A .simpler method for picking out the cropline has proven t o be sufficient to pick OUI croplines fn,m filtered images. Sudan has such pronounced tcrture as t o dlow for s i m ply picking out the largest step in texturr energy withnut use of adaptive wcifliiz. An exainple cropline compured in this iiidniier is given in Fifure 14. The henefir ofthis simpler method is increased speed.

------ ---- - -___-__-_ __ 16 01 23 Methods for Cropline Following

Results

FIGURE Cropline Found Using Texture Filters

9.0 Results ~ -~

Five data sets were used to compare the accuracy of the algorithms. with each dai;i sei representing a unique crop. field. or environmental factor. Each data set consists o f 20 iinagrs. producine a total of 100 images. The five data sets are: I i alfdfii on tlm: 2 ) alfalfa on beds: 3 ) sudaii 011 beds: 4) sudaii on tlats: 5 ) sudm on flats in high winds. The lasf data sei waz selected to illustrate factors other than texture md color which nvny afikcr segmentation.

Accuracy is ineasured as the deviation of the computed croplines Ironi ii hand-sslscied ideal cropline. Four measures are given for each data set: average error in pixclr froin the ciimpured croplines to the ideal croplines. the percent of computed cropline points that fall within 10 pixels oflhe ideal. average e m r in melers frriin the prcijsctirin of the compured croplines onto the scene and the projection ufthe ideal cropline. ;ind 111s I ~ ~ I I I -

dard devCdtion of this measure. The latter two measures provide 3 sense o f how uzetul the computed croplines are for accurately steering the vehicle.

The first data sei is a subset of that used by Ollis [ I 11. The modified adaptivc algorithm produces the best results. although the model-based method also pnduces a subsi;intiol


TABLE 1. ----_-___----_____ Performance of Color Segmentation Algorithms on Test Set from (111 (Alfalfa on

~- -

flats)

The becond dara set is a scries of images taken in El Cznrro whilc curring i n alfalfa Field5 on beds. The adaptive and color edge niutiries provide similar ;wxjyc error. but rhc adidnprive provides il greater nurnhsr of uscful cropline points.

-_--- -_-_-__-__________ -

TABLE 2. Performance of Color Segmentation Algorithms on Images of Alfalfa on Beds

The perfomlance of the modified adaptive algorithn that considers only the refimi of the image where the crop and first cut region are expected to apprur is provided its i~

point of comparison for images ofsudan on beds. The hest periiiniiance ~ c c o r d i ~ i p to both meilsurrs is the model-based method.

Contrary to expectations. the texture method performs poorly. This is due largely 111 the edge of the crop heing flatterled by the harvestcr’s cuner or1 the previous p;l\s. suppress-

-- -----_ ---- -_----___- 18 of 23 Methods for Cropllne Following

Results

12.6

ing texture information for thi, section olcrcip (which t&es up ahnut 2 0 pixtls iir i ts

widcst). and to a wbstantial shadow in the himom ofthc image.

76.8 0.74 3.76

TABLE 3.

IO.Y

Performance of Algorithms on Images of Sudan on Beds

86.5 0.67 3-25

TABLE 4.

3h.7

TABLE 5.

6Y 3 0.53 I . h l

Algorithm

hddplivr

Modified Ad;ip- live

Mndal-hared

Colrrr Edfc

Terrure 18.3

Standard

80.4 O.i(l 0.3U

Algorithm

Adaprive

Standard Avg. Error %within Avg. Error Devlation (in pixels) 10 pixels (in meiers) (in meters)

X I . ? 0.8 2.85 2.116

The texture method i s the clear choice for rhe founh d m sct. The other?. do not provide enough uscfui cropline points to he able 10 ctlrrecrly discard outlicrr.

Performance of Algorithms on Images of Sudan on Flats (Set 1 )

I I 1 - I

Modilied Adap- 33.1 2.0 0.73 1.83 tive 1 tive

I Texture

I 0.73 I 1.83 I 33. I 2.0 1 hlodel-ha*ed 37.4 1.8 0.97 ?.Oh

Culvr Edge 4x.3 I .x 0 78 0.XX

Texture 28.8 40.6 0.43 0.40 r

0.97 ?.Oh

28.8 40.6 0.43 0.40

The final data set demonstrates the difficulty o f finding an appropriate algririthni for navigating through an entire field. I t i s drawn fmm images of the same field as the founh data set, but frtim a different pass. During this pass. strong winds blew the 5 ti.+ crop wildly. obscuring the cropline.

- ~~

Performance of

Algorithm

Modified Ad;p

Model- bored

tlgorithms on Images of Sudan on Flats (Set 2. Hlgh Wlnds)

I Standard

157.6 3.63 4.56

65.25 5.41

I I I 83.34 I 1.3 2.08 3.73 1

Methods lor Cropline Following 19 of 23

~ - __ - -- - __- -__-____-__ - - - __ - - TABLE 5. Performance Of Algonthrns on Images of Sudan on Flats (Set 2. High Winds)

Standard Avg. Error %within plug. Error Deviation

Algorilhrn (in pixels) 10 pixels (in meters) 1 [in meters)

Colur Edge 174.8 h.0

The s p e d at the algorithms is compared on a Pentiuiii 111 S O 0 Mhz hiiiglc-pr(>ce>wr computer. Each algorithm is easily parallelized. enabling [he slower algorithms IC reach the 4 Hr. miniinurn withcrut the use of expensive hardwors.

le rate of the colrv edge rneth(d is cornpulrd running the algt~rithnl \vi111 ?i )x70 windows over ever\; pixel in the image. This produces highly redundant intimuation. becausc adjacent sets oiwindws h w r 9S% overlap. A spxser edge inteiisity i m n g can be produced by applying windows with no row overlap. This speeds up the ;ilgiirithin hy approximately a factor of 20. Fuflherniore. once the initial cropline is iound. i [ is not necessary fo prcxess the whole image in subsequent itcrntims. Because finding ;I color edge does not rely 011 infimnation outside of thc t w o ?Ox?O windr~wz. the nlyoritlun c x focus in on a relatively sillall refion of the h a g s once an initial cropline lias bscn found. 4 100 pixel window around the previous cropline is more than adequate fo firid

Algorithm Cycle Rate Adaptiw 30 Hz.

Modified Rdaprivc 30 Hz. Modcl-hscd

Culor Edge 0.033 Hz. I r x t u r c 1 Hr.

I the nrw cropline. This yields an additional speed-up of over 6 times (the i i i i q e h in this example have 640 columns). With these tw'o speetl-ups. the c d o r edge rnutins runs iit the minimum 4 Hz. without Ihe need for parallelization.

10.0 Conclusions -- -_-__ -

The results from the data sets. added tn thoss of Olliz [ 11 I. demonstrate that a single technique relying exclusively on RGB or texfure informalion is insufficient to accurately segment images arbitrary field types. What is needed are techniques specific;illy tuned for specific field types. The results from the above experiirients indicate posihlt. combiiiations of techniques f i r specific C I O ~ ,and field types.

l 'he modified adapfive algorithm provides the best results for alfalfa. whether i ~ n flats ur beds. There are dmwhack5, however. associated with eilcli rype of rnodified xlnptiue:

__ _-_--- -__ --__ -_----_--__ 20 of 23 Methods for Cmpline Following

Conclusions

the algorithm either requires the probahle cropline curvature tu he known a priori or is 5 l i l l susceptible (though to a lesser the image. It wriuld he helpful to have borh an cstimate o f the cuwinurr and a scnse of what ponion of the image can be discm-ded to overconic lhese limitati~~ns. The c d o r edge detector can be rifuse in providing this inf~imation. Linking edges its dcwrihrd i n section 6.0 offers a set of posiible croplines. Some of these croplines will be spurious ones. a result of the color diffrence between windrow and cut crop repion. Even rhcse spurioui boundaries are useful in estimating the curvature of the croplinc. hecause the curvature of the windrow f k ~ m the previous cut iiivlchcs that of the curreiii (.true) cropline. 'This is due to the fact that this cropline was made in the act of creating 111s windrow. The color edgc detector also can eivc a rough estiinalr o f how inuch o f the irnage can be discarded. Although the color edge detector returns an esli~iiiitc ofrhe true cropline. the results indicate that this cniplinc can have comidcruhle emor. This error i s often due 11) hare patches of crop alnng the cropline. The kiundary belweeri windrow and cut cnop. however. tends 10 be more accurate. because harvrstcrs form fairly uni- form u,indrows. This is the boundary that is of greatest inipoitance fbr deciding how much of the image can be discarded, since the goal is to prevent spurious hound;irics along thc windrow.

Sudan fields present preaterproblems. a will any field oftall criip. The colnr-hiired rechniques failed in sudan on beds because they picked Ihe hound;iry between bcnt (iver uncut crop and cut crop as the true cropline. Emor in the texture ine~hiui wiis largely due t u the presence of strong texture in the cut crop sections. More sensitive texture incthnds can conceivably differentiate between the texture in the cut crop scctiiins and the crop itself. but this is to ignore a larger problem. The images of sudaii on beds were fairly ideal because of the lack of wind. This cmnot be counted <in. Tall crop in high wind5 can completely obscure the cropline. Unfonunarely. how the cmpline is obscured is highly unsysternatic. Winds range from highly directional (blo\ving the crrip river to one side) to swirling. causing the 7 ft. crop to swirl around. The devasratinf affect ofthi, can be seen in the resulls on sudm in high winds. None ofihe techniques ~onies chise tn identifying the true cropline. and steering the harvester based on thcir ciutpurs would he a random walk.

The problem of shadows was no1 addressed here, and it is a fonnidahlc one for cropline €ollci\ving. Even shorter crops such as alfalfa cast long eniiugh shadows to require that the harvestrr not follow the shadow (if the shadow is only a few inches wide. il would be reasonable to follow ill. As the harvester passes through crop. ir pushes uncut clop over. resulting in a shadow in the next pass that falls across the crop 111 the cut rcgions. A ssgmenter would have to discriminate between shadowed crop and cut regions. This i s exceptionally difficult when cutting on beds. because longer crop stalk5 itre left behind in cut regions, which then appear similar in RGB space to uncut crop. Although the errors caused by shadows tend to be systematic. deYiating from true cropline piints by a consistent distance. the difficulty lies in identifying this distance. Once identified. it ciin be easily taken into account. One possible solution i s to use viskin guidance as a kinn of cruise control. and allow the machine operator to adjust the detected cropline to c o n - pensate for shadowing.

l h e methods presented in this work have the greatest chance of succes, in field5 of thick, shon cnip, which casts shon shadows and has few bare spots. Addirion;il sensing technrilogies will be required to enable accurate cropline following in field5 o f tiill crop.


than the origiiiiil version) lii spurious 5teps in

21 of 23

11 .O References ----- -- ---- ----__----_____

1 I 1 Billingsley. I. nnri Schotnfixh, M. Vision-(iuidance of Agrjculturill Vehicles Autonornous Robns. 2. pp. 65-76 (1495).

121 Brandoil. J. and Searcy. S. Vision Assisted Tractor Guid;u~ce for Ayicid~uri i f \kf~j. d e s . Transactions of the Society of Auirr~notivs Engineers. IYY? . piper Y3IhXl.

131 Daily, M. Color Image Segmentation Using Markov Rundom Ficlds. P r w IEEE Conf. Cornpuler Vision and Pattern Kecognition. 1989. pp. .iI)1-?12.

141 Daugrnan. 1. G Twri-dimensional Spectral Analysis oi Coi-tici Receptive l;isld Pro- files. Vision Research 00). lY80. pp. X47-X56.

151 Duda. R. and Hart, P. Pattern Classification and Scene Analysis. 147.7. J. Wile? k Sons. pp. 114-IlX.

(61 Fogel. 1. and Sagi. D. Gabor Filrcrs as Texture Discriminator. Biolofic;il Cyherrrctics (61). 1989.p~. 103-113.

[71 Gabor. D. Theory of Communication. I. Inst. Elec. Eng. (93) . 1946. pp. 4 3 4 5 7 .

181 Gerrish. John el al. Transactions of the Society of Auromorive Engineers. 14x7. paper 86 1455.

[9] Hayashi. M. arid Fujii, Y. Automatic Lawn Mower Guidancr using a Visim Syhtem. Proceedings of the USA-Japan Symprrsiuni on Flexible Automiition. New Ynrk. NY July 19x8.

[ 101 Laws. K . Rapid Texturr Identification. Proceedings ofthe SPlE ('onlerencc on Image Processing for Missile Guidance. 19x0. pp. 376-380.

111 I Ollis. M. Perception Algrrritlirns i<ir a Harvesting Robot. Doclural Thesis. CMC RI-TR-97-43 August 1997.

[ I ? ] Ollis, M. and Stentz. .4. First Results in Crop Linc Trackins. Proceedings of IEEE Conference 011 Robcitics and Automatinn ( K R A '96). Minnenpolib. MIN April I Wh. pp, 95 1-956.

[ 131 Ollis. M. m d Sientz, A. Vision-Based Perception for an Autonomous Iiarvester. Proceedings of the IEEE,RSJ International Conference on Intelligent Robotic Systelnr Septeniber. 1997.

(141 Pilmki. T,. Happold, M.. Pangels. H., Ollis. M., Filrpatrick. K.. md Stentr. A. The Demeter System for Automated Hawestinp. Proceedings nf the 8th International Topi- cal Meeting on Robotics a d Remote Systems. April, IWY.

References

[ 151 Turner. M. K. .Ic.xturc Dircriinination by Gabor Functions. Biolufical Cybernetic\ ( 5 5 ) . 1YXh.pp. 71-82.

(161 Weldrin. T. and Higgins. FV. Design 01 Multiple Crahnr Filters for Texture Ssgnicn- ration. IEEE lnt. Conf. Acoust.. Spccch. Sig. Proc.. vol. I\’. May IYYh. pp. 7-10,

[ 171 Wcldon. T.. Hisgins. UT, and Dunu. D. Efficienr Gabor Filtcr Design for Texture Segmentation. Pattern Recognition 29: I?. 1YYh. pp. ?005-?015.

[ l g ] Zhmp, Q.. Reid, J. and Noguchi, N. Agricultural Vehicle Kiivigntinn Llhing Multi- ple Guidance Sensors. l ech . Repon UILU-ENG-YY-7013.


methods for cropline following - robotics institute...suni of the square differences (ssdi of the...

Documents