Computers and Electronics in Agriculture 124 (2016) 243–253

Contents lists available at ScienceDirect

Computers and Electronics in Agriculture

journal homepage: www.elsevier .com/locate /compag

Original papers

Immature green citrus detection based on colour feature and sumof absolute transformed difference (SATD) using colour imagesin the citrus grove

http://dx.doi.org/10.1016/j.compag.2016.04.0090168-1699/� 2016 Elsevier B.V. All rights reserved.

⇑ Corresponding author at: Agricultural and Biological Engineering, Rogers Hall,1741 Museum Road, University of Florida, Gainesville, FL 32611, USA.

E-mail address: wslee@ufl.edu (W.S. Lee).

Chuanyuan Zhao a,b, Won Suk Lee b,⇑, Dongjian He a

aCollege of Mechanical and Electronic Engineering, Northwest A&F University, Yangling, Shaanxi, ChinabDepartment of Agricultural & Biological Engineering, University of Florida, Gainesville, FL, USA

a r t i c l e i n f o

Article history:Received 30 November 2015Received in revised form 13 April 2016Accepted 16 April 2016

Keywords:Chromatic mapHistogram equalizationKernel SVMPrecision agricultureTexture featureWatershed segmentation

a b s t r a c t

Recognition and detection of green immature citrus fruit more accurately and efficiently in groves undernatural illumination conditions provides a promising benefit for growers to plan application of nutrientsduring the fruit maturing stages and estimate their yield and profit prior to the harvesting period. Thegoal of this study was to develop a robust and fast algorithm to detect and count immature green citrusfruit in individual trees from colour images acquired with different fruit sizes and under variousillumination conditions. Adaptive Red and Blue chromatic map (ARB) was created and combined withthe Hue image extracted after histogram equalization (HEH). The sum of absolute transformed difference(SATD), a block-matching method, was applied to detect potential fruit pixels. After OR operation of theresults obtained from colour and SATD analysis which kept as many fruit pixels as possible, a kernelsupport vector machine (SVM) classifier was built with same learning sets used for different classificationstages to remove false positives based on five selected texture features. The algorithm was evaluated witha set of testing images, and achieved more than 83% recognition accuracy. The proposed method canprovide a more efficient way for green citrus identification in a grove using colour images.

� 2016 Elsevier B.V. All rights reserved.

1. Introduction

Citrus is one of the major agricultural products in Florida,contributing to approximately 65% of the overall citrus productionin the United States. In the 2011–12 season, Florida’s citrus yieldincreased to 170.9 million boxes, with three percent increase com-pared to previous season’s production (USDA-NASS, 2013). Citrusharvesting heavily depends on manual hand picking, and it is themost time consuming and one of the most expensive operationsin the whole fruit production chain due to the increasing costbut decreasing availability of labour. With today’s increasingcompetition, early and accurate yield forecasting of immaturegreen fruit ahead of harvesting time helps growers to identifysite-specific growth conditions of trees at an earlier stage so thatthey can properly plan application of nutrients or fertilizers duringthe fruit immaturity stages. Yield mapping can also help competi-tive farmers to promote crop yield while minimizing costs bydetermining how much labour would be needed during the

harvesting period and well allocate labour depending on the yieldprediction in advance.

Studies about fruit recognition under complex environmentshave been conducted. Machine vision is the most commonly usedmethod in fruit identification and localization. Schertz and Brown(1968) first considered mechanizing the citrus harvesting industry.After that, many studies were conducted to develop fruit detectionsystems based on machine vision and image processing.Chinchuluun et al. (2009) developed a citrus fruit measurementsystem based on machine vision to determine the number of citrusand their sizes. A 3CCD camera was used in the vision system tohelp estimate yield from every single tree by obtaining a numberof fruit and sizes, which achieved a coefficient of determinationof 0.962 between counted number of fruit and actual weight. Luand Sang (2015) worked on identifying citrus fruit within treecanopy based on colour information and contour fragments. Achromatic aberration map and normalised red (R) channel mapwere fused for the preliminary segmentation. Ellipse fitting wasused to recover the overlapped fruit. The relative error of recoveredocclusion was 5.3%. Utilization of other types of images and tech-niques were also carried out for fruit recognition. Gong et al.(2013) developed a fruit-counting algorithm using an Android

Nomenclature

ARB Adaptive Red and Blue chromatic mapB blue band in an RGB imageCDF_VALUE cumulative distribution function (CDF) value of dif-

ferent grey levelHEIGHT height of the input imageI1, I2, I3 three colour components defined in OHTA colour spaceL logarithmic transformation of imageM size of a maximal square inside each detected circleMinCDF minima of CDF value in the whole image

N decided by the size of M, where N = floor(M/20)NormLog normalised logarithm transform value for each pixelNpatch number of divided patches inside each maximal squaren size of Hadamard matrixnum number of fruit patch within the maximal squareR red band in an RGB imageRatio ratio of B and R, i.e., B/Rx, y coordinates of a centre of merged circles

244 C. Zhao et al. / Computers and Electronics in Agriculture 124 (2016) 243–253

mobile phone to estimate the yield of mature citrus from anindividual tree. The number of clustered fruit was counted by an8-connectedness chain code combined with area estimation.

Many researchers focus on mature fruit because correct recog-nition of fruit in a field is a precondition of yield estimation andfruit picking for robotic harvesting. There is no doubt that knowingearly yield estimation and fruit growth conditions is also essentialto growers. Rakun et al. (2011) combined colour, texture featureand 3D shape properties in building a computer vision basedmodel for green apples detection to estimate the number,diameter, and yield of green apple fruit. Kurtulmus et al. (2011)developed a machine vision algorithm, using colour, circular Gabortexture and ‘eigenfruit’, to detect and count green citrus fruit inRGB images with a detection accuracy of 75.3%. Sengupta andLee (2014) also proposed an algorithm to detect immature citrususing colour images. Shape features were analyzed to detect poten-tial fruit and six selected texture features were used to removefalsely detected regions. Then the scale invariant feature transform(SIFT) was carried out for further removal of false positives. Thisalgorithm reported an accuracy of 80.4% in a validation set ofimages.

Besides colour images, some research activities were alsocarried out using thermal, multispectral/hyperspectral images forgreen fruit detection. Wachs et al. (2010) developed a machinevision system using both thermal infrared and colour images todetect green apples from the background. It was shown that thelow-level features with 74% recognition accuracy had a better per-formance over the high-level features. For multispectral imaging,Kane and Lee (2007) employed a monochromatic near-infraredcamera equipped with interchangeable optical band pass filters(1064, 1150 and 1572 nm), which were decided by their previouswork (Kane and Lee, 2006). The results showed the potential touse multispectral imaging for in-field green citrus fruit detectionwith an average correct identification rate of 84.5%. Hyperspectralimaging, which contains the visible and the near-infrared regions,also used to detect green fruit. Safren et al. (2007) used hyperspec-tral imaging to identify green apples. A multistage algorithm wasproposed based on several machine vision techniques for auto-matic estimation of the yield of on-tree apples at different growthstages. The accuracy of correct detection was 88.1%, with a falsedetection rate of 14.1%. Okamoto and Lee (2009) employed ahyperspectral camera in 369–1042 nm to develop an imageprocessing method for on-tree green citrus fruit detection. Thepixel based identification accuracies were 70–85% in a validationset using three different varieties of citrus.

Although multispectral/hyperspectral images can provide awealth of information both in the visible and the near-infraredregions and thus offer the potential to provide useful clues todetect green fruit from complex background, it cannot be ignoredthat the hyperspectral imaging contains a large amount of redun-dant information that may influence the processing time forin-field fruit detection. Growers would prefer a cheaper and easier

colour camera than an expensive sensor for the early yieldmapping system used in their groves. The algorithm proposed inthis study is different from the previous work and achieved anacceptable recognition performance for the detection of immaturegreen citrus fruit which was obtained from various distances withdifferent illumination conditions.

The overall objective of this study was to develop a colourimage based algorithm to detect and count immature green citrusfruit in the field with uneven illumination, complex growth condi-tions, and different fruit sizes. Specific objectives were to:

(1) develop an adaptive red (R) and blue (B) chromatic map(ARB) to keep fruit pixels while removing as manybackground pixels as possible;

(2) develop a detection method for most similar pixels to achosen fruit template based on the sum of absolute trans-form difference (SATD);

(3) filter out false positives using texture features.

2. Materials & methods

2.1. Image acquisition

To develop an algorithm for on-tree immature citrus detection,images used in this study were captured during the daytime withvarious illumination conditions in the experimental citrus grove inthe University of Florida, Gainesville, Florida, USA in October 2010.The images were captured in the RGB (Red, Green and Blue) colourspace, with a resolution of 3648 � 2736 pixels using a typicaldigital camera (PowerShot SD880IS, Canon USA Inc., Lake Success,NY, USA), from citrus trees when immature citrus fruit were at thegreen stage. The variety selected in this study was OrlandoTangelo. The fruit scenes were randomly selected from citruscanopy on both the sunny side and the shadow side of trees. A totalof 126 citrus fruit images were obtained. Fifty-eight of them wererandomly selected for training and the rest of 68 images were usedfor testing. To make the fruit recognition system much moreefficient to meet the requirement of real-time detection while avoid-ing image distortion, a nearest-neighbour interpolation method wasused to resize the images to 912� 684 pixels, a quarter size of theoriginal rows and columns. MATLAB 2010a (The MathWorks, Inc.,Natick, MA, USA) was used for post processing in this study.

2.2. Algorithm

The proposed algorithm had three stages for detecting andcounting the number of citrus fruit from trees, as shown inFig. 1. Colour analysis, which was the main part of the algorithm,was carried out in the first stage to remove background, such assoil, branches, sky and leaves, while keeping as many as possiblegreen fruit pixels. The algorithm started by converting the resizedimages to OHTA colour space (Busin et al., 2009), and an adaptive

Stage 2Stage 1

Stage 3

Fig. 1. Flow chart for the proposed algorithm.

C. Zhao et al. / Computers and Electronics in Agriculture 124 (2016) 243–253 245

RB chromatic aberration map (ARB) was built to segment greencitrus fruit from background. In parallel, histogram equalizationwas employed and Hue in hue, saturation, and value colour spacewas extracted (HEH) to detect green citrus fruit by setting a thresh-old value obtained from the training set, to include only the regionsof interests, i.e., green citrus fruit. Since the green fruit and greenleaves have limited difference in colour, just depending on colouris not enough to discriminate them. In the second stage, a block-matching method, sum of absolute transformed difference (SATD),was carried out to detect potential citrus fruit pixels in the inputgrey scale image compared with a size of 20 � 20 fruit mask, whichwas decided according to the minimum visible fruit size in thetraining dataset. The image, with the SATD value lower than thethreshold which was also obtained from the training set, wasnamed HDM. There were still many non-citrus pixels remainedafter processing from the first and second stages. In order tofurther eliminate false positives, the last stage, which was basedon texture features analysis, was implemented and a final resultwas determined, with confirmed areas of green citrus with colour,shape and texture information.

The flow chart of the proposed method is shown in Fig. 1. Thedetails will be elaborated in the following subsections.

2.2.1. Colour-based image segmentation – adaptive RB chromaticaberration map

Colour is one of the most intuitive features to differentiate dif-ferent objects. The essential advantages of colour features are thatthey can be easily extracted from images and be used relativelywell in distinguishing various objects.

It is essential to choose an appropriate colour space in colourimage segmentation. There are several commonly used colourspaces in machine vision, for instance, RGB (Red, Green and Blue),HSI (Hue, Saturation and Intensity), Lab (L, a, and b for colour-opponent dimensions), YCbCr (luma component, blue-differenceand red-difference chroma components), YIQ (luma, and chromi-nance information), and HSV (Hue, Saturation and Value). Back-ground in the grove, such as green leaves and grass, usuallyshares a similar colour with green citrus. The colour similaritiesmay be ignored when utilizing the grey information only and thecomplete colour information had not been used to some extent.The OHTA colour space, a linear conversion of RGB introduced byOhta et al. (1980) after a colorimetric analysis of eight different col-ours, shares a better performance in colour image segmentation.OHTA colour space can be transformed by the linear transforma-tion of the R, G, and B components, which enables faster processing.

246 C. Zhao et al. / Computers and Electronics in Agriculture 124 (2016) 243–253

The orthogonal colour components in the OHTA colour space, {I1, I2,I3}, is simple calculation, and the components are independent.There are two ways in expressing the OHTA colour space usingthe RGB colour space as shown in Eqs. (1) and (2):

I1 ¼ ðRþ Gþ BÞ=3I2 ¼ ðR� BÞ=2

I3 ¼ ð2G� R� BÞ=4

8><>: ð1Þ

where I1, I2 and I3 are the orthogonal colour components in theOHTA colour space, and R, G and B are colour features in thetraditional RGB colour space.

I02 ¼ R� B

I03 ¼ ð2G� R� BÞ=2

(ð2Þ

where I02 and I03 are another form of orthogonal characteristics in theOHTA colour space.

According to the observation of the training images in the OHTAcolour space, the I02 component presented a better performance indiscriminating green citrus fruit and background. But due to theeffects of light, illumination can influence image colour to someextent. Objects with direct straight sunlight would have higherintensity value than those under shadows. The I02 component couldnot avoid the influence of different illuminations. To improve therobustness of colour-based segmentation, an improved colourcomponent was developed to associate illumination with the ratioof grey values of B and R components (Ratio = B/R). It would beidentified as a bright image if the Ratio value were lower than0.7, which was determined from the training images. Otherwise,it was regarded as a dark image and illumination enhancementwas applied to the dark images. Then an adaptive RB chromaticaberration map (ARB) was created to enhance the differencebetween green citrus and non-citrus objects. The ARB was definedin Eq. (3):

ARB ¼ R� Ratio� B ð3ÞThen the Otsu threshold method (Sezgin, 2004) was used to the

adaptive RB chromatic aberration map to segment the potentialgreen citrus fruit regions from the background.

2.2.2. Illumination enhancement based on normalised logarithmtransform

To overcome the various illumination conditions in the grove,illumination enhancement based on normalised logarithmic trans-form was applied. Logarithmic transformation aims at expandingvalues of bright pixels and compressing values of dark pixels. Thepixel intensities belonging to the dark region will be non-linearlymapped and as a result, the less bright regions’ intensity will beenhanced producing a final image (Maini and Aggarwal, 2010).Once the value of Ratio was higher than 0.7, the images wereregarded as dark images and image enhancement processing wascarried out. The judging criterion is shown in Eq. (4).

Ratio ¼ B=R> 0:7; Dark imageselse; Bright images

�ð4Þ

The general form of the logarithmic transformation is shown inEq. (5).

L ¼ c � logð1þ rÞ ð5Þwhere L is the enhanced grey value after the logarithmic transform,c is a scaling factor, and r is the grey value of the image.

To make the enhanced image uniform, normalisation for eachpixel was applied. The equation used in this study is:

NormLog ¼ L�minLmaxL�minL

ð6Þ

where

NormLog – Normalised Logarithmic transform value;minL – Minimum grey value of L;maxL – Maximum grey value of L.

2.2.3. Image segmentation based on histogram with Hue valueHistogram equalization is a commonly used technique to

enhance the contrast of images by making the change of colourmore evenly so that dark areas are brightened to show moredetails. The method proposed in this study attempted to alter thespatial histogram of an image to closely match a cumulative distri-bution function (CDF) (Wang et al., 1999). The general approach forcumulative histogram equalization is to transfer the given image toa grey scale image and creates its histogram. Then the frequency ofdifferent grey value would be counted, the CDF value of differentgrey level, named as CDF_VALUE, is calculated and the minimavalue would be named as MinCDF. New values calculated throughthe general histogram equalization were assigned for each greyvalue in the image to obtain the histogram-equalized image. Thetheory of the CDF can be expressed by Eq. (7).

H ¼ ðCDF VALUE�MinCDFÞðHEIGHT �WIDTHÞ �MinCDF

ð7Þ

where HEIGHT and WIDTH are the height and width of the inputimage, respectively.

After histogram equalization, an image of Hue component(HEH) in the HSV colour space was extracted and then used forfiltering background pixels. For the HEH image segmentation, athreshold of 64, which was obtained based on preliminary testson the training image set, was applied to select the regions of inter-est, i. e., the green citrus fruit regions.

2.2.4. Sum of absolute transformed difference (SATD) based potentialpixels detection

The sum of absolute transformed difference, SATD, is a widelyused video quality metric for block-matching in motion estimation(Xiong and Zhu, 2009). It works by taking a Hadamard transform ofthe differences between the pixels in the template and the corre-sponding pixels in the region being used for comparison. Theformula of similarity measurement are shown in Eq. (8) and (9),

Distði; jÞ ¼XMs¼1

XNt¼1

jSðiþ s� 1; jþ t � 1Þ � Tðs; tÞj ð8Þ

HDMði; jÞ ¼XMs¼1

XNt¼1

jhdmmatrix � Distði; jÞ � hdmmatrixj=256 ð9Þ

where S is an input grey scale image which needs to be searched; Tis a target image, i.e., fruit template; Dist is distance of a comparedpixel in S and a target pixel in T; hdmmatrix is the Hadamard trans-form matrix; and HDM is the sum of Hadamard transform valuebetween the input and target images.

To detect potential fruit regions, a 20 � 20 pixel template imagewas created by calculating an average of 248 citrus patchescropped from the training set that were used later for texture anal-ysis. The patch size of a 20 � 20 pixel was determined according totwo main reasons. The first one was that when calculatingHadamard matrix of the difference of target and compared images,the size of the Hadamard matrix, n, must be an integer and n, n/12or n/20 must be a power of 2. And the size 20 pixel was chosenbecause it was also smaller than the smallest citrus fruit in theimages used in this study. The results from the HDM matrixcontained the similarity value between a template patch and com-pared region after going through the whole input image. The lower

C. Zhao et al. / Computers and Electronics in Agriculture 124 (2016) 243–253 247

SATD value between a particular pixel and the template, the higherthe chance that pixel is the same as the template. A threshold of 1.4was chosen from the training dataset. Pixels whose value waslower than the threshold were kept for the higher possibility thatthey were very similar to the template. Otherwise they were trea-ted as non-fruit pixels.

2.2.5. Improved marker-controlled watershed transform with H-minima transform

In a natural growth environment, fruit randomly distributealong branches, and they may even grow in clusters. To countthe number of citrus fruit more accurately and avoid underestima-tion that would be caused by treating touching fruit as one orignoring the fruit occluded by leaves, jointed regions need to beseparated. Watershed transform is a morphology-based methodof image segmentation, which depends mostly on the estimationof image gradients. However, it should be noticed that the water-shed transform always results in over-segmentation because thealgorithm would be influenced by noise and forms redundantcatchment basins in a uniform region which develop to one exces-sive regional minima. Several studies have been proposed to over-come the problem of over-segmentation using marker-controlledwatershed transformation (Jung and Kim, 2010). It starts with amathematical morphological operation to modify geometricfeatures of an image. Then markers, extracted from the regionalmaxima of foreground objects, will be modified before applicationof the watershed transform. Also, under-segmentation in themarker-controlled watershed transformation is caused by someof the mostly-occluded or darker objects, which means theseobjects would be missing or erroneously merged in the finalsegment result, since the low-contrast edges generate smallmagnitude gradients. To avoid problems above, contrast-limitedadaptive histogram equalization (CLAHE) was used on the Valuecomponent in the HSV colour space to enhance the contrast ofthe edges in a grey scale image. To handle the over-segmentationproblem, H-minima transform with the watershed transform(Shin et al., 2012; Choi et al., 2015) has been introduced in theliterature. By using H-minima transform, all minima in the imagewhose depth is lower or equal to the h value will be suppressed.In this study, the h value was chosen to be 4 from the training data-set to minimize over-segmentation.

2.2.6. Merge multiple detection and shape analysis to further removenoise

The watershed segmentation may yield multiple detections forthe same fruit because of the over-segmentation. So these multipledetections for one fruit required merging to count accurately anactual number of fruit in the image. Before merging circles, a shapeanalysis was applied to the separated regions after the watershedsegmentation. The citrus fruit had a near round shape while theshapes of leaves were always irregular. So regions with areassmaller than one-tenth (determined from the training dataset) ofthe biggest region, which was calculated from each processingimage, was removed. And also if the ratio of major axis length andminor axis length of a single region were bigger than 4, which werealso decided from the training dataset, the region was regarded asnon-citrus and ignored in the subsequent processing steps.

The merging criterion was based on the distance between allthe centres of the circles. To merge the multiple detection circleswhile avoiding removal of touching fruit, the distance of two cen-tres was calculated. If it were less than the maximum radius of twocircles, they would be considered to belong to the same fruit andwere merged together. The coordinates of the centre of the mergedcircle were assigned as x and y coordinates of the circle withthe maximum radius. Then potential citrus fruit were locatedaccording to the merged results. The diameter of each initially

detected fruit was the major axis length of the fitting ellipse ofeach region.

2.2.7. Texture feature selection and false positives removalHowever, some leaves were falsely classified as fruit after

applying colour features and SATD analysis due to the colour sim-ilarity between fruit and leaves, irregular illumination, shadows orocclusion. Then texture features were utilized to remove falselyclassified regions.

Three types of texture features, i.e., eight features based on greyco-occurrence coefficient matrix, fifteen features from the grey-gradient co-occurrence coefficient matrix, and three Tamurafeatures, were extracted and analyzed using a total of 468 patcheswith a size of 20 � 20 pixels which extracted from fruit and non-fruit areas from the training set. A feature selection method,Greedy Stepwise, was utilized to select the most effective featuresfrom the 26 features that could yield good prediction result andreduce the number of features. It started with an empty datasetand the first addition was decided by evaluating the predictiveability of each feature individually. The best single feature thenbecame the best subset. This was then repeated until the additionof any remaining features resulted in no improvement. A 10-foldcross validation was used and the following five features have beenchosen to remove false positives, since they were the most valu-able features for ten times during the 10-fold cross-validation:(1–2) third moment and grey level range based on grey levelco-occurrence matrix (GLCM), (3) non-uniformities of gradientdistribution, (4) mean square deviation of gradient based ongrey-gradient co-occurrence matrix (GGCM), and (5) coarsenessof Tamura.

The GLCM features were extracted from the normalisedhistogram of the matrix (Pourreza et al., 2013) with the grey levelranging from 0 to 255. The third moment and grey level range wereshown in Eq. (10) and (11), respectively.

Third moment ¼Xi

ði� lÞ3pðiÞ ð10Þ

Grey level range : fmaxðijpðiÞ – 0Þ �minðijpðiÞ – 0Þg ð11Þ

where p(i) is the normalised histogram, l =P

i ip(i).The grey gradient co-occurrence matrix (GGCM) describes the

grey and gradient relationship between every single pixel in theimage (Chen et al., 2009). Non-uniformities of gradient distribu-tion, mean square deviation of gradient were defined as:

Non-uniformities of gradient distribution :

Pj

PiPði; jÞ

� �2P

i

PjPði; jÞ

h i ð12Þ

Mean square deviation of gradient :ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiX

jj� að Þ2

XiPði; jÞ

� �rð13Þ

where P(i, j) is normalised grey gradient co-occurrence matrix;a =

Pj j

Pi P(i, j).

Tamura feature is motivated by the psychological studies on thehuman perception of texture (Tamura et al., 1978). And coarsenessis one of its most important features that are designed to measurethe difference between coarse and fine textures. Fine textures havesmaller size and more repetition of texton compared with coarsefeatures.

After the processing steps described in Section 2.2.6, as many aspossible potential spherical fruit were recognized and located withcircles in any test image. A maximal square with the size of M �Mwas taken inside each detected circle. Then it was divided into

248 C. Zhao et al. / Computers and Electronics in Agriculture 124 (2016) 243–253

several small patches with the size of 20 � 20. Npatch was thenumber of divided patches, which was calculated by Eq. (14).

Npatch ¼ N � N ð14Þ

where N = floor(M/20).A kernel SVM classifier using a Gaussian Radial Basis Function

was trained using aforementioned 468 patches and applied topatches inside each maximal square. To determine the class of eachcircle, if the number of fruit patches, num, were higher than thehalf value of Npatch, the square was regarded as fruit and thecorresponding circle showed the location of fruit. Otherwise, itwas classified as non-citrus and removed from the image.

3. Results

The proposed algorithm in this study was tested with the 68validation images with various illumination conditions, differentfruit sizes, and occlusion or overlapped growth conditions.

Fig. 2. An example of R � B image and the adaptive RB chromatic aberration map: (a) OriRB chromatic map, and (e) binary image of (d) after noise removal.

3.1. Colour-based image segmentation – adaptive R � B chromaticaberration map (ARB)

Fig. 2 shows the result of an example of general RB image andthe adaptive RB chromatic aberration map, and their correspondingbinary image after removing noise and small areas. Fig. 2(a) showsan example of resized original image. Fig. 2(b) and (c) shows thegeneral RB image and its corresponding binary image after smallnoise removal, respectively. Fig. 2(d) and (e) shows the adaptiveRB chromatic map proposed in this study and its binary image ofafter noise removal, respectively. It can be seen that the ARBmap reduced some highly contrasted areas on the fruit surfaceand this can avoid missing fruit pixels. It also enhanced thecontrast of green citrus fruit and background to some extent thatmore pixels were preserved.

3.2. Histogram equalization with Hue in the HSV colour space

Histogram equalization enhanced the contrast between fruitand non-fruit portions. Then Hue component in the HSV colour

ginal image, (b) R � B image, (c) binary image of (b) after noise removal, (d) adaptive

Fig. 3. Results after histogram equalization. (a) Image after histogram equalization, (b) Hue component image after histogram equalization (HEH), (c) binary image ofbackground removal after morphological processing, and (d) result of OR operation of adaptive RB map and HEH image.

Fig. 4. The SATD based potential pixels detection results and final background removal result. (a) SATD result on input grey image, (b) regions with SATD value lower than 1.4which represent potential fruit positions, (c) final background removal results with colour and SATD analysis, and (d) colour image after removing small regions from theresult (c). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)

C. Zhao et al. / Computers and Electronics in Agriculture 124 (2016) 243–253 249

250 C. Zhao et al. / Computers and Electronics in Agriculture 124 (2016) 243–253

space was used to separate fruit and background. Fig. 3(a) showsthe results of histogram equalization. Cumulative distributionfunction was first calculated on each of the R, G and B components.Then the final histogram equalization image was generated bycombining the individual equalized R, G and B images. Fig. 3(b)shows the Hue component after histogram equalization (HEH).Fig. 3(c) shows the binary result of Hue after morphologicalprocessing. Fig. 3(d) is the result of OR operation of adaptive RBmap and HEH image.

3.3. Sum of absolute transformed difference (SATD) based potentialfruit pixel detection

Fig. 4 shows the SATD based potential pixel detection resultsand final background removal result. Fig. 4(a) shows the resultafter calculating SATD between the created fruit template andinput grey scale image. Pixels remained in Fig. 4(b) shows potentialfruit positions with SATD value lower than 1.4.

Fig. 5. The initial detection results and final decision after filtering out false positivsegmentation: blue circles indicate the potential fruit positions, (c) corresponding initial dand filtering out non-fruit regions by the shape analysis, (e) maximal squares insidinterpretation of the references to colour in this figure legend, the reader is referred to

After colour-based and SATD analysis, the final result ofbackground removal is shown in Fig. 4(c). Fig. 4(d) shows thecorresponding colour result of the final background removal. Itcannot be ignored that there still portions of leaves remained inthe image. To remove false positives, texture features wereemployed in the following stage.

3.4. Further false positives removal based on shape feature and makingfinal decision

To filter out false positives, the improved watershed segmenta-tion was applied to separate touching objects. Fig. 5(a) shows theresults of watershed segmentation. Fig. 5(b) shows the initialdetection result after segmentation. It can be seen that there wereseveral small objects ignored based on a pre-determined areathreshold, 900 pixels, which was smaller than the smallest fruitin the training dataset. Blue circles display the position of fruitdecided by the shape feature. Its diameter was the major axis

es. (a) The result of watershed segmentation, (b) initial detection results afteretection result shown on the original image, (d) result after merging multiple circlese each circle which represent fruit location, and (f) final detection results. (Forthe web version of this article.)

C. Zhao et al. / Computers and Electronics in Agriculture 124 (2016) 243–253 251

length of the fitting ellipse of each region. Fig. 5(c) is the corre-sponding initial detection result shown on the original image.Fig. 5(d) is the result after merging multiple circles and filteringout non-fruit regions by the shape analysis. Objects 7, 12 and 14in Fig. 5(c) have been removed because of their irregular shape.Fig. 5(e) shows maximal squares inside the circles that weredivided into several 20 � 20 patches. Texture features in thosepatches were extracted and a trained kernel SVM classifier wasrun to determine whether it was fruit or not. Fig. 5(f) indicatesthe final fruit recognition results after removing false positives.

To verify the applicability of the proposed green citrus detectionmethod based on colour and SATD analysis, Fig. 6 shows somedetection results with relatively big fruit size in the testing imageswith different illumination conditions. Red circles in each imageindicate the recognized fruit positions. Fig. 6(a) shows fruit undernormal illumination and was overlapped more than half of its sizeby two neighbouring fruit. Fig. 6(b) shows fruit under darker illu-mination and partial occlusion. Fruit in Fig. 6(c) were under brightillumination that were highly contrasted by the direct sunlight.Fig. 6(d) presents a complex illumination condition. One fruitwas completely under the shadow, while other two fruit were cov-ered by leaves that caused uneven fruit surface, with both brightand dark regions.

There were also other images taken from a farther distance withvery small fruit sizes compared with aforementioned images. Fig. 7shows examples with different fruit sizes, and complex growthconditions and illumination situations. Fig. 7(a) has a normal illu-mination condition and one missed fruit was occluded by leaveswith one false positive. Fig. 7(b) shows the image taken from theopposite side of the sun. All fruit were detected successfully in thisimage. The missed fruit in Fig. 7(c) were under the shadow andpartially occluded by leaves with less sunlight. Fig. 7(d) shows bothbright and dark illumination conditions in missed fruit.

Fig. 6. Detection results with relatively big fruit size. (a) Touching citrus fruit under norm(c) fruit under bright illumination, and (d) fruit under complex illumination and occlusi

Table 1 shows the detection results in the testing dataset, with atotal of 308 green citrus. The manual fruit count was determinedby the number of fruit whose area was recognized by the humaneyes more than one-third of the region. The performance of theproposed algorithm was analyzed in terms of the number of cor-rectly identified fruit using the proposed algorithm, missed fruit,and false positives due to very similar colour or texture to citrusfruit.

4. Discussion

This study investigated the effect of colour features and a block-matching method to recognize and locate on-tree immature greencitrus fruit using regular RGB colour images under various naturaloutdoor environment. Detection of mature fruit is not as challeng-ing as this study. First of all, the colour of mature fruit reveals anobvious difference with the background in the field, especiallythe most inevitable objects, leaves. But the colour of immaturegreen citrus fruit is very similar to leaves. What’s more, if greenfruit was partially occluded by a cluster of leaves or under shad-ows, it may be even very difficult for human eyes to distinguishthem from leaves. In addition, the size of immature fruit is rela-tively smaller than the mature ones in general for the same variety.This also increases the difficulty in immature fruit recognition.Some studies considered multispectral imaging with visible andnear infrared, hyperspectral or thermal images, to detect greenfruit meanwhile reduce the influence of the uneven illuminationand the complicated background. Regular colour images cannotavoid these problems and are affected even more by them. There-fore, all useful information or features need to be effectively usedto differentiate immature green citrus fruit from green leaves.

This study focused more on the first stage, background removal,by combining three different detection results that generated from

al illumination condition, (b) fruit under darker illumination and partial occlusion,on.

Fig. 7. Examples of different fruit size, complex growth conditions and illumination situations. (a) Citrus fruit with partially overlapped, (b) citrus fruit located in the oppositeside of sunlight, (c) citrus fruit under shadow or partially occluded with less sunlight, and (d) citrus fruit under brighter illumination and shadows.

Table 1Immature green citrus fruit recognition accuracy results for the testing dataset.

Manualfruit count

Correctlyidentified

Missed Falsepositives

Number of fruit 308 257 51 33Percentage (%) 100 83.4 16.6 10.7

252 C. Zhao et al. / Computers and Electronics in Agriculture 124 (2016) 243–253

an adaptive RB chromatic map, histogram equalization with Hue,and potential regions decided by SATD, respectively. The proposedadaptive RB map reduced some highly saturated areas on fruit sur-face and helps avoid excessive bright pixels filtering and keep asmany complete fruit. The Ratio enabled the feasibility of ARBmap to variable situations without prior study. Images after SATDprocess influenced little by the illumination compared with thecolour feature. Texture features used in this study were selectedfrom 26 features based on feature selection method and wereapplied to the initial detected result to filter out false positives.Among the 308 fruit in testing images, the algorithm proposed inthis study correctly recognized 257 green citrus fruit, whichachieved above 83% recognition rate. Those correctly recognizedfruit included relatively bigger (Fig. 6) and smaller sizes (Fig. 7),and various illumination conditions. These results showed the highpotential and adaptability of the proposed algorithm.

The 51 missed fruit took up to about 16% in the testing dataset.Some fruit would be missed at the first stage and these fruit cannotbe saved in the subsequent processing steps. So identifying asmany fruit pixels as possible in the first stage was essential inthe whole algorithm. In the meantime, some fruit were detectedsuccessfully at the beginning after background removal, but theywere removed in the final results. There were four reasons forthese missed fruit. Firstly, errors in the segmentation of touchingobjects would result in missing regions for the dark region or the

low-contrasted boundaries, and this would lead to two fruitcounted as one single fruit. Another reason was that fruit weretouching each other or partially hidden by other fruit or leaves. Ifone fruit was covered more than half, it could be removed afterapplying texture filter. Non-uniform illumination could also leadto missing fruit. The surface of fruit may be uneven due to irregularillumination and shadows, which may have impacted the featuresof fruit. Lastly, very small fruit may be missed, or be regarded asnoise compared with those with bigger size.

The false detection of fruit was due to colour similarity anduneven illumination conditions, since the green leaves presenteda much more similar surface features with green citrus under vary-ing illuminations. This led to the false detection of green leaves asfruit and hardly to remove these non-fruit objects.

Although this study conducted a method which mainlydepended on the colour feature and texture features to detectimmature green citrus fruit and showed high recognition results,the method still needs more improvements. To improve the qualityof original images obtained from preliminary image acquisitionsystem, the parameters of camera, such as exposure time and focaldistance, could be improved accordingly to acquire images muchmore clearly so that more features can be studied to distinguishfruit and leaves. Other sensors, for instance, a thermal camera,hyperspectral camera, Kinect or stereo cameras, could be com-bined so that more information could be available than colour.Instead of using a single colour image, other kinds of images canhelp detect more overlapped fruit with temperature contrast infor-mation, different spectral signatures, depth information of leavesand fruit. Different viewing angle from stereo cameras may con-tribute to ‘see’ some occluded fruit and finally locate each fruitfor more accurate yield mapping. What’s more, an optical filtercan be added in front of the lens to help to improve unevenillumination.

C. Zhao et al. / Computers and Electronics in Agriculture 124 (2016) 243–253 253

5. Conclusion

A recognition and counting algorithm was presented to detectand locate immature green citrus fruit in colour images taken inthe citrus grove. The created adaptive RB chromatic map was usedindependently and it was effective in saving fruit pixels withreflectance from the sunlight. The block-matching method, SATD,was able to identify potential fruit pixels that had a closer distancecompared with the template. Five texture features were selectedby a feature selection method, Greedy Stepwise. A kernel SVM clas-sifier was built based on selected five features, and was able toremove false positives. Final decision was made after false posi-tives removal and counted the number of fruit in each image.

The developed algorithm in this study was able to identify anumber of immature green citrus fruit in the canopy with a com-plex background, uneven illumination, irregular growth of fruit,and different sizes of fruit with same learning sets for differentclassification stages. Further improvement is needed for real-timeapplication in the grove, and more representative features needto be studied to remove false positives and save detected fruit frombeing removed.

Acknowledgements

The authors appreciate the help provided by Ms. Daeun Choi,Dr. Alireza Pourreza, Dr. Han Li, and Dr. Yuehua Chen in the Preci-sion Agriculture Lab at the University of Florida during this study.The authors also would like to thank the China Scholarship Councilfor financial support.

References

Busin, L., Shi, J., Vandenbroucke, N., Macaire, L., 2009. Color space selection for colorimage segmentation by spectral clustering. In: IEEE International Conference onSignal and Image Processing Applications (ICSIPA), pp. 262–267. http://dx.doi.org/10.1109/ICSIPA.2009.5478603.

Chen, S., Wu, C., Chen, D., Tan, W., 2009. Scene classification based on gray level-gradient co-occurrence matrix in the neighborhood of interest points. IEEEInternational Conference on Intelligent Computing & Int, vol. 4, pp. 482–485.http://dx.doi.org/10.1109/ICICISYS.2009.5357627.

Chinchuluun, R., Lee, W.S., Ehsani, R., 2009. Machine vision system for determiningcitrus count and size on a canopy shake and catch harvester. Appl. Eng. Agric. 25(4), 451–458.

Choi, D., Lee, W.S., Ehsani, R., Roka, F.M., 2015. A machine vision system forquantification of citrus fruit dropped on the ground under the canopy. Trans.ASABE 58 (4), 933–946. http://dx.doi.org/10.13031/trans.58.10688.

Gong, A., Yu, J., He, Y., Qiu, Z., 2013. Citrus yield estimation based on imagesprocessed by an Android mobile phone. Biosyst. Eng. 115 (2), 162–170. http://dx.doi.org/10.1016/j.biosystemseng.2013.03.009.

Jung, C., Kim, C., 2010. Segmenting clustered nuclei using H-minima transform-based marker extraction and contour parameterization. IEEE Trans. Biomed.Eng. 57 (10), 2600–2604. http://dx.doi.org/10.1109/TBME.2010.2060336.

Kane, K.E., Lee, W.S., 2006. Spectral sensing of different citrus varieties for precisionagriculture. ASABE Paper No. 061065, ASABE, St. Joseph, Mich.

Kane, K.E., Lee, W.S., 2007. Multispectral imaging for in-field green citrusidentification. ASABE Paper No. 073025, ASABE, St. Joseph, Mich.

Kurtulmus, F., Lee, W.S., Vardar, A., 2011. Green citrus detection using ‘‘eigenfruit”,color and circular Gabor texture features under natural outdoor conditions.Comput. Electron. Agric. 78 (2), 140–149. http://dx.doi.org/10.1016/j.compag.2011.07.001.

Lu, J., Sang, N., 2015. Detecting citrus fruits and occlusion recovery under naturalillumination conditions. Comput. Electron. Agric. 110, 121–130. http://dx.doi.org/10.1016/j.compag.2014.10.016.

Maini, R., Aggarwal, H., 2010. A comprehensive review of image enhancementtechniques. J. Comput. 2 (3), 8–13, Retrieved from <http://arxiv.org/abs/1003.4053>.

Ohta, Y., Kanade, T., Sakai, T., 1980. Color information for region segmentation.Comput. Graph. Image Process. 13 (3), 222–241. http://dx.doi.org/10.1016/0146-664X(80)90047-7.

Okamoto, H., Lee, W.S., 2009. Green citrus detection using hyperspectral imaging.Comput. Electron. Agric. 66 (2), 201–208. http://dx.doi.org/10.1016/j.compag.2009.02.004.

Pourreza, A., Lee, W.S., Raveh, E., Hong, Y., Kim, H.J., 2013. Identification of citrusgreening disease using a visible band image analysis. ASABE Paper No.131591910, ASABE, St. Joseph, Mich. Retrieved from <http://www.scopus.com/inward/record.url?eid=2-s2.0-84881651690&partnerID=tZOtx3y1>.

Rakun, J., Stajnko, D., Zazula, D., 2011. Detecting fruits in natural scenes by usingspatial-frequency based texture analysis and multiview geometry. Comput.Electron. Agric. 76 (1), 80–88. http://dx.doi.org/10.1016/j.compag.2011.01.007.

Safren, O., Alchanatis, V., Ostrovsky, V., Levi, O., 2007. Detection of green apples inhyperspectral images of apple-tree foliage using machine vision. Trans. ASABE50 (6), 2303–2313.

Schertz, C.E., Brown, G.K., 1968. Basic considerations in mechanizing citrus harvest.Trans. ASAE 11 (2), 343–346. http://dx.doi.org/10.13031/2013.39405.

Sengupta, S., Lee, W.S., 2014. Identification and determination of the number ofimmature green citrus fruit in a canopy under different ambient lightconditions. Biosyst. Eng. 117, 51–61. http://dx.doi.org/10.1016/j.biosystemseng.2013.07.007.

Sezgin, M., 2004. Survey over image thresholding techniques and quantitativeperformance evaluation. J. Electron. Imag. 13 (1), 146–165. http://dx.doi.org/10.1117/1.1631316.

Shin, J.S., Lee, W.S., Ehsani, R., 2012. Postharvest citrus mass and size estimate usinga logical classification model and a watershed algorithm. Biosyst. Eng. 113 (1),42–53.

Tamura, H., Mori, S., Yamawaki, T., 1978. Textural features corresponding to visualperception. IEEE Trans. Syst., Man, Cybernetics 8 (6), 460–473. http://dx.doi.org/10.1109/TSMC.1978.4309999.

United States Department of Agriculture-National Agricultural Statistics Service(USDA-NASS), 2013. Florida Citrus Statistics 2011–2012. Available online:<http://www.nass.usda.gov/Statistics_by_State/Florida/Publications/Citrus/fcs/2011-12/fcs1112.pdf>.

Wachs, J.P., Stern, H.I., Burks, T., Alchanatis, V., 2010. Low and high-level visualfeature-based apple detection from multi-modal images. Precision Agric. 11 (6),717–735. http://dx.doi.org/10.1007/s11119-010-9198-x.

Wang, Y., Chen, Q., Zhang, B., 1999. Image enhancement based on equal areadualistic sub-image histogram equalization method. IEEE Trans. Consum.Electron. 45 (1), 68–75.

Xiong, B., Zhu, C., 2009. Efficient block matching motion estimation using multilevelintra- and inter-subblock features – subblock-based SATD. IEEE Trans. CircuitsSyst. Video Technol. 19 (7), 1039–1043. http://dx.doi.org/10.1109/TCSVT.2009.2020260.