hiaolan

sesrkinheranar rae fal incomt m(SVMete

1. Introduction

n applof theber of ftill lacod is st; Xings a reture ann, 198

and Sun, 2004; Xing et al., 2005; Sun, 2008, 2010; Wang et al.,2011). In recent years visual sorting systems have been appliedfor detecting mechanical defects. They perform a multispectral orhyperspectral analysis of colour to identify and classify damagedareas and to nd correlation between spectral characteristics andphysico-chemical properties of healthy and affected tissues. Such

reliable model. It also requires enough variables are measured todescribe the samples accurately. The test stage uses signicancetests to classify new samples, where the decisions are based on sta-tistical tests performed on the object-to-model distances. Eachmodel should be checked for possible outliers and improved if pos-sible (as one would do for any PCAmodel) linear discriminant anal-ysis (LDA) is a method used in machine learning to nd a linearcombination of features which characterize or separate two ormore classes of objects or events (Yang, 2010). The resulting com-bination may be used as a linear classier, or more commonly, for

Corresponding author. Tel./fax: +48 81 7445061.E-mail addresses: pbaranow@demeter.ipan.lublin.pl, pbaranow@ipan.lublin.pl

Journal of Food Engineering 110 (2012) 345355

Contents lists available at

od

ls(P. Baranowski).apple bruising can be distinguished, i.e. browning and softeningof fruit tissue. The susceptibility of apple to mechanical damagedepends on many factors, including soil cultivation, nutrition andweather conditions in the eld during fruit growth (Woolf andFerguson, 2000).

Machine vision, which is an advanced technology to see ob-jects with an assistance of computers, has been used in manyapplications in agriculture, including pre- and postharvest productquality and safety detection, and sorting (Kim et al., 2002; Brosnan

samples into non-overlapping classes. In SIMCA the observationsare projected into each PC model and the residual distances calcu-lated. An observation is assigned to the model class when its resid-ual distance from the model is below the statistical limit for theclass. The observation may be found to belong to multiple classesand a measure of goodness of the model can be found from thenumber of cases where the observations are classied into multi-ple classes. The training stage implies that one has identied en-ough samples as members of each class to be able to build aDetection of mechanical defects iquality inspection systems. In spitereason for rejecting the highest numexisting automatic sorting systems sbruises and the manual sorting meth2002; Taoukis and Richardson, 2007dened as damage of fruit tissue awhich cause physical changes of texof colour, smell and taste (Mohseni0260-8774/$ - see front matter 2012 Elsevier Ltd. Adoi:10.1016/j.jfoodeng.2011.12.038es is very important forfact that bruising is theruit in sorting lines, thek precision in detectingill used (Leemans et al.,et al., 2007). Bruising issult of external forcesd/or chemical changes6). Two basic effects of

systems use advanced procedures for image processing and analy-sis, including principal component analysis, partial least squares,neural networks, linear discriminant analysis, support vector ma-chines (Lu et al., 1999; Lu, 2003; Peng and Lu, 2005, 2006; Grahnand Geladi, 2007; Wang et al., 2011).

Soft independent modelling of class analogy (SIMCA) is basedon making a PCA model for each class in a dened training dataset, consisting of samples with a set of attributes and their classmembership (Wold and Sjostrom, 1977). The term soft refersto the fact that the classier can identify samples as belonging tomultiple classes and not necessarily producing a classication of 2012 Elsevier Ltd. All rights reserved.Detection of early bruises in apples using

Piotr Baranowski , Wojciech Mazurek, Joanna WoznInstitute of Agrophysics, Polish Academy of Sciences, Doswiadczalna 4, 20-290 Lublin, P

a r t i c l e i n f o

Article history:Received 6 October 2011Received in revised form 20 December 2011Accepted 31 December 2011Available online 8 January 2012

Keywords:Apples and bruiseHyperspectral imagingThermal imaging

a b s t r a c t

The early detection of bruiequipped with sensors wored (10002500 nm) and tThe principal componentswere captured in particulasue and the sound ones. Thsurface provided additionaalso about its depth. Theincluding soft independensupport vector machinesimaging can improve the d

Journal of Fo

journal homepage: www.ell rights reserved.yperspectral data and thermal imaging

k, Urszula Majewskad

in apples was studied using a system that included hyperspectral camerasg in the visible and near-infrared (4001000 nm), short wavelength infra-mal imaging camera in mid-wavelength infrared (35005000 nm) ranges.lysis (PCA) and minimum noise fraction (MNF) analyses of the images thatnges made it possible to distinguish between areas with defects in the tis-st Fourier analysis of the image sequences after pulse heating of the fruitformation not only about the position of the area of damaged tissue butparison of the results obtained with supervised classication methods,odelling of class analogy (SIMCA), linear discriminant analysis (LDA) and) conrmed that broad spectrum range (4005000 nm) of fruit surface

ction of early bruises with varying depths.

SciVerse ScienceDirect

Engineering

evier .com/ locate / j foodeng

od Edimensionality reduction before later classication. This super-vised classication method provides a linear transformation of n-dimensional feature vectors (or samples) into an m-dimensionalspace (m < n), so that samples belonging to the same class are closetogether but samples from different classes are far apart from eachother. The classication model is developed from Bayes ruleassuming the probability distribution within all groups is known,and that the prior probabilities for groups are given, and sum to100% over all groups. It is based on the normal distributionassumption and the assumption that the covariance matrices ofthe two (or more) groups are identical.

SVM is a pattern recognition method that is used widely in datamining applications, and provides a means of supervised classica-tion, as do SIMCA and LDA. SVM was originally developed for thelinear classication of separable data, but is applicable to nonlineardata with the use of kernel functions. SVM are used in machinelearning, optimization, statistics, bioinformatics, and other eldsthat use pattern recognition. The Unscrambler software ver. 10.1,CAMO Software AS. Trondheim, Norway uses the algorithm basedon code developed and released under an modied BSD licenseby Chih-Chung Chang and Chih-Jen Lin of the National Taiwan Uni-versity (Hsu et al., 2009). SVM is a classication method based onstatistical learning wherein a function that describes a hyperplanefor optimal separation of classes is determined. As the linear func-tion is not always able to model such a separation, data aremapped into a new feature space and a dual representation is usedwith the data objects represented by their dot product. A kernelfunction is used to map from the original space to the featurespace, and can be of many forms, thus providing the ability to han-dle nonlinear classication cases.

The hyperspectral imaging was used on selected apple cultivarsto develop a multispectral technique which could be used forscabs, fungal and soil contamination and bruising (Mehl et al.,2002). By using either PCA or the absorption intensities at specic

Nomenclature

Ibl Intensity of dark imageIim Reectance intensity of a pixel of the hyperspectral im-

ageIwh Reectance intensity of the 99% reectance standard

white panelR Relative reectance of the hyperspectral imaget Time, sT Temperature, KBR Bruised region of the fruitNB Nonbruised region of the fruit

346 P. Baranowski et al. / Journal of Fofrequencies three spatial bands were selected capable of separatingnormal from contaminated apples. The correct classication of theapples was found to vary from 76% to 95%, depending on the cul-tivars analysed. In the PCA method, bands obtained are linear com-binations of the original spectral bands but are uncorrelated. Therst PC band contains the largest percentage of data variance. Foreach following PC band the variance decreases, much of which isdue to the noise in the original spectral data. With the objectiveof detecting bruising an automated system was developed (Lu,2003) to help the fruit industry provide better fruit for bruisingdetection and to identify and segregate new and old bruises fromthe normal tissues of apples. A bruise detection algorithm usedin this system included removing the background, normalizationto reduce the variations of reectance caused by an illuminationand the principal component analysis to enhance bruising featuresand reduce data dimensionality. According to the author spectralregion between 1000 and 1340 nm was the most appropriate fordetecting apple bruising. It was also suggested that the minimumnoise fraction (MNF) could be used instead of PCA. The PC bandsobtained are linear combinations of the original spectral bandsbut are uncorrelated. The rst PC band contains the largest per-centage of data variance. For each following PC band the variancedecreases, much of which is due to the noise in the original spectraldata. It was shown that the contaminated and damaged portions ofthe apples were more distinguishable from the normal portionswhen using the asymmetric second difference method for analyz-ing hyperspectral images of apple surface in the range of 424899 nm (Mehl et al., 2004). It was stated that by using that methodthere were no differences in the observations of the various applecultivars using the appropriate data treatment procedure.

The main problem to overcome is that existing sorting systemsare not capable of effectively distinguishing those fruits in whichbruising has occurred a short time before inspection (Xing et al.,2005). This is because themajority of these systems analyse narrowspectral ranges, e.g. reected visible light (400750 nm) or scat-tered near infrared radiation (1000 nm3 lm). They concentrateon detecting fruit browning, which in the case of fruit with earlybruising may occur indistinctly or not at all (Samim and Banks,1993; Ferguson et al., 1999; Xing et al., 2005). The application ofthe near infrared spectroscopy method (7002200 nm) has shownlimited effectiveness for bruise detection in multicoloured apples,e.g. Jonagold or Braeburn, and for early bruises (Upchurch et al.,1994; Wen and Tao, 2000; Kleynen et al., 2003; Xing et al., 2005,2007). The majority of apple bruise detection methods show de-ciencies in the case of dark skin colour or small surface area bruises.It has been revealed that inherent surface morphology and skin col-oration can signicantly affect the performance of the hyperspectralsystems if they are not properly addressed (Chen and Kim, 2004).

Because of the shortcomings of existing methods for early applebruise detection, there has been a growing interest in incorporating

SH Region on the fruit surface with the shallower bruiseDP Region on the fruit surface with the deeper bruiseLDA Linear discriminant analysisMNF Minimum noise fractionNETD Noise equivalent temperature differencePCA Principal component analysisPPT Pulsed-phase thermographySIMCA Soft independent modelling of class analogySVM Support vector machine

ngineering 110 (2012) 345355multi-range non-destructive sorting methods. A set of reectanceand uorescence image data was proposed for discriminating be-tween a healthy apple and an apple with fungal contaminationand bruising (Kim et al., 2001, 2002). Previous studies using thermalimaging inmid-wavelength (MWIR) and long-wave infrared (LWIR)for identifying tissue defects in fruit (including apple bruise detec-tion) indicated that this method offers new possibilities, providedthat the process of heat conduction in the fruit can be precisely iden-tied and the mechanism of heat contrast creation between thebruised part and sound areas on the fruit surface is understood (Hel-lebrand et al., 2000; Maldague, 2001; Fito et al., 2004; Baranowskiet al., 2008; Baranowski et al., 2009).

In this paper the results of a joint application of hyperspectralimaging (4002500 nm) and thermal imaging (35005000 nm)are presented and the effectiveness of such a system for earlybruise detection and depth evaluation is discussed.

od ESpecic aims of the paper are:

- to compare the effectiveness of the minimum noise fraction(MNF) and the principal component analysis (PCA) forextracting maximum apple bruise contrast from hyperspec-tral image sequences and from pulsed-phase method ofactive thermography (PPT);

- to distinguish between bruised and unaffected tissues aswell as deeper and shallower bruises in apples of ve stud-ied cultivars by utilizing supervised classication modelsfrom hyperspectral and thermal data.

2. Materials and methods

2.1. Materials

The fruit material used in the study came from the same orch-ard belonging to the STRYJNO-SAD Fruit Producers Association,situated 15 km from Lublin, Poland. Champion, Gloster, GoldenDelicious, Idared and Topaz apples (Malus domestica Borkh),were collected directly after hand harvesting in 2011 and storedfor 15 h at 21 C and at 80% relative humidity. The fruits were pre-liminary graded according to the size and only apples of the diam-eters 78 cm of each cultivar were studied.

2.2. Analytical measurements

To calculate fruit bulk density, each apple was weighed with anelectronic digital Mettler XS1003S balance, Mettler Inc., Switzer-land, operating at a capacity of up to 1000 g with readability of0.001 g and the fruit volume was obtained through the measure-ment of water volume which was displaced by the fruit. Addition-ally, the fruit rmness was determined using a Lloyd LRX UniversalTesting Machine, produced by Lloyd Instruments Ltd., Hampshire,UK, equipped with a tip of 11.3 mm in diameter. Three measure-ments of rmness were made on each fruit, in the pedicle area,the middle part of the apple and in the calyx area. The mean ofthese three rmness readings were expressed in N.

Soluble solids concentrations (SSC) were determined using aAtago pocket refractometer produced by Atago Co., Ltd. Tokyo, Ja-pan, at ambient temperature of 20 C. For each apple, two mea-surements of SSC were made at opposite sides of the fruit. Theobtained values were expressed in brix.

2.3. Bruising procedure

In order to check the effectiveness of detecting bruises thatreached down to various depths, the following bruising procedurewas applied. From 96 apples of each specimen, half were left with-out bruising, the rest were divided into two groups (24 apples ineach one) to be bruised at different depths. A position on the equa-torial line on the surface of each apple was chosen for bruising. Aplastic roller with a diameter of 10 mm and a thickness of 1 mmwas put in this position and a cylindrical weight of 0.2 kg wasdropped (the contact surface was the base of the cylinder) fromeither a height of 400 mm (for the group with deeper bruises) orfrom a height of 200 mm (for the group with shallower bruises).Then the fruits were stored at room temperature for 1 h beforethe hyperspectral and thermal imaging bruise assessments.

To measure the bruise depth each apple was cut through thecentre of the bruise along the maximum diameter of the damagedarea in the direction of the centre of the fruit. The depth parameterwas calculated according to the method described by Menesatti

P. Baranowski et al. / Journal of Foand Paglia (2001) as the distance along the axis perpendicular tothe surface and center of the fruit. The bruise depths calculatedwith this method was compared and veried with the methodwhich based on active thermography, described in detail in Bara-nowski et al. (2009).

2.4. Hyperspectral imaging system

The following imaging spectrographs were used in the study: avisible and near infrared (VNIR) ImSpector V10E imaging spectro-graph (4001000 nm) and a short wavelength infrared (SWIR)N25E 2/300 imaging spectrometer (10002500 nm) manufacturedby SPECIM, Finland. They were mounted 40 cm above a belt con-veyor which had the speed regulated for each camera (to performline scanning of the fruit). The illumination source consisted of four500W halogen lamps. The system enabled the apples to bescanned line-by-line as they moved through the eld of view ofthe optical system on the conveyor belt. A hyperspectral imagewas recorded for each scan. The resolution of the images acquiredby the VNIR and SWIR cameras differed. The resolution of the VNIRcamera image was 1344 (spatial) by 1024 (spectral) pixels by12 bits, which corresponds to a root mean square (rms) spot radiusof less than 40 lm and a spectral resolution of 6.8 nm (with the de-fault slit). The image from the SWIR camera had a resolution of 320(spatial) by 256 (spectral) pixels by 14 bits, which corresponds toan rms spot radius of less than 15 lm and a spectral resolutionof 10 nm (30 lm slit). The acquisition time of one scan of the fruitsurface for VNIR and SWIR cameras was 5 s. After nishing scan-ning an entire fruit, a data cube was obtained by selecting a rectan-gle covering the fruit area and eliminating background pixels fromthe images of all bands. This procedure enabled the preliminaryreduction of the amount of data that required further processing.The raw data from the cameras were calibrated to obtain thereectance R by using the following equation (Sun, 2010):

R Iim IblIwh Ibl 1

where Iim is the intensity of an image; Iwh is the intensity of thewhite reference Spectralon plate with reectance of 99% (R99), Lab-sphere Inc. North Sutton, NH, USA; and Ibl is the intensity of the darkimage (the shutter closed, light sources turned off and the lens cov-ered with a black cap).

2.5. MWIR imaging system

An active thermography system was designed, consisting of athermographic camera, an excitation source, and a system for con-trolling the heat pulse time as well as registration parameters andexternal conditions in the thermostatted laboratory (air tempera-ture controlled in the range 1530 C with the accuracy of0.5 C). The SC7600 thermographic camera (FLIR Systems, Inc.,USA) was used, which is sensitive in the Mid-wavelength Infraredrange (MWIR) of 35 lm. Using a detector (InSb) with the format640 512 enabled the recording at 100 Hz in full resolution. Thesystems thermal sensitivity NETD was 20 mK at an object temper-ature of 25 C. The spatial resolution of the camera is 1 mrad. Con-nection with a PC computer was possible via a Gigabit-Ethernetport. A lens with an angular eld of view of 11 8.8 was used.The camera was connected to a modular real time infrared systemIR-NDT (AT-Automation Technology GmbH, Trittau, Germany)which supports all measurement techniques (lock-in, pulse, tran-sient, etc.). The measurement of radiation temperature of the ap-ples was done under controlled external conditions. All series ofmeasurements were performed at 21 C air temperature and rela-tive humidity of 60% in daylight. The distance between the camera

ngineering 110 (2012) 345355 347lens and the apple surface was 0.5 m. Four infrared lamps PHILIPSPAR 38 (IR Red) (175 W each) were used as the excitation source.The infrared lamps were situated at a distance of 0.4 m from the

apple surface at the corners of a square whose sides were 0.38 mlong (Baranowski et al., 2009).

2.6. Analyzing algorithms

Two methods were used to reduce the dimensionality of thedata sets, to segregate noise components from the images andespecially to produce uncorrelated output bands for which bruisediscrimination would be possible. The rst was principal compo-nent analysis (PCA), which consists of nding a new set of orthog-onal axes that have their origin at the data mean and that arerotated so the data variance is maximized (Green et al., 1988;Boardman and Kruse, 1994). The number of PC bands can equalthe number of bands in the original image; however, only the rstfew PC bands contain the majority of uncorrelated relevant infor-mation. The second studied method was minimum noise fraction(MNF) transformation, which in fact is another, more complex, lin-ear transformation that consists of two separate PCA rotations. Therst rotation uses the PCs of the noise covariance matrix to decor-relate and rescale the noise in the data. This stage results in data inwhich the noise has unit variance and no band-to-band correla-tions. In the second stage a standard PC transformation of thenoise-whitened data is performed. Additionally, it is possible to di-vide the data space into a part associated with large eigenvaluesand coherent eigenimages, and a complementary part with near-unity eigenvalues and noise-dominated images.

To distinguish the areas of the bruising from PCA and MNFimages a script was written in ImageJ software (Rasband, 19972011). The implementation of Otsu thresholding algorithm wasused. It divides the histogram in two classes and the inter-class

Golden Delicious apple (Fig. 1a) was converted into 8-bit imageand Otsu thresholding method was applied giving thesholdedareas distinguished in Fig. 1b as red regions. The plugin in ImageJsoftware for particle nucleus counting was applied to this imagewhich enabled to distinguish regions of sizes ranging from 50 to5000 pixels. This range was suitable for creating in majority ofcases the masks of bruised regions. In Fig. 1c such a mask is addedto the thresholded image whereas in Fig. 1d this mask is put on ori-ginal image.

To get relevant information about fruit bruising in the range of35 lm, the pulsed-phase thermography (PPT) was used (Bara-nowski et al., 2009; Ibarra-Castanedo and Maldague, 2004). In this

0

200

300

400

500

600

700

23

25

27

29

31

33

35

37

0 1 2 3 4

Pow

er, W

Tem

pera

ture

,oC

Time, s

100

Fig. 2. Example of the thermal response of apple surface regions representingbruised and sound tissues to rectangular heat pulse.

348 P. Baranowski et al. / Journal of Food Engineering 110 (2012) 345355variance is minimized. The main stages of the thresholding proce-dure are presented in Fig. 1. The original 32-bit PC3 score of aFig. 1. Illustration of thresholding procedure stagmethod the heat response signal can be presented as a superposi-tion of the number of waves, each having different frequency,es for PC3 score of Golden Delicious apple.

mean of the highest values, the second highest value becomesthe mean of the second highest values, and so on.

more information related to the skin colour, especially differentia-tion of the skin colours (spots, blush). The percent of success in Ta-

od EIn this study, for creation of the SIMCA models the maximum ofrst seven PCs were considered. For creation of PCA componentsthe singular value decomposition algorithm was used, which ispreferable for not much extended data sets. This algorithm pro-duces higher accuracy results but is not suited for data sets witha high number both samples and variables since the algorithm al-ways computes all the components. The cross validation methodwas used for creating PCA components. The random method with20 segments was applied.

In this study the Mahalanobis method was used to model theclasses in the LDA models. This method was chosen because thevariability within the groups was not the same structure and linearmethod was not applicable.

In this study nu-SVC classication type was used and the kernelamplitude and phase delay. The decomposition of the response sig-nal for has been done using the CooleyTukey algorithm of the fastFourier transform. The real and imaginary parts of the Fouriertransform has been used to calculate the phase (Ibarra-Castanedoand Maldague, 2004). An individual rectangular heat pulse of thelength of 1 s was used and the characteristic thermal response ofthe object to this pulse was analysed. The frame rate of50 images/s and the number of frames in each sequence equal to500 were chosen in this method. In Fig. 2 the rectangular heatpulse characteristics (in Watts) and Golden Delicious apple sur-face temperature response in bruise and non-bruise regions arepresented. To make the difference between thermal responses ofthese regions more readable the rst four seconds of the processare presented in this gure.

2.7. The phase analysis of thermograms enables important informationto be obtained about the process of heat penetration within the studiedobjects

The classication models for bruised (BR)/non bruised (NB) andshallowly bruised (SH)/deeply bruised (DP) fruit including sepa-rately VNIR, SWIR and MWIR data as well as combinations ofVNIRSWIR and VNIRSWIRMWIR as variables were createdusing the soft independent modelling of class analogy (SIMCA),the linear discriminant analysis (LDA) and the support vector ma-chine (SVM) methods. Both training and testing data sets for thisanalysis contained 240 samples each. For each sample the areascontaining bruised or nonbruised regions were selected. The aver-age of reectivity from all the pixels in these regions was calcu-lated for all the bands separately. Each training and testingdatasets contained the equal number of samples for each cultivar.The samples within cultivars were chosen randomly. To avoid lowsignal-noise ratio, only the wavelengths ranging from 410 to999 nm for VNIR and from 1005 to 2445 for SWIR were used forclassication. The MWIR variables (41) contained heat responsedata in the rst 2 s (with time interval of 0.02 s) after individualrectangular heat pulse lasting 1 s. The VNIR variables representeddata from 46 spectral channels with approximately 13 nm incre-ments per pixel whereas the SWIR variables represented data from47 channels with approximately 31 nm increments per pixel. Thequantile normalization was performed on the data to standardizethem before classication procedure. The quantile normalizationof three distributions (concerning VNIR, SWIR and MWIR data)without a reference distribution was applied. It consisted in sortingthe distributions then setting to the average (arithmetical mean) ofthe distributions. So the highest value in all cases becomes the

P. Baranowski et al. / Journal of Fotype with the radial basis function. The nu-value assumed was 0.5and gamma value was 0.0185. The cross validation was used withtwo segments. The constant weight = 1 was applied all variables.ble 2 was calculated by segmenting (based on thresholding) all theimages belonging to particular PC of a given cultivar and evaluatingthe number of properly selected defects as a ratio of all the imagesfor this PC.

Incorporating SWIR bands into the PCA enabled not only betterdetection of the browning tissue of the bruise but also gave addi-tional information about skin spotting (Fig. 4).

The use of the MNF transform made the information about thebruise localisation and composition even more readable (Figs. 5and 6). For the VNIR range of wavelengths the most suitable com-ponents for bruise detection were MNF4, MNF5 and MNF6,whereas for the SWIR bands, the components which were the mostAssuming that the validation should reect the prediction errorof the samples they were weighted considering standard deviationof the population. The weight of 1/SDev was applied to all the vari-ables (The Unscrambler Tutorials, 2006). This procedure was ap-plied in all LDA and SIMCA models.

Analysis of the thermal response (MWIR spectral range) wascarried out in the frequency domain. Some image processing pro-cedures performed in this study (segmentation, principal compo-nent analysis) were carried out using MATLAB 7.9.0, Natick,Massachusetts, USA. The unsupervised classication models werecreated with the use of The Unscrambler software ver. 10.1, CAMOSoftware AS. Trondheim, Norway.

3. Results and discussion

Physical properties of the studied cultivars are presented inTable 1. Within analysed samples, the rmness varied from 25.1to 62.6 N and the highest mean value of 52.1 N was noted forIdared. This cultivar was characterized by the lowest value of sol-uble solid content (12.4 brix). The cultivars Topaz and Idared hadthe highest values of bulk density, at 0.1366 and 0.1237 g/cm3,respectively. The method of fruit bruising enabled considerable dif-ferences to be created between shallower and deeper bruises with-in all the studied cultivars. Shallower bruises ranged from 1.9 to3.5 mm whereas deeper bruises were between 2.9 and 4.4 mm. Itwas intended to nd out if differences of these properties betweencultivars affected the hyperspectral and thermal imaging results.

PCA of the apples of ve studied varieties revealed its useful-ness for extracting the most relevant information from multipleband imagery gathered in a few component images. Exemplaryprincipal component score images of the same Golden Deliciousapple in the VNIR and SWIR regions are presented accordingly inFigs. 3 and 4.

These were constructed by replacing the intensity value of thepixels in images with the score value of each principal component.These gures present the rst six principal components which arethe most important from the point of view of bruise detection. Theuneven illumination conditions inuenced the rst four PC scoresof VNIR hyperspectral images and the rst three SWIR PC scores.This result suggests that even illumination could be crucial forusing the rst PC scores in both regions of the spectrum.

The analysis of suitability of particular PC and MNF compo-nents was performed by examining the ability of their image dis-play to increase the thresholding ability to separate between thedamage section and the rest of the apple skin. The most suitablePC scores for bruise detection were PC4, PC5 and PC6 for the VNIRcamera (4001000 nm) and PC3, PC4 and PC5 for the SWIR (10002500 nm) camera (Table 2). PC2 and PC3 of VNIR bands displayed

ngineering 110 (2012) 345355 349independent from uneven illumination effects and could be usedfor bruise detection were MNF4 and MNF5 (Table 2). In the SWIRrange the noise caused by the low input signal of some bands

mod ETable 1Physical properties of studied fruit and bruise depth response.

Cultivar Statistics Firmness (N)

Load at 8 mmdepth (N)

Load at maximuload (N)

Idared N = 30 Mean 52.1 52.5Max 62.6 63.5Min 43.2 44.1

350 P. Baranowski et al. / Journal of Fohad an inuence on the MNF components higher than MNF5 (inFig. 6 it is seen for MNF6). These components were not suitablefor bruise detection.

By using active thermography (PPT and lock-in), the registeredsequences were analysed, as well as creating ampligrams andphasegrams for various frequencies. Compared to earlier study(Baranowski et al., 2009) in which the applicability of PPT methodfor bruise detection was evaluated, in the present study addition-ally the lock-in method was used. It occurred that both PPT andlock-in methods are useful for early bruise detection. In Fig. 7 theresults of this analysis are presented for the Topaz apple. In this

SD 6.0 5.9Golden Delicious N = 30 Mean 33.5 34.2

Max 45.2 45.3Min 25.1 25.3SD 6.9 6.8

Gloster N = 30 Mean 42.8 43.8Max 50.2 50.2Min 33.5 35.8SD 5.1 4.4

Champion N = 30 Mean 36.3 36.8Max 43.0 43.0Min 30.1 30.8SD 4.7 4.5

Topaz N = 30 Mean 43.4 44.8Max 54.1 54.1Min 38.0 39.4SD 4.7 4.5

Whole data set N = 150 Mean 41.6 42.4Max 62.6 63.5Min 25.1 25.3SD 8.3 8.2

Fig. 3. Principal component analysis (PCA) scores images for the VNSoluble solidcontent (brix)

Fruit density(g/cm3)

Shallower bruisedepth (mm)

Deeper bruisedepth (mm)

12.4 0.1237 2.5 3.613.3 0.1281 3.5 3.911.4 0.1188 1.9 3.4

ngineering 110 (2012) 345355gure, the thermogram marked with (a) is the image of the applebefore heat pulse or wave extinction (cold image). The fruit bruiseis not visible on this thermogram, which indicates that passivethermography cannot be used for bruise detection. Images in (band c) are the results of lock-in analysis. The lock-in ampligramis strongly inuenced by reections from the illumination source.Therefore, the phase analysis, which is free of this inuence is pref-erable for defect recognition.

The images in (d, e and f) are phasegrams for various phaseshifts of the same fruit obtained by pulsed-phase thermography.They were obtained with the use of the discrete Fourier transform.

0.7 0.0041 0.5 0.213.6 0.1165 2.6 3.415.4 0.1195 3.0 3.812.4 0.1140 2.2 3.11.2 0.0027 0.3 0.3

13.7 0.1170 2.6 3.015.0 0.1196 3.3 3.512.8 0.1138 2.1 2.90.9 0.0023 0.4 0.2

13.9 0.1214 2.5 4.014.8 0.1249 3.4 4.413.1 0.1174 2.0 3.50.5 0.0030 0.6 0.4

13.6 0.1366 2.6 3.814.0 0.1447 2.9 4.013.3 0.1128 2.4 3.60.3 0.0050 0.2 0.2

13.4 0.1313 2.5 3.615.4 0.1447 3.5 4.411.4 0.1128 1.9 2.91.0 0.0092 0.4 0.4

IR wavelength range of an exemplary Golden Delicious apple.

Fig. 4. Principal component analysis (PCA) scores images for the SWIR wavelength range of an exemplary Golden Delicious apple.

Fig. 5. Minimum noise fraction transform (MNF) scores images for the VNIR wavelength range of an exemplary Golden Delicious apple.

Table 2Results of PCA and MNF analysis for detecting bruises in the studied cultivars with the use of a standard image thresholding procedure.

Cultivar Numberof samples

PC score most suitable for imagethresholding

PC thresholdingresults % of success

MNF scores most suitable forimage thresholding

MNF thresholdingresults % of success

VNIR SWIR VNIR SWIR

Champion 30 PC4 PC4 89 MNF5 MNF4 93Gloster 28 PC4 PC3 88 MNF5 MNF4 90Golden Delicious 27 PC6 PC3 87 MNF4 MNF5 90Idared 30 PC4 PC3 93 MNF5 MNF4 97Topaz 27 PC5 PC5 86 MNF6 MNF4 87

P. Baranowski et al. / Journal of Food Engineering 110 (2012) 345355 351

od E352 P. Baranowski et al. / Journal of FoThe phase analysis of thermogram sequences by this method isespecially useful for the detection of defects at various depths be-neath the fruit skin. It enables not only the elimination of the dis-tortion of radiation temperature distribution resulting from theheating by halogen lamps not being homogenous, but also theidentication of defects at various depths for component frequen-cies of the heat response.

The average reectance spectra obtained for the apples of fourcultivars in VNIR and SWIR ranges are presented in Fig. 8. Consid-erable differences occurred between these regions for the majorityof wavelengths studied (4002500 nm). In this gure the typicalspectra of ROIs were selected representing sound tissue as wellas shallow and deep bruise 1 h after bruising. The characteristicwater absorption bands that 660, 970, 1200, 1470 and 1900 nm

Fig. 6. Minimum noise fraction transform (MNF) scores images for the

Fig. 7. Scores of active thermography (35 lm) of Topaz apple. Red spot indicates the cethe reader is referred to the web version of this article.)ngineering 110 (2012) 345355appear as localized minima. The carotenoids and chlorophyll pig-ments are reected through the absorption valleys around 500and 680 nm (El Masry et al., 2008). In general the samples contain-ing higher moisture contents had lower reectivity across theirspectra (Bunnik, 1978). The reectance from bruised surface, 1 hafter bruising was considerably lower than that from the normaltissue over the entire spectral region but especially from 700 to1900 nm. The difference in reectance between regions with var-ied bruise depths (shallower and deeper bruises) is also observedin some parts of VNIR and SWIR ranges but it is cultivar dependent.

To acknowledge preferable ranges for supervised classicationfrom reectance spectral characteristic in VNIR and SWIR rangesand from emission characteristics in MWIR range, the appropriateLDA, SVM and SIMCA models were created as described in Section

SWIR wavelength range of an exemplary Golden Delicious apple.

nter of the bruise. (For interpretation of the references to colour in this gure legend,

Rel

ativ

e re

flect

an

ce

Wavelength, nm

ahealthy tissueshallower bruisedeeper bruise

0.00.10.20.30.40.50.60.70.80.91.0

400 600 800 1000 1200 1400 1600 1800 2000 2200 2400

Rel

ativ

e re

flect

an

ce

Wavelength, nm

bhealthy tissueshallower bruisedeeper bruise

0.00.10.20.30.40.50.60.70.80.91.0

400 600 800 1000 1200 1400 1600 1800 2000 2200 2400

Rel

ativ

e re

flect

an

ce

Wavelength, nm

c healthy tissueshallower bruisedeeper bruise

0.00.10.20.30.40.50.60.70.80.91.0

400 600 800 1000 1200 1400 1600 1800 2000 2200 2400

Rel

ativ

e re

flect

an

ce

Wavelength, nm

d healthy tissueshallower bruisedeeper bruise

Fig. 8. VNIR and SWIR spectral characteristic curves extracted from ROI pixels of the hyperspectral images representing healthy (not bruised), shallowly bruised and deeplybruised tissue of: (a) Champion, (b) Gloster, (c) Golden Delicious, (d) Topaz apples.

Table 3Classication results of LDA and SVMmodels for distinguishing healthy (NB not bruised) and bruised (BR) tissues based on VNIR and SWIR reected and MWIR emitted radiationfor the testing set of fruit samples.

Model VNIR SWIR MWIR VNIRSWIR VNIRSWIRMWIR

NB BR Total NB BR Total NB BR Total NB BR Total NB BR Total

LDA N = 240 79 100 90 73 96 85 54 71 63 93 93 93 93 98 95SVM N = 240 69 99 84 78 83 80 93 84 88 84 86 85 93 91 92

Table 5Classication results of SIMCA model for distinguishing healthy (NB not bruised) and bruised (BR) tissues based on VNIR and SWIR reected and MWIR emitted radiation fortesting set of fruit samples.

Result of SIMCA model testing VNIR SWIR MWIR VNIRSWIR VNIRSWIRMWIR

NB BR Total NB BR Total NB BR Total NB BR Total NB BR Total

TP 56 49 53 65 50 58 56 44 50 61 65 63 63 70 67FP 44 49 47 28 47 37 42 51 46 33 28 31 34 26 30FN 0 2 1 8 3 5 3 5 4 6 7 6 3 4 3

TP (true positives) cases which were recognized by their rightful class model (classied correctly).FP (false positives) cases classied as belonging to two classes.FN (false negatives) cases classied as belonging to a wrong class.

Table 4Classication results of LDA and SVM models for distinguishing shallow (SH) and deep (DP) bruised tissues based on VNIR and SWIR reected and MWIR emitted radiation fortesting set of fruit samples.

Model VNIR SWIR MWIR VNIRSWIR VNIRSWIRMWIR

SH DP Total SH DP Total SH DP Total SH DP Total SH DP Total

LDA N = 120 90 38 64 82 44 63 24 92 58 68 58 63 100 30 65SVM N = 120 80 50 65 88 70 79 76 82 79 72 58 72 92 62 77

P. Baranowski et al. / Journal of Food Engineering 110 (2012) 345355 353

and SIMCA models, respectively. In SIMCA models true positive

Hellebrand, H.J., Linke, M., Beuche, H., Herold, B. & Geyer, M. 2000. Horticultural

ised

al

ed

od E(TP), false positive (FP) and false negative (FN) scores are obtainedat signicance level of 5%. The best total scores for LDA-95%, forSVM-92% and SIMCA-67% were obtained for models which incor-porated VNIR, SWIR and MWIR variables jointly. The analysis ofthe total scores for individual ranges (VNIR, SWIR or MWIR) indi-cated the lowest success rate for MWIR in the LDA model (63%)and the highest for VNIR in the LDA model (90%). From amongthe three studied classication methods the success score wasthe lowest for the SIMCA models (true positive scores ranged from50% to 67%, and false positive scores ranged from 30% to 47%). Therate of success (in per cent) obtained on the testing data set to dis-tinguish between shallow (SH) and deep (DP) tissue bruises forLDA/SVM and SIMCA models are presented in Tables 4 and 6. Thebest total scores were obtained in the SVM models including indi-vidual SWIR and VNIR variables (79% of success for each) and in theSVMmodel for the whole range of VNIRSWIRMWIR. For the SIM-CA models of shallow (SH) and deep (DP) bruise classication thehighest success rate was noticed for the model based on the VNIRdata (68% of success) and the whole VNIRSWIRMWIR data. Itshould be indicated that the low number of the true positive scoresin SIMCA models could be improved by increasing signicance le-vel, e.g. to 10% but in this case the number of false negative scoresincreases. The obtained results of the supervised classicationshow better performance of combination of spectral intervals(from VNIRSWIRMWIR ranges) on the classication of normalversus bruised apples but the same combination does not seemto signicantly affect the classication between deep and shallowbruises.

4. Conclusions

To detect early bruises in apples, a system was successfullyused, which incorporated the hyperspectral imaging of reected2. Both training and testing data sets contained the samples of ap-ples of the ve investigated cultivars. The Tables 3 and 5 show therate of success (in per cent) obtained on the validation set to distin-guish between sound (NB) and bruised (BR) tissues for LDA/SVM

Table 6Classication results of SIMCA model for distinguishing shallow (SH) and deep (DP) bruof fruit samples.

Result of SIMCA model testing VNIR SWIR

SH DP Total SH DP Tot

TP 39 29 68 38 75 57FP 10 19 29 60 23 42FN 1 2 3 2 2 2

TP (true positives) cases which were recognized by their rightful class model (classiFP (false positives) cases classied as belonging to two classes.FN (false negatives) cases classied as belonging to a wrong class.

354 P. Baranowski et al. / Journal of Foradiation in VNIR and SWIR ranges and infrared thermal imagingof emitted radiation in MWIR range. The whole spectrum range(4005000 nm) studied was useful for detecting bruises createdone hour before the experiment.

Hyperspectral image analysis of VNIR and SWIR wavebands waseffectively performed by application of PCA. Even better resultswere obtained by the use of the MNF transformation whose com-ponents could be preferable for image segmentation purposes.

Thermal MWIR imaging (30005000 nm) is useful for bruiserecognition when an active approach (lock-in or pulsed-phase) isapplied.

The analysis of the total scores for individual ranges (VNIR,SWIR or MWIR) indicated lower prediction values than in casewhen these ranges were included jointly into models. The createdmodels of supervised classication based on VNIR, SWIR andproducts evaluated by thermography. AgEng, Warwick, 2626.Hsu, C.W., Chang, C.C., Lin, C.J., 2009. A Practical Guide to Support Vector

Classication. Available from: .Ibarra-Castanedo, C., Maldague, X.P., 2004. Pulsed Phase Thermography Reviewed.

QIRT J. 1 (1), 4770.Kim, M.S., Chen, Y.R., Mehl, P.M., 2001. Hyperspectral reectance and uorescence

imaging system for food quality and safety. Transactions of the ASABE 44, 721MWIR ranges indicate that best prediction efciency for distin-guishing bruised and sound tissues as well as bruises of variousdepths is obtained for models incorporating these three ranges to-gether. This suggests that it would be reasonable to considerincluding MWIR range into sorting systems.

References

Baranowski, P., Lipecki, J., Mazurek, W., Walczak, R.T., 2008. Detection of watercorein Gloster apples using thermography. Postharvest Biol. Technol. 47, 358366.

Baranowski, P., Mazurek, W., Witkowska-Walczak, B., Sawinski, C., 2009. Detectionof early apple bruise with the use of pulsed-phase thermography. PostharvestBiol. Technol. 53, 91100.

Boardman, J. W., and Kruse, F. A., 1994, Automated spectral analysis: a geologicalexample using AVIRIS data, north Grapevine Mountains, Nevada: inProceedings, ERIM Tenth Thematic Conference on Geologic Remote Sensing,Environmental Research Institute of Michigan, Ann Arbor, MI, pp. I-407I-418.

Brosnan, T., Sun, D.W., 2004. Improving quality inspection of food products bycomputer vision-a review. J. Food Eng. 61, 316.

Bunnik, N.J.J., 1978. Multispectal reectance radiation by agricultural crops inrelation with their morphological and optical properties. H. Veenman & ZonenB.V, Wageningen.

Chen, Y.R., Kim, M.S., 2004. Visible/NIR imaging spectroscopy for assessing qualityand safety of agrofoods. In: Davies, A.M.C., Garrido, A. (Eds.), Proceedings of the11th International Conference on Near-infrared Spectroscopy. NIR publications,Chichester, pp. 6775.

El Masry, G., Wang, N., Vigneault, C., Qiao, J., ElSayed, A., 2008. Early detection ofapple bruises on different background colors using hyperspectral imaging. LWT41, 337345.

Ferguson, I., Volz, R., Woolf, A., 1999. Preharvest factors affecting physiologicaldisorders of fruit. Postharvest Biol. Technol. 15, 255262.

Fito, P.J., Ortol, M.D., De los Reyes, R., Fito, P., De los Reyes, E., 2004. Control of citrussurface drying by image analysis of infrared thermography. J. Food Eng. 61 (3),287290.

Grahn, H.F., Geladi, P. (Eds.), 2007. Techniques and Applications of HyperspectralImage Analysis. John Wiley & Sons Ltd..

Green, A.A., Berman, M., Switzer, P., Craig, M.D., 1988. A transformation for orderingmultispectral data in terms of image quality with implications for noiseremoval. IEEE Trans. Geosci. Remote Sens. 26 (1), 6574.

tissues based on VNIR and SWIR reected and MWIR emitted radiation for testing set

MWIR VNIRSWIR VNIRSWIRMWIR

SH DP Total SH DP Total SH DP Total

58 43 51 70 42 56 57 70 6338 55 47 28 55 42 43 28 363 2 1 2 3 0 0 2 1

correctly).

ngineering 110 (2012) 345355729.Kim, M.S., Lefcourt, A.M., Chao, K., Chen, Y.R., Kim, I., Chan, D.E., 2002. Multispectral

detection of fecal contamination on apples based on hyperspectral imagery:Part I. Application of visible and near-infrared reectance imaging. Transactionsof the ASABE 45, 20272037.

Kleynen, O., Leemans, V., Destain, M.F., 2003. Selection of the most efcientwavelength bands for Jonagold apple sorting. Postharvest Biol. Technol. 30,221232.

Leemans, V., Magin, H., Destain, M.F., 2002. On-line fruit grading according to theirexternal quality using machine vision. Biosyst. Eng. 83, 397404.

Lu, R., Chen, Y.R. & Park, B. 1999. Hyperspectral imaging for detecting bruises inapples. ASAE Annual International Meeting. Paper No. 99-3120. Toronto,Canada.

Lu, R., 2003. Detection of bruises on apples using near-infrared hyperspectralimaging. Transactions of the ASAE 46 (2), 523530.

Maldague, X.P., 2001. Theory and Practice of Infrared Technology forNondestructive Testing. John Wiley & Sons, New York.

Mehl, P.M., Chao, K., Kim, M., Chen, Y.R., 2002. Detection of defects on selected applecultivars using hyperspectral and multispectral image analysis. AppliedEngineering in Agriculture 18, 219226.

Mehl, P.M., Chen, Y.R., Kim, M.S., Chan, D.E., 2004. Development of hyperspectralimaging technique for the detection of apple surface defects andcontaminations. J. Food Eng. 61, 6781.

Menesatti, P., Paglia, G., 2001. Development of a drop damage index of fruitresistance to damage. J. Agric. Eng. Res. 80 (1), 5364.

Mohsenin, N.N., 1986. Physical properties of plant and animal materials. Gordon &Breach Science Publishers, New York, NY, USA.

Peng, Y., Lu, R., 2005. Modelling multispectral scattering proles for prediction ofapple fruit rmness. Trans. ASAE 48, 235242.

Peng, Y., Lu, R., 2006. An LCTF-based multispectral imaging system for estimation ofapple fruit rmness. Part II. Selection of optimal wavelengths and developmentof prediction models. Trans. ASAE 49, 269275.

Rasband, W.S., 1997. ImageJ. US National Institutes of Health, Bethesda, Maryland,USA, http://imagej.nih.gov/ij/.

Samim, M., Banks, N.H., 1993. Color changes in bruised apple fruit tissue. N.Z. J. CropHort. Sci. 21, 367372.

Sun, D.-W., 2008. Computer Vision Technology for Food Quality Evaluation.Academic Press/Elsevier, San Diego, California, USA, pp. 583.

Sun, D.-W. (editor) 2010., Hyperspectral Imaging for Food Quality Analysis andControl, Academic Press / Elsevier, San Diego, California, USA, 496 pp., ISBN:978-0-12-374753-2 (2010).

Taoukis, P.S., Richardson, M., 2007. Principles of Intermediate-Moisture Foods andRelated Technology. In: Barbosa-Cnovas, G.V., Fontana, A.J., Jr., Schmidt, S.J.,Labuza, T.P. (Eds.), Water Activity in Foods: Fundamentals and Applications.Blackwell Publishing, Ames, IA, USA.

The Unscrambler Tutorials 2006, Version 9.5, Camo A/S, Trondheim.Upchurch, B.L., Throop, J.A., Aneshansley, D.J., 1994. Inuence of time, bruise-type,

and severity on near-infrared reectance from apple surfaces for automaticbruise detection. Trans. Am. Soc. Agric. Eng. 37, 15711575.

Wang, J., Nakano, K., Ohashi, S., Kubota, Y., Takizawa, K., Sasaki, Y., 2011. Detectionof external insect infestations in jujube fruit using hyperspectral reectanceimaging. Biosys. Eng. 108, 345351.

Wen, Z., Tao, Y., 2000. Dual-camera NIR/MIR imaging for stem-end/calyxidentication in apple defect sorting. Trans. ASAE 43, 449452.

Wold, S., Sjostrom, M., 1977. SIMCA: a method for analyzing chemical data in termsof similarity and analogy. In: Kowalski, B.R. (Ed.), Chemometrics Theory andApplication, American Chemical Society Symposium Series 52. AmericanChemical Society, Wash, D.C., pp. 243282.

Woolf, A.B., Ferguson, I.B., 2000. Postharvest responses to high fruit temperatures inthe eld. Postharvest Biol. Technol. 21, 720.

Xing, J., Bravo, C., Jancsok, P., Ramon, H., De Baerdemaeker, J., 2005. Bruise detectionon Golden Delicious apples by using hyperspectral imaging with multiplewavebands. Biosys. Eng. 90 (1), 2736.

Xing, J., Jancso, P., De Baerdemaeker, J., 2007. Stem-end/calyx identication onapples using contour analysis in multispectral images. Biosyst. Eng. 96, 231237.

Yang, Z.R., 2010. Machine Learning Approaches to Bioinformatics (Science,Engineering, and Biology Informatics vol.4). World Scientic Publishing Co.,Pte., Ltd., London, UK.

P. Baranowski et al. / Journal of Food Engineering 110 (2012) 345355 355

Detection of early bruises in apples using hyperspectral data and thermal imaging1 Introduction2 Materials and methods2.1 Materials2.2 Analytical measurements2.3 Bruising procedure2.4 Hyperspectral imaging system2.5 MWIR imaging system2.6 Analyzing algorithms2.7 The phase analysis of thermograms enables important information to be obtained about the process of heat penetration within the studied objects

3 Results and discussion4 ConclusionsReferences