DATAINBRIEFTEMPLATE

Meta-Data

*Title: KFujiRGB-DSdatabase:Fujiapplemulti-modalimagesforfruitdetectionwithcolor,depthandrange-correctedIRdata

*Authors: JordiGené-Molaa

VerónicaVilaplanab

JoanR.Rosell-PoloaJosep-RamonMorrosbJavierRuiz-HidalgobEduardGregorioa,*

*Affiliations: aResearchGroupinAgroICT&PrecisionAgriculture,DepartmentofAgriculturalandForestEngineering,UniversitatdeLleida(UdL)–AgrotecnioCenter,Lleida,Catalonia,Spain.bDepartmentofSignalTheoryandCommunications,UniversitatPolitècnicadeCatalunya,Barcelona,Catalonia,Spain.

*Contactemail: egregorio@eagrof.udl.cat*Co-authors:

JordiGené-Mola(j.gene@eagrof.udl.cat)

VerónicaVilaplana(veronica.vilaplana@.upc.edu)

JoanR.Rosell-Polo(jr.rosell@eagrof.udl.cat)Josep-RamonMorros(ramon.morros@upc.edu)JavierRuiz-Hidalgo(j.ruiz@upc.edu)EduardGregorio(egregorio@eagrof.udl.cat)

*CATEGORY: Horticulture;AgronomyandCropScience;ComputerVisionandPatternRecognition

DataArticle

Title:KFujiRGB-DSdatabase:Fujiapplemulti-modalimagesforfruitdetectionwithcolor,depthandrange-correctedIRdata

Authors: Jordi Gené-Molaa, Verónica Vilaplanab, Joan R. Rosell-Poloa, Josep-RamonMorrosb,JavierRuiz-Hidalgob,EduardGregorioa,*

Affiliations:

aResearch Group in AgroICT & Precision Agriculture, Department of Agricultural and ForestEngineering,UniversitatdeLleida(UdL)–AgrotecnioCenter,Lleida,Catalonia,Spain.

bDepartment of Signal Theory and Communications, Universitat Politècnica de Catalunya,Barcelona,Catalonia,Spain.

Contactemail:*egregorio@eagrof.udl.cat

AbstractThisarticlecontainsdatarelatedtotheresearcharticleentitle“Multi-modalDeepLearningforFruitDetectionUsingRGB-DCamerasandtheirRadiometricCapabilities”[1].Thedevelopmentofreliablefruitdetectionandlocalizationsystemsisessentialforfuturesustainableagronomicmanagementofhigh-valuecrops.RGB-Dsensorshaveshownpotentialforfruitdetectionandlocalizationsincetheyprovide3Dinformationwithcolordata.However,thelackofsubstantialdatasetsisabarrierforexploitingtheuseofthesesensors.ThisarticlepresentstheKFujiRGB-DSdatabasewhich is composedby967multi-modal imagesof Fuji appleson trees capturedusingMicrosoft Kinect v2 (Microsoft, Redmond,WA, USA). Each image contains informationfrom3differentmodalities:color(RGB),depth(D)andrangecorrectedIRintensity(S).Groundtruth fruit locations were manually annotated, labeling a total of 12,839 apples in all thedataset. The current dataset is publicly available athttp://www.grap.udl.cat/publicacions/datasets.html.

KeywordsMulti-modaldataset;Fruitdetection;Depthcameras;RGB-D;Fruitreflectance;Fujiapple

SpecificationsTableSubjectarea Machinelearning,computervision,deeplearning,agronomyMorespecificsubjectarea

Imagefusion,Precisionagriculture.

Typeofdata Multi-modalimageswithcolor(RGB),depth(D),andrange-correctedIRintensity(S).

Howdatawasacquired TheimageswereacquiredusingMicrosoftKinectv2.Dataformat Rawimages:JPG

Rawpointclouds:MATPre-processedimages:JPG(colorchannels)andMAT(depthandrange-correctedIRchannels)Annotations:CSVandXLM.

Experimentalfactors Differentimagemodalitieshavebeenregisteredtohavepixel-wisecorrespondencebetweenimagechannels.

Experimentalfeatures Allcaptureswerecarriedoutduringthenight,usingartificiallighting.

Datasourcelocation DatawereacquiredinTarassóFarm,acommercialapplefieldlocatedinAgramunt,Catalonia,Spain(E:336297mN:4623494m31N312ma.s.l.,UTM31T-ETRS89).

Dataaccessibility http://www.grap.udl.cat/publicacions/datasets.htmlRelatedresearcharticle Gené-MolaJ,VilaplanaV,Rosell-PoloJ.R,MorrosJ.R,Ruiz-HidalgoJ,

GregorioE.Multi-modalDeepLearningforFruitDetectionsUsingRGB-DCamerasandtheirRadiometricCapabilites.ComputersandElectronicsinAgriculture(2018)162,689-698.DOI:10.1016/j.compag.2019.05.016 [1]

ValueoftheData

• First dataset for fruit detection that contains 3 differentmodalities: color, depth andrangecorrectedIRintensity.

• Thepresenteddatasetcouldbeusedinthedevelopmentandtrainingoffruitdetectionsystemswithapplicationsinyieldprediction,yieldmappingandautomatedharvesting.

• CompilationofthisdatabaseallowsfusingRGB-DandradiometricinformationobtainedwithKinectv2forfruitdetection.

DataTheKFujiRGB-DSdatabasecontainsatotalof967multi-modal imagesofFujiapplesontreesandthecorrespondinggroundtruthfruitlocationannotations.Eachimagecontainsdatafromthreedifferentmodalities: color (RGB), depth (D), and range-corrected IR intensity (S). Fig. 1illustrates three selected images fromdedataset, showingground truthannotationsand themodalitiesthatcomposeseachimage.

Fig. 1. Selection of 3multi-modal images and the corresponding ground truth fruit locations(redboundingboxes). Each imagecolumncorresponds toadifferent imagemodality:RGB,SandD,respectively.

Thisdatasetwasbuilt tobeused for training,validationandbenchmarkingof fruitdetectionalgorithms using RGB-D sensors. For instance, in [1], the deep convolutional neural networkFasterR-CNN[2]wasusedtodetectandlocalizefruitsfromthepresenteddataset.

Imagesare548x373pxandweresavedinthreedifferentfiles:• RGBhr (high resolutioncolor image):Rawcolor image.These imagesaresaved in8-bit

JPGfiles.• RGBp (projectedcolor image):Projectionof thecolor3Dpoint cloudonto thecamera

focal plane. The RGBp and the D-S modalities are obtained following the sameprocedure,allowingthecomparisonbetweenthesemodalitiesforfruitdetection.Theseimagesaresavedin8-bitJPGfiles.

• DS(depthandrange-correctedIRimage):Projectionoftherange-correctedIR3Dpointcloud onto the camera focal plane. The D channel corresponds to the depth values,while the S channel corresponds to the range-corrected IR intensity values. Thesemodalitiesaresavedinaunique64-bitMATfile.

SandDdatawerenormalizedbetween0and255–likeRGBimages-toachievesimilarmeanand variance between channels. This normalization allows a faster learning convergence ofmachinelearningalgorithms(suchasdeepconvolutionalneuralnetworks).

All images were manually annotated with rectangular bounding boxes, labelling a total of12,839applesinallthedataset.AnnotationsareprovidedinXLMandCSVformats,whereeachrow corresponds to an apple annotation, giving the following information: item, topleft-x,topleft-y,width,height,labelid.ExperimentalDesign,Materials,andMethodsThedataacquisitionwascarriedoutinacommercialFujiappleorchard(MalusdomesticaBorkh.cv.Fuji),threeweeksbeforeharvesting(85BBCHgrowthstage[3]).TheRGB-DsensorsusedweretwoMicrosoftKinectv2(Microsoft,Redmond,WA,USA),whicharecomposedbyanRGBcameraandatime-of-flight(ToF)depthsensor.Foreachcapture,thesensorprovidesa3DpointcloudwithRGBandbackscatteredIRintensitydata,andarawRGBimage.Duetotheperformanceofthedepthsensordropsunderdirectsunlightexposure[4],datawasacquiredatnightusingartificiallighting.

Fig.2.Datapreparationoutline.

Pre-processing of data was carried out to build the multi-modal images with pixel-wisecorrespondencebetweenchannels. Fig.2 showsanoutlineof thedatapreparation steps.ToovercometheIRsignalattenuation,theIRintensitydatawasrange-corrected(Fig.2a)followingthemethodologydescribedin[1].Thentheacquired3Dpointcloudswereprojectedontothecamera focal plane (Fig.2b), generating the RGB, range-corrected IR and depth projectedimages. These imagesweregeometricallywrappedand registered (Fig.2c)withRGBhr so thatdifferent imagemodalities havepixel-wise correspondence. Finally, to reduce the number offruits per image, and considering that fruit size is small comparedwith the image size, eachcapturewassplitinto9imagesof548x373px(Fig.2d).AcknowledgmentsThis work was partly funded by the Secretaria d’Universitats i Recerca del Departamentd’EmpresaiConeixementdelaGeneralitatdeCatalunya,theSpanishMinistryofEconomyandCompetitivenessandtheEuropeanRegionalDevelopmentFund(ERDF)underGrants2017SGR646, AGL2013-48297-C2-2-R and MALEGRA, TEC2016-75976-R. The Spanish Ministry ofEducationisthankedforMr.J.Gené’spre-doctoralfellowships(FPU15/03355).WewouldalsoliketothankNufriandVicensMaquinàriaAgrícolaS.A.fortheirsupportduringdataacquisition.References[1]J.Gené-Mola,V.VilaplanaV,J.R.Rosell-Polo,J.R.Morros,J.Ruiz-Hidalgo,E.Gregorio,Multi-modalDeepLearningforFruitDetectionsUsingRGB-DCamerasandtheirRadiometricCapabilites,ComputersandElectronicsinAgriculture(Submitted)

[2]Ren,S.,He,K.,Girshick,R.,Sun,J.,2017.FasterR-CNN:TowardsReal-TimeObjectDetectionwithRegionProposalNetworks.IEEETrans.PatternAnal.Mach.Intell.39,1137–1149.doi:10.1109/TPAMI.2016.2577031

[3]Meier,U.,2001.Growthstagesofmono-anddicotyledonousplants,BBCHMonograph.doi:10.5073/bbch0515

[4]Rosell-Polo,J.R.,Cheein,F.A.,Gregorio,E.,Andújar,D.,Puigdomènech,L.,Masip,J.,Escolà,A.,2015.AdvancesinStructuredLightSensorsApplicationsinPrecisionAgricultureandLivestockFarming,in:AdvancesinAgronomy.doi:10.1016/bs.agron.2015.05.002