Image Recognition and Processing of StrawberryHarvesting Robot

Hanzhi Liu, Chi Feng, Maozhu Xu

2020.2

We take thousands of strawberry photos and get a model and data set through a seriesof processing such as segmentation and training. Then by combining the recognition modelwith other subsequent operations, the strawberry harvesting robot can be realized. Ourrelated work schedule is as follows.

1 Stage one(12th,Jan to 14th,Jan)

1.1 Data Collection

On the first day, we went to Chongming to take photos of strawberries. The varieties ofstrawberries we choose are diverse, including “Hongyan”, “Zhangji”, etc. Also, their gradesof maturity and extents of overlapping are different. We sort out the photos we took, removeunnecessary photos, and establish a preliminary database. Then a script is written to convertstrawberry photos to 480x480 pixels. In general, we get 593 photos of strawberries at shortrange (Fig.1) and 101 ones of strawberries at long range (Fig.2).

Figure 1: Strawberries at Short Range Figure 2: Strawberries at Long Range

1.2 Data Classification

We learn details of the task, how to use the LabelMe tag, pycharm, anaconda navigatorand google laboratory environment, and obtain the data shared by Maozhu Xu. Next, weclassify all the pictures, take 515 of the images at short range as the training set, the remain-ing 78 ones at short range and 101 ones at long range as the test set.

1

1.3 Data Annotation

We annotate strawberry images in the training set and test set with LabelMe. Then weuse python to transfer JSON format data generated by LabelMe to COCO dataset format.After that we transfer COCO dataset format to the mask image (Fig.3) and number them inbatches.

Figure 3: Mask Images

2 Stage 2 (24th,Feb to 25th,Feb)

2.1 Study of U-net Network

We have studied the general principle, basic structure and some characteristics of U-netnetwork. Then we start to summarize the U-net framework and write the first draft of ourwork in the paper. At the same time, we also modify the introduction part of the paper.

2.2 Model Training

We select the parameters according to the loss of the test set, set the threshold, output thebinary image, and compare it with the manually marked mask image. At first, we set theepoch as 200, but it is overfitting. Its loss in the test set is quite high. According to the loss indifferent epoches, we finally determine the value of epoch and got the resulting loss valuesin Tab.1.

Table 1: Result

Symbol Value

Batch size 8Epoch 40

Train loss 0.026Test (short range) loss 0.016Test (long range) loss 0.011

We output the loss value from the training and testing of strawberry close range picturesand draw the loss curve with the increase of epoch value in Fig 4.

2

0 10 20 30 40 50 60 70 80 90 100

0

0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

0.18Loss for Every Epcho

Train Loss

Test LossLo

ss

Epoch

Figure 4: Loss Curve of Every Epoch

At last, we get a model of U-net. When given a image, this model can output the part ofimage that belongs to strawberry as shown as Fig.7

Figure 5: Input Figure 6: Output

Figure 7: Example of the Input and Output

In short, our current model can better identify which area in the image belongs to straw-berry, and can be passed to the next link for the implementation of grabbing.

However, due to other factors such as strawberry color, we can not determine whichstrawberries are most worthy of picking. Therefore, in the future work, we will combine thedepth information of the image to improve the strawberry harvesting robot.

Please refer to the appendix for the brief contents of our work for paper and the summa-rized U-net network.

3

3 Appendices

3.1 Part 1

Experimental images are obtained by photographing strawberries in Chongming, Shang-hai, China. The varieties of strawberries we choose are diverse, including “Hongyan”,“Zhangji”,etc. Also, their grades of maturity and extents of overlapping are different. Ingeneral, we have taken 593 photos of the strawberry at short range and 101 ones at longrange and made them into data sets for training and testing. Besides, the pictures are storedin JPEG format, whose resolution is 640×480.

3.2 Part 2

Then, in order to facilitate identification, we process the images into PNG format with aresolution of 480×480. In addition, we take 515 of the images at short range as training sets,and the remaining 78 ones at short range and 101 ones at long range as test sets. Through theimage annotation tool Labelme, we annotate the experimental data and generate the maskimages of the strawberry. These mask images are then used to train the model and optimizeof the model parameters. We select batch size as 8 and train 515 pictures in 40 epochs.Then strawberry test set is tested with the model, and fairly good results are received. Wecalculate that the loss in our training set is 0.026, the loss in the close view test set is 0.016,and the one in the distant view test set is 0.011.

3.3 Basic structure of U-net network

The network structure we use is deep learning segmentation network U-net, which usesFCN network for reference. Realized by pooling, the first part of the U-net is the featureextraction, which is similar to the convolution network or VGG. The second part is the uppersampling part, which is symmetrical with the front part and achieves the goal of imagesegmentation by deconvolution. In addition, feature fusion is also used in the network.To achieve better segmentation effects, the features in the former part of the lower samplingnetwork and the latter part of the upper sampling network are fused to obtain more accuratecontext information.

3.4 Features of U-net

U-net is a classic network design method and has a lot of applications in image segmen-tation tasks. The feature of it is to combine shallow features with deep ones by cascading inthe process of upsampling. For images, U-net can use deep features for location and shallowfeatures for accurate segmentation.

3.5 Difficulties in Strawberry Recognition

In our preliminary strawberry recognition, we found that the shape of strawberry ismostly unique and strange, and some strawberries are shielded from each other. At thesame time, the strawberry is in a complex environment, surrounded by strawberry leavesand low shrubs. Based on above characteristics, we think U-net is suitable for strawberryrecognition.

4