automatic determination of body condition score … · automatic determination of body condition...

Automatic Determination of Body Condition Score of Dairy

Cows from 3D Images

Processing and pattern recognition in images from a time-of-flight camera

M A R I L Y N K R U K O W S K I

Master of Science Thesis Stockholm, Sweden 2009

Automatic Determination of Body Condition Score of Dairy

Cows from 3D Images

Processing and pattern recognition in images from a time-of-flight camera

M A R I L Y N K R U K O W S K I

Master’s Thesis in Computer Science (30 ECTS credits) at the School of Engineering Physics Royal Institute of Technology year 2009 Supervisor at CSC was Stefan Carlsson Examiner was Stefan Carlsson TRITA-CSC-E 2009:009 ISRN-KTH/CSC/E--09/009--SE ISSN-1653-5715 Royal Institute of Technology School of Computer Science and Communication KTH CSC SE-100 44 Stockholm, Sweden URL: www.csc.kth.se

Abstract

The increase in number, and decrease in cost for devices giving digital three-dimensional images have implied that the number of application areas where three-dimensional images can be used has increased lately. Body condition scoring (BCS), an indirect measure ranging from one to five of the level of subcutaneous fat in dairy cattle, is widely adopted for research and field assessment or for management purposes on farms. The feasibility of using statistical measures in three-dimensional images to automatically determine the body condition score was examined at DeLaval and Hamra Farm in Tumba, Sweden. This master’s thesis describes a method for processing three-dimensional images of cows by automatic localisation of anatomical reference points. Points around the hook bones and following the spinal ridge were easy to identify. A number of parameters describing local feature statistical properties were examined to identify the ones that best describe the body condition score of dairy cattle. When the full data set testing a model combining the most significant histogram and Fourier parameters was used, 100% of predicted BCS were within 0.5 points of actual BCS and 79% were within 0.25 points. This research demonstrates the potential for using three-dimensional images for assessing BCS. The automatic method is particularly good at following BCS-fluctuations in time. Future efforts should explore ways to minimise the spread of the measured parameters and identify significant anatomical parameters that can be automatically extracted.

Sammanfattning

Automatisk bedömning av hullpoängen hos mjölkkor ur 3D-bilder

- Bildbehandling och mönsterigenkänning i bilder från en time-of-flight kamera

Ökningen av och det allt lägre priset på mätinstrument som ger digitala tredimensionella bilder har inneburit att antalet användningsområden där tredimensionella bilder kan nyttjas har ökat på senare tid. Att poängsätta hullet (eng. body condition scoring), vilket är ett indirekt mått på en skala från ett till fem av mängden underhudsfett hos mjölkkor, är vitt använt i forskning och bedömning ute i fält likväl som för skötsel av djurdriften på bondgårdar. Möjligheten att använda statistiska mätningar i tredimensionella bilder för att automatiskt fastställa hullpoäng undersöktes på DeLaval och Hamra Gård i Tumba, Stockholm. Denna rapport beskriver en metod för att behandla tredimensionella bilder av kor genom automatisk lokalisering av anatomiska referens-punkter. Punkter runt höftbenen och följandes ryggraden fanns vara lätta att identifiera. Ett antal parametrar beskrivandes statistiska lokala egen-skaper undersöktes för att identifiera de som bäst beskrev hullpoängen hos mjölkkor. När hela datamängden testades i en modell som kom-binerade de mest signifikanta histogram- och Fourier-parametrarna, låg 100% av de beräknade hullpoängen inom 0.5 poäng från det verkliga värdet och 79% låg inom 0.25 poäng. Denna undersökning visar på potentialen i att använda tre-dimensionella bilder för att bedöma hullpoäng. Den automatiska metoden visar sig särskilt bra på att följa hullpoängens fluktuationer över tid. Framtida försök bör undersöka sätt att minimera spridningen i parametervärden och identifiera signifikanta anatomiska parametrar som kan mätas automatiskt.

Table of contents

1 Introduction 1

1.1 Thesis overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

2 Cow body condition scoring 4

2.1 Body condition scoring method. . . . . . . . . . . . . . . . . . . . 6 2.2 Automatic body condition scoring using 2D tools . . . . 8 2.2.1 Back view determination of BSC by Leroy et al.. 9 2.2.2 Top view determination of BCS by Bewley et al. 10 2.3 Conclusions from earlier work. . . . . . . . . . . . . . . . . . . . . 12 3 Data acquisition setup 13

3.1 Data sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.1.1 Training set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.1.2 Validation set . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

3.2 Time-of-Flight camera for 3D images . . . . . . . . . . . . . . . 16 3.2.1 Camera data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3.2.2 Time-of-flight principle . . . . . . . . . . . . . . . . . . . . 19

3.3 Testing for optimal integration time . . . . . . . . . . . . . . . . 21

4 Methods and algorithm 24

4.1 Image pre-processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.1.1 Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 4.1.2 Segmentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

4.2 Normalisation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 4.3 Feature extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

4.3.1 Image texture analysis . . . . . . . . . . . . . . . . . . . . . . 32 4.3.2 Measures for texture analysis. . . . . . . . . . . . . . . . 33 4.3.3 Examining features in image data . . . . . . . . . . . . 35

5 Results and discussion 37

5.1 Choice of parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 5.2 Model development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

5.3 Test on new data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 5.4 Algorithm robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

6 Suggestions for future work 64

7 Summary and conclusions 67 Bibliography 70 Appendix A – Body condition scoring method 73 Appendix B – Cows used for data acquisition 78

Acknowledgements

There are many people who helped me in my master’s thesis project. First and foremost, I would like to thank my thesis advisor at DeLaval, Bohao Liao. Bohao guided me through this work, and with his constant enthusiasm concerning the project, kept me motivated. The field of computer vision was novel to me at the start of my work. I want to thank my thesis advisor at KTH (The Royal Institute of Technology, Stockholm), Stefan Carlsson, for the reassurance during the project, keeping me on track. During my thesis work, my colleagues at DeLaval provided needed advice on many topics, and their knowledge of the dairy industry greatly improved its final form. In particular, Pierre DeVilliers and Stefan Bergstrand taught me everything about cow body condition scoring and devised the system for scoring used in my work, and Kerstin Vollmer at Hamra Farm provided with the actual scoring and helped me chasing down some difficult cows. Finally I give my gratitude to 151 Valeriana, 193 Valeriana, 194 Göta, 221 Grevinna and 291 Valeriana. These lovely four-legged ladies helped me with their patience and calmness to obtain the data needed for my work.

1

Chapter 1

Introduction

How can new technology help in increasing the health of dairy cows and aid in heard management? Combining 3D-image processing

and local feature statistical techniques gives a model for an automatic body condition scoring system that can lead to a

secure method for tracking the health of the heard.

Many dairy producers have cattle that are too fat or too thin for their stage of lactation. Failure to recognize these cows and take action costs dearly for disease treatments, losses in milk production, and decreased fertility. As a preventive measure, body condition scoring (BCS) has long been a powerful tool reflecting the fat reserves carried by the animals. In practice, body condition scoring is accomplished by visual and tactile assessment of a cow by a trained evaluator. Consequently, the scoring is expensive, prone to subjectivity and cannot be performed on a regular basis due to the time commitment required. On a practical note, there is no consensus on what constitutes desired body condition for a dairy cow. Dairy scientists have not yet developed the necessary objective research to be able to advice farmers properly. Therefore, there is a need to develop methods to determine the body condition score of individual cows in an automatic way, which would be more cost effective, objective and easy to connect with data from a heard management system.

One way to do so is by using a time-of-flight (TOF) camera which gives three-dimensional information about the body shape of the cow, and combine it with image analysis techniques. The advantages of such an approach are that it does not make contact with the animal, causing no additional stress, and that shapes are detectable regardless

CHAPTER 1. INTRODUCTION

2

of the movement, background and position of the cow. The objective of this master’s thesis is to develop and evaluate a prototype algorithm for an image analysis system that could objectively determine the degree of fattening in dairy cows using this type of camera. For the development, the first challenge is to identify and concretise the shape parameters in the geography of a cow that indicate its body condition. These are placed in the hip and rump area of the cow and over the back in proximity to the hip bones. The locations of the parameters give the optimal angle from which to capture images of the animal to maximize the received information. In my system, surface shape is described by a sparse collection of 3D points. The image analysis algorithm identifies constant features of the cow in order to locate the animal. The three-dimensional data is then processed as to normalize the images and correct for posture and size of the animal, thus making the indicating parameters comparable. Recognition of free-form objects and specific shapes from range data is a challenging problem. Even when only one object or a well defined surface is present in the image, most real range images contain erroneous values from image noise and are sensitive to the viewing angle. A practical cow body condition scoring system that work with range images must therefore be robust to noise and image angle. Earlier approaches to automatic BCS use methods based on contour lines and angles at certain bony areas of the animal. Those methods obtain good results on controlled images, but the anatomical points that indicate the body condition score do not necessarily correspond to an obvious visual clue. On colour and gray-value images, whole-image approaches to object recognition have been very successful, using statistical methods such as local feature histograms [1]. This is a motive for exploring how local feature statistical measurements from range images can be used for efficient cow body condition scoring. With this master’s thesis I want to show the potential in a camera-based system for predicting the body condition score of any given cow. An automated body condition scoring system would be preferred to observational scoring because it would require less time, be less stressful on the animal, be more objective and consistent, and possibly be more cost effective. Such a system could be an important complement in a precision livestock farming system, as the demand is increasing with increased farm sizes.

CHAPTER 1. INTRODUCTION

3

1.1 Thesis overview This thesis presents a new method for evaluating the body condition score in dairy cattle automatically using a time-of-flight camera.

• In chapter 2 an introduction to cow body condition scoring is given. The importance of scoring animals is explained. I also summarise methods and results from earlier efforts to create digital ways of scoring.

• Chapter 3 describes the data and the data acquisition setup

used. It describes how the time-of-flight camera works to give reliable range information of a cow.

• In chapter 4 the algorithm developed for three-dimensional

cow-image processing and prediction of the body condition score is explained. Using the three-dimensional images obtained from the time-of-flight camera, shape parameters are found, which are relevant to the body condition score.

• In chapter 5 I describe how the region of the cow body and the

parameters that could be used for predicting the body condition score are selected. This is done by analysing three-dimensional images and parameter plots, most of which are present in the chapter. The performance of the algorithm is evaluated using image data from a test farm.

• Chapter 6 present some suggestions for future work and in the

conclusions in chapter 7 I discuss the potential in the utilised methods and equipment.

4

Chapter 2

Cow body condition scoring

By manual and visual examination a diary cow can be given a score from one to five that indicates the extent of the fat

reserves of the animal, thus creating a possibility to monitor its health and production capacity.

Body condition scoring is a method of evaluating fatness or thinness in cows according to a five-point scale, where a score of one denotes a very thin cow, while five denotes an excessively fat cow. Research and field experiments by Wilderman et al. [2] have shown that body condition influences productivity, reproduction, health and longevity. Thus, thinness or fatness can indicate underlying nutritional deficiencies, health problems, or improper heard management. When performed on a regular basis, body condition scoring is a good aid in improving the health and productivity of a dairy herd. As a mean to detect problems within the herd, it acts as an efficient tool in herd management. Body condition scoring is better for monitoring body energy reserves than body weight. The body weight can change due to changes in body fat, frame size, gut size, udder size and intake of food and water. The change can come up to as much as 15% of the cows’ weight during the course of one day only from water intake. The body condition of a normal, healthy, cow fluctuates over the lactation periods, as can be seen in figure 2.1 [23]. At calving, a re-commended body condition score is 3.25 to 3.75. At early lactation the cow increases the production of milk until peak milk production is reached. Food intake will lag behind requirements in the first six to eight weeks of lactation. During this period the cow uses its body

CHAPTER 2. COW BODY CONDITION SCORING

5

reserves to get the energy necessary to produce milk. The goal is to have a loss in the body condition of 0.5 to 0.75 in early lactation. At mid-lactation the body condition score should slowly increase to reach the same recommended value at calving at the end of late lactation. It is important not to attempt to correct the body condition of the cow during the dry period before calving as this will affect the weight of the calf more than the weight of the cow.

Figure 2.1: Normal fluctuation of body condition score over different lactation periods in

dairy cows.

Over-conditioning, or fatness, usually begins during the last three to four months of lactation, when milk production has decreased, but grain and total nutrients levels have not been reduced accordingly. At the time of calving, a cow with a body condition score over 4.0 often results in reduced feed intake and increased incidence of peripartumi problems and other difficulties at calving. A fat cow is more susceptible to metabolic problems and infections. Over-conditioned cows tend to have problems with retained placenta, gastroparesisii leading to calcium deficit, fat cow syndrome, fatty liver and mastitisiii. They might even collapse under their excessive weight. Under-conditioning, or thinness, occurs when a cow has been ill for a longer period, or if not enough energy has been added to the diet

i Occurring in, or being, the period preceding or following parturition, the act of giving birth. ii Partial paralysis of the stomach. iii Inflammation of the parenchyma of the udder.


6

during mid- and late lactation. Under-conditioning at calving with a body condition score of less than 3.0 often results in lower peak milk yield and less milk for the entire lactation. Moreover, it is a health risk in the early lactation when the cow uses much of its body reserves. Also cows should not lose more than 1.0 body score during early lactation as excessive loss of body condition in early lactation has been shown to reduce reproductive efficiency. Under-conditioning frequently lowers production and milk fat levels because of insufficient energy and protein reserves. Thin cows often do not show heativ or conceive until they start to regain – or at least maintain – body weight. In feeding these animals, care must be taken to maintain production while increasing body reserves.

2.1 Body condition scoring method

Wildman et al. developed a five-point scoring system to measure the relative amount of the subcutaneous body fat. Most body condition scoring systems in dairy cattle use the five-point scoring system with quarter point increments. Instructions for a body condition scoring system have been devised at DeLaval to asses the body condition of a dairy cow at any point during the production cycle. For accurate sco-ring, both visual and tactile appraisals of back and hind quarters are necessary. The parts considered are the thoracic and lumbar regions of the vertebral column (chine, loin and rump), spinous processes (loin), tuber sacrale (hooks), tuber ischii (pin bones), and anterior coccygeal vertebrae (tail head) which are shown in figure 2.2. A single factor may be misleading; however, all factors considered together provide an accurate score. Each condition score has been assessed in this thesis project by the criteria simplified in table 2.1. The method is subjective to the scorer, to the comparison between similar cows and the general impression of the cow. The difference between two expert scorers is of the order of magnitude of 0.25 points. It is also normal for the same scorer to score the same animal differently, with a discrepancy of 0.25 points, when she or he returns to the cow after scoring the entire herd.

iv A period of increased sexual drive in the biological estrous cycle


7

Figure 2.2: The areas considered when scoring the body condition of a cow [3]

Table 2.1: Simplified body condition score chart [4].

For an extensive description, see appendix A.

Body condition score Vertebrae at the middle of the back

Rear view (cross section) of the hook bones

Side view of the line between the hook and pin bones

Cavity between tail head and pin bone Rear view and angled view

1. Severe under-conditioning

2. Frame obvious

3. Frame and covering well

4. Frame not as visible as covering

5. Severe over-conditioning


8

2.2 Automatic determination of the body condition scoring using 2D tools

Although the benefits of regular body condition scoring are intuitive to most dairy producers, nutritionist, and consultants, relatively few dairy farms have incorporated it as a part of their dairy management strategy [5]. There are many reasons for the lack of adoption of this system, mostly related to its subjectivity and the time commitment required. It is hardly practical in a computerized heard management system. These concerns have led to a search for alternative means of assessing body energy reserves in cattle. Despite the simplicity of the idea, few research groups have published attempts to automate cow body condition scoring.

• Pompe et al. (2005) [6] used black-and-white photography and a line laser to collect a series of images from the rear of the cow. A three-dimensional analysis of the images provided an outline of the left pin, left hook, and tailhead. No statistical analysis comparing image data with BCS was reported.

• Leroy et al. (2005) [7] used ordinary two dimensional images

from the rear of the cow, to obtain a silhouette image. Their study shows that it is possible to evaluate the body score automatically with an accuracy of the result at the same order of magnitude as the error of human evaluation reaching 0.25 points between scorers.

• Some extensive work on automated body condition scoring for

dairy cattle was conducted by Coffey et al. (2003) [8] at the Scottish Agricultural College. Light lines were created on the back of the cow by using a red laser light shone through a prism. The camera was positioned at a 45˚ angle to the horizontal plane of the cows back and the laser lines were used in manual extractions of curvatures over the cow’s tailhead and buttocks. The curvature of these shapes was then modelled. In their study they found a large correlation, with a correlation coefficient of 0.55, between the tailhead curvature and observed BCS, whereas the correlation coefficient of the curvature of the right buttock as measured across the pin bone was 0.52.


9

• The most extensive study was produced by Bewley et al. (2008) [9] together with collaborators from IceRobotics Ltd. Using digital images captured from above; the angles produced by the hook bones were extracted from a contour image. 99.89% of the automatically extracted body condition scores were within 0.5 points of human score, and 89.95% were within 0.25 points.

• In the study of the body condition score of Mediterranean

buffaloes using ordinary two dimensional image analysis, Negretti et al. (2008) [10] confirmed that computerized image analysis is an effective measuring system. The Italian group also reached important conclusions showing that the automatic measurements of the angle between the back and the hook bones, together with measurements of the surface area behind the hook bones, where significantly correlated to the body condition score.

In these earlier studies, contours of the cows were measured by digital two-dimensional images captured from behind and from above the animals. The two image views contain complementary information that has been processed to function as indicators of the body condition score of dairy cows.

2.2.1 Back view determination of BSC by Leroy et al.

In their study 75 back-view images of thirty six Holstein dairy cows were used to develop an algorithm for predicting body condition score. The images were captured with a 3.34 megapixel digital camera once a week during six months at a distance of 1.5 to 2 metres so that the images contained the complete silhouette from legs to tail. Every time an image was captured, a human expert also determined the body condition score of these cows using the Wildman scoring scale ranging from 1 to 5, with intervals of 0.25 units. From the digital image of the rear of the cow, a binary silhouette of the cow was created by segmenting pixels belonging to the cow. Naturally, images were captured from different distances, there was a difference in size between the different cows, and there were differences in posture of the animals. For that reason, the images had to be transformed into the same reference system. Points at the contour corresponding to clear visual features such as the tail and the hips were identified and used for transformation by translation, rotation


10

and scaling. From the reference points 19 contour points at equal distance were used to describe the contour. A data matrix (Z) with the contour xy-coordinates and BCS values for each cow, times a set of 39 reference cows was constructed. By applying principal component analysis, the matrix of eigenvectors (V) was found. To determine the BCS of a test cow the vector containing the contour xy-coordinate and the unknown BCS value (K) vas assumed to be a sum of a vector containing average cow values at each position and a linear combination of the eigenvectors (see equation 2.1). By applying the least squares method on the parameters of the linear combination, the unknown BCS value could be predicted.

=

=

=

K

nK

K

BCS

xy

BCS

z

z

K

KK

...

1

=

++

+

= k

k

nk

k

nn

c

BCS

dz

dz

c

BCS

dz

dz

BCS

z

z

......

......

1

1

1

1

111

CVZCVZ

CVZ

BCSBCS

xyxy

•+=

•+

•+= (2.1)

The developed method was tested on a test set of images of 32 cows. It is not reported whether these cows were included in the development of the model. Of the resulting predicted BCS 42.86% of the scores were within 0.25 points of actual score, and 90.63% were within 0.5 points. The deviation between the BCS-values determined by the camera-based method and actual score was on average 0.27. A good result when the error of two humans scoring the same cow is in the order of magnitude of 0.25 units.

2.2.2 Top view determination of BCS by Bewley et al.

The feasibility of utilizing digital images to determine body condition score was assessed for lactating dairy cows at the Scottish Agricultural


11

College Crichton Royal Farm. Digital images were captured with a stationary camera from above at approximately 60 to 70 cm from the cows’ back. Simultaneously the cows were scored continuously using both the Lowman/Mulvany system, and the more widespread Edmonson/Ferguson system of scoring. Up to 23 anatomical points were manually identified in each image in order to generate a digital contour of the cow. All identifiable points around the hook bones, the pin bones, and the tailhead were used to calculate angles reflecting the shape of the contour. Correlations were calculated between BCS and the extracted angles, where the largest correlations were observed at the hook angle (HA) with a coefficient of 0.48 and at the hook posterior angle (PHA), measured in the indentation behind the hook bones, with a coefficient of 0.52. A mixed, semi linear, model of these two angles was defined as

ijijijiij PHAHAcPHAcHAccowk )(321 ×++++= µ (2.2)

where kij is the jth BCS of cow i; µ is the intercept, and cowi is a cow dependent constant for the ith cow. By regression analysis, the constants were determined for prediction of unknown BCS. In creating the model, different models were performed as a repeated measures analysis, with the cow as a random subject and first-order auto-regressive as the covariance Images were captured daily of 242 different cows, but because of lighting and setup limitations, usable images were available only for 46 of 61 days. The average number of useful images per day was 73, 30% of the possible maximum. The accuracy and robustness of the model was determined by analysis of the entire data set. A difficulty in the study was the potential error in identifying the anatomical points of interest. The human eye and hand are subject to some degree of error. As previous research has focused primarily on images of the rear of the cow, the authors suggest combining the top and rear view aiming for a three-dimensional view of the animal.

12

2.3 Conclusions from previous work The demand for an accurate and frequent scoring of the body condition of dairy cows is increasing. As manual scoring is both time consuming, expensive and inconsistent, there exists a motivation to create an automatic visual method. Efforts have been made to examine the possibility to score animals with digital cameras with positive results. Unfortunately these me-thods are not ready to be automated as they require manual identify-cation of reference points due to the changes in the silhouette caused by different postures of the animal. There is also the problem of the large amount of useless images produced when the dark animal blends into the dark background or when the animal stand in such a way as to exclude the interesting area. The difficulties that arise when using 2D image data could easily be avoided by adding the range information in a 3D image. The idea is to use the range information to identify a dark animal in a dark scene, conditions that are very common in the natural environment of a band. The three-dimensional information should also make it easier to iden-tify anatomical reference points. The earlier studies have shown that the spinal ridge, the hook bones and the tail head are reliable points of reference, hence I will concentrate on these areas. These additions should make it possible to show the potential for automating the visual body condition scoring.

13

Chapter 3

Data acquisition

Sixteen Swedish red breed cows were photographed and five were followed continuously, using a time-of-flight camera to create a

database of 3D images used for the algorithm development.

The three-dimensional images were captured with a Mesa Imaging 3D camera. To complement the 3D images with ordinary photos an 8.0 megapixel Canon IXUS 80IS digital camera was employed. The Mesa camera was connected to a portable laptop computer that had been placed on a mobile table, to make it easier to move the equipment around in the barn. Photographing was executed manually, standing on a plastic stool to reach the appropriate height. To use a hand-held setup was considered necessary at this point because the cows were only available in a free-standing position. By holding the camera I could follow the larger movements of the cow to avoid it leaving the image field. Two viewing angles were found to be good candidates. One gives the information from the dorsal and posterior parts of the cow, including the spinal processes, the hook bones, the pin bones and the tail head, as seen in figure 3.1. The second gives a lateral/dorsal image showing the area between the pin bones and the hook bones, and the edge of the spinal processes. From personal experience I found it to be to risky to photograph from the side because the cows are easily frightened and tended to kick the stool I had for standing upon. Figure 3.2 illustrates the entire data acquisition setup.

CHAPTER 3. DATA ACQUISITION SETUP

14

Figures 3.1: View from the back (left) and side (right) of the cow. Only the back view was

finally used in the project.

Figures 3.2: The setup when collecting the images with me and a cow in the research-barn.


15

3.1 Data sets

Two sets of data were collected by the same procedure. One larger set of images to be used for method development, method evaluation and for the development of a model for predicting the body condition score of the cows. A separate set of data vas collected to be used for validation of the methods and models developed. The two sets of data are summarized in table 3.1: Table 3.1: Summary of the two datasets used in method development and evaluation

Training set Validation set

No of cows 16 24

No of images 351 120

Lactation status of cows 0-3 months from calving Pregnant or recently calved

Scorer DeLaval expert scorers on first observation and Hamra farm scorer on all subsequent observations

DeLaval expert scorers

Observation period 17.6.2008 - 17.9.2008 7.10.2008

3.1.1 Training set

Twenty nine cows standing in the tied-up barn at Hamra DeLaval in Tumba were selected for scoring. Eleven of them were black Swedish lowland breed (SLB), commonly known as Swedish Holstein, and eighteen were Swedish red breed (SRB) cows. Twenty were photo-graphed initially due to their availability. For technical reasons due to contrast errors between black and white only pictures from sixteen Swedish red breed could be used for developing the image analysis algorithm. In order to follow the fluctuation of body condition score of individual cows, five cows were chosen to be kept in the tied-up barn over a period of three months for regular scoring and photographing every second week. The five cows were chosen by their spread in body condition, ranging from 2.75 to 4.25 at the time of the first photographing session. In coincidence with photographing, the body condition of the five cows was scored by two expert scorers at DeLaval on the first occasion when all twenty nine animals were scored, and by Kerstin Vollmer at Hamra Farm on all subsequent occasions. The list of cows used for data collection, the date they were photographed, and their score is found


16

in table B.1 in appendix B. The number of animals used is not statistically retrieved, but based solely on availability of cows in the tied-up barn. The cows were photographed at six occasions. Each “photograph” consists of five images captured in sequence, and at each occasion an average of two photographs were recorded for each cow. At the two latest occasions, the cows had been transferred to the VMSv-barn and the test barn, where cows run loose and I had to find innovative ways of positioning myself and the animals. The cows were scored at a maximum of three days from photographing at five of these occasions. Missing scores for images captured 23rd of July 2008 were retrieved for model development by interpolation from the scores two weeks before and after.

3.1.2 Validation set

For validation of the methods developed in this project, I needed a number of fresh cows not used in the training. Twenty nine Swedish red breed cows were photographed for scoring in the tied up barn on the 7th of October 2008. Each cow was photographed once, resulting in five images for each animal. On the subsequent day, scoring was performed by the two expert scorers from DeLaval (see table B.2 in appendix B). Additionally eleven cows were photographed, but not scored and not included in the study. Four were heifers and thus not yet fully developed to be scored as mature cows, and three had been moved the next day when the animals were scored manually. Four Swedish Lowland breed cows were also photographed for curiosity.

3.2 Time-of-Flight camera for 3D images

For this project I used the Mesa Imaging AG® Swiss Ranger SR-3000 sensor (figure 3.3). It is a complete solid-state Time-of-Flight range camera developed by CSEM (Centre Suisse d’électronique et de microtechnique) [11]. It is connected to a computer via USB 2.0 for direct measurement of real-time depth maps and is designed for operation under indoor lighting conditions.

v Voluntary Milking System


17

Figure 3.3: Photograph of the SR-3000 camera used for 3D image acquisition

3.2.1 Camera data

The camera is based on a two dimensional image sensor with a field of view of 47.5 x 39.6 degrees with a spatial resolution of 176 x 144 pixels using a CMOSvi active-pixel sensor. The technology is very similar to CCDvii-technology. The resulting output is a four dimensional representation of the view showing the intensity information at each pixel, in correlation with an ordinary digital camera. In addition, each point's relative position to the camera is given with its x, y and range (z) value, as seen in figure 3.4 and 3.5. When combining the infor-mation in the x-, y- and z-channels, it is possible to create a three-dimensional visualization of the scene. The distance measurement is based on the time of flight principle by infra red light emitters and sensors. Due to the difference in absorption and reflection on white and black, areas with great contrast in colour can be registered to be at different distances, producing significant errors. Setting the amplitude threshold, noisy pixels can be filtered if desired. The amplitude determines the amount of emitted light that is reflected back on the pixel. The integration time controls the exposure time for the acquired image and can be varied from 0.2 to 51.2 ms where the value ‘0’ corresponds to an integration time of 0.2 ms and ‘255’ to 51.2 ms respectively.

vi Complementary metal–oxide–semiconductor vii Charge-coupled device


18

Figure 3.4: The 176 times 144 pixels record the space coordinates x, y, z

and IR light intensity when capturing an image

Figure 3.5: Binary representations of the four resulting matrices

obtained when capturing an image

The images are combined into a three-dimensional representation, demonstrated in figure 3.6 as a point cloud. It should be noted that the original data is very noisy. The small fluctuations in the surface are caused by natural shakiness in my hand that holds the camera, movements of the animal and inaccuracy from the short integration

0 144 0 144 0 144 0 144

1

76

0

1

76

0

1

76

0

1

76

0


19

time used when taking the picture. Distant areas in the scene, e.g. barn details in front of the cow, create the flares seen in the image that can confuse the viewer and complicate the analysis. Very distant points create erroneous small-valued z-values appearing as peaks very close to the camera. Because of some bouncing effect, protruding areas such as the tail, the spine or areas with coarse fur show peak-shaped noise.

Figure 3.6: Point cloud of original non-processed data depicting the back of a cow.

Distant points and noise make it difficult to see the animal. The scale is in metres.

3.2.2 Time-of-flight principle

The time-of-flight (TOF) describes the method used to measure the distance to an object. It is based on the time that it takes for the infrared light with a peak wavelength of 850 nm to reflect on the object and reach the sensor while travelling at the speed of light. The functionality is the same as for sonar technology. When using pulsed light, the high velocity of light implies that the temporal accuracy must be very high – e.g. 1 ns for a spatial accuracy of 15 cm [21] The measured distance is proportional to twice the time needed for the waves to travel from the camera to the object. Image 3.7 illustrates the principle behind the distance measurement. At shorter distances for a detailed view, what is actually measured is a phase shift between the


20

outgoing signal and the detected reflected signal for a continuously modulated signal instead of separated pulses. The reflected light is monitored, and the phase of the modulation is compared with that of the sent light. The phase shift obtained is 2� times the time of flight times the modulation frequency (see figure 3.8). This in turn gives the distance to the object.

Figure 3.7: The time-of-flight principle of determining range images [22].

Figure 3.8: Determination of the distance from frequency

modulation and the resulting shift of phase.


21

3.3 Testing for optimal integration time

The integration time in the Mesa system is comparable to the shutter speed (or exposure time) in ordinary cameras. In general, images captured with a short integration time risk being very noisy whilst a long integration time can cause motion blur in live images. It is important to get as detailed, noise free, data as possible for optimal image analysis. Unfortunately it is very likely that the animal will move slightly while photographed, and since the camera is hand-held, motion blur gives an upper limit on what integration time is realistic. At long integration times there is also a risk of saturation in the image. Determination of the optimal integration time was made by observing the images captured from the first photo session. A total of 1440 images were captured with a spread in integration time from 10 to 200. Range images captured with integration time 10, 20, 50, 100, 150, 200 can bee seen in figure 3.9. All pictures captured with integration time 50, corresponding to a time of 10 ms, or less were acceptable, even though they were noisy at an integration time of 10 and 20. 10% of the pictures captured with integration time 100 were of a poor quality as were 20% of the pictures captured with an integration time of 150.

This observation was complemented by a test of the distortion and saturation in the image. The camera was mounted at a height of 10.5 centimetres from a table and at 60 centimetres from a white wall with a cross made out of adhesive tape to the vertical and horizontal. As figure 3.10 shows, an integration time of 50 gives images with less distortion in this setup. My assumption was that I was going to take the pictures from a distance of about 60 centimetres from the cows, and from these results I could roughly say that an integration time of 50 would give acceptable image noise, motion blur and distortion.


22

Figure 3.9: Depth images of cow 194 captured with, from top left to bottom right,

integration time 10, 20, 50, 100, 150 and 200.

10 20

50 100

200 150


23

Figure 3.10: Three-dimensional images of the corner between a wall with a cross and the

table captured with, from top left to bottom right, integration time 10, 20, 50, 100, 150 and

200.

10 20

50 100

200 150

24

Chapter 4

Methods and algorithm description

Processing and normalizing the images makes it possible to extract parameters that define the fatness of a cow. The parameters are used in the development of a model for prediction of the body condition

score.

Figure 4.1: Point cloud of original data showing disturbing noise and unwanted values

25

4.1 Image pre-processing

The camera produces raw data that need to be processed in order to be available for analysis. The data is noisy, see figure 4.1, with erroneous peaks, disturbing objects such as neighbouring cows and a background that has a large diversity in distances. The distance error leads to peak shaped objects appearing very close instead of very far away. Hence, not all pixels in the x-, y- and z-images belong to the cow. For analysis the data need to be filtered and passed through segmentation.

4.1.1 Filtering

The filtering function combines the two-dimensional intensity image with the range information to deal with objects that are so far away as to give misleading depth information. Objects that are far away reflect light with lower intensity than close objects. Pixels with intensity below a chosen value (500) can therefore be located and their depth value set to a distance of about the distance at the cow’s neck. In the case of images captured in this work, this distance is set at 1.6 metres from the camera. The erroneous points would hence not interfere in the regular filtering. To filter the processed image a strong seven by seven averaging filter was found optimal by visual examination of a range of cow images filtered using different filters. An averaging filter estimates the local mean and variance around each pixel (x1, x2),

∑∈

=η

µ21 ,

21 ),(1

nn

nnaNM

(4.1.1)

∑∈

−=η

µσ21 ,

2

21

22 ),(1

nn

nnaNM

(4.1.2)

where η is the N-by-M local neighbourhood of each pixel in the image. To get a strong filter that would reduce the peak-shaped noise on the surface of the cow, I chose a neighbourhood of seven by seven pixels. The function then creates a pixelwise filter using these estimates,

)),((),( 212

22

21 µσ

νσµ −

++= nnaxxb (4.1.3)

26

where 2ν is the noise variance. By default, the noise variance is set to the average of all the local estimated variances. The result of the averaging filter can be seen in figure 4.2:

Figure 4.2: Point clouds of original unfiltered (left) and filtered (right) data.

The image shows the profile of the rump from the hook bones to the

pin bones and the protruding tailhead furthest down in the image.

The last step in the filtering process is identifying all pixels where the range value is above a chosen limit, thus becoming uninteresting for analysis. These points normally belong to the interiors of the barn, the head of the animal and neighbouring cows. Their value is set to the distance of the cow’s neck. The entire filtering process is depicted step by step in figure 4.3:

27

Figure 4.3: From above, left: Range images of original data, intensity-filtered data,

average filtered data, and finally; range-filtered data.

4.1.2 Segmentation

In computer vision, segmentation refers to the process of partitioning a digital image into multiple regions (sets of pixels). The goal of segmentation is to simplify or change the representation of an image into something that is more meaningful and easier to analyze. Image segmentation is typically used to locate objects and boundaries (lines, curves, etc.) in images. In my case, I use it for identifying the posterial and dorsal surface of the cow. As a primary segmentation, a sensitive Canny edge detecting function is applied to the filtered data. The Canny method finds edges in binary images in the following way:

• A gradient image is calculated using the derivative of a Gaussian filter.

28

• In the gradient image, local maxima is located by comparing each pixel gradient with its two neighbouring gradient values in the gradient direction. Pixels that are smaller than at least one neighbour are suppressed and set to zero in the binary image.

• The method uses two thresholds, to detect strong and weak

edges. Any pixels with a gradient below the lower threshold are removed. All pixels above the higher threshold are considered as strong edges. The weak edges are the pixels where the gra-dient lies between the two thresholds. Only if they are connec-ted to a strong edge pixel they are included in the output, otherwise, if the pixel is isolated from any strong pixel, it is set to zero.

This method is therefore less likely than other simple edge detecting methods to be fooled by noise, and more likely to detect true weak edges [12]. By opening and closing, using the ready Matlab functions imdilate.m, imfill.m and imerode.m, only the points belonging to the surface of the cow are included.

Figure 4.4: Segmentation of data isolating the surface of the cow

-0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 -0.5

0 0.5 0.7

0.8

0.9

1

1.1

1.2

1.3

1.4

1.5

Filtered and segmented image

29

4.2 Normalisation

When comparing measurements, what is needed is a reference point. For instance, if one wants to compare the distance to all European capitals, one need to decide from which point to do so. If you choose Paris, it might happen that not all your measurements are made from Paris, and these measurements must be adjusted with an appropriate factor to make all distances comparable. Or, when comparing the amount of money nations spend on healthcare, you might want to cal-culate the percentage of the GNP (Gross National Product). This is the normalisation of data to make it statistically comparable. As noted, there are two challenges that are faced when normalising data. Finding a good reference point, and finding the proper normalisation factor to make the images as independent from other influence as possible. In image analysis, normalisation is also important. Leroy et al. [7] used the manually extracted points on the hips as reference and could scale, translate and rotate the image in en effort to make the image data inde-pendent of cow size, position, posture and distance to camera. Bewley et al. [9], on the other hand, computed the angle over the manually lo-cated hips. An angle is an independent measure and does not need normalisation. But they used the quality of human vision to easily find the hips as their reference point. As most feature recognition methods are based mainly on geometric information, in 3D techniques it is even more important than in 2D to carry out a good normalization of the object. In order to normalize the cow, the first step is to find the feature points, which will serve as refe-rence points for the normalization process. The feature points conside-red in earlier work for normalization are the tail head, the pin bones, the hook bones and the spinal ridge. A difficulty when taking the pic-tures from behind was the animals waving of their tails, changing the position of the tailhead. It can hence only be used as a guide, and not as a fixed feature point. When observing the images received, many showed strong occlusions in the area of the pin bones as the waving tail put the pin bones in its shadow. A system that depends on using the pin bones as a feature when capturing images from behind would therefore depend largely on the movement of the tail and be useless in most uncontrolled environments. The cow is normalized in a series of rotations and translations based on the location and relation of the feature points.

30

Figure 4.5: In the left image the tail is located. On the right the spine and hook bones have

been localized and the data translated and rotated.

4.3 Feature extraction

Two different methods of finding parameters describing the body con-dition score are generally interesting: A global method using statistical features of modified range images and surfaces, and a local method where curves and angles are fitted to contours. The latter method has showed good results on controlled images in earlier studies, but the method is limited. The anatomical points that indicate the BCS do not always correspond to obvious visual clues and are difficult to extract automatically. Figure 4.6 shows the accuracy of pinpointing the hook bones in two images of one cow, captured at the same occasion. There are also large differences between individual cows in shapes which make it reasonable to attempt a different approach. In general terms, the interest for robust feature recognition has in-creased largely lately. During the last couple of years, many new systems have been developed that can store biometrical information such as facial structure, fingerprint or voice, in order to be used for verification tasks where security is concerned. Recently, many two-dimensional feature recognition applications have been carried out with optimal results obtained for images acquired in controlled con-ditions [13]. The main constraints of these techniques are: firstly, the influence of illumination, as the shaded parts of the cow may mislead the verification pro-cess, and secondly, the changes of pose.

31

Figure 4.6: A comparison of two images of cow 151 pinpointing the hook bones

However, both humans and cows are three-dimensional, so projecting them as a two-dimensional object provokes information loss. With the development and improvement of three-dimensional data acquisition devices, mostly face recognition techniques have received more inte-rest. Nowadays, this is one of the most vigorous research areas e.g. within biometrics. Although a cow differs in many aspects from a human face, to examine 3D face features recognition tools could be very useful. Due to the novelty of three-dimensional feature recognition techni-ques, there are not many published results [14]. In general, two kinds of aspects are treated: firstly the use of range data, translating the 3D information into a 2D depth map or distance to the acquisition system, and secondly, the use of three-dimensional mesh object representation. When combining different global statistical measures, it is necessary to combine the two aspects.

32

4.3.1 Image texture analysis

Many portions of the image of a cow are devoid of sharp edges over the areas of the back and loin area, with the exception of the occlusion in front of the hook bones. In these areas, the surface could be charac-terised as exhibiting a structure analogous to the texture of cloth when looking at it from different angles, considering different properties of the animal. Image texture measurements can therefore be used for classification of the cows according to their body condition score. Tex-ture is often qualitatively described by its coarseness, in the sense of looseness or roughness in texture. The coarseness is also related to the spatial repetition period of the local structure. A larger period implies a rougher structure, whereas a small period implies an even structure. Important is to recognize that the coarseness is a relative measure of the texture in the neighbourhood of an image point. Because texture is a spatial property, measurements should be restricted to relatively uniform regions. This is the case of the back area. Because of the occlu-sion behind the hook bones, and the varied curvature close to the tail head, a smaller region of one third of the hip width, and twice the distance in length was found to be a good candidate for statistical mea-sures. To analyse the surface statistically it was necessary to make a quadratic interpolation using a mesh function to get a surface with evenly spaced values. Two texture analysis methods were examined.

1. Local feature histograms. A frequently used approach for texture analysis is based on statistical properties of the histo-gram. The descriptors commonly used based on the histogram h(z) of a region are calculated using the moments about the mean as described below:

Mean ∑−

=

=1

0

)(L

i

ii zhzm (4.3.1)

Standard deviation ∑−

=

−=1

0

2 )()(L

i

ii zhmzσ (4.3.2)

Smoothness )1/(11 2σ+−=R (4.3.3)

Skewness ∑−

=

−=1

0

3 )()(L

i

ii zhmzS (4.3.4)

Uniformity ∑−

=

=1

0

2 )(L

i

izhU (4.3.5)

Entropy )(log)( 2

1

0

i

L

i

i zhzhE ∑−

=

−= (4.3.6)

33

In addition, the width of the histogram, and the width at the half maximum value was computed, as well as the parameters of a curve, cubically fitted to the histogram.

2. Spectral measures of texture based on the Fourier spectrum are

well suited for describing the visibility of two-dimensional patterns in a depth image. As the texture coarseness is propor-tional to its spatial period, a region of rough texture, compa-rable to a skinny cow, should have its Fourier spectral energy concentrated at low spatial frequencies. Equally, smooth regions, such as the back of a fat cow, should exhibit a concen-tration of spectral energy at high spatial frequencies. Interpre-tation of spectrum features is simplified by expressing the spec-trum in polar coordinates as a function S(r,θ), where r is the frequency and θ the direction [15]. The descriptors are obtained by summing for discrete variables:

∑=

=π

θθ

0

)()( rSrS , ∑=

=0

1

)()(R

r

rSS θθ (4.3.7)

For each direction θ, S(r, θ) is considered as a one-dimensional function Sθ(r) which yields the behaviour of the spectrum along a radial direction for a fixed θ, and similarly for each frequency r, Sr(θ) is evaluated yielding the behaviour along a circle cente-red at the origin. The following descriptors of the general func-tions were computed: Mean, variance, the maximum value and the difference between the maximum value and the mean of S(r) and S(θ) respectively.

4.3.2 Measures for texture analysis

In order to find the optimal parameters to link to the body condition score, I wanted to examine many features linked to the statistical properties of the surface. In my work, I analysed four shape-specific local features:

1. Depth value. The depth values, denoted z, are considered from the xy-plane in the normalized data set and describe the shape of the back.

34

2. Subtracted depth value creating a damped image. Just as with humans, cows are shaped differently between individuals. In an effort to exclude the individual fluctuations, I tried subtracting a strongly filtered image from the originally filtered. The idea was that a strong filter only leaves the basic shape of the cow, and subtracting the images should leave only the fluctuations caused by the degree of fatness in the animal. However, dis-tance histograms are problematic when the depth range can be influences by other objects or background clutter.

3. Point gradients. For a variable less sensitive to a perfect

normalisation, it is interesting to examine the gradient. The gradient of a function of two variables, z = f(x,y), is defined as

jy

fi

x

ff ˆˆ

∂

∂+

∂

∂=∇ (4.3.8)

and can be thought of as a collection of vectors pointing in the direction of increasing values of f. The norm of these vectors at each point could be used to describe the shape of the back.

4. Surface normals. Another way to analyze the surface curvature is by surface normals. Surface normals can easily be calculated from first derivatives of the image. After the usual normalisa-tion, two components of the resulting vector could be relevant. Research has shown that a representation as a pair of angles (φ,θ) give reliable results. Given an oriented point p, with a nor-mal vector (nx, ny, nz), the normal can be defined by two angular parameters: φ and θ (figure 4.7).

Figure 4.7: Representation of normals in projective (left) and sphere coordinates (right)

35

These parameters are represented in a map N0:

23

0 : RRN →

))(

arctan,(arctan),(:)(22

0

z

xy

x

y

n

nn

n

nN

+

=θϕp (4.3.9)

φ represents the fluctuations in the xy-plane, and θ gives the fluctuations of the normal around the z-axis.

4.3.3 Examining features in image data

The four local features; range, damped range, gradient and surface normal, together with all the statistical parameters interesting for mea-surements, result in about ninety factors to analyse. To make a first exclusion and see which factors could be significant, I made a simple model of a cow with changing body condition. The model is created by using the interpolated surface from an image of cow 189, with a body condition score of 3.0. The z-value of the data is manipulated by a factor ranging from 0.8 to 1.2. Lower values simulate a fat cow by making the back flatter, and high values give a bonier and skinnier simulation of a cow. Figure 4.8 gives an idea of how the model works.

Figure 4.8: Simulating changes in body condition by manipulating the z-values of the data

Testing on the model resulted in about thirty parameters, numbered from 4 to 30, with a notable change in value between the manipulated images. Thereafter, the thirty statistical parameters were calculated for each image for potential influence on the body condition score and recorded in a separate .mat file. If any parameter value resulted as a NaN (Not a Number), the image was considered to be of insufficient quality and excluded from further evaluation.

36

An average of each parameter was calculated for each 5-image se-quence from one photograph. Correlation and cross-correlation of the parameters with the body condition score was used to analyse models for prediction. In addition, the development of the parameters was compared to the known fluctuation of the BCS during the months after calving. Only effects with a high correlation coefficient >0.5 were con-sidered for the model. The reliability of a parameter was examined by comparing the spread in values in one “photograph” with the dynamic range. Two models were developed for this set of parameters. The first model included only the linear combination of the better correlated parameters as predictions of BCS. These were numbered 4, 5, 8, 14, 18, 20, and 30. Because the relation between BCS and individual para-meters 4, 5, 18 and 20 seemed to have a quadratic relation when examining the plots visually, I created a model of second degree, attempting to include the quadratic relation.

The linear model (model 1), was defined as

iiii pcpcpcCk 30305544 ... ++++= (4.3.12)

where ki is the BCS of image i; pni the nth parameter of the ith image, cn

are the model coefficients, and C is the intercept. The quadratic model (model 2) of these parameters was defined as

2

20

'

20

2

4

'

430305544 ...... iiiiii pcpcpcpcpcCk +++++++= (4.3.13)

where c’n are the model coefficients corresponding to the squared para-meters. For both models the unknown coefficients for predicting the BCS value were found by applying the least squares method on the parameters in the entire training set of images.

37

Chapter 5

Results and discussion

Seven parameters were found to be significant. In the training set of images, the model, though inadequate because of the

model adjustment method, give results with errors comparable to the difference between two scorers.

In the discussion below, data from the training set has been used. Among cows assessed for body condition score between 27th of June 2008 and 3rd of September 2008, mean subjective BCS was 3.30 (±0.41), mode value was 3.25, with a range of 2.5 to 4.25. Of the nineteen Swedish red breed (SRB) cows primarily evaluated, three were terminated during the first months after examination, four were pregnant when observed, and two calved during the observation period, with images from both before and after calving. Mean parity7 was 2.88 (±1.35), with a range of 1 to 5. This group of sixteen cows provided the images used in the training of the BCS prediction algorithm. Within the group, the five cows observed regularly contributed with 56% of the data for the model development. Their mean BCS was 3.32 (±0.37). Two of them were pregnant at the first observation, calving before the second one, and they had a mean parity of 3.8, ranging from 2 to 5 at all other observations. Because of setup limitations with the experimental equipment, and uncertainty in a hand-held procedure, 351 images were usable out of the 380 images captured for model development. Generally for poor images, the photo angle with the cows back was too small, showing more of the cow’s rump and not sufficiently of the back.

CHAPTER 5. RESULTS AND DISCUSSION

38

5.1 Choice of parameters To represent the changes in body condition score, 90 parameters were originally thought of as possible candidates. When examining the “model” of a cow with changing BCS, only 30 showed any change when a change in the roundness of the back was simulated by manipu-lating the z-values. Examination of the “model” and of visual appraisal of a small number of images excluded the use of damped images. As damping was produced by using a strong filter on the image and sub-tractting it from the originally filtered image, the method proved to be very sensitive to the photo angle. In the algorithm, filtering is made on entire unsegmented images, and points at long range affect the values in the image differently for different filters. Since there was no possi-bility to guarantee a constant angle in the hand-held setup, the idea had to be discarded. For further analysis of useful parameters, I combined curves of the development of the parameters during the months after calving (figure 5.1) with the correlation information and other statistical data summarized in table 5.1. From the 30 remaining parameters, choosing significant parameters was conducted in three steps:

1. Correlations R(i,j) were calculated between BCS and image average parameters. The correlation coefficient matrix re-presents the normalized measure of the strength of linearity between variables [18]. Values close to ±1 suggest that there is a linear relationship between the parameters. The correlation matrix R is related to the covariance matrix C by

),(),(

),(),(

jjCiiC

jiCjiR = (5.1)

In coincidence with the correlation matrix, a matrix of p-values for testing the hypothesis of no correlation was calculated. Each p-value is the probability of getting a correlation as large as the observed value by random chance, when the true correlation is zero. If P(i,j) is small then the correlation R(i,j) is significant. I evaluated the correlation following the criteria in table 5.2. Interesting parameters in this case proved to be number 4, 5, 6, 8, 12, 14, 16, 17, 18, 20, 22, 23, 24, 27 and 30. All with |R|>0.6.


39

2. An important measure is the spread of a parameter when taking the mean of each parameter value from the five images constituting a “photograph”. The standard deviation in each take was found. Subsequently he mean of these standard deviations was calculated for each parameter, representing the absolute fluctuation of the parameter value. These were normalised by comparing them to the dynamic range. The dynamic range is the corresponding parameter range to the body score range of one to five, assuming a linear relationship between BCS and the parameter. The fraction of the spread in the dynamic range was assessed according to the criteria in table 5.2. From the first set of parameter candidates, number 6, 16, 22, 23 and 24 could be excluded due to their spread of >0.04.

3. By examination of the cross-correlation, that is, the correlation

in between the parameters, the parameters describing the same measure were detected by their strong inter-correlation. If the parameters are strongly correlated with each other one parameter is mathematically just a factor of the other, and including both would imply that the measure they represent is included twice. High correlation was noted between para-meters 5 and 6, 12 and 14, 17 and 18, as well as 27 and 30.

The choice of parameter fell on the ones with the best combination of high correlation with BCS, smallest spread, and that best followed the expected fluctuations in score during the months after calving. The analysis of the parameters resulted in the acceptance of parameters 4, 5, 8, 14, 18, 20 and 30 for the model development. These interesting parameters are mainly derived from measurements of the deviation of the surface normals from the vertical. Examples of these parameters are plotted in figure 5.1 versus the body condition score and days in milk (DIM) after calving.


40

Table 5.1.1: Correlations among BCS and image parameters 4-12, p-value, average standard

deviation within images in a photograph (Std), dynamic range (DR), standard deviation in

relation to the dynamic range

BCS 4 5 6 7 8 9 10 11 12

BCS 1 -0,6453 -0,6664 -0,6559 -0,5244 -0,7288 -0,5617 -0,2194 -0,4107 -0,7213

4 1 0,7479 0,7577 0,4693 0,7305 0,6695 0,5574 0,4940 0,6574

5 1 0,9535 0,7190 0,7831 0,7477 0,6165 0,5177 0,7876

6 1 0,6549 0,7387 0,7402 0,6074 0,4821 0,7441

7 1 0,5811 0,5333 0,5630 0,2669 0,6295

8 1 0,8811 0,4877 0,6633 0,9348

9 1 0,7151 0,6649 0,8700

10 1 0,3808 0,5546

11 1 0,6591

12 1

P (BCS) 1 1,24E-09 2,25E-10 5,31E-10 2,67E-06 5,72E-14 3,91E-07 0,0392 2,18E-05 1,28E-12

Std 0,009587 0,005691 0,03327 0,024952 0,000177 0,00015 0,002327 0,000552 0,024263

DR -0,2583 -0,1549 -0,7064 -0,2272 -0,009608 -0,004131 -0,02173 -0,006728 -0,97981

Std in DR 0,0371 0,0367 0,0471 0,1098 0,0184 0,0364 0,1071 0,0821 0,0248

Table 5.1.2: Correlations among BCS and image parameters 13-21, p-value, average

standard deviation within images in a photograph (Std), dynamic range (DR), standard

deviation in relation to the dynamic range

13 14 15 16 17 18 19 20 21

BCS 0,5164 -0,7227 0,5213 0,6081 -0,6631 0,7016 -0,2374 -0,6534 -0,5601

4 -0,4626 0,6512 -0,4859 -0,4764 0,6906 -0,6652 0,3627 0,6868 0,4888

5 -0,6827 0,7801 -0,6991 -0,6914 0,7748 -0,7904 0,3964 0,7581 0,5283

6 -0,6064 0,7249 -0,6372 -0,6317 0,6898 -0,7113 0,4464 0,6694 0,4812

7 -0,7494 0,6177 -0,7424 -0,7275 0,6421 -0,6847 0,2888 0,5854 0,3213

8 -0,7593 0,9416 -0,7751 -0,7643 0,9234 -0,9322 0,4204 0,9104 0,6827

9 -0,6652 0,8262 -0,7098 -0,6975 0,7438 -0,7875 0,5588 0,7306 0,6728

10 -0,4960 0,4586 -0,5550 -0,5383 0,4922 -0,5299 0,5902 0,4699 0,3878

11 -0,3963 0,6569 -0,4283 -0,4386 0,5056 -0,5491 0,3503 0,5404 0,6607

12 -0,8348 0,9841 -0,8690 -0,8692 0,8939 -0,9429 0,5433 0,8717 0,7356

13 1 -0,8289 0,9857 0,9795 -0,8428 0,9004 -0,3452 -0,7851 -0,4294

14 1 -0,8439 -0,8434 0,9015 -0,9424 0,4318 0,8829 0,7315

15 1 0,9970 -0,8472 0,9113 -0,4964 -0,7915 -0,4714

16 1 -0,8303 0,8996 -0,5047 -0,7751 -0,4741

17 1 -0,9848 0,3965 0,9828 0,5685

18 1 -0,4516 -0,9576 -0,6081

19 1 0,3841 0,4173

20 1 0,5846

21 1

22

P (BCS) 3,55E-08 2,29E-14 3,42E-08 1,86E-08 1,28E-11 9,40E-12 0,00905 1,27E-11 3,79E-07

Std 0,01271 1,515 0,9234 0,003212 0,001581 0,08087 0,01994 0,003679 0,05634

DR 0,2097 -57,90 15,93 0,0551 -0,0630 2,780 -0,0993 -0,1107 -0,5948

Std in DR 0,0606 0,0262 0,0580 0,0583 0,0251 0,0291 0,2008 0,0332 0,0947


41

Table 5.1.3: Correlations among BCS and image parameters 22-30, p-value, average

standard deviation within images in a photograph (Std), dynamic range (DR), standard

deviation in relation to the dynamic range

22 23 24 25 26 27 28 29 30

BCS 0,7092 0,7044 0,6273 0,4783 0,0529 -0,7654 -0,2192 -0,3589 -0,7810

4 -0,5469 -0,5472 -0,4767 -0,3130 -0,0529 0,6588 0,1606 0,2122 0,6318

5 -0,6605 -0,6598 -0,6605 -0,3585 -0,0335 0,7540 0,0521 0,2406 0,7225

6 -0,5806 -0,5849 -0,5912 -0,3141 -0,0060 0,7096 0,0449 0,1834 0,6613

7 -0,4901 -0,4888 -0,5549 -0,3038 -0,0830 0,5490 -0,1362 0,1482 0,5144

8 -0,8141 -0,8129 -0,7700 -0,5075 -0,1312 0,9268 0,1527 0,3161 0,8963

9 -0,6869 -0,6951 -0,5957 -0,4981 -0,2207 0,8153 -0,0632 0,0525 0,6928

10 -0,4077 -0,4221 -0,2595 -0,3635 -0,2837 0,4173 -0,3044 -0,1550 0,2775

11 -0,6854 -0,6868 -0,3745 -0,4629 -0,2328 0,6971 0,0177 0,0335 0,5875

12 -0,8599 -0,8659 -0,6951 -0,6071 -0,2680 0,9658 -0,0633 0,1346 0,8514

13 0,6566 0,6536 0,7182 0,5606 0,2618 -0,7404 0,2337 -0,0805 -0,6431

14 -0,8721 -0,8720 -0,7485 -0,5814 -0,2155 0,9871 0,0276 0,2124 0,9019

15 0,6801 0,6853 0,6821 0,5958 0,3089 -0,7596 0,2867 -0,0302 -0,6376

16 0,6808 0,6872 0,6635 0,6059 0,3209 -0,7625 0,3011 -0,0164 -0,6341

17 -0,7943 -0,7900 -0,7635 -0,4927 -0,1491 0,8534 0,0820 0,3320 0,8427

18 0,8201 0,8190 0,7647 0,5550 0,2097 -0,8924 0,0120 -0,2523 -0,8409

19 -0,4045 -0,4464 -0,1119 -0,4272 -0,3760 0,4156 -0,3895 -0,2566 0,2331

20 -0,8366 -0,8310 -0,7536 -0,4690 -0,1284 0,8438 0,1239 0,3416 0,8388

21 -0,7291 -0,7358 -0,4217 -0,4730 -0,2526 0,7694 0,0219 0,0318 0,6463

22 1 0,9986 0,6235 0,5386 0,2253 -0,8739 -0,0539 -0,1836 -0,7965

23 1 0,6098 0,5495 0,2413 -0,8740 -0,0324 -0,1633 -0,7881

24 1 0,3481 -0,0714 -0,6830 -0,3413 -0,4440 -0,7499

25 1 0,8289 -0,5644 0,2825 0,1823 -0,3870

26 1 -0,2081 0,4721 0,4097 0,0024

27 1 0,0588 0,2064 0,9099

28 1 0,7919 0,3840

29 1 0,5935

30 1

P (BCS) 4,46E-12 7,11E-12 4,78E-09 2,45E-05 0,661 7,62E-15 0,0661 0,00211 9,47E-16

Std 0,06222 0,04454 162,3 3698000 3146 124,2 43040 154,5 199,7

DR 1,192 0,8630 3753 34880000 2361 -4591 -188300 -953 -5544

Std in DR 0,0522 0,0516 0,0432 0,1060 1,3324 0,0271 0,2285 0,1621 0,0360

Table 5.2: Assessment criteria for determination of optimal parameters

|R(i,j)| Std /DR Assessment

> 0.7 < 0.03 Very good > 0.6 < 0.04 Good > 0.5 < 0.06 Acceptable


42

Figure 5.1: Left, parameter values versus BCS. Right, parameter values versus

the lactations stage – days in milk (DIM).

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.40.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

0.055

BCS

Para

mete

r valu

e

Value of parameter 17 versus BCS

151

193

194

221

291

-10 0 10 20 30 40 50 60 70 80 900.01

0.015

0.02

0.025

0.03

0.035

0.04

0.045

0.05

0.055

DIM

Para

mete

r valu

e

Value of parameter 17 versus DIM

151

193

194

221

291

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.450

55

60

65

70

75

80

85

90

BCS

Para

mete

r valu

e


151

193

194

221

291

-10 0 10 20 30 40 50 60 70 80 9050

55

60

65

70

75

80

85

90

DIM

Para

mete

r valu

e


151

193

194

221

291

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4

5

5.2

5.4

5.6

5.8

6

6.2

6.4

6.6

BCS

Para

mete

r valu

e


151

193

194

221

291

-10 0 10 20 30 40 50 60 70 80 90

5

5.2

5.4

5.6

5.8

6

6.2

6.4

6.6

DIM

Para

mete

r valu

e


151

193

194

221

291

-10 0 10 20 30 40 50 60 70 80 900.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

DIM

Para

mete

r valu

e


151

193

194

221

291

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.40.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

BCS

Para

mete

r valu

e


151

193

194

221

291


43

Firstly in figure 5.1, the parameter values from all images are plotted versus the body condition score (BCS) of the cows at the time. All para-meters except 18 have an inverse correlation with the body condition score. This is normal as, with increased fatness, the depth over the back of the cow decreases as the animal fills up and goes rounder and flatter. With a flatter back there is an increase in gradient vector magni-tude. With fat, and therefore flat, cows the angle is smaller. Noted when analysing the results was that when making a histogram, values in small ranges are summed to show the frequency of a value, giving a more noise-resistant measure. The uniformity and entropy of the histo-gram are inversely related and represent how much the surface of the back fluctuates. They are both a measure of variance showing the coarseness of the shape texture. The values of the function S(θ) give testimony of ordered shapes such as visible transverse spinal pro-cesses. When the back fills with fat, the spinal processes are no longer visible and the texture is no longer ordered. Additionally, the parameter values are plotted versus the number of days in milk (DIM) of the cows, where generally the inverse of the expected trend from figure 2.1 can be seen, due to their inverse rela-tionship with the body condition score. The five cows followed during the observation period are plotted separately. In the larger spread in parameter values, the values on one individual cow are relatively con-strained. It gives to believe that the spread is largely dependent on intrinsic changes between different cows, more than on method errors. Despite the spread, the expected trend is clearly notable. When only the range data was analysed, erroneous images had a large impact on the result, which could be seen as measurements clearly se-parated from the larger group of values. Faulty values will affect the correlation of the parameter with the body condition score and, even worse, have an impact on the average value from the five photos in an image, producing misleading results. The risk of using range values is that they can give large uncertainties when describing the fatness of an animal, if images of poor quality are accepted. Possibly, in a larger set of images, range data values could be used as a tool when eliminating misguiding data. However, when computing gradient norms, the data is considered as a surface function and extreme range data is filtered, so the effect of a poor quality image is damped. Compared to the overall spread in plots of the gradient norm, the values show greater uncertainty at lower BCS. When looking at the development in DIM, a strong cow-dependence is noted. For each observation, there is a


44

relatively small spread in values for individual cows, compared to the spread between the observations from different individuals, giving a clue to the cow-dependence of visual BCS. Generally, the plots of θ fea-tures show less dependence on individual cows. The means for the seven chosen parameters are summarised in table 5.3. For each parameter, with the only exemption being parameter 18 with an inverse relationship, a trend of decreasing parameter value with increasing BCS was observed. This linear relationship is also de-monstrated in figures 5.1 and 5.2. These results support the hypothesis that BCS is reflected in statistical feature parameters over the spinal processes as measured by the 3D camera. The changes between body condition scores were not perfectly linear in this data set, however. For example, the value for parameter 4 was 0.0310 for cows with a BCS of 3.5, increasing to 0.0446 for cows with a score of 3.75. This result is most likely a factor of including images of poor quality in the extrac-tion of parameters. Disregarding the effect of poor images, the stan-dard deviations for values at a low body condition score were also relatively high. This result may suggest that the changes in anatomical features below a certain level of condition vary from animal to animal, as they begin to loose fat from other areas of the body. Table 5.3: Mean, standard deviation, and number of photographs for selected parameters by

the body condition score (BCS)

Parameter 4 Parameter 5 Parameter 8

BCS n Mean Std Mean Std Mean Std

2.5 10 0,1133 0,0043 0,0893 0,0036 0,00510 0,00021

2.75 24 0,0965 0,0301 0,0763 0,0173 0,00443 0,00106

3.0 71 0,0839 0,0200 0,0743 0,0145 0,00411 0,00083

3.25 107 0,0614 0,0435 0,0576 0,0268 0,00287 0,00074

3.5 48 0,0310 0,0347 0,0539 0,0111 0,00266 0,00039

3.75 45 0,0446 0,0101 0,0415 0,0098 0,00226 0,00042

4.25 5 0,0271 0,0035 0,0300 0,0051 0,00146 0,00007

Parameter 14 Parameter 18 Parameter 20 Parameter 30

Mean Std Mean Std Mean Std Mean Std

83,16 0,72 5,234 0,089 0,0668 0,0044 10652 251

79,56 5,78 5,486 0,431 0,0538 0,0178 10210 597

77,11 5,29 5,709 0,288 0,0430 0,0123 9883 485

71,53 4,68 5,972 0,253 0,0318 0,0093 9214 376

70,59 3,74 6,014 0,186 0,0302 0,0059 9202 289

66,12 3,09 6,171 0,096 0,0266 0,0044 8820 359

55,41 1,18 6,434 0,040 0,0175 0,0018 8340 81


45

Figure 5.2: Mean and standard deviation of parameter values versus BCS

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.41

1.5

2

2.5

3

3.5

4

4.5

5

5.5x 10

-3 Mean and standard deviation for parameter 8 by BCS

BCS

Para

mete

r valu

e

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.45

5.5

6

6.5Mean and standard deviation for parameter 18 by BCS

BCS

Para

mete

r valu

e

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4-0.02

0

0.02

0.04

0.06

0.08

0.1

0.12


BCS

Para

mete

r valu

e

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.40.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09


BCS

Para

mete

r valu

e

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.450

55

60

65

70

75

80

85

90Mean and standard deviation for parameter 14 by BCS

BCS

Para

mete

r valu

e

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.40.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Mean and standard deviation for parameter 20 by BCS

BCS

Para

mete

r valu

e

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.48000

8500

9000

9500

10000

10500


BCS

Para

mete

r valu

e


46

5.2 Model development

The primary objective of this work was to develop a model to describe the body condition score by using the information obtained from the collected 3D images. The separate parameters showed a good potential for indicating the body condition score on the training set. However, a single parameter may be misleading so all parameters considered together should provide an accurate score. Two models were considered, one with a pure linear combination of parameter values and a second with quadratic terms. Four parameters were assumed to add a quadratic term to the model. Figure 4.3 show their image average value plotted versus the body condition score. In retrospect, I would not have considered the range parameters 4 and 5 to be quadratic. Surface normal parameters 18 and 20 on the other hand clearly seem to go towards an asymptotic value at higher BCS. These results suggest that the changes in the angle of the surface, as well as the entropy of this angle, in fat cows are very small. The explanation could be that the surface shape varies more for skinny cows when the transverse spinal processes start to show, increasing the entropy drastically. Unfortunately, there is a great risk in assuming non-linearity with too little data. Especially when the data in the extremes of the BCS-range come from very few animals. Taking the cow-dependence into consideration, there is an invisible spread of values at the ends of the BCS range that could actually imply a linear relationship if more cows had been used.


47

Figure 5.3: Image average parameter values for parameters 4, 5, 18 and 20 seem to have a

quadratic relation to the BCS

Results from the two models developed are shown in figure 5.4. For comparison, the body condition score was predicted by their relationship with each parameter individually. Figure 5.5 show the resulting plots. With both models, the correlation was better than for the individual parameters. The correlation coefficients for the calculated BCS with the actual BCS were 0.7987 and 0.8191 for model 1 and 2 respectively. The mean errors for the models were 0.13 in the range of 0 to 0.43 for the purely linear model, and 0.15 for the model with quadratic terms, with the errors ranging from 0 to 0.45. The models give comparable results. Because of the visually quadratic behaviour of parameters 18 and 20 and the better correlation for the second model, the remaining discussion of the results focuses on model 2. 100% of the automatically extracted body condition scores were within 0.5 points of actual score, and 79% were within 0.25 points. These are very promising results, compared to 99.89% within 0.5 points and 89.95% within 0.25 points reached by Bewey et al. [9] on the training set using manual extraction of reference points and extensive exclusion of poor data.

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.45

5.2

5.4

5.6

5.8

6

6.2

6.4

6.6

6.8

BCS

Para

mete

r valu

e

Image average value of parameter 18 versus BCS

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4-0.05

0

0.05

0.1

0.15

0.2Image average value of parameter 4 versus BCS

BCS

Para

mete

r valu

e

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.40

0.02

0.04

0.06

0.08

0.1

0.12


BCS

Para

mete

r valu

e

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.40

0.01

0.02

0.03

0.04

0.05

0.06

0.07


BCS

Para

mete

r valu

e


48

A potential concern and limitation of the availability to predict the body condition score by using 3D data and a simple model is high-lighted by the error plots in figure 5.4. In both models predicting BCS, there is a trend in the errors with increasing score. In effect, these models overpredict the body condition score of thin cows and under-predict the score of fat cows. The result is not entirely surprising, given that few cows scored at either extreme of the scale. Finding coefficients that give the best fit to the entire set will be influenced by the large number of values giving scores in the mid range. By reason, the result should be centred about the mean BCS of 3.3. When looking at the error plots, this is actually the case. In addition, the spread in some parameters for certain body condition scores have to be considered. With all images included, not compensated by the effects of the indi-vidual constitution of each cow, there is a relatively large spread in the values. The distinction between the values that belong to a fat cow, and the values belonging to a skinnier cow, is then very small. Due to the number of cows with a more moderate body condition score, the probability that the values belong to a moderate BCS is high, creating a bias model equation. When creating a model using the least square solution, making a linear adaptation of the data, the factors are weighed equally. The least squares approach to solving an overdetermined system of equations is to try to make as small as possible the sum of squares of the errors (ε = modelBCS - BCS) on either side of the fit. This is given independently of where the errors are positive and where they are negative, and be-cause of the large spread in parameter values, the best fit is given for ε > 0 at BCS < 3.3 and ε < 0 at BCS > 3.3.


49

Figure 5.4: Calculated BCS and model error versus BCS for model 1 (left) and 2 (right)

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.42.6

2.8

3

3.2

3.4

3.6

3.8

4

BCS

Calc

ula

ted B

CS

Calculated versus actual BCS for model 1

y = 0.64*x + 1.2

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.42.6

2.8

3

3.2

3.4

3.6

3.8

4

4.2

BCS

Calc

ula

ted B

CS

Calculated versus actual BCS for model 2

y = 0.67*x + 1.1

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5Model errors for model 1

BCS

Err

or

y = - 0.36*x + 1.2Norm: 1.64

Mean error: 0.13

Std: 0.11

Median error: 0.13

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5Model errors for model 2

BCS

Err

or

y = - 0.33*x + 1.1Norm: 1.56

Mean error: 0.15

Std: 0.11

Median error: 0.11


50

Figure 5.5.1: Calculated BCS and model error versus BCS for parameters 4, 5, 8 and 14

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.41.5

2

2.5

3

3.5

4Predicted BCS using parameter 4

BCS

Calc

ula

ted B

CS

y = 0.29*x + 2.3

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4-1.4

-1.2

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6Prediction error for parameter 4

BCS

Err

or

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.41.6

1.8

2

2.2

2.4

2.6

2.8

3

3.2

3.4

3.6Predicted BCS using parameter 5

BCS

Calc

ula

ted B

CS

y = 0.31*x + 2.2

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4-2

-1.5

-1

-0.5

0

0.5

1Prediction error for parameter 5

BCS

Err

or

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.42.6

2.8

3

3.2

3.4

3.6

3.8


BCS

Calc

ula

ted B

CS

y = 0.53*x + 1.5

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6


BCS

Err

or

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.42.8

3

3.2

3.4

3.6

3.8


BCS

Calc

ula

ted B

CS

y = 0.52*x + 1.6

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1Prediction error for parameter 14

BCS

Err

or


51

Figure 5.5.2: Calculated BCS and model error versus BCS for parameters 18, 20 and 30

To check for the robustness of the model, the calculated body condi-tion score from the second model, and the subjective actual body condition score, were compared against the lactation stage together with the model error (figure 5.6). The calculated score nicely follows the predicted development. The values of the scores are damped be-cause of the prediction errors, but the changes in BCS over time are naturally smooth as is expected in a live animal. The plots might be confusing because of all the data points, but let us look closely on cow 193, who has a big enough drop in actual BCS not to lay within the human error of 0.25 points. Her predicted BCS drops from 3.9 to 3.3

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4

2.5

3

3.5


BCS

Calc

ula

ted B

CS

y = 0.44*x + 1.8

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6


BCS

Err

or

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4

2.6

2.8

3

3.2

3.4

3.6

3.8Predicted BCS using parameter 20

BCS

Calc

ula

ted B

CS

y = 0.43*x + 1.9

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6


BCS

Err

or

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.42.6

2.8

3

3.2

3.4

3.6

3.8


BCS

Calc

ula

ted B

CS

y = 0.53*x + 1.5

2.6 2.8 3 3.2 3.4 3.6 3.8 4 4.2 4.4-0.6

-0.4

-0.2

0

0.2

0.4

0.6


BCS

Err

or


52

when the scores should have been dropping from 4.25 to 3.25, but the spread in each predicted score is of the order of 0.2 points which indi-cates that the observed trend is reliable. Cow 194 also show an inte-resting result. On the first observation, when scored by the two expert scorers from DeLaval, her actual BCS is lower than on the next obser-vation when she was scored by Kerstin Vollmer from Hamra Farm. The actual score then follows the expected drop in later scorings. This is due to the known difference between different scorers. Looking at the plot for the calculated BCS however, this jump in BCS is not mea-sured. Instead, the body condition score has a constant decrease, as would have been expected. So, even if the scores calculated by the developed model are incorrect, they are still sensitive to changes in the fatness of the animal. The automatic method is relative to the data of the training set of images, and the human scoring is absolute from one occasion to the other. Hence, it is possible to see in the graphs that the human scoring is more prone to subjectivity than the automatic scoring. This shows that there is excellent potential in the automatic method to monitor the changes in BCS over time. The decrease in score for the model is of the order of the recommended decrease of 0.5 to 0.75 in BCS in early lacta-tion. There is a larger subjectivity with the manual score. The human observer could be tricked to score higher than the actual value when the cow is round because of pregnancy, and lower if the animal is sick and dehydrated. The camera method, on the other hand, is objective and not influenced by other factors in time. In the automated method there is a large potential in creating a BCS-evaluation which is more sensitive to small changes than any expert in condition scoring. An interesting observation is that the spread in errors appear to de-crease as days in milk increases. In the fist month of observations, the errors range up to almost 0.5. In contrast, the errors during the last month of observations seem to be limited below 0.3 points of actual score. This indicates that the method became more robust with time. Knowing that all data was treated simultaneously, a possible explana-tion is that I became more confident at taking the pictures after some time, being more consistent with the photographing angle and the steadiness of the camera. The errors might also be influenced by the subjectivity of the human scorers, as the first scores were evaluated by different scorers than the remaining scores, and that Kerstin Vollmer could have become more confident in her scoring after some time.


53

Figure 5.6: Calculated BCS, actual subjective BCS and model errors versus lactation

stage (DIM) of the five cows followed during the observation period

5.3 Test on new data

In this section, data from the validation set is used. The previous result show a very good potential in evaluating the body condition score by 3D imaging. To see how well the method works, I wanted to examine how well it could handle cows and images not included in the training set. 120 images were accepted for BCS prediction. Four black Swedish Holstein (SLB) cows were also included only to see how strongly the intrinsic shape of the animal affects the predicted BCS. This group of cows provided the images used for the ultimate validation of the BCS prediction algorithm.

-10 0 10 20 30 40 50 60 70 80 90

2.4

2.6

2.8

3

3.2

3.4

3.6

3.8

4

4.2

4.4

DIM

Calc

ula

ted B

CS

Calculated BCS versus DIM

151

193

194

221

291

-10 0 10 20 30 40 50 60 70 80 90

2.4

2.6

2.8

3

3.2

3.4

3.6

3.8

4

4.2

4.4

DIM

BC

S

Actual BCS versus DIM

151

193

194

221

291

-10 0 10 20 30 40 50 60 70 80 90-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

DIM

Err

or

Model error versus DIM

151

193

194

221

291


54

Figure 5.7: Calculated BCS for the test set versus actual BCS compared to the calculated

BCS in the model set (left) and errors in the calculated BCS for the test set (right)

The resulting predicted scores together with the manual scores can be seen in table B.2 in appendix B. As seen in the graphs in figure 5.7, the result was surprisingly poor. Only 20% of the values had a discrepancy of less than 0.25 from the actual body condition score and 46% were within 0.5 points of the actual score. As expected, the calculated scores coincided with the actual scores around the mean BCS value of 3.3, with an over-prediction of more than one score at low BCS and an underprediciton at high scores. Especially the three cows with lowest scores are very different from expected and one would have to exa-mine these individual cows more carefully to understand the differ-rence. One contributing factor to the great difference between the result from the test set and the training set is the difference between scorers. Generally, Kerstin Vollmer who scored almost all cows in the training set tended to score less in the extremes than the more experi-enced DeLaval scorers who performed the scoring on the first and final occasion. The unusual large spread in the calculated BCS also suggests that there might have been a difference in the environment affecting the result in some extent. In an effort to find the origin of the large deviation of the calculated BCS from the expected BCS, each parameter is plotted separately in figure 5.8. Noted in the graphs is the enormous spread for some BCS-values, especially in the noise-sensitive range parameters 4 and 5. The spread is too large to be caused only by environment induced noise, and conclusions must be drawn that many unacceptable images were mistakenly included. In retrospect, I believe that the inclusion of poor images was caused by the change in spine-locating method. When using the primary method of slicing the image, very few of the vali-dation images were accepted, in comparison to all images being accepted when interpolation the surface. In the training set, this relation was constant, implying that the training set contained better

2 2.5 3 3.5 4 4.52.6

2.8

3

3.2

3.4

3.6

3.8

4

4.2

BCS

Calc

ula

ted B

CS

Calculated versus actual BCS in the test set compared to the model set

y = 0.13*x + 3.1

test set

test fit

model set

2 2.5 3 3.5 4 4.5-1

-0.5

0

0.5

1

1.5

BCS

Err

or

Errors in test set


55

quality images. I can only assume I had forgotten how I took the photos, and did it differently when collecting the test set of images. Also noted, if ignoring the large errors, is that the parameters show less change for animals in the extreme ranges of the body condition score. When developing the model, this behaviour at high scores was taken into account by adding a quadratic behaviour of some parame-ters. At low BCS however, a linear relationship was assumed. If para-meter 4 is taken as an example, and the value 0.075 is measured in an arbitrary image, the image is predicted to belong to a cow of BCS 3 because of the linearity assumed, when it could just as well have been a cow with BCS 2.25, as seen in the plot. The assumed linear relation-ship is therefore responsible for the overprediction at low BCS. From the resulting BCS of the four Swedish Holstein cows, it is clear that the method is strongly dependent on the intrinsic constitution of each individual cow. The Swedish Holstein breed is generally bonier but also significantly larger than the Swedish red breed (SRB), and the visual appearance is very different. As noted, the values of the body condition score even lay outside of the scoring range for three of the cows.


56

Figure 5.8: Mean and standard deviation of parameter values versus BCS in the test set of

images

2 2.5 3 3.5 4 4.5-0.15

-0.1

-0.05

0

0.05

0.1

0.15

0.2


BCS

Para

mete

r valu

e

2 2.5 3 3.5 4 4.50

0.02

0.04

0.06

0.08

0.1

0.12


BCS

Para

mete

r valu

e

2 2.5 3 3.5 4 4.51.5

2

2.5

3

3.5

4

4.5

5x 10

-3 Mean and standard deviation for parameter 8 by BCS

BCS

Para

mete

r valu

e

2 2.5 3 3.5 4 4.564

66

68

70

72

74

76

78

80


BCS

Para

mete

r valu

e

2 2.5 3 3.5 4 4.50.02

0.025

0.03

0.035

0.04

0.045

0.05

0.055


BCS

Para

mete

r valu

e

2 2.5 3 3.5 4 4.55.4

5.5

5.6

5.7

5.8

5.9

6

6.1

6.2

6.3


BCS

Para

mete

r valu

e

2 2.5 3 3.5 4 4.58200

8400

8600

8800

9000

9200

9400

9600

9800


BCS

Para

mete

r valu

e


57

5.4 Algorithm robustness

When creating a prototype algorithm for an automatic BCS system, the long term objective was to analyse its usefulness in a commercial product. The errors in the result may be caused by ineffective parame-ters or incorrect modelling, but may equally have their origin in the algorithm treating the images and extracting the parameters. On the other hand, the methods that are robust need to be identified for further use. The resulting parameter values in each photograph are an average from the five images captured with a pause of 0.5 seconds between each image. In this way, incoherence in parameter values caused by cow or camera movements, posture and photo angle is taken into account. The total time for one session of 2.5 seconds used in this pro-ject is a reasonable span. To examine closely the effect of the pause between images I took several photographs of cow 221 with different pause times. The result can be seen in figure 5.9, with times ranging from 0.05 to 0.9 seconds. Neither the value, nor the spread in the value between the images, depends on the time between images. It shows that there is a potential for shorter pauses between images if needed for making a faster system. To eliminate the effect of posture and movement, I believe 0.1 to be the minimum time acceptable between frames. It can be noted that fewer images were accepted with 0.9 se-conds between images. During the few seconds needed for all the frames constituting a photograph, the cow had the time to move out of the image field.

Figure 5.9: The spread in parameter values versus the time between images in a photo show

no dependence, indicating that a short pause time of 0.1 is sufficient.

The spread is plotted in relation to the dynamic range.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4Spread for parameter 4 versus time between images

time (s)

Para

mete

r valu

e

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10.75

0.8

0.85

0.9

0.95

1

1.05

1.1

1.15

1.2

1.25x 10

4 Spread for parameter 30 versus time between images

Para

mete

r valu

e

time (s)


58

When treating the images, they are primarily filtered. By applying a 7 by 7 global averaging filter, the worst noise is eliminated, but some noise is still accepted as seen in figure 5.8. A stronger filter would generally be recommended for detailed surface examinations as the ones in this master’s thesis project. In this case, however, the body con-dition of the cow is noted in the back area at large by the visibility of the spinal processes. They are of the same size as the noise after filte-ring and would be eliminated with the noise if a stronger filter was applied.

Figure 5.10: The area used for analysis is generally very flat (left). In a close up (right) in is

visible that there is significant noise in the image, generating noisy parameter values.

The area shown in figure 5.10 is the square on the back of the cow used for the feature extraction. The square is of the approximate size of 15 times 30 centimetres with a depth of only one to three centimetres. On the cow, this area provides relatively little information about the body condition score of the animal. With this taken into consideration, the results of the statistical feature method for BCS is very promising. The size of the area is only restricted by the occlusion in the image in front of the hip bones. The limitation is caused by the chosen photo angle. In effect, as the analysis was performed simulated from above, the results would be more favourable if the images were captured from that posi-tion. A larger area of the back could be analysed and the pin bones could be localised, not concealed by a waving tale. To enable statistical two-dimensional image analysis of the area, the data in the square had to be translated by interpolation to a 2D matrix. The area contains originally around 1000 data points unevenly spread out, that by interpolation are transformed into 10000 points. The quad-ratic interpolation smooth out the worst of the noise for the analysis of


59

the angle θ from the surface normal, with little or no effect on the mean range value. As a minimum number of points added between the noisy data, three interpolated values between each point gives good smoothness of the surface curvature. This corresponds to a multipli-cation of the number of points in the area by 10, as is the case of the interpolation used. If small changes are desirable to measure, this is a good ratio. For measurements of larger physical properties are of interest, the opposite would be necessary, smoothening the noisy surface. Secondly, when treating the images, a simple and fast method for seg-mentation is employed to find an isolate the cow in the image. The segmentation proved to work for all images that were accepted. During this project, I only accepted images where most of the animal, and no greater parts of a neighbouring animal, was confined inside the frame. As the filtering function set all edge values to the back wall of the image, the segmentation is not negatively affected if parts of the cow are outside of the frame. Additionally, as long as the animal of interest fills the larger part of the image, it is not affected by more than one cow placed in the image field, because of the further treatment finding the tail-candidate. However, if the camera would be placed in an above position, the segmentation would have to be adjusted as to exclude other animals trespassing into the image field. The most important part of the algorithm is the identification of the spine and the hook bones leading to the transformation and normali-sation of the images. The transform function had to be improved during the process. Originally, I chose metric values to limit the iden-tification of the spine in the images. These values were based on the position of the tail candidate and an assumed length of the cow in the image. On images where the position of the cow matched the assumed position, this method was very rapid and exact. Naturally, images with insufficient data were excluded automatically. With changes in angles, distances and lightning, it proved more robust to interpolate one to two thirds of the entire visual cow surface, finding the spine in the interpolated data. Unfortunately this increased the risk of accepting poor images with insufficient data whilst making the function inde-pendent of size and distance to the animal. A threefold interpolation is also relatively time-consuming with the risk of providing erroneous information when double valued points are averaged. To proceed with this method, double valued points should be eliminated. Generally, finding the hook bones and the spine worked perfectly, and these points are a good reference to any measurement.


60

Originally, the photo angle chosen was based on the hypothesis that the behind/above angle would provide maximum information. Unfor-tunately, occlusion in the tail area and in front of the hook bones restricted the amount of information available. Moreover, the angle with the back of the cow is difficult to keep constant as cows come in different heights, and as figure 5.11 shows, the method is unfortuna-tely still sensitive to angle. I examined all parameters in images of cow 221 captured from three different angles. The angles are numbered from one to three, with 1 almost from above, 2 being the ordinary photo angle, and 3 is a sharply acute angle. It should be noted that the chan-ges in the photo angle when testing are exaggerated and do not repre-sent the smaller fluctuations that could have occurred when I was photographing. The values for parameters 4, 5, 8, 14 and 20 differed with almost ±30% when changing the photo angle ±50% from the ordi-nary 45˚. Contrarily, the values for parameters 18 and 30 were not affected. This could be the result of insufficient calibration of the camera in the range versus the x-y measurements, creating slightly twisting the image, giving it a different appearance when aligning with the spine. Equally, the averaging filter is put on the data in the direction of photographing, having the same twisting effect. Another possibility is that the images might be aligned differently, despite three iterations. However, after three repetitions, the difference would be minimal and I have difficulties believing that it could be as large as three centimetres for the mean depth in a surface generally maximum three centimetres deep. Interestingly enough, the spread in the values does not differ with photo angle. It should be noted that the entropy parameter 18 and the structural frequency dependent parameter 30 are independent of the numerical values of the angles of the surface nor-mals. Therefore they provide the same result without having to rotate the images. That they are independent of the photo angle indicate that the area between the hook bones is identical independently of the angle, hence that the spine and hook bones are identically identified. For the very acute angle, fewer images were accepted. From this angle, the image is filled largely by the buttock of the cow, and less data from the back is available.


61

Figure 5.11: The spread in parameter values versus the photo angle show an angle

dependence for four of the parameters, for instance parameter 4 (left), but no angular

dependence for the remaining three, as seen for parameter 30 (right).

The spread is plotted in relation to the dynamic range.

5.5 Summary

The results are based on 351 images of Swedish red breed cows captured during a period of three months. Almost all images captured during the observation period were accepted for treatment. The exemption was images without sufficient data in the back area of the cow to find a spine. Therefore the survey also included some poor images with insufficient or noisy data for feature extraction. These images influ-enced the correlation between the parameters and the body condition score as well as the spread of the parameter values, in turn affecting the choice of parameters and the modelling of the data. From the 90 parameter candidates, seven were considered significant by correlation analysis and limits on spread. They are numbered after their order in the 30 parameters left interesting after analysing a model cow. Within this choice, depth measures were found to be are sensitive to image noise and poor image quality. There is a general dependency in the parameter values on the constitution of each individual cow, being the largest contributing factor to the spread in parameter values versus the body condition score. Despite the spread, the parameters are promising, showing good linear correlation with BCS, and at least parameters 18 and 20 show visual asymptotic behaviour. Both fat and

0 0.5 1 1.5 2 2.5 30

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4Spread for parameter 4 versus photo angle

Photo angle

Para

mete

r valu

e

0 0.5 1 1.5 2 2.5 30.75

0.8

0.85

0.9

0.95

1

1.05

1.1

1.15

1.2

1.25x 10

4 Spread for parameter 30 versus photo angle

Photo angleP

ara

mete

r valu

e


62

skinny animals show signs of smaller changes in parameter values, indicating that fat is removed or deposited in other areas than the one in the back used for this study. Comparing the two models, there was no explicit difference in the results from the linear and quadratic model. Both show correlations with BCS better than individual parameters, with the quadratic model showing a minimally better correlation. There results show a good potential in evaluating the body condition score by means of statistical methods. 100% of the calculated BCS came within 0.5 points from the actual score in the training set of data, and 79% came within 0.25 points. In the validation set however, only 46% came within 0.5 points and 20% below 0.25 points of the actual body condition score, indicating a need for further development of the met-hods. The spread in errors when predicting the BCS in the validation set of images is up to 1.5 points for some cows, and cows with low BCS in particular are poorly predicted. These discrepancies from the milder prediction errors for the training set of images are most probable a result of poor images with erroneous parameter values being accepted for prediction. A reasonable conclusion as most of these images were rejected when using a simpler spine-locating method. Another contri-buting factor is the difference between the scorer performing the ma-nual BCS for the learning set of data and the scorers evaluating the validation set. The later were generally more prone to score in the extremes. Importantly, for both the results in the training and in the validation set, is that the model is bias for giving a BCS of 3.3, being the mean human score in the test set, because of the way the least square solution operates. This leads to general over-prediction of the automatic BCS for thin cows and an under-prediction for fat cows. To improve the model, more cows scored in the extremes are required. An even distribution of scores in the data set could possibly be acquired by multiplication of the data for extreme BCS values. Additionally, the spread in the parameter values, generally caused by the cow depen-dency, need to be limited. Despite the errors in the model, the auto-mated method show more objectivity than human scoring when comparing the body condition score to the numbers of days in milk, following the expected trend seen in figure 2.1. Some of the spread in the data is also caused by the data being quite noisy in the square examined. The filter leaves noise of significant size for examination of the small fluctuations of the surface. Especially range and gradient values are sensitive to the noise, in particular when the surface is small with many interpolated points. The noise could


63

dominate the resulting parameter values, especially when constructing a histogram. On the other hand, the filter allows for the analysis of the spinal processes, the most significant feature in the area examined for BCS determination. They are of the same order of magnitude in size as the noise, giving notable results in the entropy of θ and in S(θ). For exa-minations of larger fluctuations in the surface, without extensive inter-polation, the filter is sufficient. An important observation is that the resulting parameter values are sensitive to the photo angle for all para-meters but the value independent entropy and S(θ). It seams that the range and gradient parameters are more sensitive to the algorithm construction than θ measures, which should be considered when selec-ting parameters. In retrospect, it is possible that only angular parame-ters should be used in the BCS prediction. The segmentation function is fast and efficient and the identification of the spine and the hook bones has en excellent hit rate of 100% in accepted images. The trans-form function finding these features is however slow as it interpolates the surface repeatedly, and double values are not excluded, contribu-ting to erroneous surfaces. Additionally, interpolating the surface leads to the acceptance of erroneous images.

64

Chapter 6

Suggestions for future work

Changing the photo angle and using a larger set of animals, together with the measurements of a larger area and more physical

parameters, will create better opportunities for a reliable automatic BCS method.

Many of the problems encountered in my master’s thesis work will be solved by attempting a different photo angle. Sufficient information would most likely be accessed by taking the images from above in a constant calibrated vertical direction. If photographing from above, the segmentation has to be adjusted as to identify when a cow is in the image field. With an edge function, the image object can be identified as one or several cows in comparison to humans or other smaller ani-mals or objects. Simply finding the oblong centre of gravity and pos-sible head and tail candidates in the image will give the direction of the cow in the image, eliminate disturbing neighbouring cows and exclude images with only parts of a cow visible. The top view render possible the localisation of the pin bones for analysis of larger parts of the animal. An above position would also facilitate a simple assembly of the camera in a feeding station or corridor where the cows pass on a daily basis. If necessary, the camera can be complemented by a supple-mentary camera of the lateral view. However one has to be aware of the difficulties of calibrating the image view for different cow heights. Future efforts to improve the existing method should firstly strive to limit the spread in parameter values. Most importantly would be to correct the parameters, or the model, with factors dependent of the physical composition of each individual cow. Another way of doing so

CHAPTER 6. SUGGESTIONS FOR FUTURE WORK

65

is by focusing on parameters that are independent of the angle and posture of the animals, e. g. texture frequency and entropy and spatial origo-independent curvatures, eliminating algorithm dependent spread. It would be interesting to evaluate the relation of the parame-ters with the length/width relationship of the cow. For texture analysis it would be interesting to compare the detailed structure with a smoot-hing data fit of the structure. However, one has to aware of the impact of the image noise. Secondly a larger data set is needed where, even in a normally distri-buted herd, there will be more cows in extreme categories. Possibly, there is also a need of duplication of the data for thin and fat cows, to simulate an even distribution of scores in the herd. Additionally, there is a need for a different approach when extracting the model, adding a condition of a resulting one-to-one relationship between the calculated and actual body condition score. The texture features showed very sensitive to small noise and are diffi-cult to analyse. Measuring physical parameters in addition to the sta-tistical analysis of the spinal processes of the cow makes it easier to verify the robustness of the algorithm by real live measurements on the animal. As physical parameters, I would suggest the spatial curva-ture, or angle, of the hook bones, and the inverse spatial curvature behind the hook bone (see figure 6.1). Furthermore, the top angle, or curvature, of the spinal processes could be measured in the transverse direction from the hook bon bones to the rump, preferably with a fre-quency analysis similar to the one in my work, to find cows with visible vertebras. These parameters must also be corrected by their dependence of the intrinsic shapes of the individual cow. This is easier with physical parameters where it is simpler to see the relationship between different body shapes.

CHAPTER 6. SUGGESTIONS FOR FUTURE WORK

66

Figure 6.1: Illustration of possible locations for physical parameters

to be measured with the TOF camera

A detail not considered in this work is the weighing of parameters that has to be put in consideration. Parameters with a low correlation but a minimal relative spread should be given a larger impact in a model than highly correlated parameters with a large relative spread in va-lues. It would also be interesting not assuming linear relationships and develop the model by e.g. neural networks. In the further development of an automated method of scoring the body condition of cows, a larger number of animals are needed. Based on the number of cows used in earlier studies, I would suggest one or two farms with two thousand animals each, with a large number of cows with a body condition score in the extremes of the range. Images should be captured al least once every day of each animal. To get a robust value representing the cow and the fatness of the cow, I suggest taking the average of the parameter values from measurements in one week. Any error caused by posture or lightning would be eliminated. It should be noted that the change of body condition under the duration of one week is minimal, not producing any significant changes in the values. Additionally, to avoid inconsistencies, it is important to let the same scorer evaluate all animals.

67

Chapter 7

Summary and conclusions

The 3D camera shows excellent potential to be used in an automated system of evaluating BCS. In particular the

fluctuations of BCS over time. With statistical parameters, an objective, if yet not entirely robust, method has been developed for body condition scoring using the 3D data.

The objective of this master’s thesis was to develop and evaluate a prototype algorithm for an image analysis system that could object-tively determine the degree of fattening, the body condition score (BCS), in dairy cows using 3D data. The algorithm works on simple principles. Image data is recorded as four separate matrices containing x, y, range and light intensity infor-mation respectively. Combining the intensity information with range data, the image is filtered and segmented to identify a cow in the ima-ge. The data that belong to the cow is rotated, translated and scaled by simple matrix transformations for normalisation. As reference points, the tail, the spine and the hook bones are localized by finding the mini-mum value in an area of the image, or the transformed image, defined by metric measures on the animal. In the normalized data, a square of is chosen symmetrically between the hook bones for texture analysis. The data in this area is interpolated to enable the use of 2D image ana-lysis methods. The statistical measures of mean, standard deviation, histogram data such as entropy and Fourier data indicating structural frequency are extracted for depth, gradient and curvature values. The curvature is described by the angular deviation between the surface normal and the vertical at each point. Each observation of a cow con-sists of five images, and the mean of each statistical value from these

CHAPTER 7. SUMMARY AND CONCLUSIONS

68

images is recorded. The values are put into the model equation deve-loped to predict the BCS of the cow. The purpose of creating an automated system for body condition sco-ring is to avoid the subjectivity of human scoring, which can differ 0.25 points on the scale of one to five for the same scorer, on the same cow, on the same day. The algorithm developed using the time-of-flight camera for 3D data is truly objective. The results show that the predict-ted BCS follow the decrease expected in a cow after calving even better than the human scoring did during the same observation period. From anatomical knowledge of the fat deposits of cows, an image angle diagonally upwards from the rear of the cow was considered optimal. Due to occlusions occurring in front of the hook bones and in the shadow of the wagging tail, large areas of the cow were difficult to analyse. Instead, an above position of the camera would be optimal, also eliminating the risk of errors originating in angle-dependent para-meters. It was also noted that there are differ-rences in the anatomy of individual cows given the same body condition score by a human scorer. This intrinsic difference is noted in the statistical measure-ments, leading to discrepancies in the BCS pre-diction. Without the consideration of the individual anatomical features of each animal, it is difficult for the indicating parameters to be comparable. Additionally, the method is sensitive to noise and the photo angle, making it less robust than expected. Nevertheless, my work resulted in a completely automated system of BCS prediction. In comparison, earlier studies have used manual identification of reference points and a strict exclu-sion of images with unwanted values. In my thesis work I approached the question of finding indication parameters by the use of statistical methods. Local feature histograms proved useful. In particular, the measure of entropy in a surface is in-dependent of the photo angle and the size of the values. Equally, the analysis of the two dimensional Fourier spectrum of angular deviation between the surface normal and the vertical, indicated the visibility or ordered shapes, such as protruding vertebras. It shows that there is a great potential in the use of local feature statistical measure-ments from range images for the determination of the body condition score in dairy cows. With this master’s thesis I wanted to show the potential in a camera-based system for predicting the body condition score of any given cow. Using a time-of-flight camera for cow body condition scoring has its

CHAPTER 7. SUMMARY AND CONCLUSIONS

69

limitations. The current camera is sensitive to contrasting white and black areas. As the majority of all cows in the world are multicoloured in black and white, this could prove to be a potential problem for a commercial application. Additionally, human scoring is mostly based on receiving the information by palpation to feel the thickness of the subcutaneous fat. This information is inaccessible in a camera-based system, and the geographical information from the three-dimensional images might be insufficient to represent the fatness of the animal. The experience of human scorers also makes it possible to adjust the score to the different physical constitution of the individual cows, even taking differences between races into account. In a camera-based system the model would have to be calibrated for each individual to reach its optimal potential, which is practically impossible. It could however reach the same accuracy as a human scorer. The result in the training set of cows gave 100% of the automatically extracted body condition scores staying within 0.5 points of actual score, and 79% within 0.25 points, which is the expected difference between two human scorers. This shows a very good potential for the camera. To apply the developed method on “any given cow” however, more work is needed to limit the spread in the measures. On a positive note, the problems when trying to give an exact BCS value are not do-minating when following the development of the condition of the cow over time. There is a great potential in creating a method tracing the changes in the condition of the cow that is unrelated to the actual score of the animal. This system could be connected with data from a heard management system, evaluating the feeding and health of each indivi-dual animal. The demand is increasing for a way of in-corporating the BCS in a precision livestock farming system, and the 3D camera shows excellent potential to act as a tool in this system. Also, for research, the principle results from employing a three dimensional camera-based system for scoring dairy cows, show great usefulness. By enabling au-tomatic measurements of the body condition score, large quantities of data can be collected an connected with productivity and feed intake in order to reach a consensus on what constitutes desired body condi-tion for a dairy cow.

70

Bibliography

[1] G. Hetzel, B. Leibe, P. Levi, B. Schiele “3D Object Recognition from Range Images using Local Feature Histograms” Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2:II394-II399, 2001

[2] E.E. Wilderman, G.M. Jones, P.E. Wagner, R.L. Boman, H.F.

Troutt, JR., T.N. Lesch. “A Dairy Cow Body Condition Scoring System and Its Relationship to Selected Production Characteristics” Journal of Dairy Science vol.65 3:491-501, 1982

[3] “Body Condition Scoring as a Tool for Dairy Heard Management” Penn State College of Agriculture. Extension Circular 363. Cooperative Extensions

[4] Adapted from: A.J. Edmondson, I.J. Lean, C.O. Weaver, T.

Farver, G. Webster. “A body condition scoring chart for Holstein dairy cows” Journal of Dairy Science 72:68- 78, 1989

[5] P.J. Hady, J. Domecq, J.B. Kaneene. “Frequency and precision of

body condition scoring in dairy cattle” Journal of Dairy Science 77:1543-1547, 1994

[6] J. C. Pompe, V.J deGraaf, R. Semplonious, J. Meuleman.

“Automatic body condition scoring of dairy cows: Extracting contour lines” Book of Abstracts, 5th European Conference on Precision Agriculture, 2nd European Conference on Precision Livestock Farming, 243-245, 2005

BIBLIOGRAPHY

71

[7] T. Leroy, J.-M- Aerts, J. Eeman, E. Maltz, G. Stojanovski, D.

Berckmans. “Automatic determination of body condition score of dairy cows based on 2D images” Precision Livestock Farming ´05: 251- 255, 2005

[8] Information received indirectly from [9] [9] J. M. Bewley, A. M. Peacock, O. Lewis, R. E. Boyce, D. J. Roberts,

M. P. Coffey, S. J. Kenyon and M. M. Schutz “Potential for Estimation of Body Condition Scores in Dairy Cattle from Digital Images“ Journal of Dairy Science, 91:3439-3453, 2008

[10] P. Negretti, G. Bianconi, S. Bartocci, S. Terramoccia, M. Verna

“Determination of live weight and body condition score in lactating Mediterranean buffalo by Visual Image Analysis” Livestock Science 113:1-7, 2008

[11] Web reference http://www.mesa-imaging.ch , last visit 11.11.2008 [12] Matlab Help file [13] W. Zhao, R. Chellappa, P.J. Phillips , A. Rosenfeld “Face

Recognition: A literary Survey” ACM Computing Surveys, Volume 35, Issue 4, December 2003

[14] K. W. Bowyer, K. Chang, P. Flynn “A survey of 3D and Multi

Modal 3D+2D Face recognition” International Conference on pattern Recognition, August 2004

[15] R.C. Gonzales, R.E. Woods, S.L. Eddins Digital Image Processing

using Matlab Pearson Prentice Hall, New Jersey, 2004 [16] A.E. Johnson. “Spin Images: A representation for 3-D Surface

Matching” PhD Thesis. Robotics Institute. Carnegie Mellon University. 1997

[17] G. Hetzel, B. Leibe, P. Levi, B. Schiele “3D Object Recognition

from Range Images using Local Feature Histograms” Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition 2:II394-II399, 2001

[18] Matlab Help file

BIBLIOGRAPHY

72

[19] Web reference http://www.DeLaval.com/Dairy_Knowledge/Efficient

CowComfort/Body_condition_score.htm, last visit 24.07.2008 [20] W. K. Pratt Digital Image Processing Wiley Interscience,

Fourth Edition, 2007

[21] Web reference http://www.rp-photonics.com/time_of_flight_measurements.html,

last visit 17.11.2008

[22] Web reference http://www.soe.ucsc.edu/classes/cmps290b/Fall07/TimeOfFlight, last visit 17.11.2008

[23] Interviews, conversations and lectures with and given by employees at DeLaval are the sources of most of the information about cows given in this thesis.

73

Appendix A

Body condition scoring method

A detailed description of how to asses the body condition score of a dairy cow [19].

BCS = 1

This cow is very emaciated (see figure A.1) and extremely rarely seen on a farm.

Figure A.1: Body condition score 1. Left: Emaciated cow. Right: No fat is deposited over the

spinal processes which has a distinct overhanging shelf effect.

APPENDIX A – BODY CONDITION SCORING

74

BCS = 1.5 This cow is too thin and is hopefully rarely seen on a farm. It will not milk well or reproduce as is probably not healthy. The vertebrae, short ribs, hook bones, pin bones, and tail head are very sharp and visible. One-half of the length of the spinal processes is visible. The ligaments are easily seen. The area around the tail head and the dish of the rump (thurl) are very dished. There are folds of skin seen between the tail head and pin bones. BCS = 2 This cow is very thin, causing low milk production and poor reproduction. It may be of good health. The spine and short ribs can be easily seen, and the short ribs appear scalloped, but individual vertebrae are not really apparent. The upper surfaces of the short ribs can be readily distinguishable by palpation. One-half to a third of the length of the spinal processes is visible and the processes have a slight overhanging shelf effect (see figure A.2). The hook bones and pin bones stand out, with no fat felt on the pin bones. The ligaments are sharp and easily seen. The areas around the tail head and the thurl area are very dished. There are folds of skin between the tail head and pins.

Figure A.2: Body condition score 2. Left: Thin cow. Right: Little fat is deposited over the

spinal processes with a slight overhanging shelf effect.


75

BCS = 2.5 It is a reasonable goal not to have more than 10 percent of the herd scoring 2.5 or less. This is the lowest acceptable condition score. A cow with a score of 2.5 has vertebrae showing but they cannot be seen as individual bones. The short ribs can be counted but are not scalloped. One-third to a quarter of the length of the transverse processes is visible. The ligaments are easily seen but not as sharp as with a BCS of 2.0. Both the hook bones and pin bones are angular but some fat can be felt on the pin. The areas around the tail head and thurl are dished. BCS = 3.0 This cow could be a healthy, high-producing cow. But, if a cow calves in at a score of 3.0 or less, she may not have enough body fat to use for high peak milk production and to carry her through until dry matter intake increases. At this score, the dish of the rump (thurl) is at the transition between being U-shaped and V-shaped (see figure A.3, left). Any cow under a BCS of 3.0 has a V-shaped thurl area. The backbone can be seen but the individual vertebrae are rounded. Covering the short ribs is half to one inch of flesh. Less than quarter the length of the spinal processes is visible. They are discernible if applying slight pres-sure and together the processes appear smooth and there is no noticeable overhanging shelf effect. There is fat covering the ligaments but they are still obvious. The hooks and pins have some fat that can be felt. The area around the tail head is dished, without sign of fat deposition, but no folds of skin are seen.

Figure A.3: Body condition score 3. Left: Average cow. Right: There is some fat on the spinal

processes, and there is no overhanging shelf effect.


76

BCS = 3.5 Dry cows and calving cows are recommended to have a body condition score of 3.5. On this cow, fat can be felt on the backbone, short ribs, and ligaments. The hook bones and pin bones are rounded. The spinal processes cannot be distinguished separately. The thurl is somewhat dished. The coccygeal (tail head) ligament is barely visible but the sacral ligament can still be seen. The area around the tail head is rounded and filled in but with no deposition of fat. BCS = 4.0 This is a fat cow. Cows calving in at this condition will eat less, lose more weight and have more metabolic problems. The spinous processes appear flat or rounded and can be distinguished only by firm palpation. There is no overhanging shelf effect and the lateral edge of the processes is rounded. This cow’s back is flat because fat has filled it in (see figure A.4). The short ribs can not be seen individually but they can just barely be felt. The hook bones and pin bones are obviously fat. The “U” between the hooks and pins is very flat with no depression. The ligaments cannot be seen. The area around the tail head is filled in and folds of subcutaneous fat are seen.

Figure A.4: Body condition score 4. Left: Fat cow. Right: The back is flat, and there is no

overhanging shelf effect.


77

BCS = 5.0 This cow is very obese and will have metabolic and breeding problems. The backbone and short ribs cannot be seen and are hard to feel. Bone structures of the vertebral column, spinous processes, hooks, and pin bones are flat and covered with flesh (see figure A.5). The thurl is totally filled in. The tail head is buried in fat.

Figure A.5: Body condition score 5. Left: Obese cow. Right: The back and the edges of the

spinal processes are rounded.

78

Appendix B

Cows used for data acquisition

Table B.1.1: List of cows used in the data collection and their score

Cow 17.06.2008 Image 08.07.2008 I 23.07.2008 I

85 2.5 X Slakt i juli

151 3.75 X kalvat 20/6 3.5 X no (3,625) X

169 3.0 X kalvat 14/6

173 2.75 X slakt sept

186 3.25 X kalvat 20/6

189 3.0 X Dr. planerad kalvning sept

193 4.25 X kalvat 29/6 3.75 X no (3,625) X

194 3.25 X kalvat 22/6 3.5 X no (3,375) X

221 3.25 X kalvat 29/6 3.25 X no (3,125) X

222 3.25 X kalvat 20/6

226 3.75 Utsläppt Dr. planerad kalvning aug

227 X Död

233 3.0 X Dr. planerad kalvning aug

282 3.5 X kalvat 23/6

285 3.25 X kalvat 27/6

291 2.75 X kalvat 8/6 2.5 X no (2,75) X

301 4.0 Utsläppt Dr. planerad kalvning sept

311 3.75 X Dr. planerad kalving okt

357 3.25 X kalvat 21/6

5660 2.75

5745 2.75

5955 2.5

6004 3.75 X

6009 3.0

6010 2.0

6013 2.25

6019 3.0

6025 2.75

6058 3.5 X

6073 3.75 X

APPENDIX B – COWS USED FOR DATA ACQUISITION

79

Table B.1.2: List of cows used in the data collection and their score, continuing

06.08.2008 I 19.08.2008 I 03.09.2008 I 17.09.2008 I

3,75 X Slaktko. Flyttad till VMS 3,75 3,5 X

3,5 X 3,5 X 3,25 X

3,25 X 3,25 X 3,25 X

3 X 3 X 3 X X

3 X Hade mastit 18/6. Streptokocker 3 X 3 X

APPENDIX B – COWS USED FOR DATA ACQUISITION

80

Table B.2: Calculated and actual BCS in a test set of new cows. The manual scores are

evaluated by expert scorers at DeLaval

Cow ID

Calculated BCS

Actual BCS Observations

123 3,53 4

133 3,50 4

147 3,79 4

164 3,46 3,75 The back is completely flat

166 3,49 3

184 3,20 3,25

189 3,18 2,25

191 3,18 3

237 3,79 4

246 3,49 4,25

252 3,56 4

258 3,78 2,5

268 3,38 -

291 3,39 2,5 Mastitis

299 3,03 3,75

300 4,15 3,75 The back is completely flat

303 3,70 3,25

306 3,48 3,5

324 3,50 3,25

334 3,37 3

350 3,57 - Heifer

351 3,65 3,25

361 3,39 3,25

364 3,25 -

370 3,47 - Heifer

374 3,48 - Heifer

377 3,20 3

385 3,31 -

389 3,66 3 After first calving

391 3,23 - Heifer

085 3,51 3

5660 11,25 - Holstein

5745 7,16 - Holstein



TRITA-CSC-E 2009:009 ISRN-KTH/CSC/E--09/009--SE

ISSN-1653-5715

www.kth.se

automatic determination of body condition score … · automatic determination of body condition...

Documents