Exploring Attributions in Convolutional Neural Networks for Cow Identification

Abstract

Face recognition and identification is a method that is well established in traffic monitoring, security, human biodata analysis, etc. Regarding the current development and implementation of digitalization in all spheres of public life, new approaches are being sought to use the opportunities of high technology advancements in animal husbandry to enhance the sector’s sustainability. Using machine learning the present study aims to investigate the possibilities for the creation of a model for visual face recognition of farm animals (cows) that could be used in future applications to manage health, welfare, and productivity of the animals at the herd and individual levels in real-time. This study provides preliminary results from an ongoing research project, which employs attribution methods to identify which parts of a facial image contribute most to cow identification using a triplet loss network. A new dataset for identifying cows in farm environments was therefore created by taking digital images of cows at animal holdings with intensive breeding systems. After normalizing the images, they were subsequently segmented into cow and background regions. Several methods were then explored for analyzing attributions and examine whether the cow or background regions have a greater influence on the network’s performance and identifying the animal.

1. Introduction

Cattle identification and the ability to precisely pinpoint the individual among many other animals in the herd are critical to modern livestock management. They enable effective tracking of animal movement, the traceability of products of animal origin, and health monitoring, including surveillance of contagious disease outbreaks. They also enhance optimization of production and performance and improve animal welfare. Traditional identification methods such as ear tags and RFID transponders are susceptible to loss, tagging errors, or fraud, necessitating the need for automated, non-invasive solutions [1]. In response, computer vision and deep learning have emerged as promising tools for animal identification, offering high accuracy and scalability. With the advances in the field of computer vision, animal recognition has undertaken new development through visual biometrics [2,3]. With the use of machine learning and a set of algorithms for image processing and recognition, researchers have successfully identified several vertebrate species [4,5] including wildlife [6,7,8,9] and farm animals like sheep [10], pigs [11,12], and cattle [13,14,15]. Particular success was achieved with the cattle recognition process using deep learning and a CNN [13,16,17]. Researchers went further in the identification of face images and cow face parts [18,19] and even distinguishing particular cattle [20,21,22].

Convolutional neural networks (CNNs) are widely used in image classification [23], object detection [24,25], and identification [26], because they can automatically learn complex patterns in data. Recent advancements in CNNs have demonstrated strong performance s in biometric recognition across multiple domains, including human face recognition, wildlife monitoring, and livestock identification [27]. However, while CNNs excel at learning distinguishing patterns, their decision-making processes remain largely opaque, raising concerns about their robustness and generalization in real-world farm conditions [28].

These networks operate largely as “black boxes”, and it is often unclear which parts of an image influence network’s predictions [29]. In particular, CNN-based animal recognition models may not solely rely on biometric features but also on environmental cues, leading to potential overfitting to background elements rather than the animal itself. This lack of interpretability becomes a significant limitation, especially when the purpose is to assess the abilities of the model to generalize, with limited data available.

It is important to emphasize that the use of computer vision and deep learning is explored for the task of identification. It involves a search process, where the goal is to pinpoint the exact match from a database. The images in the database, in general, are not available during training. For example, when identifying a cow in a farm, the system first creates a database with all cows in the farm. Then, for a new image of a cow, it checks the database to find the closest match.

In contrast, the task of recognition compares the input against predefined classes or categories. In general, the model is trained with images that represent these classes. For example, in the task of recognizing a cow in a farm, each cow represents one class, but there can be many images for one cow. The model has seen all classes during training and directly outputs the predicted class. In livestock farming, the benefits of individual recognition include health monitoring, improving welfare, and breeding [30].

The task of re-identification is recognizing an individual across different instances or contexts. It is similar to identification, but does not compare an instance to a database [31].

One of the main challenges in CNN-based animal identification is ensuring that models focus on biometric features (e.g., facial patterns, body structure) rather than on background features [32]. Previous studies have shown that animal recognition models trained in one environment often fail under changed conditions, suggesting that CNNs learn dataset-specific artifacts rather than biologically relevant animal recognition features [33]. This issue necessitates the use of explanation techniques when training the model to understand which image regions contribute most to classification decisions.

In order to help the model to generalize better, images of the animals in different environments should be taken. However, this is a challenging task due to the need to keep track of the individual cows; it appears to be also time-consuming. Hence, it would be useful if the research could apply attribution methods to detect the lack of generalization due to the background in the images. Models that do not generalize well have poor performance on unseen objects or environments. Identification systems that have poor performance are not useful in real farms.

Data Shapley values are defined in [34] as a method to reliably assess the contribution of data records to the performance of the model.

Based on the previous research in the application of machine learning in the field of cattle face recognition [18,35,36], the present paper focused on cow face identification with the aim of exploring the effect of the background on the generalization of an identification model trained on dairy cows from the Holstein breed. For this purpose, a procedure that quantifies the influence of the background in relation to the performance of the identification model was proposed. Some attribution methods were investigated in order to determine their capabilities to quantify the influence of the background without the need for intensive segmentation. The existence of such an attribution method would allow the builders of identification systems to assess the model’s ability to generalize to different backgrounds by inspecting the attribution output. Attribution methods in image tasks aim to explain model predictions by identifying the importance of input pixels or regions that contribute to the output. Techniques like saliency maps, occlusion, and gradient-based approaches visualize these contributions, offering insights into how models process visual data. However, the research cited focuses either on classification tasks and/or evaluates the results only visually. The present paper proposes a test that can numerically compare the results of a model with the output of an attribution map.

Within the current initial stage of the study, two related questions were posed: (i) Does the cow (foreground) or the surrounding background contribute more to the results of the identification task? (ii) Is there an attribution method that can quantify correctly the contribution of the foreground vs. the background for the performance of the model?

2. Materials and Methods

The present study, which is the first phase of an ongoing two-year project on cow face visual identification, included creation of datasets, training, and exploration of different methods to evaluate influence of background in process of cow identification.

An experimental dataset was created and introduced, containing images of 168 dairy cows from the Holstein breed, consisting of 760 images in total. The images were taken in several farms, situated in Southern Bulgaria in the period from 1 April to 30 June 2024. The photos were manually captured on-site in daylight with Panasonic Lumix DMC-GX8 camera (Kadoma City, Japan) with a Panasonic Leica 15 mm f/1.7 lens. Each cow has at least three images. Near-duplicate images were removed by visual inspection to make sure the test results would not be compromised. It should be noted that on some of the images more than one animal was visible. A record kept for correct identification of the individual cows. The image resolution was 908 pixels (width) and 1210 pixels (height) with a horizontal and vertical resolution of 72 dpi.

The images from the dataset were annotated after with the identity of the particular animal. In this process of the dataset preparation, we used Roboflow resources [37,38] which is a cloud-based platform that streamlines image annotation and preprocessing. The images were uploaded and segmented using built-in tools, ensuring a standardized dataset for training and evaluation. During the procedure the cow regions were manually annotated, ensuring precise bounding lines for segmentation between cow and foreground. Using standardized annotation techniques with Roboflow enhanced the model reliability, as previous studies have shown that consistent dataset preprocessing significantly improves classification accuracy [39].

The segmentation was completed for all 760 images, clearly defining foreground and background regions. Then, the background was replaced with white noise in the preparation of foreground images and the reverse for the background images, respectively.

The prepared data were afterwards split into training, validation, and test datasets with approximate ratios of 34%, 33% and 33% of the cows. When a cow was randomly assigned to one of the splits, all its images became part of the split. This ensured that the cows in the validation and the test set were not seen during training.

For the training process, the last layer of a ResNet50 [35] model was fine-tuned on an Nvidia RTX A4500 (Nvidia, Santa Clara, CA, USA) graphical processing unit with 20 GB of VRAM on a cloud-based platform. The use of a pre-trained model allowed the efficient of the network on high-performance hardware, significantly reducing the computational time. During training, the following random augmentations were applied: resized crop, horizontal flip, rotation, color jitter, affine transform, erasing, and Gaussian noise.

The loss function used was triplet loss [26], which is designed for identification. The model learns by pulling similar items closer together in an embedding space while pushing dissimilar items further apart. The loss function operates on triplets of samples, each consisting of an anchor (a reference sample), a positive sample (similar to the anchor), and a negative sample (dissimilar to the anchor).

The triplet loss function can be mathematically expressed as follows:

$$L(a,p,n)=max\left(d\right(a,p)-d(a,n)+\alpha ,0)$$

(1)

where $d(a,p)$ is the distance between the anchor, a, and the positive sample, p; $d(a,n)$ is the distance between the anchor and the negative sample, n; and $\alpha$ is a positive number that ensures a gap between the distances of positive and negative pairs.

Triplet loss is widely used in various fields, including face recognition, where it was introduced in the FaceNet paper [26]. For animal re-identification, it has been used in [31].

The following evaluation of attribution methods was proposed:

• Compute the Shapley value of the accuracy of the model for the foreground and the background;
• Compute the average contribution for a pixel in the foreground and in the background;
• Compare the previous two values.

The hypothesis was that if the average attribution was close to the Shapley value, then it could be assumed that the attribution method correctly allocated the importance according to the model.

In the definition of the Shapley value in Equation (2), i is either foreground or background; the set of players N has two elements: foreground and background; the value or the payoff v is the accuracy of the identification model.

$${\varphi }_i=\sum _{S\subseteq N\setminus \left\{i\right\}}{\displaystyle \frac{\left|S\right|!(n-|S|-1)!}{n!}}\left[v(S\cup \{i\left\}\right)-v\left(S\right)\right]$$

(2)

More specifically, the definition of $v\left(S\right)$ for the different values of S is as follows:

• Foreground: The foreground was left and the background was replaced with white noise for both the test and reference images; $v\left(S\right)$ is the accuracy of the model over these images.
• Background: The process was executed as previously mentioned, but there the background was left and the foreground was replaced with white noise.
• Empty set: $v(\varnothing )$ is set to the expected probability of being correct if the individual is guessed at random.
• Foreground and background: $v\left(S\right)$ is the accuracy of the model on the original images.

It should be noted that the model is kept fixed during the evaluation of the accuracy, and it is trained on the original images. The foreground and the background images are presented in Figure 1.

The attribution methods that were chosen for identifying the regions of the cow face images that contribute the most to the final output were gradient-based approaches [40]. Several gradient methods are in use in computer vision and machine learning, but relatively few have been proposed for triplet loss networks. In the present study, we used adaptations of Vanilla gradient, Grad-CAM [41], and EmbeddingCAM [42] in order to highlight the parts of the cattle face images that appear to be most important for the decision made by the CNN in the identification process.

The Shapley value for the foreground and background images was computed to measure the relative contributions, similar to research where the Shapley value assesses the importance of the dataset images [43]. In experimental research [40] Grad-CAM showed the highest accuracy among other Explainable AI methods (XAI). [44] suggested that particular modifications of the Grad-CAM method perform even better. However, the evaluation of the goodness of the explanations in most of the research still relies on a review by a human expert. In the current experiment, the use of Shapley values aims to give an objective measure of the attribution of a region in the image with regard to the trained model.

Vanilla gradient [45] is a technique used to interpret neural network predictions by examining the gradients of the output with respect to the input features:

$${\displaystyle \frac{\partial f_w(a;p,n)}{\partial a}}$$

(3)

where $f_w(a;p,n)=MSE(a,p)-MSE(a,n)$ is the difference between the mean squared errors (MSE) in the positive and negative images; a is the anchor image, and w are the weights of the model. This approach is being used in order to help identify which input features are most influential in the network’s decision-making process by highlighting areas where slight changes in input lead to significant changes in the output. By computing the gradients, the influence of each input pixel (in the case of images) on the final prediction could be traced, providing insights into which aspects of the data are deemed important by the model.

This method has several limitations that have been addressed in new methods [46]. Gradients are often noisy, meaning they can highlight irrelevant or uninformative pixels alongside meaningful features. This noise can make it difficult to interpret the results and may obscure the true areas of interest in the image. In Figure 2 this can be clearly seen over one of the test images.

Grad-CAM combines gradients with feature maps to produce more spatially coherent, class-discriminative visualizations [47].

The Grad-CAM for an image a is defined as follow:

$$L_a^{\text{Grad-CAM}}=\mathrm{ReLU}\left(\sum _k{\alpha }_k{A}^k\right)$$

where $A^k$ is the k-th feature map of the last convolutional layer and ${\alpha }_k$ is the importance weight computed as follows:

$${\alpha }_k={\displaystyle \frac{1}{Z}}\sum _i\sum _j{\displaystyle \frac{\partial l}{\partial A_{ij}^k}}$$

In the experiments carried out, l is defined as the average mean squared error for a over the positives minus the average mean squared error over the negatives, Z is a normalization constant that gets canceled during comparison and visualization, and ${\displaystyle \frac{\partial l}{\partial A_{ij}^k}}$ is the gradient of l with respect to the feature map activation. An example of Grad-CAM can be seen on Figure 3.

Ref. [41] adapts Grad-CAM for embedding networks by using gradient weights computed from multiple training examples. During testing, the method identifies the closest training example to the test image and uses that example’s gradient weight to generate a heatmap for the test image. This approach, called Average Grad-CAM, requires multiple training images and an index to efficiently retrieve the nearest training sample for generating heatmaps for new inputs (Figure 4).

A more recent development in [42] proposes a modification to Grad-CAM suitable for embedding networks—EmbeddingCAM. It uses a proxy point $p_c$ in the embedding space, where the index c is analogous to the class in classification. In the present experiments c ranges over the individual cows in the test dataset.

For the proxy point, the average embedding was taken for each cow c.

$$p_c={\displaystyle \frac{\frac{1}{n}{\sum }_{j\in c}{y}_j}{\left|\right|\frac{1}{n}{\sum }_{j\in c}{y}_j\left|\right|}}$$

The loss is defined as the vector product between the normalized embedding of the anchor y and $p_c$

$$L_c\left(y\right)=y·p_c$$

This method is visualized in Figure 5.

3. Results

The accuracy of the model over the original images is 92.2%; while using only foreground, it drops to 40.2%. Using only the background gives a slightly better accuracy of 50.0%. (

Table 1. The accuracy over test and reference images and the corresponding Shapley value.

	Original	Background	Foreground	Empty
Model accuracy	92.2%	50.0%	40.2%	2.0%
Shapley value	92.2%	86.3%	76.5%	2.0%

). This suggests that the model uses both the foreground and the background to make the embedding of the images, with the background having a little more importance.

In

Table 2. The ratio of the Shapley values for foreground to background compared to the gradient attribution methods.

	Foreground:Background
Shapley value	80.4%
Vanilla gradient	100.0%
Average Grad-CAM	98.8%
EmbeddingCAM	117.7%

, the ratio of the foreground to the background was compared for the Shapley values and the three gradient methods: Vanilla gradient, Average Grad-CAM, and EmbeddingCAM. For each method, the mean weight was computed for the regions defined as foreground and background. A method that has a ratio of the foreground to the background close to the Shapley value ratio would be more desirable.

4. Discussion

Deep learning and convolutional neural networks (CNNs) are successfully used in objects and patterns recognition through processing massive data [48,49]. The recognition system is based on learning of specific features that are extracted from a dataset of images [50,51]. The recent application of CNNs in veterinary medicine is focused on disease detection and prevention [36,52]; the present study focused on cow face identification. In this process, a dataset of images was created and annotated afterwards, as perceived by [53], which as an approach was also applied by [49]. The subsequent segmentation of the images into foreground and background completed for all 760 images, was found to accurately define the features needed for the analysis [54,55]. Ref. [56] argued that the segmentation technique could increase the accuracy of the experiment of recognition.

For the segmentation of the original images, the chosen method included the application of white noise for replacement of the background in the preparation of foreground images, and the reverse technique was applied for the background.

The analyses of [57,58,59] were in favor of such a multi-center approach.

The attribution method chosen in the study for determining the most important regions from the cow face images to the final output was a gradient-based approach. Several gradient methods are in use in computer vision and machine learning [60]. Among those described in the scientific literature, within the current experiment there consecutively were applied the Vanilla gradient [61], Grad-CAM [62], and EmbeddingCAM in order to highlight the parts of the cattle face images that appear to be most important for the decision made by the CNN in the recognition process. The Shapley value was computed to measure the relative contribution of values of the tested model, as described by [63], tracking the influence of the foreground and background images in the present study. In experimental research [64] Grad-CAM showed the highest accuracy among other explainable AI methods (XAI).

With heatmaps of test images, ref. [65] suggested that particular modifications of the Grad-CAM method could perform even with better interpretable results.

While gradient methods are derived using intuitive reasoning, they do not have provable guarantees on their correctness. Shapley values based attribution methods, on the other hand have game-theoretical guarantees, but they are prohibitively slow to compute exactly for most applications. The popular Python 3 package SHAP [66] has practical modifications of the Shapley-based attribution algorithms. In the current experiment the process covered only two classes, hence this exponential complexity of the computation was not perceived as a problem.

During the analysis, the experimental model showed an accuracy of about 92.2% over the test cow images. These results were comparable to other research studies on image classification with artificial neural networks, as [67] reported an accuracy of 92.9%, and ref. [59] reached an accuracy of 91.66%. These findings show that the triplet loss architecture easily outperforms previous architectures like Fisherfaces [11], which achieved 77% accuracy on the recognition task over 10 pigs, despite the much larger training.

When segmented and using only the foreground (cow), the model’s accuracy dropped significantly (40.2%). Using only the background showed a slightly higher accuracy (50.0%), indicating that the model relies on the background as well, which is not desirable in the process of cattle identification in real environment. With a Shapley value of 80.4% for the foreground to background ratio, the model determined that Average Grad-CAM values of 98.8% are the closest to the target, while using EmbeddedCAM favors foreground influence (117.7%).

From a practical perspective, improving individual cow identification accuracy could have significant implications for precision livestock farming, including the Holstein breed which represents the predominant share of dairy cows in Bulgaria. A robust AI-based identification system would enable automated health monitoring allowing correctly recorded cattle medical history reports, thus reducing the risk of misdiagnosed treatments [68]. Further benefits would allow the tracking of individual movement patterns to help identify early signs of illness, stress, or abnormal behaviors, allowing farmers to take proactive measures for ensuring high standards of animal welfare [69].

5. Conclusions

The present study, which applied a triplet loss function to identify dairy cattle based on biometric features, revealed that none of the evaluated gradient attribution methods adequately captured the Shapley value in estimating the contribution of foreground and background regions to model accuracy. The model relied more on background features for identification, while the tested methods—Vanilla gradients, Grad-CAM, and EmbeddingCAM—could not confirm this conclusively. This finding raises a problem for real-world applications of the method, as a working identification system must prioritize biometric features over environmental cues to ensure robustness and generalizability across farms.

The findings presented, while offering a novel methodology in the field of animal identification, emphasize the need for bias mitigation techniques in CNN-based livestock identification. They provide a basis for further exploration if a new gradient method for triplet loss networks or a modification of the presented methods can more accurately reflect the model’s focus on biometric features. These advances will provide an opportunity for facilitating the implementation of AI systems in the precise livestock farming, improving farm management, breeding, genetic selection, animal health, and welfare management by increasing the accuracy in animal detection and identification.