Mask R-CNN and Centroid Tracking Algorithm to Process UAV Based Thermal-RGB Video for Drylot Cattle Heat Stress Monitoring

Abstract

This study developed and evaluated an algorithm for processing thermal-RGB video feeds captured by an unmanned aerial vehicle (UAV) to automate heat stress monitoring in cattle housed in the drylots. The body surface temperature (BST) of individual cows was used as an indicator of heat stress. UAV data were collected using RGB and thermal infrared imagers, respectively, at 2 and 6.67 cm per pixel spatial resolution in Spring 2023 (dataset-1) and Summer 2024 (dataset-2). Study sites were two commercial drylots in Washington State. The custom algorithms were developed to: (1) detect and localize individual cows using a Mask R-CNN-based instance segmentation model combined with centroid tracking; and (2) extract BST by averaging the thermal-imagery pixels for each of the segmented cows. The algorithm showed higher detection accuracy with RGB images as input (F1 score: 0.89) compared to thermal (F1 score: 0.64). BST extraction with combined RGB and thermal imaging approach required corrections for alignment problems associated with differences in optics, imaging field of view, resolution, and lens properties. Consequently, thermal imaging-only approach was adopted for assessing real-time cow localization and BST estimation. Operating at one frame per second, algorithm successfully detected 72.4% and 81.65% of total cows in video frames from dataset-1 (38 s) and -2 (48 s), respectively. The mean absolute difference between algorithm output and ground truth (BST_GT) was 2.1 °C (dataset-1) and 3.3 °C (dataset-2), demonstrating satisfactory performance. With further refinements, this approach could be a viable tool for real-time heat stress monitoring in large-scale drylot production systems.

1. Introduction

Commercial dairy and beef cattle farming has been a major US agricultural activity in several states, with total cash receipts of about USD $88 billion dollars [1]. Efficient management is imperative to sustain or increase the revenue while improving quality and maximizing beef and dairy production [2]. Dairy and finishing beef cattle in the South and West of the United States are primarily housed in the drylots. These facilities are common due to their low-cost investment and easy maintenance. However, because drylots are predominantly found in hot and dry regions (e.g., Central Washington, California, and Kansas), heat stress during the summer has become a key challenge for the cattle industry. Timely identification of cattle experiencing heat is a crucial step for effective heat abatement. During heat stress, cattle cannot dissipate excess body heat caused by elevated ambient temperatures [3,4,5], triggering a range of behavioral and physiological alterations [6,7]. Additionally, cattle experiencing heat stress redirect their energy towards thermoregulation, decreasing growth and productivity [8].

Quantifying heat stress in cattle production systems, especially in the large drylots, requires rapid and non-invasive methods [9]. Traditional methods like visually observing animal behavior are subjective and laborious [10]. Weather data-driven thermal indices lack specificity [11]. Individual sensing devices such as 2D/3D accelerometers, internal body or surface temperature loggers, and thin film pressure sensors have gained popularity for tracking individual cow behaviors towards heat stress. However, they are either invasive or require extensive infrastructure (e.g., antennas, Wi-Fi, electricity) and are often cost-prohibitive for large feedlot operations.

In this context, unmanned aerial vehicles (UAVs) equipped with thermal-RGB imaging sensors (DJI Zenmuse XT, DJI Technology Co., Ltd., Shenzhen, China) could be beneficial as they offer a noninvasive approach to quantifying body surface temperatures (BST) and heat stress [12]. In recent years, consumer-grade UAVs have substantially improved autonomy and ease of operation. The UAV-based solutions have been explored in cattle production and behavioral studies for identification, enumeration, and feed intake assessment [13,14]. However, their real-time, high-throughput heat stress monitoring application has been limited [9,15].

The capabilities of UAV-based low-altitude aerial imaging and deep learning algorithms, particularly convolutional neural networks (CNNs), have been leveraged to extract meaningful information. For example, Andrew et al. [16] emphasized the advantages of using UAS imagery and deep learning algorithms for cattle localization and counting to help ranch owners monitor the cattle’s health and well-being, as well as feed and water intake. Particularly, Mask R-CNN-type algorithms have been used for automated and efficient cattle localization and counting [17]. This algorithm effectively addresses challenges associated with low contrast imagery and cattle localization in clustered animals [17,18]. However, continual animal localization and tracking after initial detection is essential for quantifying stress levels of individual cows, particularly using UAV-based aerial video streams [17].

Several studies have focused on developing and evaluating algorithms for accurately counting and tracking the cattle. Deep learning based multi-object tracking (MOT) algorithms have been widely experimented with varying levels of accuracy [19,20]. Myat Noe et al. [21] presented a Simple Online Real-time Tracking (SORT) model that demonstrated higher cow tracking success rates. Mg [22] conducted a thorough comparison of eight deep learning-based tracking algorithms (i.e., SORT, Deep SORT, Modified Deep SORT, ByteTrack, Centroid, Centroid with Kalman filter, Intersection over Union (IOU) tracking, and Customized Tracking Algorithm (CTA), to determine the most optimal method for precise and efficient individual cattle tracking. Luo et al. [23] introduced an intelligent grazing UAV that utilizes you-only-look-once (YOLO v5) detector, and the kernel correlation filter (KCF)-based automatic tracking framework to identify and locate individual cattle accurately. Most of the above efforts were, however, focused on improving the deep learning model accuracy using RGB images alone, with no significant efforts towards heat stress monitoring. Moreover, very limited studies existed for real-time cattle detection and tracking using thermal infrared imaging modality.

This study aimed to overcome the above limitations by developing an algorithm for real-time detection, tracking, and heat stress estimation of an individual cow in the commercial drylots using UAV-based thermal-RGB video streams. The specific objectives were to:

• Develop an algorithm to analyze UAV-based thermal and RGB video streams in real-time for identifying and tracking individual cows and quantifying body surface temperature as an indicator of heat stress.
• Evaluate the developed approach in commercial drylot cattle operations to provide actionable real-time insights on cattle heat stress.

2. Materials and Methods

2.1. Study Site

The study was conducted at two commercial drylot cattle operations in central Washington State (WA), USA. Both sites fall under the cold semi-arid class of Köppen climate classification, with maximum and minimum temperatures of 28.9 °C and −3.4 °C, respectively (PRISM Climate Group, 2020). Detailed weather data for these periods from nearby station (source: WSU AgWeatherNet) can be found in Appendix A,

Table A1. Weather data on the days of data collection during spring and summer at the study sites in central Washington (Source: WSU AgWeatherNet).

	16 March			26 July
	Min.	Mean	Max.	Min	Mean	Max
Air Temperature (°C)	−3.4	4.4	13.2	8.2	18.7	28.9
Relative Humidity (%)	30.4	62.3	92.9	14.2	48.1	84.3
Solar Radiation (MJ/m2)		17.6			26.5
Precipitation (mm)		0			0
Wind Speed (m/s)		1.4			1.8
Wind Gust (m/s)		9.3			4.2

. Site-1 (Figure 1a,e) had a mix of 2–6 month-old beef and dairy calves confined within pens, whereas Site-2 (Figure 1c) was an open lot pen with adult dairy cows only.

2.2. Data Collection

Data were collected at both sites on the 16 of March 2023 (Spring; year 1) and the 26 of July 2024 (Summer; year 2) using a UAV platform (DJI Mavic-3T, DJI Technology Co., Ltd., Shenzhen, China) equipped with an onboard thermal-RGB (Red-Green-Blue) imager. The RGB imager was a 48 MP wide-angle camera with 84° field of view (FOV) and 4000 (horizontal: h) × 3000 (vertical: h) pixel resolution. The thermal imager had 61° FOV with 640 × 512 pixel resolution. Data (Figure 1) were collected as images at the nadir using waypoint flight mode (Figure 1a–d) and video at an oblique angle (Figure 1e,f) during solar noon (11 a.m.–1 p.m. pacific) from an altitude of 50 m at 5 m/s flight speed. This resulted in images with a ground sampling distance (GSD) of 2 cm and 6.67 cm per pixel for RGB and thermal infrared imagery, respectively. Note that data collected in Spring 2023 were used for algorithm development, validation, and testing, while the Summer 2024 data were used for independent evaluation of the algorithm’s robustness and real timeliness.

2.3. Cattle Tracking and Body Surface Temperature Monitoring

Data collected as images and videos were utilized to develop an algorithm (Figure 2) for individual cow identification, tracking, and BST estimation. The algorithm comprised three major steps: (i) training and evaluating a CNN for segmenting and counting each cow instance [24]; (ii) tracking individual cows by assigning unique identification numbers to avoid duplicate counts; and (iii) extracting the mean BST of each tracked cow. The following section details methods associated with each of these steps.

2.3.1. Cow Detection through Instance Segmentation

For tasks involving instance segmentation, the Mask R-CNN framework [24] has been widely utilized in various applications, including cattle counting, segmentation, and body contour extraction [24,25,26]. Therefore, this study employed the same architectural framework. For model training, data from both thermal and RGB imaging modalities collected during the spring of 2023 were utilized as input. Static images collected through waypoint navigation, as illustrated in Figure 1a–d, were used directly. However, video streams were processed to extract individual frames for further analysis. The original dataset included 174 images—155 RGB and 29 thermal. Of these, 127 images were captured from a nadir perspective and 47 from an oblique angle. These images, randomly selected, featured between 5 and 84 cows each, totaling 4323 instances. This diversity in the dataset ensured model robustness across different imaging modalities and viewing angles.

Prior to model training, data were preprocessed, augmented, and annotated using Roboflow (Roboflow Inc., Des Moines, IA, USA). Initially, thermal and RGB imagery were resampled to a resolution of 1024 × 1024 pixels, ensuring no image distortion. Subsequent preprocessing steps included grayscale conversion, contrast adjustment through adaptive equalization, augmentation involving horizontal and vertical flipping, brightness adjustment (ranging from −25% to +25%), and noise level alterations affecting up to 11% of pixels. Following these steps, 344 images, encompassing 12,962 instances of cows, were annotated using a polygon method in Common Objects in Context (COCO JSON) format, with unique identifiers assigned to each label (Figure 3).

Before model development, the dataset was split into three subsets: training (91%, 312 images), validation (7%, 23 images), and testing (3%, 9 images). The training and testing tasks were performed using TensorFlow on a personal computer equipped with a Core i9-12900H processor and a NVIDIA RTX-3080 GPU. The development and execution environments were managed through Anaconda (Anaconda Inc., Austin, TX, USA), which integrated Python (v. 3.9), CUDA (v.11.2), TensorFlow (v. 2.4.0), and the Spyder Integrated Development Environment (v. 5.4.2).

The matterport implementation of Mask RCNN model for Tensorflow 2.0 (Abdulla, 2017) was adopted in this study to train with weights from the COCO 2017 dataset ‘mask_rcnn_R_101_FPN_3x.yaml’ (Lin et al., 2014). The original model architecture was unchanged, and training was performed for 100 steps with a learning rate of 0.001 across 25 epochs using the labeled dataset. The ResNet 101 backbone (He et al., 2013) facilitated the extraction and propagation of initial feature maps. Regions of Interest (ROIs) were extracted using the Region Proposal Network (RPN), with optimal ROIs selected through non-max suppression and standardized to fixed dimensions using the ROI align component. Dual fully connected layers performed classification via ‘SoftMax’ and bounding box regression. Finally, a binary mask of the individual cow was generated using a mask classifier for each instance within an image. The validation dataset was used for hyperparameter tuning. The model’s accuracy was evaluated using the test dataset through standard metrics such as Precision, Recall, and F1 Score.

2.3.2. Tracking

Tracking and duplicate count avoidance would not be a problem for images captured through a waypoint navigation system. However, for real-time video streams, it is crucial to prevent duplicated identification of the same cow. The centroid tracking algorithm was hypothesized to effectively track objects such as cattle in a series of aerially captured video frames. The algorithm (Figure 4) aids in consistently identifying individual cows over time and prevents duplicate counting. It assigns unique identifiers (IDs) to each cow detected by the instance segmentation model and tracks their movements across consecutive frames.

The algorithm initializes with a counter for unique IDs with two empty dictionaries, one to store current centroids and another to track the disappeared objects (Algorithm 1, Line 1). It begins by calculating the centroids (Algorithm 1, Line 14) of bounding boxes detected by the instance segmentation model for each video frame. If no cows are detected, the disappearance count for existing cows increases, and those absent are deregistered for an extended duration. Upon detecting new cows (‘rects’; Algorithm 1), they are assigned unique IDs, and their centroids are registered. A distance matrix measures the Euclidean distances between existing cow centroids (red dot, frame t-1) and new detections (blue dot, frame t) of the same cow (Figure 4). A linear assignment algorithm is then employed to find the optimal match, minimizing the overall distance. Matches within a predefined distance threshold of 50 pixels—determined by the frame rate, UAV speed, and cattle movement—update the cow’s centroid and reset its disappearance count. Unmatched new detections are registered as new entries. Cow ID’s exceeding this disappearance threshold are consequently deregistered. This method ensures accurate and continuous monitoring without duplicate counting and adapts to potential occlusions in dynamic settings of cattle movement in large feedlots.

Algorithm 1: Centroid Tracking Algorithm
1	Initialize nextObjectID ← 0, objects ← {}, disappeared ← {}
2	def update(rects):
3	if rects is empty then
4		for objectID in disappeared.keys() do
5			disappeared[objectID] ← disappeared[objectID] + 1
6			if disappeared[objectID] > maxDisappeared then
7				deregister(objectID)
8			end if
9		end for
10		return objects
11	end if
12	inputCentroids ← []
13	for rect in rects do
14		centroid ← computeCentroid(rect)
15		inputCentroids.append(centroid)
16	end for
17	if objects is empty then
18		for centroid in inputCentroids do
19			register(centroid)
20		end for
21	else
22		objectIDs ← objects.keys(), objectCentroids ← objects.values()
23		D ← computeDistanceMatrix(objectCentroids, inputCentroids)
24		rows,cols ← linearAssignment(D)
25		unmatchedRows ← range(D.shape[0]), unmatchedCols ← range(D.shape[1])
26		for (row, col) in zip(rows, cols) do
27			if D[row, col] > maxDistance then
28				unmatchedRows.add(row), unmatchedCols.add(col)
29			else
30				objectID ← objectIDs[row], objects[objectID] ← inputCentroids[col]
31				disappeared[objectID] ← 0
32				unmatchedRows.remove(row), unmatchedCols.remove(col)
33			end if
34		end for
35		for row in unmatchedRows do
36			disappeared[objectID] ← disappeared[objectID] + 1
37			if disappeared[objectID] > maxDisappeared then
38				deregister(objectID)
39			end if
40		end for
41		for col in unmatchedCols do
42			register(inputCentroids[col])
43		end for
44	end if
45	return objects

2.3.3. Body Surface Temperature and Stress Assessment

External body surface temperature (BST) in cattle is not always a reliable indicator of core body temperature due to the insulating effect of the hair coat, which hinders accurate estimation [4]. While core body temperature is a better reflection of heat stress, external BST can still serve as an indicator of heat stress in large livestock operations [27,28]. Therefore, heat stress on individual cows was evaluated as mean BST using: (i) both RGB and thermal images, and (ii) thermal images alone. The primary distinction between the two methods is the type of input data used for identifying and segmenting individual cows, leveraging the instance segmentation model described in Section 2.3.1.

In the combined RGB and thermal imaging-based approach, complementary information from both imaging modalities could be of significant advantage over unimodal features of thermal imaging alone. In Figure 5, the BST of the individual cow was extracted using thermal images after detection, segmentation, and tracking using RGB images as input to the instance segmentation model developed in Section 2.3.1. The output was processed to create a binary mask of the individual cow. The corresponding thermal image was then overlayed on the binary mask, and individual cow BST was extracted through element-wise multiplication (Hadamard product) [29].

This approach, however, creates a multi-modal problem. Pixel-to-pixel mapping (domain translation) of cross-domain data from source (RGB) to target (thermal) could be a challenge. The precise alignment and calibration of RGB and thermal data were hence required. The alignment problems commonly stem from differences in optics, imaging FOV, resolution, and lens properties. In this study, thermal images exhibited radial distortion due to the wide-angle lens (Figure 5c).

To address the domain translation problem, individual thermal images were first corrected using an undistortion algorithm (Algorithm 2) [30]. This step requires intrinsic parameters of the imager and lens, such as the camera matrix (K) and distortion coefficients (d). These values can be determined through a calibration process that involves capturing images of a known pattern and solving for the parameters that minimize reprojection error. However, for DJI Mavic 3T, calibration parameters were directly obtained from a GitHub repository [31].

Algorithm 2: Thermal Image Distortion Correction
	Require: Distorted image Id
	Ensure: Undistorted image Iu
1	Camera matrix K and distortion coefficients d
2		$K=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]$
5		$d=\left[\begin{array}{ccc}{k}_1& k_2& p_1\end{array}\begin{array}{cc}{p}_2& k_3\end{array}\right]$
6		Knew, ROI = cv2.getOptimalNewCameraMatrix(K, d, (w, h), 1, (w, h))
7		Iu = cv2.undistort(Id, K, d, None, Knew)
8		Crop Iu using ROI
9		if necessary then
10			Resizing Id and Iu

$$K=\left[\begin{array}{ccc}7.653e+02& 0& 3.154e+02\\ 0& 7.653e+02& 2.559e+02\\ 0& 0& 1\end{array}\right]$$

$$d=\left[\begin{array}{ccccc}-3.596e-01& 1.739e-01& 7.411e-05& 9.025e-05& -1.927e-01\end{array}\right]$$

The combined RGB and thermal imaging-based BST estimation approach can be challenging for processing real-time video streams due to the complexities involved in the synchronized matching of thermal and RGB video frames. Using an undistortion algorithm also introduces substantial computational overhead, necessitating adjustments to calibration parameters with each new thermal imaging device. Recognizing these limitations, an alternative approach that relies solely on thermal images was also evaluated. In this method, thermal infrared video streams can be directly fed into the developed segmentation model. The model processes each video frame sequentially, extracting the BST of each cow, and keeps tracking their IDs using Algorithm 1.

BST extracted following the above two approaches was validated using the ground truth BST (BST_GT). The BST_GT of individual cows was obtained by averaging the temperature of all pixels within the ROI of individual cows, manually drawn using the DJI’s thermal analysis tool. Finally, the severity of heat stress was assessed by categorizing extracted BSTs into four ranges [32], i.e., no stress (≤38 °C), mild (38–39 °C), mild to moderate (39–40 °C), and moderate to severe (≥40 °C) stress.

To ensure accuracy of BST, the DJI’s thermal imager was calibrated using a blackbody calibrator (BB701, Omega Engineering, Inc., Norwalk, CT, USA) with a target plate emissivity of 0.99. At room temperature, images of the target plate were captured using the thermal imager. The calibration process involved measuring the radiated or apparent temperature of the target plate as a function of the measured temperature from the thermal imager. The apparent temperatures ranged from 20 °C to 40 °C in 2 °C increments and then returned to 20 °C, completing a full temperature cycle. Images of the target plate were captured at each increment. The collected images were processed using DJI’s thermal analysis software (DJI Thermal Analysis Tool 3, DJI Technology Co. Ltd., Shenzhen, China) to obtain measured temperatures, which were compared against the apparent temperatures. The mean absolute difference between the apparent and measured temperatures was used to determine the calibration error, if any, for correcting the data collected in this study.

2.4. Thermal Video for Real-Time Stress Monitoring

The validated algorithm was further evaluated for real-time cow tracking and heat stress assessment using independent video streams as input. This evaluation included a 38 s video taken during the spring of 2023 and a 48 s video from the summer of 2024. For practicality, the video stream was processed using a thermal imaging-only approach. The detection accuracy of the model was assessed by manually counting the individual cows, allowing for a comparison between the algorithm’s output and the actual number of cows present in the video streams. The output BST were then categorized into different heat stress levels using methods in Section 2.3.3.

3. Results and Discussion

3.1. Segmentation, Counting, and Localization

The algorithm achieved an average detection accuracy of 84.3% on validation datasets consisting of RGB and thermal imaging modalities. Upon detecting cattle, it returned bounding boxes, providing the percentage of certainty and the count of cows (Figure 6). However, the analysis of RGB and thermal imaging modalities on the test dataset yielded distinct results in terms of accuracy and reliability (

Table 1. Confusion matrix for cow detection using RGB and thermal imaging modalities.

	RGB		Thermal
	Cow	Not Cow	Cow	Not Cow
Cow	292	60	163	54
Not Cow	9	-	129	-

and

Table 2. Evaluation metrics for RGB and thermal modalities in detecting individual cow.

Metric	Precision	Recall	F1 Score
RGB	0.97	0.83	0.89
Thermal	0.56	0.75	0.64

).

For RGB images, the confusion matrix revealed a high level of model precision, with 292 cows correctly identified (True Positives) and only 9 misclassifications of non-cow objects as cows (False Positives). This resulted in a 0.97 precision score, indicating that nearly all detections were correct. The recall rate of 0.83, although lower, still suggests that most cows in each of the images were detected accurately. This was further underscored by an F1 score of 0.89, reflecting a strong balance between precision and recall.

In contrast, the test results from only thermal imagery data were considerably less accurate. The model identified 163 cows correctly but misclassified 129 non-cow objects as cows, reducing precision to 0.56. This high error rate severely impacted the model’s reliability despite a somewhat robust recall (0.75). The overall F1 score for thermal images was 0.64, reflecting moderate effectiveness. Overall, while the mask R-CNN model excels with RGB images, its application in thermal imaging requires further adjustments or perhaps a fusion approach to enhance accuracy and reliability.

The observed differences between thermal and RGB image segmentation accuracy (Figure 6; Table 1 and Table 2) could be due to the modalities of object classes in pre-trained network weights of the COCO 2017 dataset. In this dataset, all the classes are from RGB images, and weights for the object ‘cow’ already exist, whereas training for the thermal infrared imaging modality started from scratch. Detailed color and contour features of RGB images also help segment the objects with relatively higher accuracy. However, RGB images generated by capturing reflected light off target objects are sensitive to illumination conditions and are ineffective during nighttime. This implies that the RGB layer-based BST extraction method could be ineffective for other potential applications, such as cold stress assessment during the nighttime. In contrast, although suffering from low resolution and relatively poor contrast, thermal images capture radiation emitted from objects above absolute zero, making them insensitive to illumination conditions.

The current approach of training thermal and RGB images together on the same backbone network as separate data inputs, without differentiating between image modalities, fails to fuse the useful features of thermal and RGB imagery during training. This limits information sharing between thermal and RGB images, requiring post-processing as pixel-to-pixel matching and distortion correction. Thus, integrating the domain information of both modalities during training using the image and feature-level fusion-based neural network architectures could be of a significant advantage [33].

Pertinent to the above results and limitations of this study, architectures based on Siamese networks [34] have been experimented extensively for infrared and visible light fusion [33,35,36,37]. However, most of the architectures have been evaluated for tracking pedestrians and vehicles. No specific benchmarks have been identified in the literature for tracking cows through RGB and thermal video streams. Therefore, Siamese networks fusing information from thermal and RGB images, could be a possible alternative for cow BST measurement and its application in heat/cold stress assessment.

3.2. Heat Stress Assessment

Figure 7 depicts algorithm test results in estimation of BST from thermal infrared images. The calibration of the thermal-RGB imager yielded a mean absolute difference of 0.66 °C. In UAV-based thermal imaging, temperature variations within ±1 °C are considered acceptable for detecting surface temperature changes in agricultural settings; hence, BST results were utilized with no further refinement [38].

A significant percentage (40.4%) of cattle exhibited BST below 38 °C, which is within the normal physiological range. However, even during spring, 29.5% of the cattle exceeded the 40 °C threshold, a level at which cattle are likely to experience heat stress. The remaining cattle fell within the intermediate range of 38–40 °C, suggesting varying degrees of potential heat stress.

The accuracy of the BST extraction algorithm was assessed by comparing the model-derived values against ground truth measurements obtained via manual ROI selection (Figure 8). The mean absolute difference between the extracted and BST_GT was 2.12 °C with a standard deviation of 1.96 °C in year 1, and 3.3 °C with a standard deviation of 2.8 °C in year 2. While this level of accuracy is promising for a real-time UAV-based system, further refinement of the algorithm would be necessary to achieve a higher degree of precision. Potential error sources include the challenges associated with accurately segmenting individual animals in crowded scenes, variations in emissivity across the animal’s body, and environmental factors (wind and humidity) that can adversely affect thermal measurement accuracy.

3.3. Real Timeliness of Stress Monitoring

Operating at one frame per second, the algorithm successfully detected 113 out of 156 cows in the 38 s video data collected in year 1, yielding an accuracy of 72.4%, and 89 out of 109 cows in the 48 s video in year 2, yielding an accuracy of 81.7%. This indicates the algorithm’s real-time application potential despite lower accuracy than the 89% for RGB video frames. This difference could be likely attributed to motion blur, occlusions, and dynamic thermal variations in live video frames.

The BST from the thermal infrared video feed revealed that, for year 1, 14.1% of cows had no heat stress, whereas 28.8%, 33.3%, and 23.7% of cows had mild, mild to moderate, and moderate to severe heat stress, respectively. In year 2, 28.1% of cows had no heat stress, whereas 6.7%, 10.1%, and 55.1% of cows had mild, mild to moderate, and moderate to severe heat stress, respectively. These findings align with the results from the static thermal images, highlighting the consistency of the algorithm’s performance across different data formats.

The real-time processing capability of the algorithm demonstrates its potential for practical implementation in heat stress monitoring. While the accuracy in the thermal imaging modality requires further refinement, the ability to detect and localize cows, coupled with the real-time BST estimation, underscores its value as a tool for precision livestock farming. Future work could be focused on improving model accuracy by integrating multi-modal data fusions to leverage the strengths of both RGB and thermal modalities. Moreover, although BST is not a strong indicator of core body temperature in cattle, it can provide useful insights into heat stress in large livestock operations. Further testing and refinement of measurements are needed to improve the correlation between BST and physiological changes.

3.4. Limitations and Generalizability

The accuracy and reliability of thermal imaging largely depend on various environmental factors. Relative humidity, atmospheric temperature, reflection temperature, and wind conditions, all of which can significantly affect the thermal readings captured by the camera [39]. High humidity can obscure the contrast in thermal images, making it difficult to distinguish between the animal and the background, while wind can alter the heat dissipation from the animal’s body, leading to inconsistent BST measurements [4]. Radiometric calibration can help in correcting lens distortions, sensor noise, or environmental interferences that might affect the thermal readings [38,40].

Cow-specific characteristics, including breed, lactating stage, and age, further influence the thermal emissivity response by the livestock [41,42]. Different breeds and even different individuals within a breed can have varying skin emissivity, influenced by factors like coat color, coat density, heat tolerance, and texture, which can affect BST readings [43]. Moreover, seasonal variations can also alter thermal characteristics [44].

While the algorithm developed in this study demonstrates a strong potential for real-time cattle heat stress monitoring, it can be further refined for improved performance. Integrating radiometric calibration techniques and cow breed-specific compensation methods could help mitigate some of these limitations. Future works will focus on incorporating additional sensors, such as humidity and wind speed sensors, on aerial platforms, which could provide more context for interpreting thermal data and improve the algorithm’s accuracy under varying environmental conditions. The study will also focus on refining calibration methods to ensure consistent and accurate thermal imaging across different geographic locations and climates, thus improving the overall robustness and generalizability of the algorithm.

4. Conclusions

This study successfully developed and evaluated an algorithm for near real-time heat stress monitoring in drylots using unmanned aerial vehicle equipped with a thermal-RGB imager. The study found:

• The mask RCNN model combined with the centroid tracking algorithm can accurately localize and track individual cows in real-time, avoiding duplicate counts.
• The algorithm’s detection accuracy varies between input imaging modalities, with 89% and 64% accuracy in detecting individual cows from respective RGB and thermal imagery feed.
• Real timeliness of stress evaluation using thermal imagery feed showed promising results with 72.4% (spring 2023) and 81.7% (summer 2024) accuracy in spring 2023 and summer 2024. Similarly, algorithm was able to extract the BST of individual cows with a mean absolute difference of 2.1 °C (spring 2023) and 3.3 °C (summer 2024) from ground truth and quantify heat stress.

These findings highlight the potential of the developed algorithm for precision livestock farming, enabling near-real-time heat stress monitoring and early intervention in drylots. While the algorithm’s performance is promising, further refinements are necessary to improve its accuracy and robustness, particularly in using BST as an indicator of heat stress and the accuracy of BST itself.