AI-Based Intelligent Monitoring System for Estrus Prediction in the Livestock Industry

Abstract

In order to improve a livestock breeding environment that considers securing safe cattle resources and improving productivity for the intelligent farm, we propose an animal-friendly and worker-friendly intellectual monitoring system with Artificial Intelligent (AI) technology. In order to secure safe cattle resources and increase productivity for the livestock industry, it is necessary to secure the self-activities of the cattle and predict the estrous state of target cattle as quickly as possible. For the prediction of the estrous state, it is necessary to continuously observe the cattle behavior by workers and quantify the behavior of the target cattle, but that is not easy for workers and needs a long period of continuous observation. We developed the intelligent monitoring system (IMS) with the ARM (Augmented Recognition Model) for the intelligent farm that can predict the estrus of target cattle and get activity data for individual cattle, and then the system was applied to a typical cattle farm for activity monitoring of the Korean cattle (Hanwoo). Therefore, we confirmed the target Hanwoo group with more than 400 activities among the Hanwoo groups using the ARM threshold. Thus, we verified the potential of the proposed system for tracking multiple similar objects.

1. Introduction

In the livestock industry, continuous condition monitoring of the target livestock is an essential activity for securing safe resources and productivity management. One of the main roles of herd managers was observation and monitoring the health of farm animals. They spent time to catch certain meanings of an animal’s behavior and predicted its conditions such as estrus, disease, feeding, changing of living environment, etc. In order to check the condition of the target livestock continuously, the knowledge and experience of livestock industry experts are very important, but livestock industry experts (including workers) cannot stay on the site for a long time for continuous monitoring of target livestock. Therefore, the need for an artificial intelligence monitoring system for intelligent farms based on professional knowledge and experience is increasing in the livestock industry. Timing is one of the essential factors for increasing productivity on farms. The size of the herd on current Korean livestock farms is growing, but it is becoming increasingly difficult to find experienced and trained herd managers who understand animal behavior signs. We researched and found that the intelligent monitoring system (IMS) combined with experienced experts’ knowledge could classify each animal among the herd and observe behavioral signs of estrus. This study needed a new approach that deviated from the physical method currently in use to solve the above-mentioned problems, and this study is worthwhile as an artificial intelligence-based eco-friendly method.

The monitoring system should have the functions to classify individual objects among livestock of similar shape and be able to determine whether the livestock is on the estrus state. So, we developed the ARM in advance, which has recognition functions to individualize the target cattle using AI technology. The intelligent monitoring system, including the ARM, was designed to predict the estrus cattle with breeding knowledge, and then the system was applied to a Korean cattle (Hanwoo) farm.

1.1. The Changing State of the Livestock Industry

In the Korean case, Hanwoo is a representative breed of Korean native cattle common in the Korean livestock industry. According to the National Statistics Korea (KOSIS—Korean Statistical Information Service, 2020), the number of Hanwoo being bred in Korea has steadily increased from about 1.5 million in 2005 to 2.8 million in 2018. The number of large livestock farms with 100 or more animals has increased more than five times over the last two decades, from 1155 households in 2005 to 6001 households in 2020 (Figure 1).

Thus, livestock farms are gradually becoming larger. However, compared to the growing scale, the growth rate in the number of professional workers participating in breeding in the livestock industry, who are essential at livestock sites, is low [1,2]. Professionals (such as veterinarians or experts in food safety and productivity) concerned about the management of livestock have an interest in the abnormal livestock behaviors exhibited during specific periods (e.g., estrus, disease, feeding, change of living environment). Hence, the meanings of specific behaviors of target livestock are continuously being researched.

Executives in the livestock industry know that the construction of healthy and safe breeding environments for livestock and work environments for the workers are crucial for the development of the industry. The livestock experts and workers of large-scale livestock farms are involved in various tasks for the efficient management of individual livestock. In most small farms that have a small number of livestock, the workers can easily predict the estrus of target livestock by observation and experience. Moreover, they can easily perform artificial insemination and management procedures based on these observation results. However, the optimal timing to artificially inseminate cattle is around 20 h after the start of specific behaviors (estrus signs) such as mounting activity. Thus, in large-scale livestock farms, it is difficult to perform various observations and management tasks for individual livestock because of the large number of target livestock that workers are responsible for. In particular, although the scale of breeding environments is gradually increasing, the number of employees in the livestock industry is decreasing. Therefore, proper breeding management of target livestock that is required in the livestock production and consumption process without increasing the tasks of workers in the livestock industry has been established as an important element [3].

To adapt to this trend in the livestock industry, livestock farms are increasingly applying ICT, which requires them to solve various problems related to workers, executives, target livestock, and many other factors [4].

1.2. The Livestock Activity Monitoring

In the case of animals, they all have similar shapes in appearance, and because they behave according to their self-awareness as living individuals, it is difficult to predict their behavioral characteristics. The data for analysis of animals’ behavioral characteristics are mostly collected through real-time observation using cameras. When acquiring the activity images of animals, the observer is excluded as much as possible to avoid influencing the natural behaviors of the animals.

In the image acquisition process, the activities of the animals can be detected using various sensors, such as infrared sensors, and the acquired image information and the detected patterns from the sensors are used as information for observing the detailed behavioral characteristics of the target animal. This image acquisition process must minimize factors that may influence the animal’s behavior and secure the target animals’ rights.

In terms of industrial applications, the method can be applied for observation and monitoring of health conditions of livestock resources, specific, abnormal, or unusual behavior of the livestock, such as estrus, and efficient management of livestock and their environment. The animals’ behavioral characteristics are analyzed by experts’ visual checks with the acquired images later, and the meanings of the behavioral characteristics are investigated [5].

The method of observing livestock is also very similar to the environment of animal observation. In the case study of Hanwoo, the estrus manifestation timing of Hanwoo appears between 6:00 p.m. and 6:00 a.m. with a 55.9% probability [6]. It has been found through the research of various conditions and visual methods to check estrus that close observation in intervals of 4 to 5 h per day in the early morning and late evening is most effective [7]. Estrus often begins at inconvenient times, such as at night when it is difficult to detect estrus through direct observation. In large-scale livestock environments, the probability of detecting estrus manifestation is quite low because it is difficult for the worker to directly distinguish and specify the target Hanwoo that has manifested estrus because most Hanwoo have similar appearances [8].

A number of studies have been conducted in relation to the detection or prediction of estrus manifestation, and the methods of attaching accelerometers to the neck and ankles of Hanwoo to detect the mounting motion or temperature sensors to the tail to detect the change in body temperature were used for the prediction of estrus manifestation sign. However, because these methods may interfere with cattle behavior and cause problems in animal welfare, we used AI technology, which is image-based object classification. Related to the reasons, instead of attaching various electronic devices and sensors to animals and collecting information from them, the method using AI based on image information is more appropriate in terms of securing healthy livestock resources as well as animal rights, contributing to the establishment of safe livestock farms.

For the safe management of livestock resources and the establishment of an effective breeding environment, information that can determine and verify the objective state of animal objects needs to be properly defined, collected, and shared with livestock professionals, field workers, and executives.

2. Intelligent Monitoring System for the Intelligent Farm

In the introduction, we mentioned the need for estrous prediction monitoring on large-scale livestock farms. An intelligent farm system should be operated based on livestock management and should be able to perform the duties of workers instead.

As shown in Figure 2 [9,10,11,12], the solution currently applied in livestock farms for establishing an intelligent breeding environment is to attach sensors, such as temperature and acceleration sensors, to the neck, tail, and ankles of the livestock to classify individual objects through each sensor’s unique identifier [13,14,15,16]. In this way, information on the activity of target objects and behavioral characteristics such as estrus is acquired. However, this method requires the attachment of devices for detection that directly and indirectly affect the safety of livestock. As a result, it increases device damage due to livestock activity and increases management costs and stress on the livestock. This issue leads to the secondary problems of animal welfare, the acquisition of safe livestock resources, and the management of the livestock environment.

We present the Augmented Recognition Model (ARM) for estrus prediction by classifying individual objects into livestock groups for management and collecting activity data on individual objects to establish a safe breeding environment. In addition, to solve the above-mentioned problems, the monitoring system can be implemented based on AI technology, and we have developed IMS for an estrous prediction that can perform the decision of estrous in advance and livestock monitoring among workers’ duties.

The IMS has the function of distinguishing target objects within a vision-based image of similar objects’ activity characteristics and detecting estrus livestock among the distinguished objects.

In this study, an AI-based recognition algorithm (You Only Look Once: YOLO) was used for the image-based identification of individual objects of similar livestock [17]. However, although the YOLO algorithm can recognize objects with different features of different classes as individual objects, as shown in Figure 3a, in a group of livestock classes having similar features, it cannot recognize similar objects of the same class as unique objects, as shown in Figure 3b.

The recognition model using the YOLO algorithm is operated by using image data to recognize the object of observation contained in the image. The monitoring system with the YOLO algorithm consists of a camera (an image acquisition device) in the environment to be observed, and the hardware system can be easily composed. Therefore, ARM developed based on the YOLO algorithm can recognize and classify the target object in the image acquired through the camera without a separate observer and can store activity information about the identified target object.

3. Method of IMS

As shown in Figure 4, the IMS consists of a CCD, and the PC is equipped with an ARM. The ARM distinguishes the individual livestock in the real-time images through the CCD and calculates and collects the position coordinates of the individual livestock at the time. At the same time, when the mounting motion of the target livestock occurs in the image, the weight factor (k) according to the mounting motion is combined with the activity data of the target livestock. The estrus livestock is determined and predicted among the target livestock using the accumulated activity data and weight factor (k) during a specific time scope (3 days).

The livestock we have to observe is an animal similar in shape and form. Therefore, it must be distinguished and recognized as an individual livestock. In addition, recognized individual livestock can be displayed with ID. Therefore, the ARM developed can classify similar livestock in the video by ID, as shown in Figure 5. And it can collect data about the position and activity information of the target livestock.

4. Estrus Prediction Using Livestock Activity Data

As shown in Figure 2, the method of collecting activity data that has been recently highlighted involves attaching high-performance acceleration sensors for ID classification to cattle and collecting gait information through wireless communication. Using the method, the activities of the target Hanwoo can be measured directly, but it is not recommended in terms of animal welfare because the sensors must be directly attached to the target Hanwoo. Moreover, the meaning of the collected data is highly limited because the data are collected only for the gait information of the target Hanwoos. Therefore, since the image acquired through the camera installed in the farm can be processed into information of various meanings, the activity of the Hanwoo was converted into position data and turned into information. In the development of ARM, individual Hanwoo were defined as groups of similar shape, and activity data of individual Hanwoo were collected and defined into information such as the activities of mounting-motion, moving-path, and specified motion.

For the ARM, the image coordinates for the unit time were converted into real-world positions for the Hanwoos, which were distinguished as individual objects from the images acquired in real-time by applying the multiple object tracking technologies, as shown in Figure 6. Then, the activities of the target Hanwoo were converted into data by synchronizing the data for the image acquisition frame times of ARM into cumulative moving distance information. As shown in Figure 6, to collect the location information of individual livestock, the movement distance was measured using the Euclidean distance method for the change of the acquired coordinates (x, y) to (x1, y1) from P1 to P3 of Hanwoo No. 6.

The ARM collects the movements of individual Hanwoos over time and the coordinate (X, Y) values of the index (ID) that distinguishes individual objects, as shown in Figure 6. The collected data are quantified and stored as the activity information of individual Hanwoos. The changes in position coordinates of individual Hanwoo in the images can be changed to length changes in a real environment and can be used as actual activity data.

The coordinate value of the collected individual Hanwoo can be used to calculate the movements of individual Hanwoos based on the frame changes (frame counter) of input images, and then the coordinate changes based on the pathway of individual Hanwoos can be summed, and that data can be changed to activity data for the Hanwoo’s activities analysis.

To calculate the actual movements of target objects individually, the changes of the center coordinate P (x, y) of the recognition area (red square box) of individual target objects, detected in 2D images, were measured every second. The movement distances were calculated by Equations (1) and (2) and then summed every second. The distance data was accumulated on the ID set during the monitoring time.

$$\mathrm{L}=\sqrt{x^2+y^2}$$

(1)

$$\mathrm{D}={\sum }_{t=0}^{3600}\sqrt{{dx}^2+{dy}^2}$$

(2)

The total activity value, M(t), at a time (second, t), can be determined by an accumulation of the input image frame count numbers as follows:

$$\mathrm{M}(t)=\sum _{}^{}D(x,y,t)$$

(3)

Therefore, the activity of each individual object at a specific time (t) can be identified, and if M(t) is larger than the specific threshold k, it can be considered that the change in activity is large. The threshold k is normally used as the average activity value for the target object per 1 day.

$$\mathrm{A}(t)=\left\{\begin{array}{c}1,M\left(t\right)>k\\ 0,otherwise\end{array}\right.$$

(4)

A(t) can be used to determine at which t there were many movements and abnormal behaviors, such as the mounting activity by the target object.

5. Learning Unique Information for Analysis of Target-Behavioral Characteristics

The measurements of target livestock activity proposed in the IMS are saved as the change in movement over time, based on data obtained through image information. We checked the usefulness of the system that proposes the prediction of estrous Hanwoo using acquired data and the changes in the measured movements.

In general, the activity of Hanwoo increases significantly during estrus. Previous research using the step meter found that the number of steps during estrus increased 2.4 to 5 times compared to the number of steps during ordinary times [18]. Furthermore, when depressed, Hanwoos dislike moving and either lie alone and always have difficulty standing, or they remain standing. When their appetite is reduced, they exhibit sluggish behavioral characteristics [19].

In this study, additional experiments were performed on the possibility of predicting the estrus time of Hanwoo using ARM. To that end, images of cattle mounting activity were collected to define the estrus condition. The ARM was trained with them, and a recognition algorithm for detecting specific behaviors in cattle was added, as shown in Figure 7.

After recognizing individual objects in a group of similar objects in the step for detecting individual objects, the ARM was trained with various images of mounting activities that can be recognized as mounting by the mounting motion recognition algorithm. In this way, the ARM was improved to enable the detection of mounting activities from various angles and applications. The objective was to verify the usefulness of ARM by training the AI training model with one case to observe estrus conditions, which should be observed by workers, in order to apply the data acquired through ARM to breeding sites.

The objective was to check the usefulness of ARM by training the AI training model with one kind of case—estrus conditions, which should be observed by workers—in order to apply the data acquired through ARM to breeding sites.

6. IMS for Livestock Industry

6.1. Field Experiments with IMS

To verify the usefulness of the IMS to collect and utilize the activity data of individual livestock through the classification of similar objects, the IMS was installed in a real Hanwoo farm, and a control group for comparison was composed to collect activity data.

As shown in Figure 8, the real breeding environment of the Hanwoo farm was monitored. In group A, three normal-state Hanwoos were assigned, and in group B, one Hanwoo that manifested estrus and two normal-state Hanwoos were assigned. A camera for observation was installed at the center of the farm to create an experimental environment, and images were acquired through the camera. The ARM added a class of Hanwoo objects to the recognition objects used in the intelligent monitoring system, and the Hanwoo object class was trained using normal-state Hanwoo images and mounting images. The two Hanwoo classes classify similarly shaped Hanwoo into individual Hanwoo, and classify them for recognizing the activity data and estrous activity of the separated. For the estrus prediction, over the 12,000 images of Hanwoo objects and mounting activities, 10,000 images were used as the training data, and 2000 images were used as data for verification and as the weights of the developed ARM.

For the estrus prediction, over the 12,000 images of Hanwoo objects and mounting activities, 10,000 images were used as the training data, and 2000 images were used as data for verification and as the weights of the developed ARM. The specifications of the hardware system for the experiment were an i7-9700F CPU @3.0 GHz, 32 GB of RAM, and a NVIDIA TITAN RTX (40 GB), and the image sensor had a resolution of two Mega-pixel.

The AI learning method for the IMS followed a general type of YOLO_V5. The external area of the barns neighboring to the left and right in the images in Figure 9 was arbitrarily blocked (black area) so that the ARM could not detect changes in the external area that could affect the recognition.

6.2. Experimental Result

The activity data collected through the intelligent monitoring system, which utilized image information, was applied through the classification of individual objects in the Hanwoo groups, as shown in Figure 10.

The activity data of the individual Hanwoos, consisting of six Hanwoos, were collected. The ARM detected and indexed the Hanwoos of groups A and B. Group A has Hanwoos No. 1, 2, and 3, and Group B has Hanwoos No. 4, 5, and 6.

Figure 10 shows the cumulative activities for monitoring times for the six Hanwoos. It is necessary to compare the changes in activity in a specific section (0–350 frames). When the activities of Group A with Hanwoos No. 1, 2, and 3 are compared with the activities of Group B with Hanwoos No. 4, 5, and 6, the graph for the Hanwoos that belong to Group A does not show large changes in general, whereas in Group B, the activities of Hanwoos No. 5 and 6 clearly show changes. For the prediction of estrus, the threshold (400–600 activities) for identifying the changes in activities can be compared through the reference area shown in Figure 10, and the threshold (k) can be changed for the different types of cattle through the experience and knowledge of livestock experts. According to a comparative study on the estrous characteristics of Hanwoo, it can be confirmed that Hanwoo is in estrus if her activity level exceeds 600. If activity levels are between 400 and 600, the probability of Hanwoo being in estrus is 60% or less. If the activity level is below 300, it can be determined that Hanwoo is not in an estrus state. Therefore, the range of activity data for estrous prediction can be set based on the characteristics of various target livestock species, as shown in Figure 10, when applying the estrous prediction system to various types of livestock. Additionally, considering the characteristics of the previous models (acceleration sensor, infrared sensor), and the concurrent system, it is not necessary to use wearable or implantable sensors to collect the livestock’s activity information, so it can be collected in real-time and managed in a convenient manner, which can save the management time and effort of the worker.

In addition, when the activities of individual Hanwoos in the measurement sections were compared, as shown in Figure 11, the activities of three Hanwoos in Group A were significantly lower than those of the Hanwoos in Group B.

There is a Hanwoo that manifested estrus (Hanwoo No. 6) in Group B. The IMS specified two target Hanwoos that showed sharp changes in activity to determine estrus manifestation based on activity data among the three Hanwoos in Group B. The IMS can predict target Hanwoos for estrus manifestation on the basis of the acquired data and judgment criterion (threshold).

The activity data of Hanwoos in Group A, as shown in Figure 11, show that Hanwoos No. 1 and 2 have low activities in general, whereas the activity of Hanwoo No. 3 increased at a specific time. When we observed the image in this section, feeding and other activities were identified.

In contrast, among the Hanwoos in Group B, Hanwoo No. 4 does not have many sections in which the activity changes sharply in general. However, rapid changes in activity were observed in the entire data section for Hanwoos No. 5 and 6. As mentioned previously, one Hanwoo object in Group B was predicted to manifest estrus; thus, it was predicted that there would be a large activity change in Group B. This could be quantitatively determined by adjusting the threshold, as described previously, using the collected data. Furthermore, by examining the changes in activities of Hanwoos No. 5 and 6 in Group B, it can be found that the sections in which activity increases are similar.

Comparing the activities of the two Hanwoos (No. 5 and 6), as shown in Figure 12, confirms that the clear characteristics of change are similar. It is necessary to further examine the correlation between the two Hanwoos, No. 5 and 6, by comparing the data. It can be seen that the overall cumulative activity data of Hanwoo No. 5 is higher than that of Hanwoo No. 6, as shown in Figure 12. It can also be confirmed that the activity is much higher when compared with other Hanwoos as well. Activity change was defined as an important condition for judging estrus manifestation. Thus, predicting Hanwoos with a high possibility of estrus manifestation based on the acquired data can be performed by comparing the data of cumulative activity using the movement distance of Hanwoo objects for the total observation time such that Hanwoos No. 5 and 6 are the target Hanwoos for estrus manifestations, as shown in Figure 13.

Hanwoo No. 5 and 6, for which estrus manifestation was predicted, showed peculiar behavioral characteristics such as tucking their chins on other Hanwoos nearby or wandering around the barn to perform a mounting activity. An increase in the activity due to behaviors to avoid the estrus manifestation target Hanwoo is information with insufficient meaning to accurately determine the target Hanwoo for estrus manifestation.

When comparing the cumulative activities in Figure 12, which was plotted from the experimental data, it can be determined that Hanwoo No. 5, with high cumulative activity, is the target Hanwoo for estrus manifestation by comparing the cumulative activity of Hanwoo No. 5 that interacted with Hanwoo No. 6, predicted to manifest estrus. However, the activity of another Hanwoo that interacts with the target Hanwoo for estrus manifestation may not show a large difference depending on the barn environment.

If two Hanwoos are specified for interaction, cumulative activity may be effective as comparison data for specifying the target livestock for estrus manifestation, but it is somewhat insufficient for determining a specific Hanwoo for estrus manifestation.

As shown in Figure 14, the IMS can detect mounting activity through ARM with the pre-learning of Hanwoo class. Thus, in this experiment, Hanwoo No. 6 showed continuous mounting activities to Hanwoo No. 5, which were recognized as ARM. Therefore, the ARM finally demonstrated that it could determine the Hanwoo for estrus by repeatedly detecting mounting activities and by comparing the overall increasing data of the activity.

Finally, we could verify the usability of the ARM by deciding the degree of prediction that can determine the target Hanwoo object for estrus manifestation by combining the weight of 0.7 for the detection of mounting activity and the weight of 0.3 according to the cumulative activity in consideration of the prediction degree to determine the Hanwoo expected to manifest estrus. Therefore, the IMS showed the possibility of predicting the target cattle for estrus manifestation through quantification of the cumulative activity and mounting activity detection data during the monitoring period. It also showed the possibility of sufficiently replacing the observation job for close checking of livestock conditions by workers in a breeding environment.

7. Discussion

The IMS was proposed to predict the estrus of target livestock by classifying target objects and changing object coordinates. In addition, the data obtained from the change in activity can be utilized to process the result of activity scope within the barn into data that can be analyzed, as shown in Appendix A. The position color of visualization is the staying time of target livestock. The area of red-color means long staying time and blue-color area is short staying time. When the acquired data were checked, six frames of Hanwoo objects were unrecognized among the 17,500 frames of acquired images in total, but it was considered that there was no problem with the validity of the data.

Figure A1 shows the pathway graph of Hanwoo objects, which expresses the pathway according to the movement in the x (length of the barn) and y (width of the barn) directions on the z-axis. It can be seen that Hanwoo No. 1 has few movements, staying in one place for a certain time. It was represented in red-color when the actual moving occupancy times in the barn were accumulated. Hanwoos No. 5 and 6 were found to have a large scope of activity inside the barn, and they showed larger changes in pathways than those of other Hanwoos. Through such data analysis, we can not only predict the estrus manifestation of Hanwoo objects but also determine the health condition of the target livestock and process them as data for analysis of behavioral characteristics in specific time slots, thereby utilizing them as information of various meanings.

In related research that analyzed the behavioral characteristics of red foxes, motion sensors and acquired image information were used to acquire activity information through machine learning technology to exclude the impact on passive activities due to human intervention [20]. Thus, changes in the activities of livestock can be caused by various factors, including disease and animal welfare problems, and can be made on the basis of the acquisition of standardized data to observe, measure, and evaluate the behavior of livestock. Moreover, it has been shown that this can be used as an important indicator to determine the state of animal breeding.

The IMS system of this study does not require workers’ observation activities compared to the existing method of collecting livestock activity data using attached sensors [21,22,23,24,25] and has the advantage of managing livestock safety resources while reducing workers’ various functions and roles.

8. Conclusions

This study researched an intelligent monitoring system that can replace the function of an expert in the livestock industry according to the industrial change demand of the livestock industry.

The IMS has the function of collecting activity information and converting data to analyze the activity contents of the target object recognized through ARM. Based on the collected information and the result of ARM’s recognition, an IMS that can predict the timing of the estrus of the target livestock in advance was implemented.

By conducting an experiment with Hanwoo, we presented a solution for problems in terms of (1) recognition of individual objects in groups of similar objects and (2) prediction of manifestation of specific changes (e.g., behavior) exhibited by individual objects, which were limitations of previous generated AI-based recognition models.

Using the collected data, we verified the applicability of the proposed method for the effective implementation of intelligent monitoring, combined with the results of research in other technical fields. However, the determination of the weight factor (k) for the monitoring of various livestock to determine estrus needs to be more clearly verified through future experiments.

In addition, it was verified that the proposed method can be effectively utilized for various industrial cases subject to complex constraints such as animal welfare, limitations in the environment of observation, and unmanned measurements.