Multitarget Intelligent Recognition of Petrographic Thin Section Images Based on Faster RCNN

Abstract

The optical features of mineral composition and texture in petrographic thin sections are an important basis for rock identification and rock evolution analysis. However, the efficiency and accuracy of human visual interpretation of petrographic thin section images have depended on the experience of experts for a long time. The application of image-based computer vision and deep-learning algorithms to the intelligent analysis of the optical properties of mineral composition and texture in petrographic thin section images (in plane polarizing light) has the potential to significantly improve the efficiency and accuracy of rock identification and classification. This study completed the transition from simple petrographic thin image classification to multitarget detection, to address more complex research tasks and more refined research scales that contain more abundant information, such as spatial, quantitative and category target information. Oolitic texture is an important paleoenvironmental indicator that widely exists in sedimentary records and is related to shallow water hydraulic conditions. We used transfer learning and image data augmentation in this paper to identify the oolitic texture of petrographic thin section images based on the faster region-based convolutional neural network (Faster RCNN) method. In this study, we evaluated the performance of Faster RCNN, a two-stage object detection algorithm, using VGG16 and ResNet50 as backbones for image feature extraction. Our findings indicate that ResNet50 outperformed VGG16 in this regard. Specifically, the Faster RCNN model with ResNet50 as the backbone achieved an average precision (AP) of 92.25% for the ooids test set, demonstrating the accuracy and reliability of this approach for detecting ooids. The experimental results also showed that the uneven distribution of training sample images and the complexity of images both significantly affect detection performance; however, the uneven distribution of training sample images has a greater impact. Our work is preliminary for intelligent recognition of multiple mineral texture targets in petrographic thin section images. We hope that it will inspire further research in this field.

1. Introduction

In recent years, the introduction of big data and artificial intelligence algorithms has been a major development in geological research fields. Machine learning and deep learning algorithms have been used to uncover hidden relationships in massive structured and unstructured geoscience data [1,2,3,4,5,6], identify geochemical anomalies [7,8,9,10,11,12], and predict potential mineral resources [13,14,15,16,17]. Furthermore, with the development of deep learning, image-based computer vision technology has made it possible to intelligently identify and classify rocks by their optical properties [18,19,20,21,22].

An important rock identification method is observing and studying the optical properties of mineral composition and texture in petrographic thin sections under a polarizing microscope [23]. This process requires the professional ability and expert experience of observers; thus, some scholars have used machine learning and deep learning algorithms to classify rocks with promising results [20,24,25]. For instance, Młynarczuk et al. [24] used the nearest neighbor method, K-nearest neighbor algorithm, closest approach mode method, and optimal spherical domain method to classify nine kinds of rock flakes and achieved satisfactory accuracy. Shu et al. [25] compared manually selected features and automatically extracted features based on the k-means method to classify rock images. They found that k-means has greater robustness and flexibility in the classification effect of rock images, albeit requiring a large number of training sets as support.

With advances in deep learning methods in computer vision, rock image classification research methods have shifted toward deep learning. Deep learning models generally adopt mature convolutional neural network models, such as the Visual Geometry Group Network (VGG) [26], UNet [27], Residual Network (ResNet) [28], Dense Convolutional Network (DenseNet) [29], Inception-v3 [30] and Faster Region-based Convolutional Neural Network (Faster RCNN) [31]. These models have demonstrated excellent performance in computer vision and have also achieved good classification results in rock image classification. Classification studies of rock images can be divided into two categories according to the image scale: macroscopic hand specimen rock and microscopic petrographic thin section images. While some scholars have conducted research on rock image classification from a macroscopic perspective [25,32,33,34,35], others have studied petrographic thin section image classification from a microscopic perspective [24,36,37,38,39]. For instance, Zhang et al. [33] classified granite, phyllite and breccia images based on the Inception-V3 deep learning model and transfer learning method. They used a total of 571 rock images and achieved a satisfactory overall classification effect despite the small size of the training set. Xu and Zhou [39] used the UNet model to classify and recognize pyrite, chalcopyrite, galena and sphalerite in petrographic thin section images. After data enhancement, they obtained a recognition success rate of more than 90% for each mineral. Bai et al. [40] used the VGG model to classify petrographic thin section images of dolomite, andesite, oolitic limestone, granite, lithic sandstone and quartz sandstone, with 1000 samples in each category. They achieved an accuracy rate of 82% on the test set. Polat et al. [37] used the transfer learning method to classify 1200 rock thin sections of 6 kinds of volcanic rocks based on ResNet50 and DenseNet121 models, with an average classification accuracy of more than 98%. These results demonstrate the potential of transfer learning and data enhancement in achieving good results even with small data sets. In these classification studies, deep learning models have been used to classify rocks by learning the features in rock images or petrographic thin section images, which usually contain only one rock type and one target [33,36,37,38]. To classify multitarget images, the tedious process of standardizing the images into single-target images of the same size and resolution is usually needed.

The above studies of rock and petrographic thin section images have largely focused on mineral composition identification, with few investigations into mineral textural and structural characteristics. In fact, it is not only the mineral composition, but also the mineral textural and structural characteristics that are crucial to determining the rock type and its genesis. Oolitic texture is a kind of texture widely existing in sedimentary rocks. It is composed of small ooids (spherical or ellipsoidal particles composed of a core and a shell surrounding the core) with high porosity and permeability. Therefore, oolitic shoal reservoirs are important targets for oil and gas exploration [41,42]. However, previous studies on intelligent recognition of oolitic texture only focused on whether a petrographic thin section image has oolitic texture or not [40], but did not pay attention to information such as the size, shape, aspect ratio, sorting index, etc., that is bearing in ooids, and is important for reconstructing the paleogeographic environment. Intelligent identification of these minimal units is more challenging than mineral composition identification, as it requires multitarget localization and classification, and thus meets the requirements of multicomponent intelligent recognition in petrographic thin section images.

In this study, the Faster RCNN model was applied to multitarget identification of oolitic texture on petrographic thin section images. The training and test sets were composed of sedimentary rock petrographic thin section images, which contain multiple ooids. This study has two main characteristics: (1) it transitions the research based on petrographic thin section images from simple classification to multitarget detection, making it more consistent with real situations and the research task more complex; (2) the research scale is more refined and contains more abundant information, such as the spatial, quantitative and category target information, which is of reference significance for multitarget mineral intelligent recognition on petrographic thin section images. The present study offers potential benefits to users of the model who can utilize the automatically detected ooids as a reference for their own research on oolitic texture, thereby reducing the time and effort required for manual identification and measurement. Furthermore, this study lays the groundwork for future research on oolitic texture and related fields, including mineralogy, geology and paleontology, which may lead to novel discoveries and insights.

2. Materials and Methods

The microscopic image data of oolitic texture are from the “Micro image data set of some rock-forming minerals, typical metamorphic minerals and oolitic thin sections” of the public database ScienceDB [43]. Each microscopic image contains multiple ooids.

The object detection model can accomplish both localization and classification tasks and, thus, multitarget recognition. The object detection model generates a large number of anchor boxes on the original image by direct or indirect methods and calculates the intersection over union (IOU) between these boxes and the ground truth box, as shown in Figure 1. Positive and negative samples are determined according to the threshold of the IOU ratio, and the positive and negative samples are classified and trained by regression. Therefore, the object detection task is decomposed into classification and regression tasks [44,45,46].

Object detection models can be divided into two categories: one-stage and two-stage models. One-stage models are generally faster, but two-stage object detection models have shown higher accuracy and better small target detection capabilities than one-stage models [44]. This study applies the classic Faster RCNN model in the two-stage approach to multitarget identification of oolitic texture in petrographic thin section images due to the varying mineral composition and texture scales.

2.1. Faster RCNN

The Faster RCNN model, as depicted in Figure 2, consists of a feature extraction backbone network, region proposal network (RPN), region of interest (ROI) pooling and classification and regression. The calculation process can be divided into the following four steps [31]:

• Backbone network: The backbone network comprises a series of convolution, batch normalization, activation function and pooling operations. It is used to extract image features and generate feature maps.
• Region proposal networks (RPN): A large number of anchor boxes are indirectly generated on an image, and the IOU ratio between each anchor box and the ground truth box is calculated. The anchor boxes are labeled as positive samples and negative samples according to the IOU threshold, and the positive and negative samples are classified and trained by regression. The final output is 300 relatively accurate region proposals (ROI).
• ROI pooling: The region proposal output in the previous step is projected onto the feature map. The feature maps of these region proposals are different sizes, so it is necessary to standardize them and output feature maps with the same size of 7 × 7, which is convenient to connect the subsequent fully connected layer.
• Classification and regression: More accurate classification and regression are performed on the feature map output in the previous step.

2.1.1. VGG16

During the 2014 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [47], the VGG Lab at the University of Oxford introduced the VGG series network model architecture [26]. The primary innovation of the VGG series is the utilization of small convolution kernel continuous convolution, as opposed to large convolution kernel convolution operation. This approach extends the network depth while ensuring the same receptive field.

The VGG models are characterized by their simple and elegant structure, as well as their impressive performance. The VGG model comprises five VGG-blocks (VGG-block1, VGG-block2, VGG-block3, VGG-block4, VGG-block5), five max pooling layers and three fully connected layers (FC). The varying number of convolutional layers in each VGG-block constitutes VGG-block1, VGG-block2, VGG-block3, VGG-block4 and VGG-block5. VGG11, VGG13, VGG16 and VGG19 are composed of different combinations of VGG-blocks. Among the VGG series, VGG16 is the most classic. In this study, we selected VGG16, which utilizes a 3 × 3 convolution kernel size as the backbone for feature extraction.

Figure 3 illustrates the architecture of VGG16, which comprises five VGG-blocks. VGG-block1 and VGG-block2 each consist of two convolution layers that employ a 3 × 3 convolution kernel size. In contrast, VGG-block3, VGG-block4 and VGG-block5 each contain three convolution layers that also utilize a 3 × 3 convolution kernel size. Subsequently, three fully connected layers are connected to the output.

2.1.2. ResNet50

The residual network model (ResNet) is a classical convolutional neural network model proposed by He et al. [28]. It emerged in the ILSVRC and won the classification, localization and detection category. The residual block in the network structure is the primary feature that distinguishes ResNet from the previous convolutional neural networks. The introduction of the residual block expands the depth of the convolutional neural network and develops a network structure with dozens or hundreds of layers [28].

In the standard convolutional network, the input is x, and after the convolutional layer, the input is transformed into y. The mapping that this standard convolutional network needs to learn is F, where y = F(x). In the ResNet convolution network, the input is x. After going through the convolution layer, the input x on the shortcut connection (which may need to be transformed) is added, and the output after feature transformation is y. The mapping that this convolution network needs to learn is F, where F(x) = y-x, as the learning mapping is the difference between output and input. Therefore, the network structure corresponding to F(x) is referred to as the residual building block. Compared with the standard convolutional network, the residual network has added shortcut connections, as shown in Figure 4.

ResNet is stacked with one convolutional layer, four residual modules and one fully connected layer. Common ResNets are ResNet18, ResNet34, ResNet50, ResNet101 and ResNet152. ResNet50 is used in this study, as shown in Figure 5. ResNet50 contains five blocks. Conv1 has only one convolution layer, and the convolution kernel is 7 × 7. Conv2 has 3 residual modules; Conv3 contains 4 residual modules; Conv4 contains 6 residual modules; and Conv5 contains 3 residual modules. Each residual module has a shortcut connection and contains 3 convolution layers; the convolution kernel sizes are 1 × 1, 3 × 3 and 1 × 1. The backbone network of the Faster RCNN model applied in this study is based on ResNet50.

2.1.3. Transfer Learning

Deep learning models have been shown to be heavily reliant on large data sets [48]. When the size of data sets is limited, deep neural networks with a large number of parameters (e.g., ResNet50) cannot be fully trained, which will lead to overfitting and other problems. To avoid overfitting problems and enhance the generalization ability of the model, transfer learning and data augmentation are usually adopted. Transfer learning generally uses the network weights that have been trained on other large data sets as the initial weights of new tasks. The data enhancement method generates additional image data by translation, flipping, cropping and brightness adjustment of the original image, which is then used to supplement the training data set [48,49].

2.1.4. Average Precision

In the binary decision problem, the classification decision is to classify the samples as positive or negative. Four results are generated by comparing the predicted value with the real value. The decision result of the classifier can be represented by a confusion matrix [50], as shown in

Table 1. Confusion matrix table.

	Actual Value: True	Actual Value: False
Predicted values: Positive	TP	FP
Predicted values: Negative	FN	TN

.

T indicates that the sample is correctly classified, F indicates that the sample is incorrectly classified, P denotes the sample is classified as positive and N denotes the sample is classified as negative.

The meanings of TP, FP, FN and TN are as follows:

TP (true positive): A positive sample is correctly classified as positive.

FP (false positive): A negative sample is incorrectly classified as positive.

FN (false negative): A positive sample is incorrectly classified as negative.

TN (true negative): A negative sample is correctly classified as negative.

Recall: the probability of correct positive samples among all positive samples.

$$\mathrm{Re}\mathrm{call}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$$

(1)

Precision: the probability of positive samples correctly classified among all positive samples returned.

$$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$$

(2)

Intersection-Over-Union (IOU): IOU is the ratio of the intersection to the union of the predicted box and the ground truth box. In region proposal networks, samples with an IOU greater than 0.7 are considered positive, while those with an IOU less than 0.3 are considered negative. The IOU threshold is crucial for calculating the average precision (AP), which is a key metric for evaluating the quality of object detection. In the evaluation of the validation set and test set, an IOU threshold of 0.5 is utilized to calculate the AP.

Average precision: indicates the average accuracy of different recall rates. This index is used to evaluate the performance of ooid detection in this study. The prediction boxes in the test set were sorted according to the confidence from high to low, and the correct rate and recall rate were calculated successively. The Precision-Recall (PR) curve with the recall rate as the abscissa and the correct rate as the ordinate was drawn, and the area enclosed between the rounded PR curve and the coordinate axis was calculated by the integration form as the AP value of this category.

2.2. Experiments

The ooids in the original microscopic images were annotated by LabelImg [51], an open source Python library, and the annotated images were used for training and testing the Faster RCNN. The code has been modified from the source code obtained from GitHub [52]. To improve the computational efficiency and speed up the operation, all the training was completed on a GPU. To accelerate the convergence of the model, the weights that were trained on the PASCAL Visual Object Classes [53] data set were selected as the initial weights. For ablation experiments, VGG16 and ResNet50 were employed as the backbone of the Faster RCNN model. The hardware and software used in the experiment are shown in

Table 2. Hardware and software configuration.

Hardware/Software	Series/Version
CPU	[email protected] GHz
GPU	2080ti
DRAM	32 G
SSD	1.5 T
OS	Windows10 Professional
Python	3.7.1
Torch	1.8.1
Torchvision	0.9.1

below.

The performance of a model reaches saturation under ideal conditions as the size of the dataset increases, beyond which it no longer improves. Additionally, the optimal state of the model’s performance is achieved with an increase in model complexity, but further increases in complexity lead to a decrease in performance [54]. The size of the dataset and the complexity of the model are closely linked to the model’s performance. Complex models require larger datasets to fit the model, and they tend to predict better performance. Warden [55] suggests that for image classification tasks in deep learning, a general rule of thumb is to have 1000 images per classification. However, this requirement can be reduced significantly by using transfer learning and introducing a pre-trained model in a classification task. In the same domain, ten or twenty new categories may suffice. However, if the categories are not in the same field, more training images are necessary.

There are a total of 103 images in the original data set, of which 56 single-polarized microscopic images are selected as experimental data. We chose a random sampling method to reduce human intervention while making it simple and easy to implement. Ninety percent of the images are randomly selected as the training set (81%) and validation set (9%), and the remaining 10% are selected as the test set. The training set images were data enhanced, and the data set was expanded to 315 images by cropping, translation, rotation, flipping, adding random noise and adjusting brightness. Figure 6 shows the comparison between the original image and the image after data enhancement. After the training, the weight with the smallest loss value is selected as the weight of the test set to test and verify its performance.

3. Results

The experimental results of this study showed that the multitarget detection of ooids based on Faster RCNN has good performance. In the ablation test of the Faster RCNN model with VGG16 and ResNet50 as backbones, the AP values for VGG16 and ResNet50 in the test set were 87.83% (Figure 7) and 92.25% (Figure 8), respectively. The performance of ResNet50 as the backbone of the Faster RCNN model in the test set was analyzed. Figure 9, Figure 10, Figure 11, Figure 12, Figure 13 and Figure 14 show the test results of ooid detection in the test set using ResNet50 as the backbone. It was observed that when the features of ooids in the microscopic image were distinct from the background, the detection performance was good, and the ooids were accurately located (Figure 9 and Figure 10). Conversely, when the composition of the image was relatively complex and the difference between the ooids and the background was not obvious, misclassification occurred (Figure 11, Figure 12 and Figure 13), and the uneven distribution of samples in the training set was found to have a significant impact on the detection performance, with cases of missed detection occurring when the number of similar samples in the training set and the test set was small (Figure 14).

In Figure 9, the detection model revealed a large number of ooids with moderate size proportions, small differences and uniform distribution, which were significantly distinct from the background. The detection result was highly intuitive, enabling the accurate identification of complete ooids in the image, as well as those partially displayed ooids located at the edge.

In Figure 10, the model detected a large number of ooids with moderate size proportions, small differences and uniform distribution, which were distinct from the background. The checking model yielded satisfactory results, with complete ooids near the center of the image correctly identified. However, a few misclassifications occurred at the image′s edges.

In Figure 11, the composition complexity of components is low, and oolitic features are relatively obvious, while in Figure 12 and Figure 13, the oolitic texture features are more complex, and the size of oolitic particles differs greatly from the background, although not as much as in Figure 9 and Figure 10. The main ooids were correctly detected in all three figures, and large ooids at the edge were localized and detected.

The features of complete ooids in Figure 14 are not as obvious as those in Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13, and the contrast with the background is not as pronounced, leading to a potential misclassification. Nevertheless, the ooids in Figure 11, Figure 12 and Figure 13 are also highly similar to the background. Although misclassification may occur in the image edge, the main ooids were still detected and identified without error. Comparing Figure 9, Figure 10, Figure 11, Figure 12 and Figure 13 revealed that the detectability of ooids is not solely dependent on the distinctiveness of their features and the contrast with the background. A comparison of the original training data set shows that the number of samples with high similarity to the test in Figure 14 in the training set is small, suggesting that the distribution of training samples can have a significant impact on the detection performance.

4. Discussion

The potential causes of misclassification in detection can be attributed to a variety of factors, including the following:

• The complexity of image components and the lack of distinction between target texture and background features can lead to misclassification in detection. This is exemplified by the petrographic thin section image classification of Bai et al., (2019) [40], where the similarity between oolitic limestone and dolomite thin sections was so high that misclassification occurred. To address this issue, incorporating such samples into the training process may enable the model to learn the differences between them and thus reduce misjudgment.
• The uneven distribution of training image samples can have a significant impact on detection. This effect was demonstrated by the training set analysis, which revealed that the training set similar to Figure 9, Figure 10 and Figure 11 accounted for 54%, the training set similar to Figure 12 and Figure 13 accounted for 32%, and the training set similar to Figure 14 accounted for 14%. This skewed distribution of training set samples indicates that the model may not have sufficient learning experience, leading to better performance on simple images than on complex images.
• The limited size of the training image data sets can lead to insufficient generalization of the model. Data augmentation has been employed to address the issue of small data sets, thus improving the model′s generalization capacity [38,39]. After data augmentation, the overall result was promising, which shows the potential of this method. With the increase in data sets, the generalization ability of the model can be further enhanced, thus improving the detection performance.

When the number of samples is large, deep learning models can be constructed from the very beginning and yield satisfactory classification results. Conversely, when the number of samples is insufficient, transfer learning methods such as appropriate initial weight selection can be employed to expedite the model convergence, and the data set can be augmented and supplemented through data augmentation, ultimately leading to satisfactory classification results. These results suggest that complex deep learning models can be applied to small data sets with the help of appropriate transfer learning and data augmentation.

In this study, a deep learning model was applied to detect multiple targets of ooids in microscopic petrographic thin section images. Object detection was found to be more complex and difficult than classification, with an average precision of 92.25% using ResNet50 as the backbone. Most of the prediction targets had high confidence. However, the uneven distribution of samples in the training set and the component complexity of microscopic image samples both impacted the detection performance, with the former having a greater influence. The results of this experiment suggest that ooid detection methods may provide useful insights for multitarget recognition on microscopic petrological thin section images. Our future work will focus on applying these technologies to intelligently recognize the other mineral textures and structures that reflect the genesis and formation environment of rocks.

5. Conclusions

By analyzing the mineral composition, texture and structural features, petrographic thin section images can be employed to classify rock types, elucidate petrogenesis, invert paleoenvironments and facilitate other relevant studies. In this study, Faster RCNN object detection algorithms were applied to identify multiple objects in petrographic thin section images using oolitic texture images from the public database ScienceDB. Transfer learning was used to accelerate the model convergence, data augmentation was used to enhance the generalization ability of the model, and GPU was utilized to improve the computational efficiency. The following conclusions and understandings are drawn from the experimental research.

• The AP value of the ooids test set using ResNet50 as the backbone was 92.25%, indicating good overall detection performance. This object detection model was found to be robust and generalizable, as it was able to identify both complete ooids in the middle of the image and partial ooids at the edge.
• The uneven distribution of samples in the training set and the complex composition of microscopic images affected the detection, with the former having a greater effect. Deep learning was used to learn features from the training set and make predictions on the test set, but the uneven distribution of samples caused the distribution of the learned features to deviate, resulting in missed detection in the prediction process. The complexity of the microscopic image composition, with a small difference between the target and the background, also contributed to misclassification and thus affected the detection performance, although to a lesser extent than the uneven distribution.
• This study sought to transition from the classification of petrographic thin section images to multitarget detection, incorporating richer content such as spatial, quantitative and categorical target information, as well as more complex tasks. The research scale was further refined, transitioning from rocks to the textures and structures within them, providing a reference for multitarget intelligent recognition on petrographic thin section images.