Efficient Hair Damage Detection Using SEM Images Based on Convolutional Neural Network

Abstract

With increasing interest in hairstyles and hair color, bleaching, dyeing, straightening, and curling hair is being widely used worldwide, and the chemical and physical treatment of hair is also increasing. As a result, hair has suffered a lot of damage, and the degree of damage to hair has been measured only by the naked eye or touch. This has led to serious consequences, such as hair damage and scalp diseases. However, although these problems are serious, there is little research on hair damage. With the advancement of technology, people began to be interested in preventing and reversing hair damage. Manual observation methods cannot accurately and quickly identify hair damage areas. In recent years, with the rise of artificial intelligence technology, a large number of applications in various scenarios have given researchers new methods. In the project, we created a new hair damage data set based on SEM (scanning electron microscope) images. Through various physical and chemical analyses, we observe the changes in the hair surface according to the degree of hair damage, found the relationship between them, used a convolutional neural network to recognize and confirm the degree of hair damage, and categorized the degree of damage into weak damage, moderate damage and high damage.

1. Introduction

Hair is an important part of human body image. Modern people’s aesthetic desires for beauty, along with the continuous growth and diversification of hair and the continuous development of hair shape, color, and texture allows people to use their own personality to change their image to follow the ever-changing trends. Hair is composed of 1–8% external hydrophobic lipid epidermis, 80–90% α-helix or β-sheet conformation of parallel polypeptide chains to form water-insoluble keratin, less than 3% melanin pigment, and 0.6–1.0% trace elements, 10–15% moisture, etc. The normal cuticle has a smooth appearance, reflecting light and limiting friction between hair shafts. It is responsible for the shine and texture of hair. The keratin layer of hair becomes fragile and cracked under the influence of the external environment, temperature, humidity, along with chemical and physical treatments, thus affecting hair quality. Although most people’s hair is prone to various damage problems because hair is inconvenient to observe, it is impossible to conduct a detailed analysis, and there are few studies related to hair damage.

With the widespread application of photomicrography in the field of hair microstructure analysis, it has become easier to observe hair details that were difficult to clearly observe with traditional microscopes. Clinically, microscopic analysis can be used as a tool to assess hair damage, as an indicator of health status, by identifying the morphological characteristics of hair damage, and it can be analyzed in a qualitative manner. Hair’s exposure to various chemical agents and physical effects, such as detergents, dyes, combing, ultraviolet rays, etc., will affect hair. Researchers have conducted related studies accordingly. However, there is limited research that quantifies the degree of hair damage based on the morphological characteristics of hair. Traditionally, it is difficult to distinguish hair damage in detail using only manual observation with an optical microscope. With the development of microscopy techniques and the application of scanning electron microscopy (SEM) in multiple fields, the microscopic details of hair are easier to find. For example, Coroaba A et al. [1] used scanning electron microscopy (SEM), energy-dispersive X-ray spectroscopy (EDX), and other physical and chemical methods to identify the effects of alopecia areata on the structure and composition of human hair. Lima et al. [2] used a heating iron to heat hair and observed hair images under a scanning electron microscope to evaluate the damage to the epidermis and cortex of the hair fiber by heat loss. Although these manual methods use scanning electron microscope imaging technology to analyze hair microscopic images, they do not perform detailed quantitative classification of hair damage.

In this paper, we propose a novel hair damage detection network. Based on artificial intelligence algorithms, were used to analyze scanning electron microscope (SEM) hair surface image damage (Figure 1) to automatically identify and categorized the damage. At the same time, we established a new hair damage data set based on SEM microscopic image data. To summarize, the contributions of this work are as follows:

• We created a new hair microscopy data set based on SEM (Scanning Electron Microscope) image data and performed a quantitative analysis to classify the degree of hair damage under the categories: weak damage, moderate damage, and high damage.
• We proposed a novel and effective convolutional network model for hair damage detection: RCSAN-Net (residual channel spatial attention network).
• We designed and introduced a channel and spatial attention mechanism into the hair damage detection model to gather hair features to improve the accuracy of detection and recognition.

2. Related Work

Hair micro-detail detection and recognition can find use in many disciplines such as medicine and forensics. In the context of forensic science, microscopic hair analysis (a qualitative method) has shown effective discrimination [3,4], and detailed examination of hair under the microscope is helpful for forensic identification. Clinically, microscopic analysis can be used as a tool to assess hair damage, as an indicator of health status [5,6], but the analysis needs to be performed under certain conditions. Hair is subjected to various physical and chemical agents, including detergents, dyes, combing, and ultraviolet rays, which will change the structure of the hair [7,8,9]. The morphological characteristics of damaged hair can be detected by qualitative methods for clinical research. Most studies focus on the identification of hair damage, and few studies quantify the degree of hair damage based on morphological characteristics [10,11,12]. It is worth noting that Kim et al. proposed a classification system for hair damage. The system is divided into five damage levels, expounding the influence of external factors on hair damage [13]. After that, Lee et al. extended it to 12 classifications [14]. Observing the images of hair from a scanning electron microscope (SEM), the researchers assessed the severity of hair structure irregularities subjectively and refined the degree of hair damage into 12 levels. In these studies, visual assessment was used as the main qualitative technique for hair damage. Few people have developed a more objective hair damage analysis and identification system, most of the research focuses on the manual analysis of morphological features detected by optical microscope and analysis using commercial software [15,16], while there are few studies on automatic and efficient detection of hair damage.

The detection and recognition of microscopic hair features is a challenging task. In recent decades, with the continuous development of microscopic examination technology, the microstructure of the hair that can be observed is clearer, and more precise, and the sample preparation is simple and high-resolution, which makes the study of the microscopic morphology of hair more in-depth. The manual detection method is still the most important practical detection method, but it usually takes at least one year to train qualified professional detection personnel. This is time-consuming and laborious because it requires the microscopist to observe the fiber morphology under the microscope for a long time, and the subjective and detection accuracy depends on the experience of the microscopist. For example, Enrico et al. [17], used SEM image data to study the relationship between hair damage that may be caused by repeated cosmetic treatments and the absorption of cocaine from a wettable solution into the hair matrix (simulating external contamination). Over the years, researchers have been developing a fast and accurate method to automatically detect the degree of hair damage and have made many useful attempts. These methods mainly include optical microscope and scanning electron microscope (SEM) observation analysis of microscopic hair images, and the application of image processing and computer vision technology to autonomously and efficiently detect the degree of hair damage.

In recent years, with the emergence of convolutional neural networks and a large number of applications, Methods based on image processing and computer vision are the current research hotspot, greater number of researchers have focused on the field of hair detection. For example, Umar Riaz. et al. [18] proposed a convolutional model that uses convolutional neural network algorithms from an unconstrained perspective and uses only texture information to achieve complete hair analysis (detection, segmentation, and hairstyle classification). Chang. et al. [19] proposed a smart scalp inspection and diagnosis system based on deep learning used to detect and diagnose four common scalp and hair symptoms (dandruff, folliculitis, hair loss, and oily hair). As a part of scalp health care, it is used for scalp and hair physiotherapy, and is an effective inspection and diagnosis system. In the early days, people mainly used image processing technology to measure the geometric parameters of hair, such as hair diameter and hair uniformity. Recently, some scholars have begun to use computer vision technology to extract abstract detailed features from hair microscopic images in order to recognize them. For example, Jiang et al. [20]. Proposed a convolutional network algorithm based on artificial intelligence: XI-Net, which performs feature analysis, extraction, and classification of hair microscopic SEM images to deal with forensic criminal investigation cases. According to this algorithm, the forensic scientist can quickly extract the desired hair detail features, and the investigation efficiency is greatly improved, speeding up the detection of the case.

3. Materials and Methods

3.1. Convolutional Neural Network

With the emergence of Lenet [21], a great success in handwriting font recognition, convolutional networks began a new chapter in automatic image recognition. Since AlexNet [22] won the ImageNet [23] image recognition algorithm award, deep convolutional neural networks have dominated image classification. With this trend, research has shifted from engineering manual functions to engineering network architectures. For example, VGG-Net [24] proposed a modular network design method that uses stacking the same type of network block strategy, which simplifies the workflow of network design and simplifies the transfer learning of downstream applications. Inception [25], expands the depth and width on the convolutional layer, extracts the information of multiple scales in the image, and obtains more features. At the same time, the 1×1 size convolutional layer is used for dimensionality reduction, which reduces the amount of calculation. ResNet [26,27] introduces skip connections, which greatly alleviates the problem of vanishing gradients in deep neural networks and allows the network to learn improved feature representations. Properly changing the type and number of layers in the residual block (Figure 2) can improve the performance of the network. ResNet has become one of the most successful CNN architectures and has been adopted by various computer vision applications.

3.2. RCSAN-Net: Residual Channel Spatial Attention Network

We proposed the RCSAN network as shown in Figure 3 based on the residual network [28]. We deal with complex and difficult-to-detect hair damage feature regions by stacking multiple residuals and a residual attention block formed by combining attention blocks, thereby establishing an efficient network framework for hair damage detection.

The RCSAN network was constructed by stacking multiple residual attention modules. Previous studies usually only focused on one type of attention, such as scale attention [29] or spatial attention [30], and could not extract the precise features of the target area. In our attention mechanism model, each module was divided into two parts: residual block and channel-spatial combined attention block. The residual block convolutional network extracts the global features of the image, and then enters the attention module to pay attention to the local detailed features. However blindly adding the number of attention models does not improve the overall network performance. In our network model, we use skip connection to connect the residual block and the attention module.

After our model receives the image, it uses standard convolution calculation to generate a feature map in the early stage, enters the attention mechanism module, extracts the multi-channel and spatial information about the feature, generates an attention feature map, and then enters the next round of calculation. Attention to the feature extraction process is shown in Figure 4.

3.3. Attention Mechanism Module

As we all know, attention plays an important role in human perception. Humans use a series of parts to glimpse and selectively focus on the prominent parts in order to better capture the visual structure. In the convolutional neural network, adding an attention mechanism can effectively improve the performance of a CNN in large-scale classification tasks.

When processing hair SEM microscopic image data, due to the small differences in hair surface characteristics, ordinary algorithm models cannot accurately identify the subtle differences in hair damage. In this study, we designed an attention mechanism module suitable for hair microscopic images. To improve the accuracy of hair SEM microscopic image detection and damage feature identification. The attention module includes, a channel attention module and spatial attention module.

3.3.1. Channel Attention Module

Each channel graph of a feature can be regarded as a class-specific response, and different semantic responses are related to each other. By taking advantage of the interdependence between channel mappings, we can emphasize interdependent feature mappings and improve specific semantic feature representations. Therefore, we added a channel attention module to enhance the interdependence between feature channels. The structure of the channel attention module, is shown in Figure 5.

We reshape the original feature A ∈ R^C×H×W into A ∈ R^C×N perform matrix multiplication between the transpose of A_i and A_j, and then pass the softmax function to generate the attention feature K ∈ R^C×C. In addition, perform matrix multiplication between the transposes of K and A and reshape their results to R^C×H×W. The final output Channel attention feature map M_c (F) ∈ R^C×H×W was as follows:

$${\mathit{M}}_{\mathit{c}}(\mathit{F})=\beta {\displaystyle \sum }_{i=1}^C\left(k_{ij}{A}_i\right)+A_j$$

(1)

where K_ij measures the j^th channel’s impact on the i^th channel, and the scale parameter β gradually learns a weight from zero. The final feature of each channel is the weighted sum of the features of all channels and the original features, which models the remote semantic dependence between feature maps.

3.3.2. Spatial Attention Module

Different from channel attention, the focal point of spatial attention is the specific area information part of the feature map, which supplements the channel attention. The spatial attention map is generated by using the spatial relationship between the elements. The structure of the spatial attention module, as shown in the Figure 6. When calculating spatial attention, in order to retain the feature information from channel attention to the maximum, we adopt a dual pooling method combining maximum pooling and average pooling to generate two 2-dimensional feature maps: Fa and Fm. They are connected to generate effective feature descriptors, and after standard convolutional layer calculations, generate a spatial attention feature map M_s (F) ∈ R^H×W:

M_s (F) = σ (f^7×7([avgpool(f); maxpool (f)]))

(2)

where σ denotes the sigmoid function and f^7×7 is the filter size in the convolution operation.

4. Results

4.1. Datasets

We collected hair samples from 50 men and 50 women in the 20–60 age group and used a Hitachi S-4700 SEM (scanning electron microscope) to generate hair microscopic images. First, the sample hair was screened, classified, and cut. The hair section was attached to the conductive tape coated with carbon on both sides and fixed on the metal aluminum rod, and platinum is used as the metal target. The spray coating machine performed sputtering coating under a high vacuum. Then, an SEM (model: Hitachi S-4700) was used to scan the coated hair sample at 15 kV. Scans of multiple sections of each axis were performed, to ensure that the changes in the surface of the hair sample seen were uniform and not isolated. The damage to the hair shaft under the scanning electron microscope was divided into three grades: weak damage, moderate damage, and high damage (Figure 7). We used the degree of cuticle damage as the criterion to judge hair damage. The sample with no obvious cracks on the cuticle of the hair surface was defined as having weak damage; the sample with severe cracking and warping of the cuticle was defined as having moderate damage; cuticles that had almost completely disappeared from the sample, were defined as having high damage.

We collected the data of 500 hair samples from 100 testers, each hair data sample was evenly divided into three equal parts, and the 5 mm segment in the center was selected for observation, and classified as a weak damage, moderate damage, and high damage image. Five hundred images for each hair damage category were produced. Since the collection of hair SEM microscopic images is complicated and expensive, it is impossible to collect data on a large scale. To this end, we used data enhancement algorithms to enhance the collected hair SEM images, as shown in the Figure 8. For example, we used image inversion, rotation, cropping, adding Gaussian noise and other algorithms. We created a hair SEM data set that has a total of 15,000 microscopic hair images of various types, containing:

• Five thousand microscopic images of weakly damaged hair.
• Five thousand microscopic images of moderately damaged hair.
• Five thousand microscopic images of highly damaged hair.

4.2. Implementation Details

In the experiment, we divided the data into a training set and a test set, 80% as a training set and 20% as a test set. We crop the image size to 224 × 224 by using center cropping. The network was trained using a stochastic gradient descent algorithm (SGD) [31]. We set the initial learning rate to 0.1 and it was gradually decreased. We used a Nesterov momentum [32] of 0.9 and a 10-4 weight decay without dampening for all the weights. Our model parameters are shown in

Table 1. The Configuration parameters of SACN-Net.

Layer	Output Size	Attention
Conv1	112 × 112	7 × 7, 64, stride 2
Max pooling	56 × 56	3 × 3 stride 2
		(1 × 1, 64)
Residual Unit	56 × 56	(3 × 3, 64) × 1
		(1 × 1, 256)
Attention Module	56 × 56	Attention × 1
		(1 × 1, 128)
Residual Unit	28 × 28	(3 × 3, 128) × 1
		(1 × 1, 512)
Attention Module	28 × 28	Attention × 1
		(1 × 1, 256)
Residual Unit	14 × 14	(3 × 3, 256) × 1
		(1 × 1, 1024)
Attention Module	14 × 14	Attention x1
		(1 × 1, 512)
Residual Unit	7 × 7	(3 × 3, 512) × 3
		(1 × 1, 2048)
Global average pooling	1 × 1	7 × 7 stride 1

.

4.3. Comparison with Other Advanced Methods

We chose other excellent network models, for example, the classic AlexNet algorithm, the widely used VGG network model, the lightweight CNN network MobileNet, and the excellent performance ResNet and other state-of-the-art methods to compare with our proposed model algorithm. The model we proposed is based on ResNet50. In the performance test comparison, we compared the 34-layer and 50-layer ResNet networks. The experiments shown in

Table 2. Comparison of accuracy of different models.

Mode	Accuracy
AlexNet	0.8310
VGG16	0.9053
Inception	0.9239
MobileNet	0.9143
ResNet34	0.9255
ResNet50	0.9474
SACN-Net	0.9838

show that our proposed SACN-Net has better accuracy compared to the other models.

In the model test, we also added a test of the attention module, Figure 9 shows the attention characteristics of the hair damage area, and a comparative test of the channel attention module only, the single use space attention module, and the channel space attention module combined with the position change method regarding the overall performance of the model. We chose to use different attention modules in the model, and there was a big gap in performance, using a single attention module because it could not accurately identify the target, in the fusion of multiple attention modules, the location of each attention module will also result in performance differences.

In Top-1, in the benchmark test as shown in

Table 3. Classification error (%) on SEM Hair dataset.

Attention Type	Top-1 Err. (%)
Channel Attention	5.03
Spatial Attention	4.25
Channel & Spatial Attention	2.92
Spatial & Channel Attention	2.47

, we found that the performance is the best based on the channel first and then the spatial attention module.

5. Discussion

We focused on how to detect hair damage autonomously and efficiently. We created a system based on artificial intelligence convolutional neural network that could autonomously detect the degree of hair damage. This could be used for personal daily care and maintenance, medical health diagnosis and other directions. We are currently developing a handheld machine that can detect the degree of hair damage, using the hair damage detection algorithm to quickly diagnose hair damage. When collecting hair samples, we only considered adults 20–60 years old, and did collect hair samples from minors or people over 60. The reason was that people in this age range care more about daily hair care and have more hair damage. In turn, thid helped our hair damage sample collection. We observed SEM (scanning electron microscope) images, considering the possible influence of hair color on the degree of hair damage, and we collected some brown, brown and red hair samples. After observation and analysis, we found that the relationship between hair color and hair damage is weak. When making the data set, we compared the microscopic characteristics of hair at 400× and 800× (Figure 10). First, we used the 400× hair image under SEM observation. The display effect was not satisfactory, the hair surface texture could not be displayed accurately, and it was difficult to use for model training. Finally, we chose the 800× hair image as the standard of the data set and made our own SEM hair data set.

In the model, we explored the proposed attention mechanism model to find the best performance of the model. After many revisions, it is found that the dual attention mechanism module, especially the channel first attention method, has the highest efficiency in identifying hair damage features. Figure 11 shows the attention feature extraction of our attention mechanism module.

6. Conclusions

In this paper, we created an SEM based microscopic hair image data set of 50 male and 50 female hair samples in the 20–60 year age range and divided the data into weak damage, moderate damage, and high damage according to the degree of damage to the cuticle on the surface of the hair. We mainly focused on the degree of hair damage. When collecting hair samples, we adopted a random collection method. The samples include hair samples such as dyed hair, curly hair, and straight hair. During microscopic observation, the middle section of each hair was used as the observation area. We reduced the variability of the samples. We also proposed a new residual convolutional network based on the spatial attention module and verified it with our data set. Experiments showed that our network model is effective and robust. In this article, we only discussed the degree of hair damage, without the diagnosis of the cause of the hair damage. In future research, we will focus on the cause of hair damage and analyze it.