YOLOv7-Based Anomaly Detection Using Intensity and NG Types in Labeling in Cosmetic Manufacturing Processes

Abstract

The advent of the Fourth Industrial Revolution has revolutionized the manufacturing sector by integrating artificial intelligence into vision inspection systems to improve the efficiency and quality of products. Supervised-learning-based vision inspection systems have emerged as a powerful tool for automated quality control in various industries. During visual inspection or final inspection, a human operator physically inspects a product to determine its condition and categorize it based on their know-how. However, the know-how-based visual inspection process is limited in time and space and is affected by many factors. High accuracy in vision inspection is highly dependent on the quality and precision of the labeling process. Therefore, supervised learning methods of 1-STAGE DETECTION, such as You Only Look Once (YOLO), are utilized in automated inspection to improve accuracy. In this paper, we proposed a labeling method that achieves the highest inspection accuracy among labeling methods such as NG intensity and NG intensity when performing anomaly detection using YOLOv7 in the cosmetics manufacturing process.

1. Introduction

Pandemics and other forms of infectious disease outbreaks are unique examples of manufacturing risk, characterized by high uncertainty, increased contagion, and long-term disruption to manufacturers, supply chain actors, end users, and consumers [1]. The outbreak of the COVID-19 pandemic has had an enormous impact on various sectors, including manufacturing. The numerous restrictions imposed to control the spread of the virus have significantly disrupted manufacturing operations and supply chains across the globe. The unprecedented nature of the pandemic has created significant uncertainty and volatility in the manufacturing industry, requiring changes in working conditions and significant adjustments within manufacturing facilities. Developing sustainable products and processes has become essential for the survival of manufacturers in the current competitive market and Industry 4.0 era. The COVID-19 pandemic has intensified the impact of automation [2]. Humanity’s digitalization journey has begun, with the majority actively embracing smart technologies and their benefits [3].

However experienced they are, humans can make mistakes. Manual assembly and inspection tasks are particularly susceptible to human error, which can reduce the quality of the final product [4]. Humans are fallible, no matter how experienced they are. Fatigue, distraction, subjective judgment, or differences in individual expertise can all contribute to inconsistencies and inaccuracies in quality inspections, and human error can lead to acceptance of defective products or rejection of non-defective products, affecting overall quality control and customer satisfaction.

Detecting product defects is essential for manufacturing quality control [5]. Human inspection can increase personal expenses. Inspections often need to be performed by skilled workers, and it costs more to get their expertise. Training and retaining qualified personnel can be costly for a company, especially if the inspection process requires specialized knowledge or certification, and furthermore, reliance on human inspectors can lead to increased labor costs, especially if overtime or additional personnel are needed to meet inspection requirements.

Manual quality inspections often require a dedicated space or area where inspections can be performed. Allocating space for inspection activities can be challenging, especially if floor space is limited or additional inspection stations need to be set up, which can increase overhead costs associated with renting or building inspection facilities, equipment, and storage space.

Artificial Intelligence (AI) has been successfully applied in industry for decades, from the emergence of expert systems in the 1960s to deep learning today [6]. Machine vision technology has been integrated into manufacturing workplaces to achieve an efficient and high-quality production mode for manufacturing [7]. Leveraging deep learning technology for quality inspection provides solutions that minimize issues related to human error, personal costs, space costs, and time costs, enabling companies to accelerate R&D, improve quality, reduce errors, and sustain supply chains through demand forecasting and outcome simulation to generate higher margins in the face of fierce competition [8].

By harnessing the power of deep learning algorithms, smart factories can improve quality control processes, optimize efficiency, and mitigate a variety of issues that traditionally plague quality inspection procedures. One important benefit of implementing deep learning technology in smart factories is the ability to minimize problems caused by human error. Quality inspection tasks often rely on human judgment, which can lead to inconsistencies and mistakes, but by integrating deep learning algorithms into the inspection workflow, smart factories can use advanced computer vision systems to analyze vast amounts of data and images. Because these algorithms can be trained using a wide range of data sets, they can learn complex patterns and detect even the smallest deviations that human inspectors might miss. As a result, smart factories can achieve consistently high accuracy, minimizing human error and its associated consequences.

In addition to reducing human error, deep learning technologies also contribute to minimizing personal costs in quality inspection. Traditional quality control processes often require a significant number of human resources, making it costly to recruit, train, and retain qualified personnel. However, by moving to a smart factory model, the integration of deep learning algorithms can automate and streamline inspection tasks. Intelligent machines and robotic systems with deep learning capabilities can take over repetitive and time-consuming inspection processes, significantly reducing the need for a large workforce, which in turn can optimize operational costs and resource allocation while minimizing personal costs.

The application of deep learning in smart factories can dramatically reduce the cost of time, a key factor in quality inspection. Traditional inspection processes can be time-consuming, leading to delays in production schedules and potential customer dissatisfaction. Deep learning algorithms allow data to be analyzed in real time to make quick and accurate decisions. Smart factories can quickly process large amounts of data and images to quickly identify defects or quality issues, allowing for timely intervention and resolution. As a result, inspection time is minimized, leading to increased throughput, improved production schedules, and increased customer satisfaction.

You can also integrate deep learning into your smart factory to continuously improve and optimize your quality inspection processes. These algorithms can learn from real-time data and adapt to changing conditions to continuously improve accuracy and performance. Smart factories can leverage machine learning capabilities to detect trends, predict potential quality issues, and implement proactive measures. This proactive approach minimizes defects, improves product quality, and drives continuous improvement throughout the production cycle. When training a YOLOv7-based model with a supervised learning method, it finds the labeling method with the highest accuracy and proceeds with anomaly detection to improve the accuracy of detecting defective products [9].

This paper is organized as follows. In Section 2, we describe the related work that helped us in our research, and in Section 3, we describe the architecture of our research. Section 4 displays the experimental results, including the research environment and process, and Section 5 concludes.

2. Related Work

2.1. Plastic Manufacturing Process

The widespread use of plastics as the main material for industrial products has attracted much attention [10,11]. The injection molding process of plastics involves injecting resin into a mold, where the melt cools and solidifies to form a plastic product. It is usually a three-step process consisting of filling, packaging, and cooling stages [12]. However, conventional plastic injection molding processes suffer from inconsistent product quality and large variation.

Molding conditions or process parameters play an important role in plastic injection molding. The quality of the molded part, such as strength, warpage, and residual stress, is greatly affected by the processing conditions, and the molding conditions also affect the productivity, cycle time, and energy consumption of the molding process. Molding conditions are closely related to other factors such as material, part design, and tooling that determine the quality of plastic products, and consist of important parameters such as melt temperature, mold temperature, filling time, packing time, and packing pressure.

The quality of the molded part depends not only on the properties of the plastic material, but also on the process parameters. Optimal process parameters reduce cycle times and improve product quality. In practice, process parameter settings are mainly based on the experience of plastics engineers, which does not always guarantee proper process parameter values. Because plastics exhibit complex thermo-elastic properties, it is difficult to set the right molding conditions to achieve the desired product quality. Therefore, process parameters are often adjusted through a lot of trial and error. The trial-and-error approach can prove to be costly and time-consuming [13].

2.2. Supervised Learning

Supervised learning is a machine learning task that learns a function that maps inputs to outputs based on example input-output pairs. In supervised learning, both normal and abnormal samples are present in the training data set, and these two samples are used together to train a detection model. The trained model identifies test samples as either normal or abnormal [14].

Supervised learning infers a function from labeled training data, which consist of a set of training examples. The supervised machine learning algorithm is the following algorithm. The input data set is divided into training and test data sets. The training data set has an output variable that needs to be predicted or classified. All algorithms learn some kind of pattern from the training data set and apply it to the test data set for prediction or classification [15].

Supervised machine learning involves predetermined output attributes in addition to the use of input attributes. The algorithm attempts to predict and classify the predetermined attributes, and the accuracy and misclassification of the algorithm, along with other performance measures, depend on the number of predetermined attributes that are correctly predicted or classified. When the algorithm achieves an acceptable level of performance, the learning process stops. According to [Libbrecht and Noble] [16], technically, supervised algorithms first perform analytical operations using the training data and then construct conditional functions for mapping new instances of the attributes. As discussed earlier, the algorithm requires a pre-specification of the desired results and maximum settings for the performance level [17]. Considering the approach used for machine learning, it has been observed that about 66% of the training subsets are reasonable and help to achieve the desired results without requiring more computational time [18].

2.3. Yolo

In recent years, deep learning (DL) applications and systems have blossomed [19]. Among them, object detection, a key use in image processing, has grown rapidly since 2012 with unprecedented advances in convolutional neural networks (CNNs) and their variants [20,21]. Object-based algorithms embed semantic information in groups of pixels with similar characteristics such as color, texture, brightness, and shape rather than in individual pixels [22].

Object detectors are broadly classified into two categories: two-stage object detectors and one-stage object detectors. While two-stage detectors mainly focus on selective region proposal strategies with complex architectures, one-stage detectors focus on proposing all possible spatial regions with relatively simple architectures to detect objects at once. The performance of all object detectors is evaluated through detection accuracy and inference time. In general, the detection accuracy of two-stage detectors outperforms one-stage object detectors. However, the inference time of the one-stage detector is better than the two-stage detector. In addition, with the advent of the one-stage detector You Only Look Once (YOLO) and its architectural successors, detection accuracy is improving significantly, sometimes even better than two-stage detectors.

YOLO is an algorithm that uses neural networks to provide real-time object detection. YOLO is mainly adopted in various applications due to its fast inference speed rather than considering detection accuracy. For example, the detection accuracy of YOLO, a one-stage detector, and Fast-RCNN, a two-stage detector, are 63.4 and 70, respectively, but the inference time is about 300 times faster for YOLO [23]. YOLO uses a unique approach. YOLO uses a clever convolutional neural network (CNN) to detect objects in real time. The algorithm implements one neural network on the entire image, then sections the image into multiple sections and estimates bounding boxes and probabilities for each region. These bounding boxes are then weighted according to the estimated probabilities. In YOLO, the CNN predicts multiple bounding boxes and probabilities for those boxes at any given time. It trains on real images and optimizes performance directly. Improving real-time object detector accuracy can improve recommendation hint generation and hint generation recommender systems, as well as standalone process management, and reduce the need for human input [24].

2.4. Anomaly Detection

Anomaly detection is the process of identifying outliers that have unexpected patterns with respect to the normal data distribution [25]. Nowadays, it has become essential to monitor the health of manufacturing environments to avoid unexpected repairs, downtime, and to detect defective products that can cause large losses [26]. Anomaly detection for industrial processes is essential for industrial process monitoring and is an important technique to ensure production safety. There are numerous real-world applications for anomaly detection, including sensor fault identification, product manufacturing quality control, network intrusion detection, and pandemic geospatial modeling. Effective anomaly detection on the factory floor improves availability, product quality, worker safety, and reduces rework costs [27].

Manufacturing companies use cameras and laser sensors to create data on product surfaces or product conditions, and then utilize these data to automatically perform quality inspections, including statistical methodologies, image processing methodologies, and methodologies utilizing machine learning models. Statistical methods collect and analyze data produced in the field to improve quality control and processes, helping workers and managers make meaningful decisions. Statistical methods help correlate, organize, and interpret data, and statistical analysis reveals underlying patterns in a data set. Machine learning is commonly presented as an approach used in smart manufacturing inspection and has implications for quality management systems in industry. Histogram analysis performs various statistical analyses such as mean, geometric mean, standard deviation, and median. Due to its simplicity, this method has been widely used for low-cost, low-level analysis in a variety of problems. Autocorrelation analysis measures the correlation between an image and the display vectors in the image by repeatedly using patterns or textures on the surface of a product, such as wood or textile products. Anomaly detection and timely assessment by machine vision systems can enable the industrial sector to take an innovative leap forward.

Typical applications of anomaly detection research include big data anomaly detection, Internet of Things (IoT), Wireless Sensor Networks (WSN), network anomaly/intrusion/attack detection, manufacturing anomalies, and video surveillance anomaly detection. With the development of new technologies such as sensor technology and information and communication technology, vast amounts of data are continuously collected over time. Data recorded in chronological order constitute a time series, and typical examples of time series are data collected such as hourly temperature of a machine, daily stock price of a company, weekly interest rate, monthly sales of a store, and annual Gross Domestic Product (GDP) of a country. Time series anomaly detection is one of the fundamental tasks in time series analysis, and much research has been focused on this area of study in the last few decades.

3. Yolov7-Based Anomaly Detection Using Intensity and NG Types

3.1. System Architecture

In this paper, we compared the F1 score, mAP of labeling NG in cosmetic containers by dividing them into Strong NG and Weak NG according to the strength of the product NG, labeling each type of NG by subdividing the Region of Interest(RoI) into Scratch, Pollution, Point, and Edge by separating the top and side of the cosmetic container, and labeling each type of NG without separating the RoI. The performance of each labeling method is evaluated for accuracy using the under- and over-check rates. To improve the performance, we return to the preprocessing after the performance evaluation and repeat the process to compare the final performance.

Figure 1 displays the architecture of anomaly detection in the painting process during the cosmetics manufacturing process. The process begins with the acquisition of a raw data set of the cosmetic container using a camera, lens, and optical illumination. These components ensure clear and consistent image acquisition, enabling reliable data for subsequent analysis and processing. Once acquired, the raw dataset is subjected to preprocessing techniques to enhance the data and prepare it for further analysis. Preprocessing includes operations such as Fourier transform, wavelet transform, and normalization. After preprocessing, the dataset is classified into meaningful images suitable for training by identifying and extracting relevant features from the images such as shape, size, and color of cosmetic containers.

After preprocessing, image augmentation techniques are applied to improve the generalization ability of the model, fill in gaps in the dataset, and increase diversity. These techniques involve applying various transformations to existing images, such as rotating, scaling, flipping, and adding noise, to create new training examples. Enriching the dataset makes the deep learning model more robust and able to handle changes in container shape. The labeled dataset is used to build accurate knowledge about normal cosmetic containers. Each diversified image is assigned an appropriate label to indicate whether it represents a normal or abnormal container. By dividing the labeling into three methods (labeling by intensity of NG, labeling by type of NG without distinguishing RoI, and labeling by type with distinguishing RoI), we find out which labeling method produces higher accuracy when performing anomaly detection in cosmetics manufacturing process. The labeled information is trained using the YOLO algorithm. By learning from the labeled data, the model performs Anomaly Detection to accurately detect and distinguish anomalies such as irregular shapes, deformities, or defects that deviate from the expected normal cosmetic container shape. Anomaly Detection with YOLO algorithm is used to analyze cosmetic containers in real time.

3.1.1. Dataset

Figure 2 is a photograph of the top and side of a plastic cosmetic container that has undergone an injection molding process. The raw dataset consists of images of cosmetic containers taken using a camera, lens, and optical illumination. These images form the basis of the anomaly detection process. The obtained data is classified into meaningful images that can be used to train a deep learning model. This classification involves identifying and extracting relevant features or RoIs within the images. In the context of cosmetic containers, these RoIs include areas such as the sides of the container, the top of the container, and certain components that are important for inspection. By focusing on these meaningful images, the model can learn to analyze and detect anomalies in specific areas of the container, improving accuracy and efficiency.

Table 1. Cosmetic Container Spec and Lab Setup.

Side	Top	Distance between Products
30.15 × 28.11 (mm)	25.86 × 25.86 (mm)	80 (mm)

displays the experimental dataset, a cosmetic container Spec, and the configuration of the experimental environment.

3.1.2. Meaningful Image Classification

Framework Darknet [28] can be used to train convolutional neural networks (CNNs) for image classification, object detection, and segmentation, among other tasks. To perform image classification to train a CNN, a high-quality training dataset containing both normal and anomalous images must be prepared. The dataset should be carefully curated to include a range of images that represent normal operating conditions of the system being monitored, as well as images that contain anomalies. Once the dataset is ready, a CNN can be trained for image classification using the Darknet framework. The network is trained using a supervised learning approach where it learns to map input images to their corresponding labels, and during training, the neural network adjusts its parameters to minimize the difference between the predicted and actual labels. Once the network is trained, it can be used to classify new images as normal or anomalous. If an image is classified as an anomaly, you can trigger an alert or perform further analysis to determine the nature of the anomaly.

To achieve high accuracy in image classification, it is important to choose the right network architecture and hyperparameters. The network architecture should be designed to handle the specific features of the images being classified, and the hyperparameters should be tuned to optimize performance on the training dataset. By accurately identifying objects and scenes in an image, anomalies that deviate from the expected pattern can be detected, allowing for timely intervention and preventing further problems.

3.1.3. Pre-Processing

Image preprocessing is a set of methods used to enhance the quality of an image for subsequent processing purposes [29]. Quantum leaps in performance have been realized in the last decade [30]. Different image preprocessing methods are required to perform different tasks such as image sharpening, contrast enhancement, and cloud removal [31]. Data preprocessing is an important part of a deep learning project and is a large part of the overall analytics pipeline [32]. It aims to improve the quality, consistency, and relevance of the data so that the model can learn effectively and produce accurate results. Preprocessing can be a simple stretching of histograms or a more complex approach such as denoising or filtering [33,34]. Image preprocessing typically consists of enhancement (i.e., improving image quality) and restoration (i.e., removing degraded regions); the former is a more subjective process, while the latter is an objective process that models the degradation (based on prior knowledge) and applies an inverse process to recover the original signal. In the context of deep learning for anomaly detection in cosmetic containers, preprocessing plays an important role in preparing the image data before training the model. Preprocessing aims to improve data quality, reduce noise and inconsistency, and provide optimal representations for training deep learning models, ultimately leading to more accurate anomaly detection in cosmetic containers. The following preprocessing techniques were used in this paper.

• Fourier Transform
  The Fourier transform consists of decomposing a signal or image into a sum of fundamental signals, which has the property of being easy to implement and observe [35]. Since these fundamental signals are periodic and complex, the amplitude and phase of the system can be studied. The Fourier transform is a powerful mathematical tool for analyzing the frequency content of signals and functions. It finds a wide range of applications in various fields, providing insight into the fundamental properties of signals, facilitating filtering operations, and enabling efficient data compression techniques.
• Wavelet Transform
  Wavelet transform is a mathematical tool used to analyze signals and data in both the time and frequency domains. It provides a localized representation of a signal by decomposing it into a series of wavelet functions called wavelets. Unlike the Fourier transform, which uses a fixed sinusoidal basis function, the wavelet transform uses wavelets that are localized in both time and frequency, allowing for more precise analysis of signals with transient or localized features.
  Wavelet functions are mathematical functions that are localized in both time and frequency, and are typically derived from a mother wavelet through scaling and transformation operations. The mother wavelet acts as a building block, and by expanding or compressing it and moving it through time, a set of wavelets with different sizes and positions can be obtained.
  Wavelet transforms have a wide range of applications across multiple domains, including signal processing, image compression, noise reduction, feature extraction, and time-frequency analysis, and provide a powerful tool for analyzing signals with localized or time-varying characteristics, allowing for more detailed and adaptive representations compared to traditional Fourier-based methods.
• Normalization
  Normalization is a common preprocessing technique used to standardize the scale or range of input data, which involves transforming data in such a way that it has a consistent scale and distribution that helps improve the performance and convergence of many machine learning algorithms. The normalization process involves adjusting the values of a feature or set of features to fall within a certain range or follow a certain distribution, with the goal of bringing features to a similar scale and removing any bias or variation that may exist in the original data. Normalization is especially important when dealing with features that have different units of measure or vary widely in range, as without it, certain features on a larger scale can dominate the learning process and bias the model’s behavior for those features.

3.1.4. Image Augmentation

Deep learning requires sufficient defect data for training, but in general production sites, there are problems such as data shortage, which means that it is difficult to obtain a large amount of defect data or defect samples for training, and data imbalance, which means that certain classes of data are much more or less than other classes. If training is performed with an insufficient amount of data, it may display insufficient inspection accuracy for field operation, so image augmentation is performed as a way to overcome this.

The purpose of image augmentation is to introduce variability to the train dataset and increase diversity so that the model can better generalize to unseen data. Flipping, rotating, scaling, cropping, and adding noise to the image can be utilized to improve the robustness of the model to different lighting conditions, viewpoints, and opening directions. By augmenting the training dataset with transformed versions of the original image, the model can learn to recognize anomalies even when they occur under different circumstances. Leverage deep learning models such as Cycle GAN [36] and SRGAN [37] to generate anomaly data and training data to increase data diversity to address data scarcity, data imbalance, and enhance generalization capabilities.

3.1.5. Labelling Processes

Labeling plays a pivotal role in supervised-learning-based anomaly detection, enabling effective training, quantitative performance evaluation, iterative improvement, real-world applicability, generalization, adaptability, and decision support. Accurate labeling provides the basis for training accurate models to detect anomalies with high precision and recall. Various industries can leverage the power of labeling to build robust anomaly detection systems that drive efficiency, reliability, and improved outcomes, and by recognizing the importance of labeling, organizations can unlock the full potential of supervised learning-based anomaly detection and solve critical problems in their domains. To improve accuracy in supervised-learning-based anomaly detection for the cosmetics manufacturing process, different labeling methods are utilized to find the optimal labeling method.

Figure 3 displays the sequence of a specific labeling process. First, select an initial data point or region as the starting point for the labeling process. Then, define regions of interest in the data set you need to label using criteria such as the top and sides of the container, color, and texture. After defining the regions of interest, analyze the characteristics of the pixels or data points within the regions of interest to determine whether they represent normal or abnormal behavior. Assign a label or category to each region based on the results of the analysis. Once the initial regions are labeled, expand the labeling to include similar regions or patterns that may have been missed in the initial analysis. The process of expanding, analyzing, and labeling repeats until all relevant data points or regions are labeled. Repeat this iterative process until it converges, or until a certain level of accuracy or completeness is achieved. Review the labeled areas to ensure consistency and accuracy and validate the results using appropriate performance metrics such as F1 Score, mean Average Precision (mAP), etc.

3.1.6. Anomaly Detection

Figure 4 is the type of defective product that is determined by the inspection system. During the injection molding process for manufacturing cosmetic containers, the RoI is divided into the top and side areas of the container. The goal is to perform anomaly detection and classify the anomalies into specific categories such as Scratch, Pollution, Point, and Edge. A YOLO algorithm is used for this task. Anomaly detection is an important step in ensuring the quality of cosmetic containers. The YOLO algorithm is trained to detect and classify different types of errors that can occur during the injection molding process by using information learned from the rest of the container, except for the putting lines caused by the mold.

The classes defined for the above are as follows:

• Scratches
  This category refers to surface defects that result in noticeable marks on the top or sides of the container due to abrasions or scratches.
• Pollution
  Pollution is when the surface of a container is contaminated or scratched by foreign objects or debris.
• Point
  Point anomalies are localized irregularities, such as small dots or spots on the top or sides of a container that can affect its appearance or function.
• Edge
  Edge anomalies include irregularities or deformations along the edge of a container, which can compromise its overall quality or structural integrity.
• Swelling
  Swelling is the expansion of the surface of a part. Swelling can occur if the product’s molding temperature is too high, the molding pressure is too high, or the material overheats during the molding time. It can also occur if the product is poorly designed and the material does not cool properly.

The classification also distinguishes between StrongNG and WeakNG anomalies. This distinction helps to identify the severity or strength of the detected error.

• StrongNG
  Indicates a larger, more serious issue that requires immediate attention.
• WeakNG
  Anomalies that are confused with dirt or are relatively minor in nature.

By utilizing the YOLO algorithm, the system can effectively detect and classify these anomalies in real time. The algorithm analyzes the input image, identifies the RoI (top and side regions of the container), and applies object detection techniques to find and classify anomalies within a given class. The algorithm’s ability to simultaneously detect and classify multiple objects in an image is ideal for this task.

The YOLO-based anomaly detection system enables manufacturers to improve the quality control process during the injection molding stage of cosmetic container production. By accurately identifying and classifying anomalies, production problems can be quickly resolved, reducing the number of rejects and improving quality. By using the YOLO algorithm and defined anomaly classes for scratches, contamination, points, and edges, and distinguishing between StrongNG and WeakNG anomalies, effective anomaly detection and quality control in the injection molding process can be achieved, contributing to the production of superior cosmetic containers.

4. Experimental Results

4.1. Experimental Environments

Table 2. Software Development Environment.

OpenCV	CUDA	cuDNN	Framework	Working Distance	Conveyor Belt Speed
v6.4.16	v11.1	v8.2.0	Darknet	Top: 135 mm Side: 145 mm	150 mm/s

and

Table 3. Hardware Development Environment.

LENS	Lens Mount	C Mount
	Resolution	2464 × 2056
	Pixel Size	3.45 × 3.45
	Optical Format	2/3″
CAMERA	Name	MG-A500M-22
	Sensor Format	2/3″
	Mono/Color	Mono
	Dimension	29 × 29 × 40 (mm)
IPC	Processor	Intel Xeon Fold 5220R 6-Core Processor 2.20 Ghz
	RAM	90 GB
	GPU	Tesla V100-SXM2-32 GB
Optical Lightning	Name	ADQL4-300
	LED Count	84 EA
	Spec	300 × 300 × 100 (mm)

describe the Development Environment used to perform Anomaly Detection. OpenCV is an open source library that is widely used for developing image processing and computer vision algorithms. CUDA is a programming model and platform for NVIDIA’s parallel computing architecture, which leverages GPU acceleration to increase computational performance. We use OpenCV version 6.4.16 and CUDA version 8.2.0 to perform image processing and computer vision tasks. We utilize the Darknet framework to perform deep neural network (DNN)-based object detection and classification tasks. Darknet is an open-source DNN framework that is used to implement high-performance object recognition algorithms. In our study, we set a working distance of 135 mm from the top and 145 mm from the side. The working distance determines the depth of focus (DoF). The speed of the conveyor belt is set to 150 mm/s. This speed is one of the components of the automation system and is set to regulate the speed of object movement and processing.

The Lens and Camera used in the study were chosen to capture high-quality video. Lens is a lens system that utilizes optical properties to capture video. Camera is a device that collects and converts footage into digital form, providing high resolution and an appropriate frame rate. An IPC is a dedicated device for performing image processing, which is used for efficient processing and analysis of image data. The IPC applies algorithms and models developed in research to perform image processing tasks. Optical lighting is used to provide proper lighting conditions during image acquisition. In this study, we configured a proper lighting environment by adjusting the light intensity, color temperature, etc. This is one of the factors that affect the quality and Signal-to-Noise Ratio (SNR) of raw data. The above hardware development environment was assembled for raw data acquisition and used for image acquisition, signal processing, and analysis in the course of the study.

4.2. Data Collection and Processing

We studied a deep-learning-based EEG reading that can determine whether a cosmetic container has defects such as scratches and pollution during the cosmetic manufacturing process. In the context of anomaly detection in the cosmetic container manufacturing process, data collection and processing play an important role in identifying and mitigating container defects. A total of five cameras and one optical lens were utilized to perform real-time anomaly detection using top and side images of the container. The purpose of data collection and processing is to identify deviations from expected standards and take corrective action. By identifying and addressing defects early in the manufacturing process, cosmetic container manufacturing companies can improve product quality, reduce waste, and increase efficiency. The data collected can also be used to optimize the manufacturing process by identifying areas for improvement, such as adjusting the speed of the production line or changing the materials used in the manufacturing process. The cosmetic containers being studied are those that are in the process of being painted after the injection process.

4.3. Performance Matrix

In this study, we use performance metrics such as F1 Score, mAP, and Accuracy to evaluate the performance of YOLOv7 based on Anomaly Detection.

Accuracy is the ratio of TPs and TNs among the total samples identified. (TP = correctly predicted positives, FP = incorrectly predicted positives, TN = correctly predicted negatives, true negatives that are incorrectly predicted positives).

$$Accuracy=\frac{TP+TN}{TP+FN+FP+TN}$$

(1)

Precision is the percentage of samples that the classification model determines to be positive that are actually positive. Precision indicates how accurate the results detected as positive are.

Recall is the proportion of positive samples that the classification model identifies as positive out of the actual positive samples. Recall indicates how well the classification model identifies the true positive class without getting worse. (TP = correctly predicted positives, FP = incorrectly predicted positives, TN = correctly predicted negatives, true negatives that are incorrectly predicted positives)

$$Precision=\frac{TP}{TP+FP}$$

(2)

$$Recall=\frac{TP}{TP+FN}$$

(3)

Precision and Recall can be used interchangeably, with higher values for both metrics indicating a better model. F1 score is the harmonic mean between precision and recall, which is a metric that considers both accuracy and recall simultaneously. The F1-score is calculated from the total number of True Positives (TP) and False Positives (FP) in the failure data.

$$F1Score=2\ast \frac{Precision\ast Recall}{Pricision+Recall}$$

(4)

Average Precision (AP) is a metric used in multi-class classification problems such as object detection. It is obtained by calculating the area under the precision-recall curve of the precision-recall curve to find the AP for a multi-class problem. AP takes into account the imbalance between classes and evaluates the performance for all classes in the aggregate. performance across all classes. These metrics play an important role in evaluating the performance of machine learning models and are used to compare different models, optimize model parameters, and measure progress on various tasks.

Evaluating the performance of an algorithm in a multi-class classification model, the Average Precision (AP) for each class is calculated, then summed up and divided by the total number of object classes to obtain the mAP. This metric facilitates fair comparisons among different models and serves as a crucial measure to assess the overall effectiveness of an object detection algorithm. By considering both precision and recall, mAP provides a comprehensive evaluation of an object detection model’s performance.

4.4. Results

Figure 5 displays the training graph when labeled by the severity of the defect. Labeling NGs by intensity achieved F1 score of 53% and mAP of 41%.

Figure 6 displays the Training graph when labeled by defect type. The method of separating RoIs and labeling them by NG type achieved a 65% F1 score and 70% mAP.

Figure 7 displays training graph for labeling defects by type without RoI. Labeling by NG type without distinguishing RoI achieved an F1 score of 72% and a mAP of 76%.

Table 4. Labeling Training Results for Intensities.

Labeling Methods	Threshold	Total Number of Images	TN	FN	TN Rate	FN Rate
Intensity	0.5	939	69	68	7.35%	7.24%
Intensity	0.5	175	34	1	17.13%	1.71%
Intensity	0.5	480	55	1	11.46%	0.00%
Intensity	0.5	615	135	0	21.95%	0.00%
Intensity	0.5	1320	169	0	12.80%	0.00%

displays training results for labeling defects by Intensity. In this study, we ran tests using 3592 pieces of data over the course of five studies, labeled by the Intensity of the NG. After running the tests on the entire dataset, we analyzed the results to understand the performance of the system. The Final results were a TN rate of 12.8% and a FN rate of 0%.

Table 5. Labeling Training Results for Types.

Labeling Methods	Threshold	Total Number of Images	TN	FN	TN Rate	FN Rate
Type	0.5	1266	66	1	5.21%	0.08%
Type	0.5	485	44	1	7.59%	0.17%
Type	0.5	615	35	248	2.74%	19.42%
Type	0.5	4640	200	8	4.31%	0.17%
Type	0.5	1315	51	0	3.88%	0.00%

displays training results for labeling defects by Type. In this study, we ran tests using 8321 pieces of data over the course of five studies, labeled by the Type of the NG. After running the tests on the entire dataset, we analyzed the results to understand the performance of the system. The Final results were a TN rate of 3.88% and a FN rate of 0%.

Table 6. Labeling Training Results for Types without RoI Distinction.

Labeling Methods	Threshold	Total Number of Images	TN	FN	TN Rate	FN Rate
Type without RoI	0.5	1295	66	1	5.10%	0.08%
Type without RoI	0.5	195	20	44	2.74%	19.42%
Type without RoI	0.5	580	44	1	7.59%	0.17%
Type without RoI	0.5	824	45	0	5.46%	0.00%
Type without RoI	0.5	306	5	2	1.63%	0.00%

displays training results for labeling defects by Type without RoI Distinction. In this study, we ran tests using 3200 pieces of data over the course of five studies, labeled by Type without RoI Distinction. After running the tests on the entire dataset, we analyzed the results to understand the performance of the system. The Final results were a TN rate of 1.63% and a FN rate of 0%. A total of 15,113 data were obtained, and vision inspection was performed by dividing the RoI into the top and side of the cosmetic container. Based on these results, labeling by NG type without distinguishing RoIs achieved the highest F1 score of 72% and mAP of 76%, meaning it had the highest overall accuracy of the three methods. This suggests that focusing on labeling by NG type without considering the RoI leads to better results compared to the other two methods. The mAP and F1 score combined provide valuable insight into the precision, recall, and overall effectiveness of the model. mAP and F1 score combined provide valuable insight into the precision, recall, and overall effectiveness of the model. The method that labeled RoIs by class without distinguishing between them had the highest f1 score and mAP. It achieved high precision, recall, and accuracy in object detection or classification tasks, displaying good localization, accurate prediction, and robust performance across different classes. The second best performing labeling method is labeling by intensity class. In the area of anomaly detection, choosing an appropriate labeling method is crucial to accurately identify and classify anomalies. This paper aims to compare three labeling methods: labeling by NG intensity, labeling by NG type, and labeling by NG type without RoI classification. Through a comprehensive evaluation, we find that the labeling by NG type without RoI classification method outperforms the other methods by achieving the highest F1 score and mAP.

• Labeling for NG Intensity
  Labeling by NG intensity involves labeling anomalies based on the severity or intensity of the anomaly. This method focuses on quantifying the degree of anomaly, but may lack specificity in identifying the exact type of anomaly. It provides important information about the severity of the anomaly, but may not be suitable for accurate classification tasks.
• Labeling for NG Type
  NG labeling by type involves classifying anomalies based on a specific type or class. This method aims to identify the underlying patterns and characteristics of anomalies so that you can better classify and understand different anomaly types. Labeling anomalies based on a specific class enables more accurate anomaly detection and classification.
• Labeling for NG Type without RoI Distinction
  The NG type-specific labeling without RoI classification method focuses on labeling anomalies according to their specific type without considering their RoI or location in the image. This method primarily aims to accurately classify anomalies based on their type without explicitly specifying their spatial location.
• Superior labeling performance by NG type without RoI classification
  After thorough evaluation and analysis, we found that the NG type-specific labeling method without RoI classification achieved the highest F1 Score and mAP compared to other labeling methods. The reasons behind the superior performance are as follows.
• Precision and recall
  This labeling method focuses on accurately classifying anomalies by type, ensuring high precision and recall. Anomalies can be accurately classified regardless of their spatial location.
• Flexibility
  The lack of RoI classification gives you the flexibility to identify anomalies in different areas of the image, and allows the model to capture anomalies that can occur in different areas, giving you a comprehensive understanding of different anomaly types.
• Improved model performance
  The NG type-specific labeling method without RoI classification provides a clear focus on anomaly types, allowing the model to learn specific patterns and features associated with each anomaly class. This enhanced learning contributes to better detection and classification performance, resulting in higher F1 scores and mAPs.
  In anomaly detection, the choice of labeling method has a significant impact on the accuracy and efficiency of the model. After evaluating different labeling methods, we found that labeling by NG type without RoI classification outperforms labeling by NG intensity and labeling by NG type in terms of F1 Score.

In a supervised-learning-based anomaly detection system for defective cosmetic containers, labeling methods play an important role in accurately identifying and classifying anomalies. We used three labeling methods: labeling according to the intensity of an anomaly (NG) by distinguishing regions of interest (RoI) on the top and sides of cosmetic containers, labeling according to the shape, and labeling according to the type of anomaly without distinguishing RoI.

To evaluate the performance of these labeling methods, we prepared a table to compare the unconfirmed and overconfirmed rates for each anomaly type regardless of RoI. The undetected rate represents the rate at which anomalous data was mistaken for normal, indicating that the system failed to detect a defect. On the other hand, the over-check rate represents the percentage of anomalous data that is misclassified as normal, which means that defects are not detected.

As a result, the labeling method by anomaly type without RoI distinction achieved the highest accuracy in supervised-learning-based anomaly detection with the lowest false positive and false negative rates. The false positive and false negative rates for this labeling method are 1.63% false negative and 0% false positive. This means that the system successfully identified the majority of anomalous data as anomalous and correctly recognized normal data as normal. The second best performance was observed with the labeling method for each anomaly type, with a 4.11% false positive rate and a 0.15% false negative rate. Although this method had a slightly higher miss rate than the best performing method, it still performed well in accurately identifying abnormal and normal data. The labeling by anomaly strength (NG) method ranked third with a 12.8% missed rate and 0% over-check rate. Although this method has a higher false positive rate than the other two methods, it scored a perfect overidentification rate, which means it has a low risk of misclassifying abnormal data as normal.

The results of the study suggest that the labeling method for each anomaly without RoI distinction outperformed the other methods in terms of accuracy, achieving the lowest false positive and false negative rates. This labeling approach effectively identifies and classifies anomalies on cosmetic containers, minimizing the likelihood of defective products entering the market. Implementing an accurate and reliable anomaly detection system is crucial to maintaining high quality standards in the manufacturing industry. By adopting a labeling approach by anomaly type without RoI distinction, manufacturers can improve the accuracy and efficiency of their quality inspection processes, resulting in improved overall product quality.

4.5. Discussion

Both [6] and our proposal have in common the use of YOLO to detect objects and supervised learning to detect anomalies. However, ref. [6] utilized YOLOv4 and this paper utilizes YOLOv7. Because YOLOv7’s convolutional neural network architecture and new object detection and classification methods are efficient, YOLOv7 has the advantage of being more accurate and faster than YOLOv4. YOLOv7 is based on the convolutional neural network architecture of YOLOv4 and introduces new object detection and classification methods. YOLOv7 is also trained on a larger dataset than YOLOv4, resulting in more accurate results.

5. Conclusions

While researching and experimenting with anomaly detection using a YOLOv7-based supervised learning approach, we successfully found what we believe to be the optimal labeling method. This method greatly improves the accuracy and efficiency of the anomaly detection process. By utilizing the YOLOv7 framework, which combines deep learning and object detection techniques, we were able to train a robust model that can detect anomalies with high precision and recall. In conclusion, through extensive research and experimentation on anomaly detection using the YOLOv7-based supervised learning approach, we successfully discovered and implemented an optimal labeling method. By utilizing this labeling method, anomaly detection of cosmetic containers can be effectively performed, which improves operational efficiency, reduces downtime, and improves overall quality and performance.

The supervised-learning-based anomaly detection in this paper relies heavily on labeled data, which are expensive and time-consuming to acquire. In addition, supervised learning models are designed to learn from labeled examples, which means that they can only recognize patterns that are explicitly displayed during training, which means that in many domains, they cannot detect the presence of underlying patterns or anomalies that are not known or labeled in advance. Therefore, to address these issues, we seek to compensate for the shortcomings of supervised learning by ensuring that the training data contains a variety of anomaly types, including known and potentially unseen anomalies, so that the model can learn generalized patterns that can be applied to a wider range of anomalies, and by building more comprehensive and diverse datasets and using unsupervised or semi-supervised anomaly detection techniques to complement supervised learning methods. Semi-supervised learning techniques have the advantage of being particularly useful for data exploration and preprocessing tasks. By adopting unsupervised learning as a preliminary step to supervised learning, it can help researchers better understand and preprocess data to improve model performance and gain more reliable insights by identifying outliers, detecting anomalies, reducing data dimensionality, and revealing hidden relationships or structures within the data. It can detect new or previously unseen anomalies based on deviations from normal patterns without relying on labeled data, and can act as a safety net to catch anomalies not considered during the labeling process. We plan to conduct future research not only on supervised learning, but also on unsupervised learning techniques and learning through semi-supervised learning.