Recommendation Method Based on Glycemic Index for Intake Order of Foods Detected by Deep Learning

Abstract

In this paper, we propose a recommendation method for food intake order based on the glycemic index (GI) using deep learning to reduce rapid blood sugar spikes during meals. The foods in a captured image are classified through a food detection network. The GIs for the detected foods are found by matching their names or categories with the information stored in the database. If the detected food name or category is not found in the database, the food information is found from a public API. The food is classified into one of the food categories based on nutrients, and the median GI of the corresponding category is assigned to the food. The food intake order is recommended from the lowest to the highest GI. We implemented a web service that visualizes the food analysis results and the recommended food intake order. In experimental results, the average inference time and accuracy were 57.1 ms and 98.99% for Mask R-CNN, respectively, and 24.4 ms and 91.72% for YOLOv11, respectively.

1. Introduction

Blood sugar, which refers to the amount of glucose in the body, is a critical factor in maintaining health. Blood sugar spikes may strain the body’s ability to regulate glucose effectively [1]. The consumption of high-carbohydrate foods is a factor that easily increases blood sugar spikes [2,3]. Since carbohydrates digest faster than fats and proteins, glucose in carbohydrates is directly absorbed into the bloodstream during digestion. Therefore, frequent consumption of high-carbohydrate meals can lead to repeated blood sugar spikes, and this increases the risk of insulin resistance, inflammation, diabetes, and cardiovascular disease [4,5]. On the other hand, dietary fiber and fat undergo relatively slow and complex digestive processes compared to carbohydrates. Foods including dietary fiber or fat can delay carbohydrate uptake [6]. Therefore, the increase in blood sugar can be delayed by eating foods with dietary fibers or fats before ingesting carbohydrates [7,8,9]. The Glycemic Index (GI) [10,11] quantifies the rate at which a food containing a specific amount of carbohydrates raises blood sugar levels. The blood sugar more rapidly increases as the GI increases. To prevent blood sugar spikes, an optimal food intake order can be determined by referencing GIs.

Previous research has been conducted for food intake recommendations. A study [12] proposed a health management method to suggest personalized dietary plans based on health conditions, medical history, and nutritional needs. An intake recommendation system [13] provided users with a personalized and healthy meal, taking into account both preferences and medical prescriptions. An ontology-based food recommendation system used semantic web technologies to generate personalized meal plans based on health conditions, preferences, and allergies [14]. A nutrition and preference-based system combined collaborative and content-based filtering to recommend meals aligned with health goals [15]. An ontology-based system for elders [16] has been proposed to connect health conditions (e.g., hypertension, diabetes) to medical data for providing personalized nutritional plans. A diabetes management app analyzed real-time blood glucose levels to suggest low-carbohydrate, high-fiber diets for effective health management [17]. However, these previous studies did not provide a food intake order for blood sugar management.

Increases in blood sugar can be managed by adjusting the food intake order. Automated food analysis is possible through food detection based on an image using a neural network. The food intake order can be provided for preventing blood sugar spikes using the food GIs from the analyzed food information. However, the recommendation of food intake order has to take into account certain considerations: new food types are difficult to detect because they have not been included in the network’s training, and for certain food types, GIs may not be found in food databases.

In this paper, we propose a recommendation system for food intake order based on GI using an object detection network. Food types in a given image are detected using the object detection network. The system provides a manual input interface for foods that are not detected by the detection network. After that, the food GIs are found in a GI database. If a GI cannot be found for a certain food, its category is identified based on nutrients through an Application Programming Interface (API) for food nutrition information. Then, the GI is determined as the representative GI of the corresponding category. The system suggests the food intake order based on ascending GIs. The recommended food intake order and other food information are visualized and provided to users through a web application.

The main contributions of the proposed method are as follows:

• Recommendation system for food intake order based on GI: We implemented a recommendation system for food intake order using an object detection model based on GIs. The proposed method can suggest a food intake order to prevent blood sugar spikes.
• GI Estimation for foods that are not found in databases: We proposed a GI estimation for foods whose types or GIs are not found in the GI database. The undetected foods are classified into the one of the predefined categories based on major nutrients, and then the corresponding GIs are determined by representative values of their categories.
• Visualization of food information analysis: A user-friendly web application was designed to visualize food information including food types, GIs, and a recommended food intake order. Users can conveniently check this information by uploading a food image.

2. Related Work

2.1. Food Detection and Classification Based on Deep Learning

Food detection using object detection has been noticeably improved across various environments with the introduction of deep learning technology. Food detection methods based on deep learning have been studied as follows.

Suddul et al. [18] conducted a food detection study using EfficientNetV2 [19]. They applied data augmentation techniques such as rotation, shifting, shearing, zooming, and flipping to address the issue of the imbalanced dataset.

Zhang et al. [20] proposed a CNN-based deep learning model to recognize food types and to assess quality. They generated new samples by mixing two existing images and labels with weighted combinations to increase the dataset size. Additionally, they applied transfer learning and semi-supervised learning to enhance model performance.

Pan et al. [21] proposed a deep learning framework for multi-class classification of food ingredients. They optimized a CNN-based model using transfer learning. This approach significantly enhanced the accuracy of multi-class classification.

Kiourt et al. [22] conducted an analysis of deep learning approaches for food recognition. The authors developed a new CNN architecture called PureFoodNet inspired by the VGG structure. This model introduces an optimization technique using Nesterov momentum to enable fast and stable training.

Shen et al. [23] developed a system for recognizing food images and estimating nutritional information using a pre-trained CNN model. They proposed a method to analyze food calories, attributes, and ingredients using a pre-trained CNN model. Word2Vec was adopted to process crawled data from texts that provide information on calories, attributes, and ingredients. This system with a client-server architecture automatically analyzes upload images to estimate the nutritional information.

Freitass et al. [24] conducted a study on supporting food monitoring and diet management through a CNN model in a mobile application. This system enables users to easily grasp the nutritional information of daily food intake. The researchers compared three CNN architectures—Small Xception [25], MiniVGGNet [26], and SimpleNet [27]—and integrated the most efficient model into the system. Additionally, they expanded the training dataset through data preprocessing and augmentation techniques and evaluated each model’s performance using cross-validation. In the validation results, Small Xception achieved the highest accuracy. Small Xception was integrated into a mobile application to automatically log and monitor nutritional information about food intake.

Kim et al. [28] proposed a method for classifying food types and estimating food intake by comparing pre- and post-meal images using the deep learning object detection network Mask R-CNN [29]. The authors adopted Mask R-CNN to detect food types and food regions. Homography transformation [30] was applied to align the meal plate regions between the two images. Each 3D food shape was determined as one of three forms—spherical, conical, or cuboid—to calculate a food volume through the corresponding geometric formula.

Although the previous studies for food detection and classification have achieved significant advancements in accuracy and efficiency, most of them do not focus on food information related to human health such as GI. In addition, these studies have difficulty handling untrained or rarely encountered food types. Therefore, it is difficult to effectively utilize the previous methods to provide food information.

2.2. Food Detection Models

Mask R-CNN [29] is an extension of Faster R-CNN [31] that performs pixel-level segmentation of objects in addition to object detection. While Faster R-CNN focuses on predicting bounding boxes, Mask R-CNN is designed to also predict segmentation masks that include precise object boundaries. Mask R-CNN adds a mask prediction branch to the Faster R-CNN structure, enabling simultaneous prediction of the bounding box, class, and mask for each object. Additionally, it replaces Region of Interest (RoI) Pooling with RoIAlign to improve coordinate precision, thereby enhancing the accuracy of mask predictions.

YOLO [32] has been developed to address the issues of high computational complexity and unsuitability for real-time processing in R-CNN models, which use complex pipelines for object detection. It improves computational efficiency by processing the input image only once through a CNN network to simultaneously predict the location and class of objects. By dividing the image into square grids, each grid cell predicts the bounding boxes and class probabilities for objects.

3. Proposed Method

3.1. Food Detection Through Object Detection Model

The proposed method detects food items in an image through Mask R-CNN [29] or YOLOv11 [33]. Mask R-CNN detects objects through two stages for region proposal and object detection, so this model can achieve precise and detailed food identification. On the other hand, YOLOv11 integrates region proposal and object detection into a single network to detect objects faster than Mask R-CNN.

Figure 1 shows the network architecture of Mask R-CNN for food detection. Mask R-CNN consists of a backbone network, a region proposal network (RPN), a RoIAlign module, and heads. The backbone network consists of a ResNet and a feature pyramid network (FPN). The ResNet converts input images into feature maps with high-level semantic information. The FPN generates a hierarchical pyramid of feature maps at varying resolutions by processing the feature maps across multiple scales for robust object detection irrespective of object sizes. The RPN identifies RoIs within the input image that likely contain objects. The RoIAlign module normalizes the sizes of the RoIs to align feature maps. The heads process the feature map corresponding to the RoI for detecting bounding boxes, classifying object types, and segmenting object regions.

Figure 2 illustrates the YOLOv11 network structure. YOLOv11 consists of a backbone network, a neck network, and heads. The backbone network extracts feature maps in the same way as Mask R-CNN. The YOLOv11 backbone has convolution layers, a C3K2 module, and a spatial pyramid pooling-fast (SPPF) module. The C3K2 module enhances computational efficiency by optimizing gradient flow between layers and by generating intermediate feature maps with rich semantic information. The SPPF module integrates features from multiple scales to preserve both global context and fine-grained details. The neck network improves the representation of the feature maps to improve the accuracy of object detection. The resolution of the feature maps is upsampled to improve detection for tiny objects. Then, the feature maps with different resolutions are concatenated to effectively merge detailed local features and broader contextual information. The C2PSA module attends to image regions likely to contain objects. The head processes the enhanced feature maps from the neck network to detect objects. The C3K2 module refines the feature maps and the CBS blocks stabilize data flow through batch normalization and enhance non-linear transformations using a sigmoid linear unit (SiLU) [34]. YOLOv11 outputs object detection results in the form of bounding boxes, object types, and object regions.

3.2. Food Intake Order Recommendation Based on GI

High GI foods cause rapid and pronounced spikes in blood sugar levels in a short time, leading to sharp fluctuations that can affect energy levels and metabolic stability. In contrast, low GI foods raise blood sugar more slowly and gradually. Therefore, low GI foods can provide a more sustained energy release and reduce the risk of blood sugar variability [35]. In other words, consuming low GI foods first can more effectively manage the rise in blood sugar. Blood sugar spikes can be prevented by consuming fiber-rich vegetables first [9]. Therefore, the GI is utilized as a criterion to determine the food intake order to prevent blood sugar spikes. After detecting foods through the food detection model, the GIs corresponding to the foods are found in the public food database [36]. Then, the food intake order is presented in ascending order based on their food GIs. The suggested food intake order by the proposed method can help delay the rise in blood sugar for preventing blood sugar spikes.

GI Estimation Based on Food Category

It is impractical to train the model on whole food types due to limitations in food data collection. Therefore, the food detection model may sometimes not detect some foods which were not included in the training. For addressing this issue, a food detection model is additionally introduced using a single common label for all food types. The model using the common label detects foods through common features of foods. Therefore, it can accurately detect foods whose types are not trained. Figure 3a,b show food detection results through the detection models trained with the multi-class labels for food types and the common label, respectively. Whereas the model trained with multi-class labels detects only rice, the model trained with the common label detects all food objects.

The GI database does not contain the GIs of all foods. Therefore, it may be difficult to determine an accurate GI for certain foods in the image. To address this issue, the missing GI is estimated by referencing the corresponding food category. The foods are classified into dietary fiber, protein, fat, carbohydrate, and fructose. Although dietary fibers and fructose are found in small amounts in foods, they significantly impact blood sugar increases. Therefore, the food classes of vegetables and fruits are categorized as dietary fiber and fructose, respectively, based on the nutrients they contain. For the others, the category is determined as one of protein, fat, and carbohydrate based on their primary nutrients. Figure 4 shows food samples for the categories and

Table 1. Examples of food category determination.

Food Type	Food Class	Food Nutrient Information (g per 100 g of Food)					Food Category
		Dietary Fiber	Protein	Fat	Carbohydrates	Fructose
Carrot	vegetables	2.60	1.03	0.18	4.26	1.63	Dietary fiber
Pine nuts	nuts	4.10	15.96	59.28	19.38	0.00	Fat
Soybeans	legumes	10.50	17.99	6.79	12.71	0.00	Protein
Rice	grains	0.00	2.40	0.00	35.70	0.00	Carbohydrates
Peach	fruits	0.59	0.04	0.00	13.10	3.96	Fructose

presents the determination of food categories for some foods in the database. The median GIs for the food categories are obtained from the foods in the GI database.

Table 2. Food category information in GI database.

Food Category	Number of Food	Minimum GI	Maximum GI	Median GI
Dietary fiber	9	14	39	23
Protein	15	24	53	37
Fat	8	13	27	24
Carbohydrates	169	23	96	57
Fructose	55	22	80	50

presents the number of foods, minimum GI, maximum GI, and median GI for the food categories. When the GI of a certain food is not found in the database, the corresponding category is determined by referencing its food nutrients and class, which are searched through a food nutrition information API (Application Programming Interface) [36]. Subsequently, the GI is determined as the median value of the corresponding category.

The foods in the image are sometimes not found through the food detection model. The food region may be detected through the model with the common label but its type may not be detected. In addition, the type of food may be identified, but its corresponding GI may not be found in the database. For universal application, the recommendation system must be able to address both scenarios. In other words, the system must include functions to estimate the GI of unknown foods. Figure 5 shows two flows for addressing these scenarios. The first scenario is that both detection models, which are trained with multi-labels and the common label, respectively, detect the food objects, but GIs of some foods are not found in the database as shown in Figure 5a. The GIs are determined by searching food information through the food nutrition information API. Another scenario is in the case of detecting certain foods only by the model trained with the common label as shown in Figure 5b. In this scenario, these food names are inputted manually. The GIs are found in the database or through the food nutrition information API. After obtaining the food nutrition information, the foods are classified into the food categories, then the median GI of the corresponding category is assigned to the food for suggesting the food intake order recommendation. For foods within the same category, the intake order is determined based on their carbohydrate content, ranked from lowest to highest.

The difference between the actual and the estimated GIs is evaluated as shown in

Table 3. Comparison between actual and estimated GIs.

Food Name	Actual GI	Estimated GI
Grape	43	50
Oats	57	57
Tomato	31	23
Cashew nuts	27	24

. The estimated GIs are generally close to the corresponding actual GIs.

3.3. Web Application

A web application is implemented based on the proposed recommendation method for food intake order to intuitively view results on various devices. The web application is implemented using the Python web framework to ensure compatibility with the object detection model. Figure 6 shows the flow of the recommendation of food intake order through the web application.

Figure 7 provides image upload and preview interfaces. These interfaces are implemented using HTML5, CSS3, and JavaScript. When the user interacts with the ‘Select Photo’ button, a stored food image can be selected or a new food image can be captured. The food image is previewed and sent to the server when the user clicks the ‘Submit’ button. The web application is designed to optimize the mobile user experience, featuring CSS transitions for smooth visual feedback during interactions.

After the food image is sent to the server, the object detection model detects the food objects and identifies the food names and bounding boxes. Subsequently, these GIs are obtained from the GI database or estimated based on the corresponding food categories. The food information for the food names, the GIs, and the bounding boxes are organized in JSON format as shown in Figure 8. The user-uploaded image is immediately removed from the server after analyzing the food information. Figure 9 shows the data flow between the server and client for food analysis. After that, the food information is sent to the client. The client generates a HTML page using HTML5 and CSS3 that visualizes the food types, GIs and the recommended food intake order, as shown in Figure 10.

4. Experiment

4.1. Dataset

In order to create datasets for experiments, we manually captured 523 Korean meal images over a certain period while also collecting the corresponding food types. The bounding boxes of the foods in the captured image were manually labeled using LabelMe [37]. Figure 11 shows the samples of the meal images. To increase the amount of food images and balance the distribution of food items across categories, we also utilized 1150 Korean food images provided by an open database (AI-HUB) [38], including labels with bounding boxes and food types. Of the collected images, 1339 images were utilized as the training set, 167 images were designated as the validation set for cross-validation, and the remaining 167 images were used as the test set. Furthermore, data augmentation was applied to prevent overfitting, increasing the size of the training set to 5356 images.

Table 4. Food category specification for experiment.

Food Category	Number of Food
	Before Data Augmentation			After Data Augmentation
	In Captured Images	From Open Database	Total
Dietary fiber	1058	200	1258	5032
Protein	592	200	792	3168
Fat	29	300	329	1316
Carbohydrates	481	250	731	2924
Fructose	34	200	234	936
Total	2194	1150	3344	13,376

presents the food item information for the experiments.

4.2. Performance Comparison Evaluation of Mask R-CNN and YOLOv11

The performances for inference time, accuracy, and IOU were evaluated using the test set with 167 images. SGD [39] and AdamW [40] were used as optimizers for Mask R-CNN and YOLOv11. The batch size and initial learning rate were 4 and 0.001 for Mask R-CNN, respectively, and 8 and 0.001 for YOLOv11, respectively. The models were trained for 400 epochs.

Table 5. Comparison of food detection performance between Mask R-CNN and YOLOv11.

Model	Detection Time (ms)	Accuracy	IoU	Precision	Recall	F1-Score
Mask R-CNN	57.1 ± 14.6	98.99%	93.88%	98.20%	97.56%	97.88%
YOLOv11	24.4 ± 17.5	91.72%	93.37%	93.95%	91.04%	92.47%

shows the model performances for Mask R-CNN and YOLOv11. YOLOv11 was twice as fast as Mask R-CNN for the food detection. In Table 5, ±symbols indicate the ranges of the detection times. The differences in IOU and accuracy were approximately 0.5% and 7%, respectively. In terms of object detection performance, YOLOv11 demonstrated slight improvement over Mask R-CNN. Figure 12 and Figure 13 show the food detection results including bounding boxes, food types, and detection scores by Mask R-CNN and YOLOv11, respectively.

Data augmentation is applied to model training to increase the number of the images. Data augmentation effectively increases the amount of data by applying transformations to images without distorting the original data [41]. This leverages the principle that the performance of object detection through deep learning is proportional to the volume of the training dataset. Image transformations are probabilistically applied for the data augmentation: changes in hue, saturation, and brightness; horizontal flipping; image rotation; image scaling; and perspective transformations.

Table 6. Performance improvement through data augmentation.

Data Augmentation	Accuracy	IoU	Precision	Recall	F1 Score
No	85.94%	86.43%	87.58%	86.16%	86.16%
Yes	91.72%	93.37%	93.95%	91.04%	92.47%

shows improvements in the food detection performance through data augmentation.

5. Discussion

5.1. Detection Results for Similar Foods

Even for visually similar foods, the food detection model can accurately differentiate between them because it extracts the main features for identifying objects through multiple layers. Figure 14 shows the detection results for foods with similar red tones.

5.2. Selection of Food Detection Model

Since YOLOv11 is faster than Mask R-CNN in food detection, YOLOv11 is more suitable for real-time applications. Consequently, YOLOv11 was adopted as the primary model for the proposed system. However, Mask R-CNN can be utilized as a secondary model to detect food objects that are not detected by YOLOv11. Figure 15 shows an example demonstrating the improvement in food detection using Mask R-CNN. All foods are detected by YOLOv11 trained with the common label, but some food types cannot be identified by YOLOv11 trained with multi-class labels. Mask R-CNN detects these undetected foods.

5.3. Comparison with Previous Methods

We implemented a system that detected foods using the food detection model and provided a food intake order based on the GIs to prevent blood sugar spikes. The improvements of the proposed method compared with the previous methods are as follows.

• Recommendation system for food intake order based on GI: The proposed method can accurately identify foods in a given image and find their corresponding GIs, which are indicators of the amount of carbohydrates in foods. The food intake order is presented based on GI, according to a theory that suggests consuming low carbohydrate foods can delay blood sugar increases. Therefore, the proposed method can help prevent blood sugar spikes.
• GI estimation for foods that are not found in database: It is necessary to estimate food GIs when they are missing from the database. The proposed method searches the nutritional information of their corresponding foods using the public food information API. Subsequently, the categories are determined based on their main nutrition, and the representative GIs of these categories are assigned to the missing foods.

  Table 7. Intake order recommendation results for foods with unknown GIs.

  Food Type	Food Class	Food Nutrient Information (g per 100 g of Food)					Food Category	Estimated GI	Intake Order
  		Dietary Fiber	Protein	Fat	Carbohydrates	Fructose
  Pork cutlet	meat	0.0	22.73	8.18	18.18	0.00	Protein	37	3
  Pickle	vegetable	2.1	0.48	0.48	3.56	10.78	Dietary fiber	23	1
  Rice	grains	0.9	2.50	0.20	46.20	0.00	Carbohydrates	57	4
  Miso	grains	0.4	12.5	6.25	50.00	0.23	Carbohydrates	57	5
  Salad	vegetable	2.7	7.95	3.51	5.30	2.81	Dietary fiber	23	2
  Sweet potato	grains	1.5	0.80	6.80	54.00	1.50	Carbohydrates	57	6

  shows the intake order recommendation results for foods that are not in the GI database in Figure 16. The recommended intake order based on the GI estimation is as follows: pickles, salad, pork cutlet, rice, miso, and sweet potato. This food intake order appears to effectively suppress blood sugar spikes to some extent because it aligns with the order of foods with lower carbohydrate content.
• Visualization of food information analysis: We implemented a user-friendly web application that provides comprehensive food information, including food names, GIs, and food intake order, all presented in an intuitive and easy-to-understand format.

6. Conclusions

We proposed a recommendation method based on GI for the intake order of foods detected by a deep learning model. YOLOv11 or Mask R-CNN was employed to detect foods in the images. A web-based application was implemented to provide the immediate visualization of recommendations for food intake order and food information across various devices. The user-uploaded food image was sent to the server. The food types was detected using the food detection model. For foods that were not identified, the user could manually enter their names. After that, the corresponding food GIs were found in the database. If a food GI was not found in the GI database, it was assigned as the representative value of its category, which was determined based on its nutrients searched through the public food information API. In the experimental results, the food detection accuracies were 98.99% and 91.72% for Mask R-CNN and YOLOv11, respectively. The proposed system allows users to obtain the optimal food intake order to prevent blood sugar spikes. This method is expected to guide individuals toward healthier eating practices.