A Human Body Simulation Using Semantic Segmentation and Image-Based Reconstruction Techniques for Personalized Healthcare

Abstract

The global healthcare market is expanding, with a particular focus on personalized care for individuals who are unable to leave their homes due to the COVID-19 pandemic. However, the implementation of personalized care is challenging due to the need for additional devices, such as smartwatches and wearable trackers. This study aims to develop a human body simulation that predicts and visualizes an individual’s 3D body changes based on 2D images taken by a portable device. The simulation proposed in this study uses semantic segmentation and image-based reconstruction techniques to preprocess 2D images and construct 3D body models. It also considers the user’s exercise plan to enable the visualization of 3D body changes. The proposed simulation was developed based on human-in-the-loop experimental results and literature data. The experiment shows that there is no statistical difference between the simulated body and actual anthropometric measurement with a p-value of 0.3483 in the paired t-test. The proposed simulation provides an accurate and efficient estimation of the human body in a 3D environment, without the need for expensive equipment such as a 3D scanner or scanning uniform, unlike the existing anthropometry approach. This can promote preventive treatment for individuals who lack access to healthcare.

1. Introduction

Personalized healthcare has received worldwide attention due to technical improvement [1]. Both special groups (e.g., company or research center) and ordinary individuals are able to access wearable devices involving real-time monitoring sensors [2,3,4]. To be more specific, a smart watch (e.g., Apple watch) can be used to track the irregular pulse of a person, so that potential heart anomalies can be detected in real time [5]. In addition, a wearable tracker, such as a Fitbit, can be used to track the physical activity of an individual [3]. Given that most OECD countries have become aging societies and their lifestyle is rapidly changing towards single households [6,7], the growth of the personalized healthcare market is not in question.

There are two types of personalized healthcare in terms of point in time of treatment. Wearable devices used to monitor anomalies in health can be considered as proactive treatment. This is because real-time information can mitigate the reaction time of a first responder [8]. According to the personal health devices (PHDs) standard [9], proactive treatment can be illustrated as activities that allow people to conduct self-monitoring of health conditions and to share that information with healthcare professionals and other caregivers. Proactive treatment consists of two major types of devices: detecting anomalies using users’ health data, and curing discomfort. An example of the former is the Advanced care and alert portable telemedical MONitor (AMON) [10], which continuously collects vital signs and alerts the medical center of anomalies; an example of the latter is the Quell, by Neuro Metrix Inc., which relieves chronic leg pain via transcutaneous electrical nerve stimulation [11].

On the other hand, preventive treatment is associated with maintaining the individual’s health [12]. The most popular approaches in this category are home training and dietary therapy [13]. In fact, under the COVID-19 pandemic, home training and dietary therapy are becoming more common and popular in many countries, including the U.S., Germany, France, Japan, China, and South Korea [14]. According to Fortune, the market size of home fitness equipment in the U.S. in 2020 was about USD 3.55 billion, and by 2028, is expected to grow to USD 14.74 billion [15]. Similarly, the market size of dietary supplements in 2020 was about USD 61.20 billion, and by 2028 is expected to be USD 128.64 billion. Generally speaking, this is known as a more effective approach in terms of cost and fatality compared with proactive treatment [16].

In preventive treatment, it is critical to understand the correlation between anthropometric variables and diseases, such as diabetes, hypertension, dyslipidemia, and coronary artery disease [17]. Once the correlation is analyzed, it can be used to build a prediction model of human diseases. This implies that diseases can be prevented by changing anthropometric variables in terms of decreasing waist circumference, waist–hip ratio, and body mass index (BMI). Additionally, a number of studies for predicting disease were conducted. Sung et al. [18] proposed cardiovascular disease (CVD) prediction models. The prediction model was based on the deep learning method (RNN−LSTM), which is widely used to analyze time-series datasets. This technique improved the prediction accuracy compared to conventional Cox proportional hazard regression models [19]. In addition, various companies have participated in developing and commercializing services that can diagnose disease and health abnormalities by analyzing users’ health record data; e.g., Apple received FDA approval for a deep learning algorithm that detects atrial fibrillation using their smartwatch [4]. Even though recent technologies in preventive treatment, which are combined with artificial intelligence (AI), have been developing rapidly, additional research is needed to provide a result that can represent a cross-section of the population, and the privacy and security of the data used should be supplemented, which can reveal the users’ medical history details [20].

This study aims to propose a human body simulation that predicts and visualizes the individual’s 3D body change based on 2D images taken by a portable device (e.g., a smart phone). Notice that 3D body models provide more accurate volume information and body shape than conventional 2D body images, so they can be used to precisely calculate health indicators such as BMI. The main part of the proposed simulation consists of two major subprocesses in the anthropometric measurement module: (1) A semantic segmentation process [21,22,23], which segments the human image from 2D imagery taken by a smart phone, and (2) an image-based reconstruction process [24,25], which constructs the 3D body model. Experiments were conducted to obtain optimal conditions in the aforementioned subprocesses to solve limitations, such as the expensive equipment requirement in 3D scanner-based anthropometry and direct anthropometry, while deriving reliable anthropometric variables. According to the experiments, the p-value of the paired t-test conducted between the measurement given by the proposed simulation and the actual measurement is 0.3483, so the proposed simulation can accurately estimate the human body in the 3D environment (see Section 4 for more detail). As a result, the proposed simulation enables not only the reconstruction of an accurate 3D body model from 2D images, but also the visualization of bodily changes in accordance with a prescribed exercise plan through the 3D body model. This will contribute to eliminating inequalities for people who are not provided with expensive healthcare services.

The contributions of this study are as follows. This study presents a specific method for preprocessing anthropometric measurements to better utilize the existing 3D body modeling techniques, and designs a selection module for precise 3D body part implementation from 2D images. In addition, it is significant that a module for predicting body changes according to a given exercise plan is developed and implemented as a single simulation program.

The organization of the paper as follows: Section 2 briefly introduces existing studies associated with 3D modeling of the human body and image segmentation, while Section 3 introduces a three-phase simulation-based modeling approach for the visualization and prediction of a 3D human body. Section 4 demonstrates the proposed simulation with experimental analysis and test results, while Section 5 concludes the paper and indicates future work associated with this study.

2. Three-Dimensional Modeling of the Human Body

As the demand for 3D human body modeling increases in multiple industries (e.g., clothing, healthcare, gaming, and industrial design), researchers are intensively conducting research in the field of human body shape modeling (HBSM). According to Baek and Lee [26], the HBSM techniques can be categorized as follows: (1) Direct model creation/acquisition modeling of the 3D human body via depth-related information or a map of per-pixel data, e.g., RGB−D data; (2) template-based model scaling, which modifies template models to elicit the desired 3D human body model; (3) statistical-based model synthesis, deforming pose and shape by learning datasets; and (4) image-based reconstruction, extracting parameters that elicit the 3D human body model from 2D images.

The direct model creation/acquisition technique is the modeling of a 3D model via surface spatial data obtained by a 3D scanner, which is based on photogrammetry, structured light, and laser scanning techniques. In photogrammetry, multiple lenses are used to obtain the spatial data of an object. Once two charge-coupled devices (CCDs) located at a parallax angle take two different object images, the technique calculates the distance between a base plane and points on the object by using their geometrical relationship [27]. The laser scanning technique with triangulation [28] and the structured light-based technique using contour mapping (e.g., moiré topography) are also widely used techniques due to their modeling accuracy and cost efficiency [29].

Template-based model scaling generates a 3D model by scaling a body part categorized by the deformation of applied types: (1) vertical deformation part, such as waist height, and inside leg length, (2) joint girth (circumference) deformation, such as knee girth and wrist girth, and (3) girth deformation between two joints or regions, such as thigh girth and waist girth [30]. The major advantage of template-based model scaling compared to the other techniques is its effectiveness, because it generates a body model from a template, while expensive equipment (e.g., a 3D scanner) is unnecessary.

Statistical-based model synthesis was devised to overcome the inefficiency of the direct model creation/acquisition technique in terms of cost and labor. Instead of observing and measuring all body parts every time, this technique models a homogeneous face model database with principal component analysis (PCA) in analyzing shape variation [31]. Once the database is constructed, the dominant variables that determine the 3D model face are analyzed and extracted. By generating these variables, the objective 3D body is modeled. In this technique, Shape Completion and Animation of PEople (SCAPE) and Skinned Multi-Person Linear (SMPL) techniques are widely used to understand shape variations under different body shapes [32,33].

Image-based reconstruction focuses on modeling a 3D body model from an existing 2D image. Since it does not require the numerous image collection efforts necessary for the other three techniques, it is known as the most cost-effective technique. One of the popular models is DeepCut [34], using a convolutional neural network (CNN) to reconstruct the posture of a human body via partitioning and the detection of body parts. Bogo et al. [35] proposed an advanced framework called SMPLify based on CNN and SMPL. In this technique, SMPLify identifies 2D body joint locations using DeepCut, and the derived 2D body joints are used to calibrate the pose ($\theta$) and shape ($\beta$) parameters of the SMPL model. Furthermore, to fit the pose and shape to 2D body joints, a minimizing objective function is used that represents the sum of joint-based data terms, $E_j$($\beta ,\theta ;K,J_{est}$), pose priors, $E_a\left(\theta \right)$ and $E_{\theta }\left(\theta \right)$, a shape prior, $E_{\beta }\left(\beta \right)$, and a penalty term, $E_{sp}\left(\theta ;\beta \right)$. Equation (1) represents the objective function of SMPLify, which implies camera parameter $\left(K\right)$, scalar weights $\left({\lambda }_{\theta },{\lambda }_a,{\lambda }_{sp},{\lambda }_{\beta }\right)$, and the 2D distance between estimated joints ${(J}_{est})$:

$${E\left(\beta ,\theta \right)=E}_j\left(\beta ,\theta ;K,J_{est}\right)+{\lambda }_{\theta }{E}_{\theta }\left(\theta \right)+{\lambda }_a{E}_a\left(\theta \right)+{\lambda }_{sp}{E}_{sp}\left(\theta ;\beta \right)+{\lambda }_{\beta }{E}_{\beta }\left(\beta \right)$$

(1)

However, in terms of estimating the human body shape, the SMPLify framework only uses the connection length between two joints to fit the 3D model, so that the derived result can have insufficient accuracy. Thus, Lassner et al. [36] added an image silhouette (S) and a model silhouette $\left(\widehat{S}\right)$ to the SMPLify framework. Thus, Equation (1) is modified with the bi-directional distance between $S$ and $\widehat{S}$, which is shown in Equation (2):

$$E_S\left(\overrightarrow{\theta },\overrightarrow{\beta },\overrightarrow{\gamma };S,K\right)={\sum }_{\overrightarrow{x}\in \widehat{S}\left(\overrightarrow{\theta },\overrightarrow{\beta },\overrightarrow{\gamma }\right)}dist({\overrightarrow{x},S)}^2+{\sum }_{\overrightarrow{x}\in S}dist({\overrightarrow{x},\widehat{S}(\overrightarrow{\theta },\overrightarrow{\beta },\overrightarrow{\gamma }))}^2$$

(2)

In contrast to the other techniques, the framework called HMR [37] has advantages in terms of training process and output results. First, the HMR framework directly derives 3D mesh parameters from image pixels, while the aforementioned frameworks only estimate 2D joint locations and 3D model parameters. Next, compared to outputting only the 3D skeleton, it is more appropriate in terms of applied shapes, as well as poses for outputting 3D meshes. To make this possible, the HMR framework consists of two steps: (1) the image is encoded through the CNN process; then, (2) the encoded latent vector $\Theta =\{\theta :pose,\beta :shape,R,t,s:camera\}$ is sent to the 3D regression module to learn to represent the 3D human shape. At this time, for the 3D shapes representing the vector, the latent vector $\Theta$ is inferred by minimizing the re-projection error.

3. Human Body Simulation Using Semantic Segmentation and Image-Based Reconstruction Techniques

This study aims to introduce the human body simulation using the HBSM technique mentioned in Section 2. In particular, the image-based reconstruction technique is utilized because of its temporal efficiency in constructing the 3D body model, as well as smoothness in the simulator-driven environment. The image-based reconstruction technique, which is known as the most cost-effective technique, focuses on building a 3D body model from an existing 2D image, without requiring numerous image collection efforts. Figure 1 shows an overview of the proposed simulation that consists of three modules: (1) anthropometric measurement, (2) model selection, and (3) prediction adjustment (see Figure 1).

Once the user inputs their front picture and background picture via a camera in their smart phone, the image-based reconstruction is conducted via the framework shown in Figure 1. From the given input image data, the framework measures anthropometric variables and compares relevant body models by calculating differences to demonstrate the closest and realistic human body change under different exercise plans.

In Figure 1, the anthropometric measurement module extracts anthropometric variables via two processes termed image segmentation and 3D body shaping. In particular, to reduce noise in the input image of 3D body shaping, the DeepLabv3 pretrained model is used to segment only the subject’s body image from the original input image. Next, the HMR framework (i.e., a selected image-based reconstruction method) is used to produce the 3D human body shape model. The measured anthropometric variables via ten body parts—i.e., circumferences of the waist (natural indention and omphalion), bust, wrist, neck, thigh, and hip; arm length; and biacromial breadth—are delivered to the model selection module, which identifies the most similar 3D model by calculating the SAD between the extracted anthropometric variables and the data of the anthropometry database (ADB) with historical anthropometric variables and its 3D model file. The ADB database is referenced in the fifth and sixth Size Korea Surveys. The surveys were conducted on a nationally representative sample of 29,592 individuals, with direct measurements taken from 6016 individuals in three-dimensional shapes for each city and county in the country. Direct measurements were taken on individuals aged 7 to 90 years old, stratified by sex. In direct measurement using a human measuring instrument, 139 human dimensions and 3D shape measurements using a full-body scanner were taken. These measurements were taken on 177 human dimensions (body: 73, foot: 19, head: 45, hand: 19, direct measurement: 21). The prediction adjustment module estimates the variation in anthropometries over time, which is affected by the user’s exercise plan. The changed anthropometric variables are then used to select the adjusted 3D model.

3.1. Anthropometric Measurement

This module is devised to enhance time efficiency by resolving the limitations of traditional direct measurement approaches, such as Martin’s anthropometric technique [38] and the measurement techniques using the 3D scanners mentioned in Section 2, while maintaining measurement accuracy. Figure 2 represents the process of the anthropometric measurement module.

Initially, two input images (i.e., front body image and background image) are obtained from a smart phone camera to generate a 3D model that is more similar to the actual subject body in the 3D body shaping process, and to measure the anthropometric variables accurately. To make this possible, the image segmentation process is conducted. In particular, considering the objective of the module to measure the subject’s anthropometric variables from images, the semantic segmentation method is selected, since, in semantic segmentation, only the important parts are extracted when the spatial resolution is reduced by the pooling operation, so that detailed information about the original image is disregarded. This result can cause difficulty in the classification of the outer pixels of an object (e.g., to classify human pixels from natural images, the outer pixels of a human can be classified as background class). To resolve this problem, DeepLabv3 [39] is adopted in the proposed module. The model design of DeepLabv3 involves several critical components: (1) Atrous convolution, which allows the model to capture multi-scale contextual information without losing resolution, crucial for accurately segmenting the human body in images; (2) Xception Architecture [40], as the backbone network, providing a deep and efficient feature extraction mechanism by leveraging depthwise separable convolutions to improve performance and speed; and (3) Down-Sampling strategies to address the challenge of resolution loss, helping to retain important spatial information and ensuring that the segmented output preserves the details necessary for accurate anthropometric measurements.

The processing flow of the model involves several steps. Initially, the input images are preprocessed to normalize and resize them to a consistent resolution. Next, feature extraction is performed using the Xception backbone to capture detailed representations of the input images. Atrous Spatial Pyramid Pooling (ASPP) is then applied, which uses Atrous convolutions at multiple rates to capture context at different scales; and finally, the decoder refines the segmentation map, producing a high-resolution output. There are specific constraints on the input images to ensure accurate segmentation: for the front image, the subject should be clearly visible, with minimal occlusion and appropriate lighting; for the background image, a plain or simple background is preferable to reduce the complexity of the segmentation task. However, DeepLabv3 is robust enough to handle varying background complexities to a reasonable extent. Background complexity can affect segmentation, but the robustness of DeepLabv3, trained on diverse datasets like MS-COCO [41] and PASCAL VOC [42], helps mitigate these effects. The model’s ability to generalize across different backgrounds ensures reliable performance in varied real-world scenarios.

After the human segmentation process, the 3D body-shaping process, which uses the segmented subject image, is conducted. This process uses the aforementioned HMR framework to construct 3D models that are based on SMPL model fitting to segment the subject image.

Lastly, using vertices in the constructed 3D model area to be measured (e.g., length between vertices of the crown of the head and sole are used in measuring stature, and perimeter of vertices on waist are used in measuring waist circumferences), the stature is set to the reference value, and other anthropometric variables (i.e., circumference of the waist (natural indention and omphalion), bust, wrist, neck, thigh, and hip; arm length; and biacromial breadth) are inferred from their proportional relationship to stature.

3.2. Model Selection

This module visualizes the 3D model from the most similar model to the subject’s body using the ADB with anthropometric variables that are given by the anthropometric measurement module.

As described in Figure 3, eleven anthropometric variables (i.e., stature; circumferences of natural indentation, omphalion, bust, wrist, neck, thigh, and hip; arm length; and biacromial breadth) derived from the anthropometric measurement module, gender, and age of subject are used as inputs of this module.

A body type of the subject’s age and gender group (e.g., male in his 40s) is provided (e.g., male in his 40s reverse triangular (40 rtm) and average (40 avgm)). The comparison stage is conducted to identify the best 3D representation with the minimum sum of absolute difference (MINSAD) between the anthropometric variables of the subject’s body parts and the stored anthropometric variables in ADB (see

Table 1. Sample anthropometric data of subjects.

3D Model File (.obj)	30e_3dm	54e_3dm	57e_3dm	107e_3dm	…	62e_3dm
Stature (cm)	170	172	170	168	…	182
Waist circumference (Natural indentation; cm)	84	85	85	89	…	93
Waist circumference (Omphalion; cm)	82	86	84	90	…	93
Bust circumference (cm)	101	93	95	100	…	106
Wrist circumference (cm)	18	17	17	18	…	18
Neck circumference (cm)	37	38	41	39	…	42
Arm length (cm)	56	57	56	54	…	61
Thigh circumference (cm)	55	55	53	62	…	59
Biacromial breadth (cm)	45	45	47	43	…	45
Hip circumference (cm)	95	92	91	97	…	97
Upper arm circumference (cm)	38	42	35	29	…	31
Body type	40avgm	40avgm	40avgm	40avgm	…	40avgm

for more detail). Equation (3) is used in this process, where i is a row index in ADB, and j is a column index in ABD. In the equation, $X_{i,j}$ is the $j$th body part in the $i$th row in ADB, and $Y_j$ is the $j$th body part of the subject’s anthropometric data. This module identifies an appropriate 3D model by computing MINSAD for all relevant body types in ADB.

$$\underset{i}{\mathit{argmin}}\left(SA{D}_i={\sum }_{j=0}^{10}\left|X_{i,j}-Y_j\right|\right)$$

(3)

3.3. Prediction Adjustment

Figure 4 shows a pseudo code of the prediction adjustment module. Once anthropometric variable set (A) from the model selection module is loaded, the exercise effect is chosen under the selected exercise plan (e.g., walking, and leg cycling 60 min, 6 days a week in 12 weeks) in

Table 2. Exercise Effect Database (EEDB).

Exercise Plan	Plan A		Plan B			Plan C				Plan D		Plan E
Exercise code *	a, b, c, d, e, f	a, b	a, b, e, f, g, h, i, j, k			c				a, c, h, o, p, q	r, s	r, s
Gender	M		M			M		F		F		M		F
Week	12		10			6				12		12	48	12	48
Exercise per week	3		4			3				3	4	6
Iteration per set	8~12		8			6~8				8~12
Set number	1	3	3	6						2~3
Minutes per exercise											30	60
Weight (%)			2.02	2.63	2.19					−3.37	−4.02	−1.30	−1.90	−0.50	−1.80
Waist circumference (%)										−2.32	−5.86	−1.90	−3.20	−1.90	−1.60
Bust circumference (%)	1.53	2.20
Upper arm circumference (%)			2.27	4.65	4.76	1.89	1.76	1.89	1.76
Thigh circumference (%)	−0.98	1.30	2.94	1.50	6.35
Hip circumference (%)	−2.33	−2.33

* See Table 3 [43]; data obtained from ^A Rhea et al. [44], ^B Ostrowski et al. [45], ^C Munn et al. [46], ^D Sarsan et al. [47], ^E MCTiernan et al. [48].

. The module calculates the body measurement value shown in Section 3.1 through the Exercise Effect Database (EEDB) values described in Table 2. Equation (4) is used for the calculation.

$$A_{predict,i}=A_{original,i}{E}_{n,i}$$

(4)

where $A_{original,i}$ is the $i$th part of the original anthropometric variables; $A_{predict,i}$ is the changed body measurement computed from $A_{original,i}$; and $E_{n,i}$ is the $i$th part of the affected part of anthropometric variables given by the $n$th exercise plan of EEDB (see Table 2 and

Table 3. Exercise code and primary worked muscle [43].

Exercise Code	Exercise Name	Primary Worked Muscle
a	Bench press	Pectoralis major, anterior deltoids, triceps brachii
b	Leg press	Rectus femoris, vastus intermedius, vastus medialis, vastus lateralis
c	Biceps curl	Biceps brachii, brachialis
d	Pull-down	Latissimus dorsi, brachialis, brachioradialis
e	Seated row	Latissimus dorsi, trapezius, rhomboids, erector spinae
f	Back extension	Erector spinae
g	Squat	Rectus femoris, vastus intermedius, vastus medialis, glutes, vastus lateralis
h	Leg extension	Vastus medialis, vastus lateralis, rectus femoris, vastus intermedius
i	Stiff-leg deadlift	Biceps femoris, semimembranosus, semitendinosus, gluteus maximus
j	Leg curl	Biceps femoris, semimembranosus, semitendinosus
k	Calf raise	Gastrocnemius
l	Shoulder press	Anterior deltoids
m	Upright row	Trapezius, anterior deltoids, medial deltoids
n	Close grip bench press	Triceps, middle pectoralis major, middle pectoralis major
o	Arm extension	Outer triceps, medial triceps, anconeus
p	Twisting Oblique	Upper rectus abdominis, lower rectus abdominis serratus muscles, exterior oblique
q	Abdominal crunch	Upper rectus abdominis, middle rectus abdominis
r	Walking	-
s	Leg cycle	-

). In EEDB, each exercise plan shows the change in weight, waist, bust, upper arm, thigh, and hip circumference based on the type of exercise, gender, and how the exercise is performed (i.e., week, exercise per week, iteration per set, set number, and minutes per set). Through this procedure, the predicted anthropometric variables go through the model selection module again, and finally result in visualizing the 3D model and its body type.

3.4. Data Collection

There are two main simulator factors in this study: (1) anthropometry, and (2) exercise variables. In this section, we focus on each factor, and how each factor works on the simulator.

3.4.1. Anthropometry

Anthropometry has been studied in various fields, such as clothing, ergonomics, and physiology, for a long time. In this section, we introduce anthropometric measurement techniques from direct measurement to recently studied techniques using a 3D body scanner. In the past, anthropometry focused on studying systemic correlations of human body parts. One of the representative anthropometries, body surface area (BSA), has received attention. Therefore, a number of techniques used to measure BSA were developed: coating, surface integration, and triangulation [49]. Each technique has been widely used for some time; however, they still contain limitations that make them very laborious and time consuming. To solve these limitations, 3D scanners are widely used in anthropometric measurements, such as the Civilian American and European Surface Anthropometry Resource (CAESAR) [50]. Of course, 3D scanner techniques have advantages over manual measurements, such as Martin’s anthropometric technique, in terms of labor and time effectiveness. Nevertheless, according to Daanen et al. [51], scan-derived body circumference measures are slightly larger than values manually obtained. Manual measurements and scan-derived measurements are calculated as the performance measure of accuracy. Therefore, we collected anthropometries based on Martin’s anthropometric technique and the 3D scanner technique.

To collect anthropometric data, we referred to the fifth and sixth Size Korea Surveys that were conducted under ISO 15535 [52], in which 29,592 people were measured directly, and 6016 people were measured as 3D shapes for each city and county nationwide. Direct measurements were taken from male and females aged 7 to 90 years old, and from another set of males and females aged 8 to 75 years old. In the direct measurements using a human measuring instrument, 139 human dimensions and 3D shape measurements using a full-body 3D scanner were measured, with 177 human dimensions (body: 73, foot: 19, head: 45, hand: 19, direct measurement: 21). Among these dimensions, exercise plan and affected body area are also considered using these anthropometrics, and various body type classification techniques exist, such as those of Sheldon et al. [53] and Rasband and Liechty [54]. However, since Sheldon’s classification technique uses skinfold variables, which are hard to measure, we classified the 3D model into Rasband’s, which is widely used in the clothing industry, and classified based on length and circumference variables. According to Rasband and Liechty [54], body type can be classified into eight types (ideal, triangular, inverted triangular, rectangular, hourglass, diamond, tubular, and rounded). To classify these body types, using the cluster analysis technique, each body type was considered in terms of pattern and size cluster. The pattern cluster is the body type of the group that it represents, and it can also represent which parts are the same or different in each part of the human body compared to the standard body type. On the other hand, from the formation of the size cluster, it provides information on what size distribution the human characteristics of observers belonging to the pattern cluster represent. After the formation cluster process, the interpretation of the formed cluster was conducted. Interpreting the cluster meaning process is performed according to the characteristics of the center position (average) of the cluster; i.e., if the first factor in a cluster represents a ‘torso’, and the value is very large, the characteristic of the ‘torso’ of the subjects in the cluster can be said to be people with a very thick torso. The meaning analysis of the cluster ensures that the factor score follows the standard normal distribution; the area of the probability density function (pdf) is equally divided into 10 sections, meaning that the initial 10% is the section with the smallest factor score from the first section. That is, the first observation refers to a case where the corresponding factor score is −1.28, and when any one factor belongs to each section, the display technique is P1 to P10. In this way,

Table 4. Subject groups clustered by size.

	Factor
Cluster	Height	Circumference (Thickness)	Torso Length	Shoulder Width	Ratio (%)
1	P9 (1.16)	P5 (−0.44)	P6 (0.14)	P6 (0.23)	21
	Tall height and the rest are normal.
2	P5 (−0.11)	P6 (0.17)	P7 (0.47)	P2 (−1.24)	21
	Short torso with very narrow shoulders.
3	P5 (−0.22)	P7 (0.33)	P2 (−1.24)	P6 (0.123)	28
	Thick and very short torso.
4	P3 (−0.55)	P5 (−0.11)	P8 (0.74)	P8 (0.59)	30
	Short height and long torso with wide shoulders.

lists the clusters that are formed for size characteristics, while

Table 5. Example of body types of subjects.

Body Type	Ratio (%)	Detailed Characteristic (Compared to Standard Body Type)
Inverted triangular	27.7	Slightly thick torso, wide shoulder width, small head, short hip length
Small rectangular	28.5	Thin torso, narrow shoulder width, long arm length, big head, short hip length, long limbs
Triangular	20.8	Normal torso, narrow shoulder width, very short arm length, very small head, long hip length, slightly short limbs
Large triangular	23.0	Very thick torso, wide shoulder width, very big head, long hip length, very short limbs
Total	100.0

lists the resultant body types for people in their 40s. Therefore, we can classify the subject body type and interpret what it means.

3.4.2. Exercise Plan

We wished to determine the changes in the body through the anthropometries described in Section 3.4.1. To predict relevant body changes, it is necessary to set appropriate independent variables that can cause such change. There are various independent variables that can be used to predict body change, such as dietary habits and exercise. Thus, we needed to establish independent variables that are more closely related to body change, so we set an exercise that shows different changes depending on the area where we exercise as an independent variable. To establish the exercise plan data, we used the results of the previous body shape change experiments according to the exercise effects. The exercise plans for each experiment were first classified into aerobic exercise and anaerobic exercise, according to the type of exercise, such as bench press or walking. The classified experimental data were again classified according to the condition of the amount of exercise, which is a variable representing how much exercise is performed over a set period, and aerobic exercise depends on how many minutes per week the exercise is performed over a period and how many sets per set of anaerobic exercises are performed over a period. In the case of anaerobic exercise, it was set in consideration of the results of existing studies [44,45,46,47,48] that revealed the difference between the number of exercise sets and the exercise effect with exercise iterations per set. By selecting these variables for the exercise plan, each plan can derive different body shape changes; therefore, we selected anthropometries that related to body change, which were weight, waist circumference (omphalion), bust circumference, thigh circumference, and hip circumference.

4. Experiments

4.1. Scenario

Experiments were conducted considering the optimal conditions of the human segmentation process for image-based reconstruction methods in the anthropometric measurement process. The experiments were conducted to verify the necessity for the human segmentation process, and if human segmentation processes were required, to determine the architecture and training datasets of the human segmentation pretrained models. In addition, 174 subjects with various demographic characteristics were selected to prevent inaccuracies in the experiment that could occur if subjects were confined to a specific age and gender, considering the characteristics of the experiment conducted through 2D images and directly measured anthropometric variables of the subjects.

Table 6. Summary of subject demographic characteristics.

Gender	Age	Number of Subjects	Ratio (%)
Male	20–29	42	24
	30–39	5	3
	40–49	10	6
	50–59	15	9
	60–69	13	7
	70–	8	4
Female	20–29	31	18
	30–39	3	2
	40–49	13	7
	50–59	19	11
	60–69	12	7
	70–	3	2
Total		174	100

summarizes the demographic characteristics of the subjects.

4.2. Results

First, to determine the necessity of the human segmentation process, a paired t-test was performed on the actual measured body size values and the body size values inferred through the HMR-based technique without the human segmentation process.

Table 7. Anthropometric variables of the direct measurement approach and the HMR-based technique without segmentation.

Anthropometric Variable	Direct Measurement	HMR without Segmentation
Stature (cm)	169.09	169.09
Waist Circumference (Natural Indentation; cm)	83.41	84.90
Waist Circumference (Omphalion; cm)	76.90	74.23
Bust Circumference (cm)	96.94	93.95
Wrist Circumference (cm)	15.27	15.11
Neck Circumference (cm)	43.98	42.12
Arm Length (cm)	51.20	50.74
Thigh Circumference (cm)	57.01	54.56
Biacromial Breadth (cm)	48.52	46.95
Hip Circumference (cm)	96.94	95.86

describes the difference between the actual body measurements (direct measurement) and the body measurements based on the 3D model created from the original 2D image without removing the background under the HMR-based technique (i.e., no segmentation). In the case of body size (i.e., stature), which can be confirmed relatively accurately even in 2D images, accurate values were estimated, but for other body parts that included volume information, differences of as little as 0.90% (arm length) and as much as 4.30% (thigh circumference) were observed. The reason for this result is that during the process of creating a 3D model, the size parameters of body parts that require volume information, such as thighs or arms, are predicted relatively inaccurately because they are inferred together with the surrounding background. When a paired t-test is conducted between the two values in Table 7, the p-value of the paired t-test at α = 0.05 is 0.0268, which means that there is significant difference between the two measurement methods. This result confirms that the HMR-based technique without the human segmentation process is an inadequate methodology for predicting body dimensions by generating 2D images into 3D images.

In the semantic segmentation combined HMR frameworks, we can consider which pretrained model for human segmentation will be selected for the rapid measurement process. The considered pretrained models were (1) the Xception architecture [40] using MS−COCO [41] and the PASCAL VOC training–validation dataset, (2) Xception architecture using MS-COCO and the PASCAL VOC training–augmented dataset, (3) Mobilenetv2 [55] architecture using MS−COCO and the PASCAL VOC training–validation dataset, and (4) Moblienetv2 architecture using MS-COCO and the PASCAL VOC training–augmented dataset. As mentioned in Table 7, the prediction accuracy of body dimensions that can be identified even in 2D images, such as stature, showed that accurate predictions were possible in all methodologies. However, in body parts requiring information on volume, differences were observed depending on the methodology, with the highest difference of 3.02% on average observed in neck circumference, excluding stature, and the lowest difference of 0.52% observed in arm length.

According to the measured results described in

Table 8. Anthropometric variables of the direct measurement approach, and each condition.

Anthropometric Variable	Direct Measurement	Condition (1)	Condition (2)	Condition (3)	Condition (4)
Stature (cm)	169.09	169.09	169.09	169.09	169.09
Waist circumference (Natural indentation; cm)	83.41	82.81	82.9	82.04	82.01
Waist circumference (Omphalion; cm)	76.90	78.38	78.23	77.77	77.77
Bust circumference (cm)	96.94	94.89	93.95	98.06	97.14
Wrist circumference (cm)	15.27	15.07	15.11	15.12	15.21
Neck circumference (cm)	43.98	42.03	42.12	43.14	43.32
Arm length (cm)	51.20	50.84	50.74	51.1	51.05
Thigh circumference (cm)	57.01	54.88	54.56	55.78	56.69
Biacromial breadth (cm)	48.52	47.98	47.95	48.75	48.57
Hip circumference (cm)	96.94	96.01	96.86	95.62	95.5

, the p-values of the paired t-tests at α = 0.05 are 0.0682, 0.09, 0.3483, and 0.3434, respectively, which means that the measurement technique using pretrained models (1), (2), (3), and (4) exhibits no significant difference to the direct measurement approach. However, considering that the shape of the body can vary according to each anthropometric variable, the human segmentation process based on conditions (3) and (4) is optimal to modeling the body model through the image-based reconstruction process. Figure 5 shows examples of segmented images for each condition. In Figure 5, there is a minor difference between the actual body dimensions and the dimension values from the inferred 3D image because the background images of the armpits and between the legs, where the gap between the body and the clothes is relatively narrow, were not accurately removed. Although these aspects need improvement, we can still confirm that more accurate body dimension prediction is possible through the proposed segmentation process than the existing HMR approach.

In addition, we conducted computational complexity experiments on Xception architecture and Mobilenetv2 architecture, which are the main methodologies used in semantic segmentation. In the process of converting 513 × 513 image inputs into 3D models, Xception architecture was applied to conditions (1) and (2), and Mobilenetv2 architecture was applied to conditions (3) and (4). The results of measuring the FLOPs (Floating Point OPerations), number of parameters (#param.), and GPU utilization (%) for the two architectures are described in

Table 9. Computational complexity of semantic segmentation with 513 × 513 image inputs under MobilenetV2 architecture and Xception architecture.

Computational Metrics	MobilenetV2 Architecture (Conditions (1) and (2))	Xception Architecture (Conditions (3) and (4))
FLOPs	35.5 G	9.8 G
# param.	22.8 M	2.1 M
GPU usage (%)	65.6%	22.3%

. The experiment was conducted on the Dell Precision Workstation (Intel Xeon ^® Silver 4210 Processor, Nvidia RTX Quadro 4000, and Ubuntu 20.04), Austin, TX, USA.

In Table 9, conditions (3) and (4) using Mobilenetv2 show 9.8 G FLOPs, 2.1 M parameters, and 22.3% GPU usage, which are all more efficient than the Xception architecture in terms of computational complexity. This is because the Mobilenetv2 architecture has a smaller number of filters and a shallower network structure than the Xception architecture and requires less computational demand and resource usage by applying the Linear Bottleneck structure.

The results in Table 8 and Table 9 and Figure 5 show that the HMR framework is suitable for creating 3D models using Mobilenetv2. Accordingly, additional experiments were conducted to secure the general applicability of the DeepLabv3 model with the Mobilenetv2 architecture. That is, the experiments were conducted on combinations of the Atrous rates of the Atrous convolutions in the ASPP (Atrous Spatial Pyramid Pooling) module, which is used to detect information of various scales and integrate the global context in the image in the DeepLabv3 model in order to derive the optimal structure of the ASPP module. An example of each Atrous rate of the Atrous Convolution is shown in Figure 6. The experiment was conducted under the same hardware conditions as the experiment in Table 9, with three conditions for the Atrous rates of the Atrous Convolution in the ASPP module: (1) single Atrous rate of 12, (2) two Atrous rates of 6 and 12, and (3) three Atrous rates of 6, 12, and 18. Semantic segmentation was performed and the IoU (Intersection of Union; %), FLOPs, and #param under each condition were measured. The experimental results are shown in

Table 10. Experimental results of semantic segmentation with ASPP module conditions: (1) ASPP module with Atrous rate of 12; (2) ASPP module with Atrous rate of 6 and 12; and (3) ASPP module with Atrous rates of 6, 12, and 18.

Metrics	ASPP Condition (1)	ASPP Condition (2)	ASPP Condition (3)
FLOPs	8.8 G	9.1 G	9.8 G
# param.	1.9 M	2.0 M	2.1 M
IoU (%)	72.4%	72.7%	73.1%

.

The experimental results show that applying the ASPP module (ASPP condition (3)) consisting of three-layer Atrous Convolution with Atrous rates of 6, 12, and 8 has the highest IoU of 73.1%. However, in terms of computational complexity, ASPP conditions (1) and (2) were shown to have superior performance with 8.8 GFLOPs and 1.9M FLOPs and 9.1 GFLOPs and 2.0M #param, respectively, compared to 9.8 GFLOPs and 2.1M for the ASPP condition (3). Considering these results, it was confirmed that setting the conditions of the ASPP module according to the specifications of the computational hardware device used in the process of segmenting the user’s 2D image, rather than specifying one methodology, allows for a wide range of applications to various computational hardware (such as low-cost PCs and mobile devices).

As a result, the selected measurement method to process human segmentation based on Mobilenetv2 architecture, before modeling the 3D human model using HMR framework, can produce reasonable results, while solving the limitations of the existing measurement method, such as expensive 3D scanners, long measurement time, and limited clothing conditions for measurement.

5. Conclusions

This study proposes a human body simulation that can predict body changes induced by selected exercise plans for disease prevention, and even aesthetic purposes. The proposed simulation is based on a body measurement technique using a 3D body model modeled from 2D images, which has the advantage of being time- and cost-effective, compared to conventional expensive 3D scanners and time-consuming direct measurement approaches. Previous studies have attempted to reconstruct three-dimensional images from two-dimensional images. However, this study is notable for its approach of measuring each part of the body through a three-dimensional model created for the purpose of visualizing three-dimensional changes in the body according to an exercise plan. This is achieved by aggregating existing studies to create an exercise effect database. Experiments conducted on subjects with various demographic characteristics to verify the validity of the proposed simulation demonstrate that the proposed simulation is sufficiently accurate and versatile. The paired t-test result between constructed images and actual anthropometric measurement was 0.3483 in terms of the p-value. In addition, considering that the proposed simulator can predict body changes according to various exercise plans, it is expected to be widely used in the recently growing field of preventive treatment and personalized healthcare.

This study has the possibility of expansion through future studies in body change prediction. First, the prediction of body shape change through the proposed simulator is based on the existing body change experiment, which only considers exercise plans. However, since body shape changes are significantly affected by eating habits, through future research, the simulator is expected to provide more accurate and realistic predictions of body shape changes.