*6.3. Memory Usage*

In the evaluation of memory consumption, data were collected for the metrics: MEM\_USED, which represents the calculation of the total memory used; MEM\_FREE, which is the memory that is not being used; MEM\_AVAILABLE, which estimates how much memory is available to start new applications without swapping (it may include memory space that is being used for buffers or cache); MEM\_SHARED, which is the memory mainly used by TMPFS which is the file system that keeps all files in virtual memory; MEM\_BUFFERS\_CACHED, which is the sum of the memory buffers and cache; SWAP\_USED and SWAP\_FREE metric, which represent respectively the used and free amount of virtual memory's swap space, that allows the system to use a part of the hard disk as physical memory. All these metrics were monitored using the "free" tool in both the Minikube and K3S environments.

In the evaluation of memory utilization in Minikube, the MEM\_USED metric in Figure 14 has its behavior mirrored with that of the MEM\_AVAILABLE metric, while the MEM\_USED increases throughout the experiment, the MEM\_AVAILABLE decreases in an inversely proportional trend. MEM\_USED has a consumption increase of around 70% at the end of the experiment, even applying rejuvenation (i.e., cluster termination and restart between cycles). Such an action drops the memory usage temporarily, but when the cluster is started again, the system restores the same memory usage level observed at the end of the previous cycle. The MEM\_FREE metric has a drop close to 48%. The MEM\_BUFFERS\_CACHED metric has a drop of around 41%. The SWAP\_USED metric also behaves inversely to the SWAP\_FREE metric, while the SWAP\_USED has a 20% increase at the end of the experiment and SWAP\_FREE a drop of 11%. The MEM\_SHARED metric in both Minikube and K3S behave similarly, maintaining a regularity between 48 to 179 MB of consumption.

**Figure 14.** Memory consumption in Minikube.

In the evaluation of memory utilization in K3S, the MEM\_USED metric in Figure 15 showed behavior similar to that observed in Minikube. MEM\_USED has a consumption increase of around 61% at the end of the experiment, even applying rejuvenation. The MEM\_FREE metric has a decrease of close to 79%. MEM\_BUFFERS\_CACHED has an increase of around 12%, which differs from the behavior in Minikube. The SWAP\_USED has an increase of 8% when it reaches the end of the experiment and the SWAP\_FREE a decrease of 8.5%.

For these memory consumption metrics, both in Minikube and in K3S, linear regression calculations on MEM\_USED were performed to estimate the moment when the system would reach its upper limit for RAM usage, which in these cases is 8 GB. To confirm that estimate, we also computed the linear regression for MEM\_FREE, which is another way to indicate the exhaustion of the resource, leading to system downtime and, consequently, the interruption of service provision. Similar regression estimates were carried out for the swap space usage.

**MEMORY CONSUMPTION**

**Figure 15.** Memory consumption on K3S.

Equation (1) of the linear regression was obtained for the MEM\_USED metric in the Minikube environment (MU\_Minikube), shown in Figure 14. From this equation, it is possible to observe, as a function of MU\_Minikube, that the 8 GB limit is reached after 170 h (i.e., 7 days and 2 h) of continuous execution of the workload used in the experiment. For the SWAP\_USED metric, also exposed in Figure 14 for the Minikube environment, the linear regression Equation (2) was obtained.

$$M\!\!\!\!\!\!\!\!\!\!\!\!\!M\_{\text{Minikube}} = \\$\!\!\!900.84 + \text{23.98072} \times T\_{\text{stress}}\tag{1}$$

In Equation (2), it is possible to observe that the upper limit of the SWAP\_USED metric of the Nginx environment (SU\_Minikube), which in this case is 5.8 GB, is reached after approximately 551 h of experiment, or 22 days of the same, so that the resource was completely exhausted.

$$SII\_{\text{Minikube}} = -221.43413 + 10.37255 \times T\_{\text{stress}} \tag{2}$$

For the MEM\_USED metric of the K3S environment (MU\_K3S), the linear regression Equation (3) was obtained which, through it, it is possible to observe that the upper limit of 8GB of resource for the MEM\_USED metric is reached after 187 h (i.e., 7 days and 8 h) of workload execution.

$$MUL\_{\rm KSS} = 2482.70 + 29.67105 \times T\_{\rm stress} \tag{3}$$

Finally, the SWAP\_USED metric of the K3S environment (SU\_K3S) had the linear regression Equation (4) obtained, which allows the visualization of resource exhaustion, which has a total of 5.8GB, after 603 h (i.e., 25 days and 1 h) of workload performed in the experiment.

$$SIL\_{\rm KBS} = -120.24857 + 9.30782 \times T\_{\rm stress} \tag{4}$$

#### *6.4. Evaluation and Discussions*

When evaluating the results presented in Figures 10 and 11, it can be seen that most of the CPU consumption happens through the USR metric in the K3S environment, while the SYS metric does the highest consumption in the Nginx environment. This growth behavior of the USR metric in K3S was recurrent even after applying Software Rejuvenation every cycle, unlike the Nginx environment that maintains stability in the consumption of its CPU utilization metrics.

The results presented in Figures 12 and 13 show similar behavior in the use of disk usage metrics in the K3S and Nginx environment, differing only that in Nginx, the READ metric presents an interruption when it reaches 4,000,000 KB/s, returning in the fifth cycle with a total utilization close to 10%. In the K3S environment, this READ metric does not suffer an interruption but presents a linear growth from one workload cycle to another. In both scenarios, the READ metric generally presents a linear growth representing a greater need for reading from disk at each cycle.

Figures 14 and 15 show similar behavior related to memory consumption metrics in the Nginx and K3S environments, respectively. In both, there is a linear growth of the MEM\_USED metric and in the SWAP\_USED metric, and the opposite behavior of the MEM\_AVAILABLE and SWAP\_FREE metrics. Thus, with Equations (1) and (2) obtained from linear regression, it is possible to glimpse the effects arising from software aging related to memory consumption even after the application of a potential software rejuvenation action, that is, the cluster termination.

The results presented in this section are the observation of software aging phenomenon for a private cloud system hosting a UAM-ODT platform for UAM management. It is crucial to emphasise that those findings point to the dangers of system breakdowns and performance declines brought on by signs of software ageing. However, the timing of those events relies on the nature and volume of the workload that the system must handle, in addition to the hardware and software requirements of particular Kubernetes system. The ageing phenomena would be delayed and the failures caused by resource exhaustion would follow if the system had more resources available or a lighter burden than that used in this experiment. This reality does not lessen the significance of assessing the software ageing in those systems and organising countermeasures. Evaluating these scenarios using other software rejuvenation approaches and complementary metrics related to software aging are the most promising steps that could be taken in future work.

Regarding how to avoid the observed software aging phenomenon in the UAM-ODT infrastructure, in general, there are several strategies that can be used to avoid or mitigate software aging. These can include: (i) regularly updating and patching the software to fix bugs and security vulnerabilities; (ii) monitoring the performance of the software and identifying potential problems before they occur; (iii) implementing automation and management tools to help manage the software and its dependencies; (iv) using modular, microservice-based architectures to make it easier to update and maintain individual components of the system; (v) using containerization technologies, such as Docker, to package the software and its dependencies into a self-contained environment that can be easily deployed and managed. These are just some examples of strategies that can be used to avoid software aging in a cloud system. Currently, the technique to avoid software aging is monitoring the performance of the software and identifying potential problems before they occur. Further investigation on how to adopt the software rejuvenation techniques in optimal and automatic manner will be an interesting extension for research into the UAM-ODT system in which the services for UAM management using ODT are constant and at zero downtime.

#### **7. Conclusions**

This paper presented a comprehensive study on the effects of software aging problems on Kubernetes in container orchestration system in a digital twin cloud infrastructure for UAM-ODT systems. The behaviours of Kubernetes software were analysed in an accelerated lifespan experiment utilising both Nginx and K3S tools. The operations for establishing and terminating pods were carried out in real time, allowing us to monitor the usage of computational resources (such as CPU, memory, and I/O), the performance of the Nginx and K3S environments, and the response time of an application hosted in those environments. In particular settings and for specific metrics, such as virtual memory utilisation, software ageing effects were detected, indicating a memory leak that is not entirely cleansed when the cluster is halted. The study's findings help to understand the phenomenon of software ageing in digital twin computing infrastructures built on Kubernetes, which is at the very beginning of current research on software ageing issues for highly reliable and fault-tolerant digital twin computing infrastructures.

**Author Contributions:** Conceptualization, methodology, and supervision, R.M.; project administration, formal analysis and investigation, R.M., J.A., J.-W.L., D.M. and T.A.N.; resources, funding acquisition and investigation, E.C., J.-W.L. and D.M.; validation, J.C., J.L.; writing—original draft, J.C., R.M. and J.A.; writing—review and editing, J.A., R.M., J.L., E.C., J.-W.L., D.M. and T.A.N.; All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2021R1A2C2094943). This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Education (2020R1A6A1A03046811). This work is supported by the Korea Agency for Infrastructure Technology Advancement(KAIA) grant funded by the Ministry of Land, Infrastructure and Transport (Grant RS-2022-00143965).

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.
