Generative AI-Enabled Energy-Efficient Mobile Augmented Reality in Multi-Access Edge Computing

Abstract

This paper proposes a novel offloading and super-resolution (SR) control scheme for energy-efficient mobile augmented reality (MAR) in multi-access edge computing (MEC) using SR as a promising generative artificial intelligence (GAI) technology. Specifically, SR can enhance low-resolution images into high-resolution versions using GAI technologies. This capability is particularly advantageous in MAR by lowering the bitrate required for network transmission. However, this SR process requires considerable computational resources and can introduce latency, potentially overloading the MEC server if there are numerous offload requests for MAR services. In this context, we conduct an empirical study to verify that the computational latency of SR increases with the upscaling level. Therefore, we demonstrate a trade-off between computational latency and improved service satisfaction when upscaling images for object detection, as it enhances the detection accuracy. From this perspective, determining whether to apply SR for MAR, while jointly controlling offloading decisions, is challenging. Consequently, to design energy-efficient MAR, we rigorously formulate analytical models for the energy consumption of a MAR device, the overall latency and the MAR satisfaction of service quality from the enforcement of the service accuracy, taking into account the SR process at the MEC server. Finally, we develop a theoretical framework that optimizes the computation offloading and SR control problem for MAR clients by jointly optimizing the offloading and SR decisions, considering their trade-off in MAR with MEC. Finally, the performance evaluation indicates that our proposed framework effectively supports MAR services by efficiently managing offloading and SR decisions, balancing trade-offs between energy consumption, latency, and service satisfaction compared to benchmarks.

1. Introduction

Mobile augmented reality (MAR), which seamlessly combines virtual information into a person’s real-world environment, is becoming a prominent application within mobile multimedia networks (MMNs) by utilizing artificial intelligence (AI) [1]. For example, various deep learning (DL)-based AI object detection algorithms, such as YOLO, SSD, and Fast R-CNN, are employed to identify objects from images in MAR. While these DL algorithms provide precise object detection, they demand significant computational power from MAR devices that have limited processing capabilities and battery life. This leads to increased processing delays and quick battery depletion. To address this issue, multi-access edge computing (MEC) in MMNs provides additional computing resources to mobile devices (MDs) through computation offloading [2].

Since both MDs and MEC servers have heterogeneous computing and networking capabilities, designing efficient offloading management to optimize energy efficiency, minimize latency, or balance both is challenging [2]. Particularly, determining whether to offload images based on current networking and computing conditions has been extensively studied [3,4,5]. However, a significant issue in MAR is that frequently transmitting high-resolution images to the MEC server results in high latency and energy consumption. To address this, recent works [6,7,8] have considered resolution management to balance accuracy and latency/energy trade-offs, recognizing that image resolution impacts the object detection accuracy. Thus, maintaining an acceptable detection accuracy in MAR while reducing the bitrate remains a challenging trade-off.

To address this, recent research in MMNs has explored the potential of generative AI (GAI), such as super-resolution (SR) neural models, to enhance image quality by increasing the resolution of lower-quality frames, thereby reducing the bitrate of network transmission [1] (GAI can produce high-quality, naturalistic content in various formats, including images, videos, and 3D content [1,9]. Among the various GAI solutions, SR technologies such as EDSR and RCAN convert low-resolution images into high-resolution ones using machine learning (ML)-based AI schemes.). However, GAI technology demands substantial computational power and can introduce latency, which may significantly burden the MEC server if there are numerous offloading requests from MAR clients for MAR services [10]. To the best of our knowledge, no studies have investigated the use of GAI technology for MEC-assisted MAR. Therefore, a thorough investigation is necessary to determine how to efficiently utilize GAI while managing offloading decisions, presenting an opportunity to improve upon conventional approaches.

In this study, we propose a novel offloading and SR control scheme for energy-efficient MAR in MEC by leveraging GAI. Specifically, we adopt SR as a promising GAI technology and conduct an empirical study to verify that the computational latency of SR, including SRGAN and EDSR, increases with the upscaling level. From this empirical study, we highlight a trade-off between increased computational latency and enhanced service satisfaction when upscaling images for object detection, as it boosts the detection accuracy. This makes it difficult to decide whether to choose SR for MAR while simultaneously managing offloading decisions. As a result, to balance the trade-off between energy efficiency and latency in MAR with MEC, our approach considers both the computational latency for SR at MEC servers and the computing and networking conditions of MDs to determine whether to offload the image to the MEC server. If offloading is decided, SR management is conducted. To achieve this, we rigorously formulate analytical models and design an elaborate cost optimization problem that balances this trade-off by controlling the offloading decisions and SR management of each MAR client. Since the problem is a non-convex optimization problem, we leverage several relaxation techniques to develop a near-optimal yet feasible algorithm. Numerical analysis reveals that the proposed scheme significantly improves the balance among the computation latency, energy consumption of each MD, and the satisfaction of service quality from enforcement of service accuracy compared to benchmarks. This creates an incentive to apply GAI in future MAR services.

The rest of this paper is organized as follows: Section 2 reviews related work and offers an overview of previous research in this area. In Section 3, we describe the proposed system model. Section 4 details the problem formulation and the solution for the proposed scheme. Section 5 explains the evaluation setup and presents our results, including a comprehensive analysis and comparison with other approaches. Finally, Section 6 summarizes our findings and discusses possible future research directions.

2. Related Works

See Table 1, Our work relates to MEC-based MAR systems where numerous research on the MEC framework has been appeared and improved from the first MEC platform, which was established by IBM and Nokia Siemens Network [11]. Consequently, in recent years, MEC technology has been applied in various fields, especially in the AR field [12,13].

Table 1. The summary of related works.

Topic	Reference Number	Proposed	Pros	Cons
MEC-based MAR System	[7,8]	A novel resolution and power control scheme for computation offloading of MDs to MEC	Finding both the optimal transmission power and video frame resolution	Do not consider the GAI technique although resolution control is considered
	[2,6]	MEC-based MAR system optimization with MD cache management and power control	Present a comprehensive overview of the overall process of local cache-enabled MAR in MEC and provide a thorough analysis of latency and energy considerations	Do not select direct and detailed management for the offloading optimization
	[14,15]	Optimization methods of task offloading scheme for single-user in MEC-based MAR system	Present the fundamental optimization on MEC task offloading for the mobile device and suggest various types of solutions	Consider single-user scenario only
	[5,16,17]	Task offloading schemes for MEC-assisted MAR system to optimize service latency	Present various end-to-end latency modeling of MEC-based MAR application and suggest various solution approaches for minimizing latency	Consider the latency only and it to be topic-intensive
Offloading management for MAR	[18]	EARNet, a distributed edge system orchestration approach for mobile web AR in 5G networks	Location-aware task scheduling model and service migration model	This solution can be adapted to web-based AR only
	[13,19,20]	Optimization methods of task offloading scheme for MEC-based MAR	Consider both single-MEC system and multi-MEC system, consider both service latency and energy consumption of MDs, and suggest various solution approaches for joint optimization problem	Although considering service accuracy as one of the main factor on the cost trade-off, do not consider the factors that may affect the accuracy, such as other object detection algorithms or GAI
	[21]	User preference-based energy-aware MAR task offloading system	Adopt experimental results on factors affecting MAR client energy efficiency and preference-based offloading strategy	Insufficient comparison with the other preference-based methods
MAR with GAI	[22]	An edge-assisted system that provides distortion-tolerant image recognition for mobile AR with imperceptible system latency	Enables distortion-adaptive image recognition to improve the robustness against image distortions	Do not consider the heterogeneity of the mobile device
	[23]	A motion-guided video-to-video translation framework using a diffusion model	Uses motion information to prevent the regeneration of the redundant areas for content consistency	Focuses on the performance of the GAI model, not the overall system
MAR with SR control	[24]	GAN-assisted multi-SR model framework	Analyze the impact of SR on object recognition strictly	Topic-intensive and model-specific
	[25]	A MAR performance optimization model based on HD restoration	Reduce application cost and ensure the high quality of customer perception	Do not consider the impact of SR energy consumption requirement on service QoS

2.1. MEC-Based MAR Systems

The main advantage of the MEC environment is the powerful resources that the MEC server provides in terms of computation, network bandwidth, and storage. Many of the research studies on MAR applications have tried to take these powerful resources since most MAR applications require heavy resources. Ahn et al. proposed a centralized orchestration scheme for energy-efficient MAR applications in MEC [7]. They analyzed a trade-off between the accuracy, latency, and energy consumption of each MD as intertwined costs to achieve energy efficiency under QoS constraints. In addition, they considered resolution control to improve QoS through improving service accuracy [8]. Seo et al. considered the local cache-enabled MAR with MEC environment [2]. The local cache is one of the approaches that can reduce the server load and service latency, so the MAR cache control scheme offers more efficient latency and energy consumption utilization. Also, they suggested a novel joint mobile cache and power management scheme for energy-efficient MAR services [6]. By adapting power control, they could more precisely adjust the trade-off between latency and energy consumption.

2.2. Offloading Management for MAR

Task offloading is one of the key techniques of an MEC system. It allows MAR users to access the MEC server’s resources easily. Furthermore, MAR users no longer have to be concerned about the energy consumption of the MD since the MEC server processes MD’s tasks. Many of the early research studies on MEC offloading schemes focused on how to utilize specific objectives with a simple scenario such as the computing resource and storage capacity of the MEC server or energy consumption of the MD [5,14]. For instance, Tang et al. presented a joint optimization task offloading method for both system response time and energy consumption in a single-device to one mobile edge server scenario [14]. They constructed the fundamental optimization on MEC task offloading for the MD side. Unlike the above, Hoa et al. proposed a dynamic offloading scheme for an edge computing-assisted Metaverse system to involve a virtual service provider (VSP) [5]. This aims to minimize the latency of the whole Metaverse service process from a VSP to users with the help of UAVs in an edge computing environment to meet the latency requirements of Metaverse users.

However, in the real MAR environment, multiple MDs connect to one base station or server, so the research works on the MEC offloading scheme for MAR systems need to design complicated and complex models and solutions to address multi-user environments [13,16,17,18]. Ketyko et al. provided a general model of the system considering the end-to-end computational latency of MEC applications [16]. They presented complex multi-user offloading problems related to resource sharing and load balancing, and then suggested a solution to this problem. Do et al. proposed a MEC federation system for MAR with a delay optimization solution based on the Markov Decision Process (MDP) and Deep Deterministic Policy Gradient (DDPG) model to reduce the latency of the communication computation, as well as queuing [17]. Ren et al. proposed a distributed edge system orchestration scheme for a web-based MAR service [18]. This paper suggested a management system for network dynamics with respect to user mobility and workload balance to improve service performance such as message efficiency, scheduling latency, etc. Chen et al. suggested a joint optimization of task offloading and resource allocation scheme for energy-efficient MAR [13]. The main goal of this paper is to minimize the energy consumption of user terminals in the task offloading process. This paper designs a novel AR application task model consisting of several dependent subtasks and implements a Deep Reinforcement Learning (DRL)-based optimization solution.

These research works still focused on the specific factor and goal. However, in the real MAR environment, users’ requirements are more complicated. Most MAR users require lower latency, lower power, and more accurate AR services and applications. With this trend of demand, service providers are also forced to consider more factors. Therefore, recent research works on the MEC framework for MAR have considered not only the architecture that one MEC server takes multiple MDs but also the impact of the trade-off among multiple factors, such as computation latency, energy consumption of clients or MEC server, and service accuracy. First of all, Zhang et al. considered the mobility of mobile devices used by mobile users [15]. They aimed to minimize both task completion latency and energy consumption in the long term with stochastic computation tasks and dynamic network conditions. To solve this complicated problem, they also adopted the DRL approach. Also, they focused on the ask offloading placement problem for AR overlay rendering in multi-party MAR system [19]. Long et al. proposed a task offloading algorithm under end-edge-cloud three-tier architecture [20]. In this paper, a genetic algorithm-based solution was suggested to optimize energy consumption and latency. They first presented the observation about performance bottlenecks of edge devices and explained the necessity of splitting the AR overlay rendering pipeline and then suggested a DRL-based offloading decision solution for optimizing both QoS and service cost. With these studies, we can conclude that the recent MAR applications on the MEC system should consider QoS, energy consumption of MAR users, and service latency. Wang et al. designed an energy-aware edge-based MAR system that enables MDs to dynamically change their configuration parameters, such as CPU frequency and computation model size, based on their user preferences, camera sampling rates, and available radio resources at the edge server [21]. In this paper, their proposed algorithm can minimize the per-frame energy consumption of multiple MDs without degrading their preferred MAR performance metrics, such as service latency and detection accuracy.

2.3. MAR with GAI

While GAI technologies have been widely applied across various fields, there has been insufficient research on their application to computation offloading in MAR [1]. Liu et al. present CollabAR, an edge-assisted system that provides distortion-tolerant image recognition for mobile AR with imperceptible system latency [22]. They implemented accurate image recognition with correlation among mobile AR users under the image distortion problem by generative adversarial networks (GANs), one of the most popular generative AI models. Hu et al. focused on the diffusion model, which generates images from text-based guideline input [23]. They proposed VideoControlNet, a motion-guided video-to-video translation framework, to generate various videos based on the given prompts and the condition from the input video. This framework uses motion information to prevent the regeneration of redundant areas for content consistency.

2.4. MAR with SR Control

In particular, SR is a well-known GAI solution that improves the object detection accuracy and reduces network latency by enabling the delivery of low-resolution images due to upscaling capabilities. According to [24], GAN-based SR models like YOLO and Retina can generate images nearly identical to the originals and effectively reduce service delays by minimizing computation overhead. Also, Dang et al. [25] suggested a video super-resolution optimal configuration choice model based on the energy consumption requirements of mobile AR applications. Their algorithm predicted the PSNR of input video and then drew out the optimal model configuration to improve the accuracy of SR model. As shown above, adapting SR to the MAR environment is one of the great approaches to enhance MAR performance. Therefore, in this paper, we introduce a novel approach by applying these SR technologies to MAR services. Considering their impact, we design a GAI-enabled MAR framework.

3. System Model

3.1. Proposed Framework

In this section, we outline the system architecture for a multi-user SR-supported MMN environment utilizing MEC, as illustrated in Figure 1. The system model consists of multiple MDs labeled as $N=1,2,\dots ,n$, along with a single MEC server linked to the core network with a network bandwidth of B and a computing capacity of $f_{MEC}$. Each MAR client device is assigned a specific network bandwidth $B_n$ and possesses distinct computing resources $f_n$. MAR tasks are represented as a vector comprising task workload density ${\omega }_n$ defined as CPU cycles per bit, offloading data size $d_n$, and the resolution level of source data $R_n$. Here, we assume that each MAR client executes only one MAR application per decision process, and this application may differ from those of other clients. Consequently, each MAR client has a unique MAR task workload and its data size. In this framework, MAR clients can offload their tasks to the MEC server to manage time-sensitive and computation-intensive MAR tasks given the MDs’ limited battery life and computing power. During the offloading process, the MD can transmit low-resolution source images to the MEC server, where the resolution is enhanced using SR to improve accuracy, thereby increasing MAR service satisfaction. Therefore, throughout the entire MAR process, including offloading and SR, MAR clients should optimize offloading and SR control for efficient MAR services.

Figure 1. Proposed system architecture.

The proposed system model consists of four main components, as shown in Figure 1. The communication module gathers information from MAR clients, such as the data needed for offloading and SR decisions. The decision controller then calculates the optimal offloading and SR choices, represented by binary variables $s_1$ and $s_2$, which indicate whether to offload or not, and whether to apply SR or not, respectively. Once the optimal values of $s_1$ and $s_2$ are determined, the decision controller sends the results to the MDs. The MAR clients then execute their MAR tasks based on the received decisions. The MEC server processes the offloaded MAR tasks and applies SR as required. This process flow is detailed in Figure 2. Importantly, as illustrated in Figure 2, some MAR clients may choose to offload their MAR tasks (4.1), while others may process their tasks locally (4.2). In the flow of 4.1, MAR clients can take advantage of SR from the MEC server to enhance the resolution of the offloaded images, provided that the additional computational latency introduced by the SR process is within acceptable limits. If the overall latency exceeds acceptable limits, MAR clients will offload their tasks without requesting SR. In this context, the primary objective of the proposed framework is to balance the overall energy consumption, acceptable latency, and service satisfaction from the MAR service throughout the process. The key notations used in this paper are summarized in Table 2.

Figure 2. Task flow in proposed system.

Table 2. Parameter description.

Symbol	Description
n, N	Index of MAR client, number of clients
B, $B_n$	System bandwidth, bandwidth allocated to client n
$f_n$, $f_{MEC}$	Available CPU frequency of client n and MEC server
${\omega }_n$, $d_n$	Task workload and input data size of client n
$R_n$	Source image resolution of client n (defined as the number of pixels)
$s_1$, $s_2$	Offloading decision factor, super-resolution decision factor

3.2. Analytical Models

In this subsection, we present analytical models to evaluate the overall latency and energy consumption of the proposed system. To achieve this, we first conduct an empirical study to analyze the computational latency of SR in relation to different source resolution levels.

3.2.1. SR Latency Analytics

As observed, SR is a type of GAI technology that uses neural networks to upscale low-resolution images to high-resolution images. While SR enhances the accuracy of AR services by providing higher-resolution images, it also imposes a significant computational burden, which can increase the overall latency in MAR. Understanding the relationship between SR latency and other parameters is crucial for optimizing the efficiency of MAR services.

SR latency is significantly affected by the SR model used, the size of the input image (or its source resolution), and the available computing resources. Given that computing resources are limited and a more advanced SR model demands additional resources, the source resolution becomes the key parameter for influencing the SR latency. To optimize the SR computation latency, it is essential to understand how the resolution of the source image impacts this latency. Figure 3 illustrates the empirical results on how input image resolution influences SR computation latency. In this study, we employed two SR models: EDSR, based on the ResNet architecture, and SRGAN, based on the GAN framework. The $Urban100$ dataset from [26] was used for experimentation. As shown in Figure 3, increasing the resolution of the source image leads to a higher SR computation latency, following a consistent pattern across both SR models for the same upscaling factor. This allows us to define the relationship between the input image resolution R, denoted as the number of pixels per height, and the SR computation latency $l_{SR}$ (ms). Here, $l_{SR}$ increases as a function of R because the computational workload grows with R. However, due to the complexity of analytically determining this relationship, a data-driven approach is more practical. This involves using offline training with empirical data to model the relationship. We employed regression-based modeling, a common technique seen in areas like mobile CPU property modeling, including CPU power and temperature variations [21]. Thus, we conclude that $l_{SR}$ can be modeled as

$$l_{SR}=\alpha R^2+\beta R+\gamma ,$$

(1)

where $\alpha ,\beta ,\gamma$ are hyperparameters decided by the SR model. This equation (Equation (1)) makes quantifying the latency of SR computation possible via input image resolution.

Figure 3. The impact of input image resolution on the SR computational latency.

3.2.2. Network Latency and Energy Consumption

The network latency and energy consumption are determined by the MAR client’s offloading data size. We consider the wireless network with an orthogonal frequency-division multiple access (OFDMA), and we assume that the MEC server provides a total bandwidth fairly over MAR clients such that $B_n$ for client n equals $B/N$. Let $t{r}_n^{up}$ and $t{r}_n^{down}$ denote the transmission rate of uplink and downlink for client n and base station, respectively. Then, the uplink and downlink transmission rates can be calculated by

$$t{r}_n^{up}=B_n{log}_2(1+{\displaystyle \frac{p_n{h}_{up}}{{\sigma }^2}}),$$

(2)

$$t{r}_n^{down}=B_n{log}_2(1+{\displaystyle \frac{p_m{h}_{down}}{{\sigma }^2}}),$$

(3)

where $p_n$ and $p_m$ denote the transmission power of MAR client n and base station, respectively, $h_{up}$ and $h_{down}$ denote the channel gain of uplink and downlink, respectively, and ${\sigma }^2$ denotes the noise power. Based on Equations (2) and (3), the transmission latency of uplink and downlink is calculated by

$$l_n^{up}={\displaystyle \frac{d_n}{t{r}_n^{up}}},$$

(4)

$$l_n^{down}={\displaystyle \frac{d_m}{t{r}_n^{down}}},$$

(5)

where $d_m$ is the output data size after execution in the MEC server. The transmission energy consumption of each MAR client is composed of input data size and energy consumption per bit. We do not consider downlink energy consumption since our framework aims to minimize the energy consumption of whole clients except for the MEC server. Thus, transmission energy is given by

$$e_n^{up}=l_n^{up}{p}_n.$$

(6)

3.2.3. Computation Latency and Energy Consumption

With SR decision factor $s_2=\{s_2^1,s_2^2,\dots ,s_2^n,\dots s_2^N\}\in \{0,1\}$ for all n. In this context, $s_2^n=1$ indicates that the MAR client requests the SR process to upscale the image, while $s_2^n=0$ signifies that the MAR client opts not to use the SR process. Computation in MEC is composed of an offloading task and an SR task defined as Equation (1). Let $l_{MEC}^c$ denote the latency of the MEC computation about client n, which is calculated by

$$l_{MEC}^c={\displaystyle \frac{{\omega }_n}{f_{MEC}}}+s_2^n(\alpha R_n^2+\beta R_n+\gamma ),$$

(7)

where ${\omega }_n$ denotes the data size of the task workload offloaded from MAR client n. Therefore, the latency and energy consumption of computation are defined by

$$l_n^c={\displaystyle \frac{{\omega }_n}{f_n}},$$

(8)

$$e_n^c={\displaystyle \frac{{\mu }_n}{2}}{\left(f_n\right)}^2{\omega }_n,$$

(9)

where ${\displaystyle \frac{{\mu }_n}{2}}$ denotes the energy coefficient of MAR client n’s device chipset. The energy consumption of computation is defined as the product of the energy coefficient, the square of CPU frequency, and the size of task workload [13].

3.2.4. Total Latency and Energy Consumption

Total latency and energy consumption can be described via the sum of the network part and computation part. Therefore, the total latency and energy consumption of MAR client n is calculated by

$$l_n=s_1^n(l_n^{up}+l_n^{down})+(s_1^n{l}_{MEC}^c+(1-s_1^n)l_n^c),$$

(10)

$$e_n=s_1^n{e}_n^{up}+(1-s_1^n)e_n^c.$$

(11)

where offloading decision factor is defined as $s_1=\{s_1^1,s_1^2,\dots ,s_1^n,\dots s_1^N\}\in \{0,1\}$ for all n. In this context, $s_1^n=1$ indicates that the MAR client requests task offloading to compute MAR tasks with the MEC server’s resources.

4. Problem Formulation and Solution

4.1. Problem Formulation

Based on the total latency and energy consumption model (10) and (11), we formulate the multi-objective optimization problem to balance the energy consumption of MAR clients, total latency, and the service satisfaction, which are the trade-off relationship. Such a trade-off relationship is expressed in multi-objective optimization problems by assigning weighted sums to each metric, as widely modeled in the literature, such as in [21]. Here, we define the service satisfaction of MAR client n, denoted as $Q_n$, as

$$Q_n=s_2^n{\displaystyle \frac{S_n}{R_n}}$$

(12)

where $S_n$ represents the satisfaction attained by MAR client n from utilizing the SR, which enhances the detection accuracy by upscaling the resolution. However, since the input resolution, $R_n$ is already sufficiently high to achieve acceptable accuracy, the additional satisfaction gained from the SR will be diminished. Moreover, in our problem, latency does not necessarily have to be minimal but does not exceed an acceptable latency $L_n$. To formulate this concept, we design a simple latency cost function as

$$cost\left(x\right)=\left\{\begin{array}{cc}0\hfill & \mathrm{if}-L_n\leqq x\leqq 0\hfill \\ \theta x\hfill & \mathrm{if}x>0\hfill \end{array}\right.$$

(13)

where $x=l_n-L_n$ is given. This cost function means that the cost value is 0 if the total latency $l_n$ is smaller than the latency constraint $L_n$. If not, the cost value increases linearly via hyperparameter $\theta$. Then, we can design the joint optimization problem of task offloading $s_1$ and SR decision $s_2$ for energy-efficient MAR. Since the problem has the integer variables of $s_1$, and $s_2$, it is a non-convex optimization problem. To solve this problem, we relax the integer variables of $s_1$ and $s_2$ into a continuous value. Then, we can design the multi-objective optimization problem as follows:

$\mathbf{P}\mathbf{1}:$

$$\underset{s_1^i,s_2^i}{\min }\hspace{1em}\hspace{1em}\sum _{i=1}^N{w}_1{e}_i+w_{2}cost(l_i-L_i)-Q_i$$

(14a)

$$\mathrm{s}.\mathrm{t}.\hspace{1em}\hspace{1em}0\le s_1^i,s_2^i\le 1,s_1^i-s_2^i\ge 0.$$

(14b)

where $w_1$ and $w_2$ are the weight factors for balancing energy consumption and latency as well as service satisfaction. In the constraints, constraint (14b) means the value of offloading decision factor $s_1^i$ and SR decision factor $s_2^i$ is between 0 and 1, and SR works only if the MAR client declares the offloading since the local computing does not need to control the resolution of source images from the SR process. The detail of resolution levels is described in Table 2.

4.2. Optimization Solution

Since the latency cost(.) function in (13) is non-differentiable, we need to convert this cost(.) function into the $max(.)$ form to translate the problem into LP by using the relaxation technique, which is $max(0,\theta T)$, where $l_i-L_i$ is T. Then, we transform the $max(.)$ form into an affine function by introducing an auxiliary variable $z_i=max(0,\theta T)$. Finally, the $\mathbf{P}\mathbf{1}$ can be translated to as follows:

$\mathbf{P}\mathbf{2}:$

$$\underset{s_1^i,s_2^i,z_i}{\min }\hspace{1em}\hspace{1em}\sum _{i=1}^N{w}_1{e}_i+w_2{z}_i-Q_i$$

(15a)

$$\mathrm{s}.\mathrm{t}.\hspace{1em}\hspace{1em}0\le s_1^i,s_2^i\le 1,s_1^i-s_2^i\ge 0$$

(15b)

$$\hspace{1em}0\le z_i,\theta T_i\le z_i$$

(15c)

As shown above, variable $z_i$ must also be optimized to solve $\mathbf{P}\mathbf{2}$. It makes our problem more complicated. To address $\mathbf{P}\mathbf{2}$, we illustrate a lemma for solving our problem in an alternative way.

Lemma 1. 

$\mathbf{P}\mathbf{2}$ is a linear programming (LP) with respect to optimization variables $s_1^i$, $s_2^i$, and $z_i$, respectively.

Proof. 

Regarding $s_1^i$, the objective function of $\mathbf{P}\mathbf{2}$ takes a linear form because the other variables are constants relative to $s_1^i$. Additionally, constraint (15b) is affine with respect to $s_1^i$, and the remaining constraints are independent of $s_1^i$. As a result, $\mathbf{P}\mathbf{2}$ is a linear program (LP) in terms of $s_1^i$, since both the objective function and the inequality constraint are affine functions. A similar argument applies to $s_2^i$ and $z_i$.    □

According to Lemma 1, we can solve $\mathbf{P}\mathbf{2}$ using the block coordinate descent algorithm [27]. That is, for a given initial value of other optimization variables except the target optimization variable, the optimal $s_1$, $s_2$, and z can be calculated via the $Simplex$ $algorithm$ (SA). Consequently, we can obtain all optimal values after an iteration that solves one variable by fixing each other until the result of the objective function converges below the threshold. The procedure of the proposed block coordinate descent algorithm is summarized in Algorithm 1.

Algorithm 1 Proposed optimization algorithm.

Input: B, $f_n$, $d_n$, $R_n$, $p_n$, ${\omega }_n$, $h_n$, $L_n$, $\sigma$, $f_{MEC}$, $\alpha$, $\beta$, $\gamma$, ${\mu }_n$, difference threshold $df$ Initialize: Initial $s_1$ and $s_2$ are initialized in 1. Initial cost $C_0$ is a large number. Output: Optimal $s_1$, $s_2$, R, z    Initialize t = 0    for all i in $1,2,\dots ,n$ do       while True do          $s_1^i$← solving $Prob.2$ with fixed $s_2^i$, $z_i$          $s_2^i$← solving $Prob.2$ with fixed $s_1^i$, $z_i$          $z_i$← solving $Prob.2$ with fixed $s_1^i$, $s_2^i$          Calculate current cost $C_{t+1}$          if $|C_{t+1}-C_t|<df$ then               $s_1^i\ast \leftarrow s_1^i$               $s_2^i\ast \leftarrow s_2^i$               $z_i\ast \leftarrow z_i$               break          end if          $t=t+1$       end while    end for    return $s_1^{*}$, $s_2^{*}$, $z_i*$

As demonstrated in [28], the block coordinate descent method used in Algorithm 1 exhibits a sublinear convergence rate. Additionally, each block, formulated as the LP problem, can be efficiently solved using the SA, which has polynomial complexity denoted as $O\left(N^3\right)$, where N represents the number of decision variables (here, the number of MDs). Therefore, the method is practically deployable, as the number of MDs connected to a single MEC server is typically within a reasonable range.

4.3. Discussion for Practical Considerations

As a practical consideration, we conduct experiments to analyze the SR effect on the MAR system in terms of the latency and object detection accuracy. We implement all modules, including our proposed framework, on the MEC side using Python and PyTorch. Specifically, to establish a practical MAR and SR environment, we implement the object detection module using the OpenCV library and the ultralytics YOLOv8 model [29], while the SR module is implemented using the EDSR model [30]. The client node sends object detection requests to the proposed framework, and after the MEC server node processes the object detection tasks based on the SR decision from our framework, the client displays the results using the OpenCV library. The overview of our implementation is shown in Figure 4.

Figure 4. The result of object detection on implemented framework.

To evaluate the effect of SR on object detection, our implementation collects three metrics from the object detection results using ultralytics YOLOv8 built-in methods: image preprocessing time, inference time, and confidence score. We compare the original low-resolution video case to the video with the SR case through the three metrics above. Image preprocessing time and inference time are used to analyze the impact on the object detection task’s latency regarding whether SR is applied or not, and confidence scores are used to analyze the impact on the object detection task’s accuracy. The higher the confidence score, the higher the accuracy. The comparison results about performance metrics are shown in Table 3. As shown in Table 3, the task execution time increases when SR is applied, and the confidence score also increases. That is, applying SR in the MAR system causes improvement in terms of accuracy but degeneration in terms of service latency. Also, depending on the trade-off between the latency decrement and the accuracy increment, users’ satisfaction with MAR applications such as QoS may vary. Therefore, the satisfaction model design in our framework is effective in a practical MAR environment.

Table 3. The performance metrics’ comparison of YOLO implementation.

	Image Preprocessing Time	Inference Time	Confidence Score
Original video	Avg: 0.67 ms, Std: 0.95 ms	Avg: 1.53 ms, Std: 0.175 ms	Avg: 76.21, Std: 7.57
Video with SR	Avg: 7.72 ms, Std: 0.479 ms	Avg: 1.81 ms, Std: 0.1 ms	Avg: 88.47, Std: 9.12

5. Performance Evaluation

5.1. Simulation Setup

In this section, we present simulation results to validate the effectiveness of the proposed optimization scheme, comparing it to five benchmarks.

• Benchmark 1—All local [31,32,33]: All MAR clients compute MAR tasks locally, without MEC offloading and SR.
• Benchmark 2—All offload only [31,32,33]: All MAR clients offload MAR tasks to the MEC server without the SR process.
• Benchmark 3—All offload and SR: All MAR clients request not only task offloading but also the SR process.
• Benchmark 4—Minimum SR: Offloading decisions are optimum, but only the clients that have the lowest resolution data request SR tasks. It is one of the rule-based approaches for SR control.
• Benchmark 5—Random [31,32,33]: The number of clients that request task offloading and SR is randomly decided.

In our simulation, the total number of mobile devices is set to [5, 10, 15, 20, 25] [34] to analyze the tendency from the number of clients (i.e., MDs). Each MD is assigned a single MEC server with the wireless channel bandwidth $B=75$ MHz [13] and the computation resource $f_{MEC}=15$ GHz. Also, each MD is allocated bandwidth resources and computation resources. The transmission power ranges from $p=50$ mW to 100 mW randomly, and background noise power $\sigma =-100$ dBm [34]. The workload density of computation tasks ${\omega }_n$ is randomly distributed between 500 and 1000 cycles per bit [13], and the data size of video frame size $d_n$ is randomly distributed between 500 and 3000 KB [20]. Each local device used by the MAR client has its own computing resource $f_n=1$ to $1.8$ GHz and a delay threshold of AR application $L_n=20$ to 25 ms. The data size of offloaded tasks is 80% of the original tasks, although the proposed algorithm adopts the binary offloading decision in this paper. This is because the MAR tasks consist of a sequence of subtasks, and some subtasks cannot be offloaded to the MEC server [13].

5.2. Performance Evaluation

To evaluate the impact of weight parameters for latency and energy consumption, we first evaluate the change of latency and energy consumption with different weight parameters $w_1$ and $w_2$ in fixed client number 15. We identify trends by measuring the latency and energy consumption with the proposed algorithm while changing only as the weight parameters $w_1$ and $w_2$ in the above experimental environment. As shown in Figure 5, the weight parameter for latency $w_1$ increases and the weight parameter for energy consumption $w_2$ decreases, and the computation latency decreases and the energy consumption of MAR clients increases. On the contrary, as $w_2$ increases and $w_1$ decreases, the energy consumption of MAR clients increases, and the computation latency decreases. This means that latency and energy consumption are in a trade-off relationship regarding weight parameters $w_1$ and $w_2$. Since this paper aims to balance the latency, energy consumption of MAR clients, and service satisfaction, we choose weight parameters $w_1=0.5$ and $w_2=0.5$ for performance evaluation.

Figure 5. The impact of latency and energy consumption on the change of weight parameters.

Next, we evaluate the performance of the proposed algorithm and benchmarks in terms of latency, energy consumption, and total cost that we defined with two different SR methods, EDSR and SRGAN. First of all, Figure 6 shows the result of the proposed total cost model from the proposed algorithm and benchmarks with the EDSR SR model. The proposed algorithm achieves the minimum values at the cost function compared to benchmarks. This is because the proposed cost model considered not only latency and energy consumption but also the satisfaction of service quality obtained from video resolution $Q_n$ by balancing all those factors. This is why the proposed algorithm achieves the minimum cost despite the proposed algorithm not gaining the definite minimum value in latency and energy consumption. Finally, the reason why benchmark 4 achieved a higher total cost than our proposed algorithm is that it is difficult to accurately adapt the satisfaction caused by SR process with a rule-based algorithm. Therefore, we can conclude that optimizing the offloading and SR decision leads to the balance of the computation latency, energy consumption, and service quality.

Figure 6. Total performance of proposed cost model per number of MAR users (MMN clients) with EDSR SR model.

Figure 7 and Figure 8 show the detailed results in terms of latency and energy consumption from the proposed model and benchmarks with the EDSR SR model. As shown in Figure 7, as the number of clients increases, the latency also increases, except for all local computing environments named benchmark 1. The main reason is that the offloading strategies including the proposed algorithm are greatly affected by the available network and computing resources of the MEC server. In other words, as the number of MAR clients increases, the allocated bandwidth and computing resource for each client decreases. However, the latency of benchmark 1 only affects the local computing resource of each client. In Figure 8, we present the energy consumption generated by different offloading and SR strategies. It can be seen that the number of clients does not greatly impact energy consumption. This is because the energy consumption of MAR clients is largely affected by the task computation energy rather than the transmission energy. It means that a client’s energy consumption is mainly determined by their available computing resources. Also, it can be seen that the proposed algorithm shows an almost identical average value as another offloading strategy. This is because all offloading strategies including the proposed algorithm, benchmark 2, and 3 adopt the same binary offloading strategy, and the proportion of task computation energy is larger than transmission energy.

Figure 7. Latency of proposed cost model per number of MAR users (MMN clients) with EDSR SR model.

Figure 8. Energy consumption of proposed cost model per number of MAR users (MMN clients) with EDSR SR model.

Figure 9, Figure 10 and Figure 11 present the average results for scaled proposed cost, latency, and energy consumption when using the SRGAN model. The SRGAN model requires more computational resources but provides more accurate SR results compared to the EDSR model. As illustrated in Figure 9, all strategies result in a higher cost than EDSR, due to increased satisfaction. However, the average latency also rises, as shown in Figure 10. Despite this, Figure 10 and Figure 11 indicate that the trends in latency and energy consumption are almost identical to those observed with the EDSR model.

Figure 9. Total performance of proposed cost model per number of MAR users (MMN clients) with SRGAN SR model.

Figure 10. Latency of proposed cost model per number of MAR users (MMN clients) with SRGAN SR model.

Figure 11. Energy consumption of proposed cost model per number of MAR users (MMN clients) with SRGAN SR model.

6. Discussions for Future Works

Our proposed framework has proven to be useful in improving the performance of the MAR system; however, there are some points for discussion or further development.

• Expansion on the GAI aspect: In this paper, we focus on the SR aspect of GAI to improve MAR performance. However, there are many GAI solutions to adapt to the MAR system to improve MAR performance, and the effects of these GAI solutions are designed differently from SR. In the real-world MAR environment, different GAI solutions can be applied depending on the different requirements of the application since other GAI solutions have their unique advantages and requirements [1]. For example, the diffusion model, one of the GAI algorithms, generates images from text descriptions, even when the training dataset does not include the specific images described. This algorithm can be applied to various MAR applications that require flexible qualities or specifications. If the proposed framework with SR is advanced, it may be possible to consider expanding to other GAI technologies.
• Expansion on the problem formulation and optimization method: In this paper, we formulated a multi-objective optimization problem to balance the trade-off between latency, energy consumption, and service satisfaction. We solved this problem using the block coordinate descent method, as it allows us to address the problem mathematically with minor modifications. However, our approach has the limitation of not enabling real-time decision making. Since the real-world MAR environment is dynamic, solutions that lack real-time adaptability are not applicable to practical MAR systems. To address this, our problem can be extended to a real-time decision-making framework using a Lyapunov-based, round-wise drift-plus-cost minimization approach. Additionally, key real-world factors such as heterogeneity among MEC servers and user mobility should be incorporated. As the problem becomes more complex and advanced, the optimization solutions will also require more sophisticated techniques. As discussed in Section 2, heuristic algorithms or AI-based solutions may be well suited for solving these more complex scenarios.
• In-depth understanding of SR adaptation to MAR framework: Although we aim to enhance the user experience for MAR users through the application of SR, there is limited analysis on the effects of SR in MAR systems, such as distortions or loss of detail in SR results, as well as the overall impact of SR on users. Unfortunately, few studies have applied SR to MAR systems, and those that have often underestimated the challenges associated with SR [24], despite extensive research addressing these challenges in other fields [35,36]. For instance, some research suggests novel training methods using a distortion-aware network, as higher distortion in source images can degrade the performance and accuracy of SR. These studies explain that image distortion in AR is caused by the degradation process [35], changes in viewing range, and sphere-to-plane projection [36]. Moreover, as mentioned earlier, SR offers unique advantages that set it apart from other GAI technologies [1], and clearly demonstrating these differences will have important implications. In our future work, we plan to conduct more detailed analyses and evaluations to address these gaps.

7. Conclusions

In this study, we investigated GAI-enabled energy-efficient MAR in MEC environments, where SR is employed as one of the promising GAI technologies. Specifically, to balance the energy consumption of each MAR user, overall latency, and service satisfaction, the proposed algorithm designed a joint management framework for offloading and SR decisions in MEC-assisted MAR systems. This framework considered the trade-off between SR overhead in terms of the computational latency and the improved satisfaction resulting from higher image resolution. In our performance evaluation, we validated the effectiveness of the proposed algorithm in optimally managing offloading and SR decisions, demonstrating its ability to balance latency, energy consumption, and service satisfaction compared to benchmarks. Additionally, we discussed the practical deployment of the proposed scheme by presenting accuracy improvement tests related to practical object detection and SR implementation. As a future work, we plan to extend our research into developing a real-time decision-making algorithm utilizing Lyapunov-based round-wise drift-plus-cost minimization problems, leveraging the virtual queue concept. Specifically, more complex aspects such as heterogeneity among MEC servers and user mobility will be considered. Furthermore, a more rigorous implementation of real-world MAR applications will be explored to evaluate the extended work.