An Advanced Job Scheduling Algorithmic Architecture to Reduce Energy Consumption and CO2 Emissions in Multi-Cloud

Abstract

Cloud Computing is one of the emerging fields in the modern-day world. Due to the increased volume of job requests, job schedulers have received updates one at a time. The evolution of machine learning in the context of cloud schedules has had a significant impact on cost reduction in terms of energy consumption and makespan. The research article presents a two-phase process for the scheduling architecture of cloud computing where PMs are the main working unit and users are supplied to the PMs based on the work abilities of the PM in terms of resources. A minimum cost is desired in the preliminary phase of the allocation of the user to the PM. A clustered approach utilizing k-means and Q-learning was imposed to migrate the users from one PM to another PM based on Quality of Service (QoS) parameters. The proposed work has also incorporated CO2 emissions as a major evaluation parameter other than energy consumption. To support resource sharing, the deployment model is a multi-cloud model. The proposed work is evaluated against other recently proposed state of the art techniques on the basis of QoS parameters and the proposed work proved to be better in terms of efficiency at the end of the draft.

1. Introduction

Cloud Computing (CC) has been one of the most emerging areas in the computation world where the users are associated with a cloud platform to get services from it. For any computation platform, the requests are viewed as jobs and are processed under the datacenter of CC. A CC data center is a physically measured component that uses computational elements to process requests from users. Physical Machines (PMs) are the main handler of the jobs and receive instructions from the data center. For example, Netflix is a cloud platform that provides a significant number of movies to its associated and authorized users and Netflix uses hardware resources to store and process the computation of the jobs. A data center can consume the same amount of energy as that consumed by 25,000 households if they are left in working mode for an hour. Hence, energy consumption and carbon emissions are a few of the serious issues in the CC architecture. In general, a CC architecture is made up of three layers namely Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). The data center handles the jobs and assigns the jobs at the IaaS layer through the SaaS layer. In the early days of development, a CC architecture was only designed to speed up the efficiency of the computation but due to increasing load and energy consumption through cloud data centers, energy efficacy has become a vital issue. Jobs are supplied to the datacenter with a constraint deadline and hence several algorithm architectures illustrate the operation based on execution time [1]. For example, Minimum Execution Time (MET) selects the PM based on the time contracts that are associated with the job. When it comes to connected jobs, viz. if the dependency of the job is on the outcome of another job, it also becomes vital to execute the dependent jobs first before executing the primary job. Based on the connected job models, the CyberShake algorithm was introduced to the world in early 2013 and has received updates over time [2]. The CyberShake algorithm uses a dependency map based on Heterogenous Earliest Finish Time (HEFT) to check the overall cost of execution. The CyberShake algorithm utilizes Statistical Machine Learning (S-ML) for the evaluation of the total cost of the job over different physical machines or data centers. Machine learning has spread into various sectors of computation including cloud computing and big data analytics [3]. Machine learning works on the behavior pattern of the entire process and requires a bulk amount of data to learn and to reach a conclusion. The ordinal measure of machine learning, Minimum Execution Time (MET), and the CyberShake algorithm are provided below.

1.1. MET

MET is one of the simplest algorithm architectures for the scheduling of jobs in a cloud server. MET looks for the minimum execution time based on the total number of buffers in the buffer list and a total number of jobs in each buffer. Based on the computational complexity of the job, the total time to execute the job is calculated. Each PM submits its cost to the scheduler and based on the minimum cost, the PM is selected for the allocation. One of the major advantages of this algorithm architecture is its computational complexity but at the same time, it does not create or utilize the history of allocations. These characteristics bar the algorithm architecture from expanding in terms of elasticity [4].

1.2. ML in Cloud Computing

The cloud computing industry has been completely upended by the emergence of multi-clouds as a successor to CC architectures. Using this idea, the performance of different cloud designs can be improved by having multiple cloud components pool their resources together.

In multi-cloud computing, where resources are frequently underutilized due to a lack of effective resource allocation methodologies, efficient resource allocation is a crucial challenge. The current methods have difficulty optimizing both available and utilized cloud resources, which can result in costly performance inefficiencies for end users. The purpose of this work is to develop and apply a novel learning pattern that utilizes both ideal and active machines to optimize resource use.

The major objectives have been outlined to this end.

• To begin, a labeling architecture will be created so that the system may be trained to recognize labels of interest.
• Second, a Q-learning-based pattern of behavior analysis is used to help make decisions at any stage of the work distribution process.
• Finally, the suggested method will be investigated for quality of service (QoS) characteristics and compared with state of the art algorithms to demonstrate its superior performance.

When applied to multi-cloud computing environments, the proposed learning pattern has the potential to greatly increase the efficiency with which computer resources are allocated. Compared to standard practices, the learning pattern can save time, cost, and effort by making use of both unused and underutilized cloud computing resources. The overarching goal of this effort is to present a more efficient and effective method for allocating multi-cloud computing resources that will be cost-effective for end users and boost overall performance.

The rest of the paper is organized in the following manner. Section 2 contains a literature survey covering job allocation and handling architectures. It also incorporates modeling improvements through machine learning and the evolution of machine learning in the field of multi-cloud architecture systems. Section 3 describes the proposed system model along with the implementation design. The evaluation of the results and the comparison with other state of the art techniques is demonstrated in Section 4 and the paper is concluded in Section 5.

2. Related Works

In recent years, cloud computing’s popularity has skyrocketed due to its capacity to supply customers with versatile and inexpensive computing resources. Multi-cloud computing, in which many cloud providers are employed to host applications, increases the potential benefits of cloud computing. The benefits include increased scalability, dependability, and cost-effectiveness. However, there are also substantial obstacles to overcome when it comes to optimizing resource allocation in multi-cloud computing. There is a need for more advanced methodologies that can optimize resource allocation across several clouds in a way that is more energy-efficient, cost-effective, and environmentally friendly than traditional approaches to multi-cloud computing resource allocation.

While several researchers have developed models for allocating resources in multi-cloud environments, these approaches frequently have their own set of problems.

Some models, for instance, rely on a single person making all the decisions, which might introduce instabilities and prevent the system from expanding. Some models may not account for the heterogeneity of cloud resources, which can lead to inefficient use of those assets. The majority of the currently available models also fail to account for fluctuations in the job, which can lead to subpar performance and unnecessary resource consumption. The contributions of several researchers in this field are as follows.

Hu et al. (2018) proposed a multi-objective scheduling algorithm for multi-cloud environments to reduce the workflow makespan and scheduling cost of data centers. Their approach utilized Particle Swarm Optimization to customize job scheduling based on location and order of data transmission. Although their simulation study showed promising results, their algorithm did not consider the impact of renewable energy sources or the carbon footprint of data centers [1]. Xu and Buyya (2020) proposed a workload shift approach that addressed the CO2 emissions issue by shifting jobs among multi-clouds located in different time zones. They also utilized renewable energy sources, such as solar energy, to reduce the usage of non-renewable energy. Their approach was effective in reducing CO2 emissions by 40% while maintaining a near-average response time for user requests. However, their approach did not consider load balancing or resource utilization, which are important factors in multi-cloud environments [2]. Cai et al. (2021) proposed a distributed job scheduling approach for multi-cloud environments that considered multiple objectives, including cost, energy consumption, time consumed, throughput, load balancing, and resource utilization. Their approach utilized intelligent algorithms based on the aforementioned objectives, as well as a sine function for model implementation. Although their simulation analysis demonstrated high scheduling efficiency with enhanced security, their approach did not consider the impact of renewable energy sources on energy consumption or CO2 emissions [3]. Renugadevi and Geetha (2021) developed a model for a geographically distributed multi-cloud environment that utilized solar energy as the main source of energy. Their model considered electricity prices, favorable location, CO2 emissions, and carbon taxes in energy management and resource allocation. The type of task was customized in response to the deadline of the task, which resulted in adaptive management of the multi-cloud model and workload algorithm. However, their approach did not consider load balancing or resource utilization, which are critical factors for optimal performance in multi-cloud environments [4]. Gaurang Patel et al. (2015) present study of task scheduling algorithms and modification of load balanced min-min algorithm. The proposed algorithm is based on survey of load balancing algorithm for static meta-task scheduling in grid computing. It select task based on maximum completion time and improve resources utilization and makespan [5]. Zhang et al. (2021) proposed a distributed deployment approach for tasks in multi-cloud environments, based on reinforcement learning. Their approach performed two steps: job offloading and task scheduling based on cloud center decisions. Their simulation analysis showed that their approach had the least latency in a heterogeneous environment, compared to existing approaches. However, their approach did not consider the impact of energy sources on energy consumption or CO2 emissions [6]. Sangeetha et al. (2022) utilized deep learning Neural Networks to control routing capabilities in multi-cloud environments to manage space and task allocation. Their approach minimized the delay associated with data processing and storage across clouds. Their simulation analysis demonstrated improved performance in terms of delay and costs associated with resource allocation in multi-cloud environments. However, their approach did not consider the impact of renewable/non-renewable energy sources on energy consumption or CO2 emissions [7]. Cao et al. (2022) presented the carbon footprint of data centers as a major source of intensive CO2 emissions and suggested that data centers increasingly switch to renewable energy to minimize the negative effects of CO2 emissions and boost energy circulation. They further suggested the integration of Artificial Intelligence to address the challenges and put the framework in place in real-world scenarios. However, they did not propose a specific approach to integrate AI or address the challenge of load balancing or resource utilization in multi-cloud environments [8]. Jun-Qing Li et al. (2021) proposed a hybrid technique of greedy and simulated annealing for scheduling crane transportation process. The objective was to reduce the completion time and energy consumption. It works in two steps: the first step is the scheduling of jobs and the second step is the assignment of machines [9]. Yu Du et al. (2022) develop a multi-objective optimization algorithm for distribution algorithm estimation and deep q-network to solve the problem of scheduling job shops. The problem was the processing speed of the machine, idle time, setup time, and transportation between machines. The proposed model was validated on CPLEX. The results showed that the proposed method performed effectively for job shop scheduling [10]. Ju Du (2022) presented a deep Q-network (DQN) model for multi-objective flexible job shop scheduling for crane transportation and setup times. It included 12 states and 7 actions for the scheduling process. The result showed that the proposed method produced an effective and efficient output. DQN also has the option to use dispatching rules according to situations [11]. Haider Ali et al. (2021) explored IOT and its application in the area of smart cities and also discussed wireless sensors. The authors also discussed the multiprocessor chip on-chip (MPSoC) used in daily IOT-based gadgets. At the end of this survey, the author also mentioned the future directions [12]. Umair Ullah Tariq et al. (2021) proposed a novel energy-aware scheduler with constraints on Voltage Frequency Island (VFI)-based heterogeneous NoC-MPSoCs deploying re-timing integrated with DVFS for real-time streaming applications. The authors proposed R-CTG which is a task-level timing approach integrated with non-linear programming and a voltage-level approach. The comparison results showed that the proposed method performed better in terms of latency and energy consumption [13].

More recent efforts have aimed to optimize multi-cloud resource allocation using machine learning and deep reinforcement learning (DRL) approaches. However, there is still a large knowledge gap in the application of DRL to multi-cloud resource allocation, especially in the areas of energy efficiency, CO2 emissions, and cost optimization.

We present a unique DRL-based methodology for optimal resource allocation in multi-cloud computing that accounts for energy efficiency, CO2 emissions, and cost. We consider the dynamic and complicated character of multi-cloud computing systems, and our proposed model uses DRL algorithms to learn optimal resource allocation techniques in real time. Extensive experimental and simulated results show that our suggested approach provides significant improvements over the current state of the art methods in terms of energy efficiency, cost, and carbon dioxide (CO2) emissions.

3. Materials and Methods

The proposed work is a novel work model that aims to reduce the overall energy consumption and carbon emission of the processing that must be performed to complete the supplied number of jobs. The proposed work uses the following definitions to illustrate the proposed work architecture; the important notations and symbols used in the proposed model are shown in

Table 1. Important Notations and Explanations.

Notation	Explanation
PMS	Physical Machines
QoS	Quality of Service
CC	Cloud Computing
IaaS	Infrastructure as a Service
PaaS	Platform as a Service
SaaS	Software as a Service
MET	Minimum Execution Time
HEFT	Heterogenous Earliest Finish Time
S-ML	Statistical Machine Learning
MIPS	Million Instruction Per Second
DC	Data center
ZL	Job list
S	Total number of clouds
R	Associated RAM
C	CPU utilization capacity
L	Location of jth
D	Total number of data centers in one cloud
N	Total number of PMs under one cloud
U	Total number of users under one cloud
dx and dy	Location of the data center
ux and uy	User location
BFD	Best Fit Decreasing
MBFD	Modified Best Fit Decreasing
VM	Virtual Machine
${\mathrm{E}}_{\mathrm{C}\mathrm{P}\mathrm{U}}$	Energy consumed by the CPU in the physical computer
${\mathrm{E}}_{\mathrm{m}\mathrm{e}\mathrm{m}\mathrm{o}\mathrm{r}\mathrm{y}}$	Energy consumed by the memory in the physical server
${\mathrm{E}}_{\mathrm{f}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{d}\mathrm{s}\mathrm{e}\mathrm{r}\mathrm{v}\mathrm{e}\mathrm{r}}$	Fixed power consumed by the server
${\mathrm{E}}_{\mathrm{d}\mathrm{y}\mathrm{n}\mathrm{a}\mathrm{m}\mathrm{i}\mathrm{c}}$	Dynamic energy consumed
${\mathrm{P}}_{\mathrm{d}\mathrm{y}\mathrm{n}\mathrm{a}\mathrm{m}\mathrm{i}\mathrm{c}}$	Dynamic power consumed
${\mathrm{T}}_{\mathrm{C}\mathrm{O}2}$	Total emission of CO2
$\mathrm{E}{\mathrm{C}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}$	Total power consumed by the machines
I	Intensity of emission of greenhouse gases
M	Total number of instructions under 1 PM
Ece	Execution cost at jth PM
S	Storage required for mth instruction
Ecs	Total consumed energy in the storage of one instruction at jth PM
Cee	CO2 emitted in execution of mth instruction set on jth PM
Ces	CO2 emitted in the storage
EC	Energy consumption
CO2	Carbon emission
D	Distance between the user and the location of the datacenter
M	Total supplied instruction set from users
T	Current state value
t − 1	Previous state value
$\mathrm{E}{\mathrm{C}}_{\mathrm{i}\mathrm{u}}$	Unit cost of execution one MIPS
$\mathrm{E}{\mathrm{C}}_{{\mathrm{i}}_{\mathrm{u}\mathrm{s}}}$	Unit cost of storage of the executed instruction set
$\mathrm{C}\mathrm{O}{2}_{\mathrm{i}\mathrm{d}\mathrm{o}\mathrm{l}}$	Amount of emission released in idol state
$\mathrm{C}\mathrm{O}{2}_{{\mathrm{i}}_{\mathrm{u}\mathrm{s}}}$	Amount of emission released in storing the executed instruction set
$\mathrm{A}$	Unit cost to produce EC amount of energy to be consumed
$\mathsf{{\rm B}}$	Malicious processing cost
MOS	Multi Objective Scheduling
MOIA	Many Objective Intelligent Algorithm

.

The model proposed is a novel approach for reducing energy consumption and carbon emissions during the processing of a certain number of tasks. The model is based on a set of definitions and architectural components that assist in characterizing the relationships between the many relevant elements.

The ‘Z’ symbol represents the concept of a job, which is central to the model. A job is a Million Instructions Per Second (MIPS) sequence that is given to a data center (DC). The job set is denoted as $Z\in \left\{\mathrm{1,2}\dots \dots g\right\}$, ranges from 1 to $g$, $\mathrm{g}\ne \mathrm{\infty },$ and Zg $\subseteq \forall \mathrm{D}\mathrm{C}.{\mathrm{Z}}_{\mathrm{L}}$ where Z is the total number of given jobs, $\mathrm{Z}\mathrm{g}$ is the subset of Z_L for a given data center and Z_L DC indicates that the work list associated with each DC may vary. The DC divides jobs into execution machines, also referred to as PMs (processing machines). A PM is responsible for the sequential execution of one or more jobs. One PM may have more than two jobs in a row under the suggested approach. The model incorporates numerous clouds in addition to jobs and PMs. ‘s’ represents the number of clouds. A cloud is a cluster of PMs that have common attributes, such as RAM, CPU utilization capacity, and location.

Based on the component module, the following is used to represent a cloud structure:

$$Clou{d}_s\subseteq {\int }_{i=1}^d{\int }_{j=1}^n{PM\left(R,C,L\right)}_{i_j}dn$$

(1)

This equation demonstrates that a cloud is a collection of physical machines (PMs), with each $(PM$) aving its own RAM, CPU usage capacity, and data center location. $Clou{d}_s$ Denotes the cloud’s representation as a set.

The sum of PMs is represented by the integration symbols in the equation. The outer integral reflects the total number of data centers, ‘$d$,’ whereas the inner integral represents the number of PMs within each data center, ‘$n$’. The variable i indicates the data center number, and the variable ‘j’ represents the PM number for that data center. The notation ‘${PM\left(R,C,L\right)}_{i_j}$’ denotes the characteristics of the jth PM in the ith data center. ‘$R$’ Indicates the RAM connected to the PM, ‘$C$’ represents the CPU usage capacity, and ‘$L$’ reflects the PM’s location inside the data center.

Consequently, this equation is utilized to represent the cloud structure in the suggested model. The cloud s represents the collection of PMs (Processing Machines) associated with the ith data center’s RAM R, CPU utilization capacity C, and location L. The value of s represents the number of clouds in the system as a whole. The usage of this equation facilitates the definition of the cloud’s structure in a more ordered and systematic manner, which is essential for the effective administration of data centers and jobs.

Figure 1 show the important component of multi-cloud environment. The proposed work is divided into two subsequent segments from here. The first segment performs a pollination process to manage the $J_{L_{i_j}}$ based on the R, C, and L and executes the jobs based on the best fit decreasing (BFD) algorithm that was proposed by Baker in 1981 which has been further modified as the Modified Best Fit Decreasing (MBFD) in the later stage of the development of the CC architecture [2,14]. The first segment is termed as the preliminary allocation of the job to the data center execution machines based on the pollination process with improved fitness values of the pollination and the second segment is termed as advanced allocation which utilizes the propagated Q-learning method for further illustration. The illustration is as follows.

3.1. The Preliminary Allocation

The preliminary allocation utilizes the attribute set{R,C,L} as the primary measure considering CyberShake and MBFD as the base of allocation. To do so, the proposed work can be viewed with the help of the process diagram that is shown in Figure 2.

The preliminary allocation phase is divided into six steps and the number mentioned in this figure resperents the flow of working as follows:

• The user sends a request to the CC regardless of knowing the intercommunication architecture of the cloud. The cloud architecture has multiple clouds that are interconnected to each other to share resources.
• The CC transfers the request to the IaaS layer.
• The IaaS layer broadcasts the requirement to the job manager and the job manager sorts the jobs based on CPU utilization in descending order. The IaaS layer broadcast the sorted requirements to the data centers and the data centers compute the overall expected cost based on the CPU utilization.
• IaaS passes the responses of dc to the job manager and the job manager views the minimum cost that has been offered by dc.
• The job manager also evaluates the location compatibility of the user and the data center and the job manager returns the evaluated values to the IaaS layer.
• With the help of SaaS allocation architecture, the job is allocated to the minimum cost holder dc.

The pre-allocation phase can be algorithmically written as in Algorithm 1.

Algorithm 1: Preliminary Allocation

Require: UL dcd where UL is the user list and dcd is the datacentre description

Output allocation a returns the allocation architecture

tu = UL.COUNT fetch total number of users

CU = UL.CPUU fetch CPU utilization from user demands

RU = UL.RAMU fetch ram demand from the users

Nr = norm(CU, RU) perform normalization over the parameters

Demand = nr. CU + nr.Ru

SD = Sort (demand, de) sort the demand vector into descending order

dcc = 1;Initialize data center counter to 1

for1 usr = 1:tc

While dcc ≠ dcd. Count

pma = dcd.PMs pmc = pma.count;min(ec) = max.assign maximum value to energy consumption as minimum value pmcr = 1;pm counter

while pmcr ≠ pmc

Compute ec = EC (pma[pmcr]) computer energy consumption

If 1ec ≤ min (ec)

[ux,uy] = location (usr) fetch location of users

[dcx,dcy] = location(usr);

Dist = √(dcx − ux)2 + (dcy + uy)2(1)

If 2dist ≤ th where th is the location threshold

Min(ec) = ec

Allocationt = allocate(usr,dcc.pmcr)

End if1

End if2 endwhile2

End while1 end for1

Output: SF

Once the allocation is performed for a user to a PM, the PM starts executing the job assigned to it and by the end of the execution, the following parameters have been evaluated:

• Distance from user to PM.
• Total consumed energy in the allocation as well as in the execution against total number of supplied instructions.
• The total amount of CO2 emission in the processing of the job at the PM.

Each cloud data center has its own energy consumption profile. To estimate the energy consumption of a multi-cloud environment, we can use the following equation:

EC_total = ∑ DCi (E_{fixed_DCi} + E_{dyn_DCi})

(2)

where:

• EC_total: total energy consumption of the multi-cloud environment.
• DCi: a specific data center in the multi-cloud environment.
• E_{fixed_DCi}: the fixed energy consumption of the data center, which is independent of the actual workload and is related to the power consumed by the cooling and networking equipment.
• E_{dyn_DCi}: the dynamic energy consumption of the data center, which is directly related to the actual workload and is caused by the power consumed by the servers and other computing resources.

To compute Ed_{yn_DCi}, we can use the following equation:

E_{dyn_DCi} = ∑ a∈CNiE_{dyn_a} + ∑b∈SNi_{Edyn_b}

(3)

where:

• E_{dyn_DCi}: the dynamic energy consumption of data center DCi.
• a: a compute node in the data center that is used to run cloud services and virtual machines.
• CNi: the set of compute nodes in data center DCi that are used to run cloud services and virtual machines.
• E_{dyn_a}: the dynamic energy consumption of compute node a, which is proportional to its CPU utilization and the power consumed by its CPU and memory.
• b: a storage node in the data center that is used to store data and provide access to it.
• SNi: the set of storage nodes in data center DCi that are used to store data and provide access to it.
• E_{dyn_b}: the dynamic energy consumption of storage node b, which is proportional to the power consumed by its disk drives and other storage components.

To compute E_{dyn_a} and E_{dyn_b}, we can use the following equations:

E_{dyn_a} = T·[(Pcores_a,j + 1_P_cores_a,j) · k · u_a + [(j + 1) · P_cores_a,j_j · P_cores_a,j + 1]]

(4)

E_{dyn_b} =T·[(P_disks_b,j + 1_P_disks_b,j) · k · u_b + [(j + 1) · P_disks_b,j_j · P_disks_b,j + 1]]

(5)

where:

• E_{dyn_a}: the dynamic energy consumption of compute node a.
• E_{dyn_b}: the dynamic energy consumption of storage node b.
• T: the time interval for which the energy consumption is being estimated.
• P_cores_a,j: the power consumed by the CPU of compute node a at a time interval j.
• P_disks_b,j: the power consumed by the disk drives of storage node b at time interval j.
• k: a constant that reflects the power consumption per unit of CPU utilization or disk I/O activity.
• u_a: the average CPU utilization of compute node a over the time interval T.
• u_b: the average disk I/O activity of storage node b over the time interval.
• T_j: the time interval index, which varies from 1 to N, where N is the total number of time intervals in T.

The above equations can be used to estimate the energy consumption of a multi-cloud environment by computing the fixed and dynamic energy consumption of each data center and summing them up over all data centers. Note that the accuracy of these equations depends on the accuracy of the input parameters, such as the power consumption profiles of the computing and storage resources, the CPU utilization and disk I/O activity levels, and the time interval T.

The above study shows that about 70% of the energy is consumed when the physical machine in the cloud data center is in an idle state in comparison to fully utilized physical machines [15,16,17]. The enormous consumption of energy results in massive emissions of carbon dioxide. This results in the excellent efficiency of resources and carbon footprint being dependent on the emission of CO2 from different sources such as active physical machines, idle physical machines, and CO2 emitted from other sources. Equation (3) shows the relationship between the energy consumed by the machines and the emission of CO2. Therefore, the energy consumed by the machines is directly related to the emission of carbon dioxide given as follows [18,19,20]:

$${\mathrm{T}}_{\mathrm{C}\mathrm{O}2}=\mathrm{E}{\mathrm{C}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}\times \mathrm{I}$$

(6)

where ${\mathrm{T}}_{\mathrm{C}\mathrm{O}2}$ is the total emission of CO2, $\mathrm{E}{\mathrm{C}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}$ is the total power consumed by the machines, and I is the intensity of emission of greenhouse gases. In the given equation, $\mathrm{E}{\mathrm{C}}_{\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}}$ is obtained from

Table 2. Result illustration for variation in user base.

Number of Users	Q-Learning Energy Consumption in mj	Q-Learning CO2 Emission	Q-Learning Cost	Neural Energy Consumption	Neural CO2 Emission	Neural Cost
50	0.02106822	6.42986867	521.572554	0.0220352	6.5413934	545.511394
60	0.02217979	14.430218	242.5145	0.02217979	17.130218	242.5145
70	0.02164342	10.5698643	551.419352	0.02213468	10.8097756	563.935287
80	0.02360281	11.2820382	541.583354	0.02360281	14.283821	541.583354
90	0.02106822	6.42986867	521.572554	0.0220352	6.5413934	545.511394
100	0.02217979	14.430218	242.5145	0.02217979	17.130218	242.5145
110	0.02126149	6.1577718	721.998875	0.02200738	6.3383637	747.327726
120	0.02122007	6.65439454	526.109269	0.02231988	6.99928378	553.376878
130	0.02112321	8.23285148	158.480955	0.02219698	8.24468813	166.537091
140	0.02165894	9.42446206	90.8682777	0.022725	9.44535424	95.3408492
150	0.02206507	11.6133645	550.580614	0.2271847	17.4150556	566.884408
160	0.02166065	5.5407717	901.246574	0.02201759	5.63207531	916.097766
170	0.0239655	1.80370788	665.621701	0.0239655	1.80370788	665.621701
180	0.02159285	2.97493584	845.613912	0.02204145	3.03674146	863.181919
190	0.02206986	7.80330226	707.091422	0.02212858	7.82406631	708.972944
200	0.02196233	10.0295133	191.443118	0.02282014	11.0697247	198.920631
210	0.02201432	8.5942338	517.707915	0.02201432	11.5942338	517.707915
220	0.02215512	15.159949	171.925346	0.02215512	0.11159949	171.925346
230	0.02222979	2.95209659	624.968445	0.02288803	3.03950965	643.474075
240	0.0213401	0.00819311	291.881689	0.02208191	0.00847791	302.02796
250	0.02217912	0.4468032	351.464532	0.02254749	0.45422409	357.301957
260	0.02140226	1.18694688	219.056545	0.02218325	1.23025979	227.050143
270	0.022298	1.67893868	183.636069	0.022298	1.67893868	183.636069
280	0.02302263	0.56100439	357.686264	0.02302263	0.56100439	357.686264
290	0.02213323	71.0216162	698.751199	0.0224206	71.9437083	707.823267
300	0.02270107	72.7395481	637.170575	0.02302616	73.781194	646.294994
310	0.02211887	1.66328916	284.414536	0.02211887	1.66328916	284.414536
320	0.02204071	0.58939459	131.359452	0.02204071	0.58939459	131.359452
330	0.02217318	9.09875558	772.71284	0.02300474	9.43998375	801.691681
340	0.02201554	0.47459599	255.240777	0.02201554	0.47459599	255.240777
350	0.0237244	0.1483026	522.671016	0.0237244	0.1483026	522.671016
360	0.02216749	7.70194038	681.416068	0.02264763	7.86876229	696.175353
370	0.02229621	3.36930451	730.644765	0.02274257	3.43675669	745.271993
380	0.02249852	79.2656184	560.80046	0.02293822	80.8147721	571.760648
390	0.02249672	0.63418433	184.635617	0.02249672	0.63418433	184.635617
400	0.02284044	2.72712565	538.181407	0.02345898	2.80097843	552.755798
410	0.022519	1.28845628	283.535246	0.022519	1.28845628	283.535246
420	0.021824	72.6149194	542.407396	0.02290201	76.2018021	569.200124
430	0.02243758	67.4184839	725.64322	0.02275454	68.3708582	735.893879
440	0.02210143	2.19684776	247.249702	0.02210143	2.19684776	247.249702
450	0.0220649	0.20084786	74.7594037	0.0220649	0.20084786	74.7594037
460	0.02268973	7.71539389	711.468934	0.02314231	7.86928955	725.660301
470	0.02123597	61.9374793	669.197134	0.02210871	64.4829288	696.699182
480	0.02210045	0.47457086	150.650028	0.02210045	0.47457086	150.650028
490	0.02199189	0.42806465	415.814566	0.02257911	0.43949459	426.917409
500	0.02222279	0.00588392	160.808541	0.028222279	0.00588392	160.808541
510	0.02206507	11.6133645	550.580614	0.2271847	17.4150556	566.884408
520	0.02215512	15.159949	171.925346	0.02215512	0.11159949	171.925346
530	0.02222979	2.95209659	624.968445	0.02288803	3.03950965	643.474075
540	0.0213401	0.00819311	291.881689	0.02208191	0.00847791	302.02796
550	0.02217912	0.4468032	351.464532	0.02254749	0.45422409	357.301957
560	0.02140226	1.18694688	219.056545	0.02218325	1.23025979	227.050143
570	0.022298	1.67893868	183.636069	0.022298	1.67893868	183.636069
580	0.02302263	0.56100439	357.686264	0.02302263	0.56100439	357.686264
590	0.02213323	71.0216162	698.751199	0.0224206	71.9437083	707.823267
600	0.02270107	72.7395481	637.170575	0.02302616	73.781194	646.294994
610	0.02211887	1.66328916	284.414536	0.02211887	1.66328916	284.414536
620	0.02204071	0.58939459	131.359452	0.02204071	0.58939459	131.359452
630	0.02217318	9.09875558	772.71284	0.02300474	9.43998375	801.691681
640	0.02201554	0.47459599	255.240777	0.02201554	0.47459599	255.240777
650	0.0237244	0.1483026	522.671016	0.0237244	0.1483026	522.671016
660	0.02216749	7.70194038	681.416068	0.02264763	7.86876229	696.175353
670	0.02229621	3.36930451	730.644765	0.02274257	3.43675669	745.271993
680	0.02249852	79.2656184	560.80046	0.02293822	80.8147721	571.760648
690	0.02249672	0.63418433	184.635617	0.02249672	0.63418433	184.635617
700	0.02284044	2.72712565	538.181407	0.02345898	2.80097843	552.755798
710	0.022519	1.28845628	283.535246	0.022519	1.28845628	283.535246
720	0.021824	72.6149194	542.407396	0.02290201	76.2018021	569.200124
730	0.02243758	67.4184839	725.64322	0.02275454	68.3708582	735.893879
740	0.02210143	2.19684776	247.249702	0.02210143	2.19684776	247.249702
750	0.0220649	0.20084786	74.7594037	0.0220649	0.20084786	74.7594037
760	0.02268973	7.71539389	711.468934	0.02314231	7.86928955	725.660301
770	0.02123597	61.9374793	669.197134	0.02210871	64.4829288	696.699182
780	0.02210045	0.47457086	150.650028	0.02210045	0.47457086	150.650028
790	0.02199189	0.42806465	415.814566	0.02257911	0.43949459	426.917409
800	0.02222279	0.00588392	160.808541	0.028222279	0.00588392	160.808541
810	0.02140226	1.18694688	219.056545	0.02218325	1.23025979	227.050143
820	0.022298	1.67893868	183.636069	0.022298	1.67893868	183.636069
830	0.02302263	0.56100439	357.686264	0.02302263	0.56100439	357.686264
840	0.02213323	71.0216162	698.751199	0.0224206	71.9437083	707.823267
850	0.02270107	72.7395481	637.170575	0.02302616	73.781194	646.294994
860	0.02211887	1.66328916	284.414536	0.02211887	1.66328916	284.414536
870	0.02204071	0.58939459	131.359452	0.02204071	0.58939459	131.359452
880	0.02217318	9.09875558	772.71284	0.02300474	9.43998375	801.691681
890	0.02201554	0.47459599	255.240777	0.02201554	0.47459599	255.240777
900	0.0237244	0.1483026	522.671016	0.0237244	0.1483026	522.671016
910	0.02216749	7.70194038	681.416068	0.02264763	7.86876229	696.175353
920	0.02229621	3.36930451	730.644765	0.02774257	3.43675669	745.271993
930	0.02249852	79.2656184	560.80046	0.02293822	80.8147721	571.760648
940	0.02249672	0.63418433	184.635617	0.02249672	0.63418433	184.635617
950	0.02284044	2.72712565	538.181407	0.02545898	2.80097843	552.755798
960	0.022519	1.28845628	283.535246	0.022519	1.28845628	283.535246
970	0.021824	72.6149194	542.407396	0.02290201	76.2018021	569.200124
980	0.02243758	67.4184839	725.64322	0.02275454	68.3708582	735.893879
990	0.02210143	2.19684776	247.249702	0.02210143	2.19684776	247.249702
1000	0.02284044	2.72712565	538.181407	0.02745898	2.80097843	552.755798

and I is the emitted CO2 by each physical server. According to [14,15], I is 12.62 MT CO2/month in the cloud data center for 100 servers. The total emission of CO2 is shown in Table 2. Furthermore, the consumed energy and carbon emission in the allocation and processing of the job at a PM per machine is realized further using Equations (7) and (8) as follows:

$$\mathrm{E}\mathrm{C}={\sum }_{\mathrm{m}=1}^{\mathrm{M}}{\mathrm{I}}_{\mathrm{m}}\times \mathrm{e}\mathrm{c}{\mathrm{e}}_{\mathrm{j}}+\left({\mathrm{s}}_{\mathrm{j}}\times {\mathrm{I}}_{\mathrm{m}}\right)\times \mathrm{e}\mathrm{c}{\mathrm{s}}_{\mathrm{j}}$$

(7)

where M is total number of instructions under 1 PM ‘j’, ece is the execution cost at the jth PM, s is the storage required for the mth instruction, and ecs is the total consumed energy in the storage of one instruction at the jth PM. In a similar fashion, CO2 emission is computed by Equation (8).

$$\mathrm{C}\mathrm{E}={\sum }_{\mathrm{m}=1}^{\mathrm{M}}{\mathrm{I}}_{\mathrm{m}}\times \mathrm{c}{\mathrm{e}\mathrm{e}}_{\mathrm{j}}+\left({\mathrm{s}}_{\mathrm{j}}\times {\mathrm{I}}_{\mathrm{m}}\right)\times \mathrm{c}\mathrm{e}{\mathrm{s}}_{\mathrm{j}}$$

(8)

where cee is the CO2 emitted in the execution of the mth instruction set on the jth PM and ces is the CO2 emitted in the storage.

The proposed algorithm architecture introduces a separation mechanism that is attained using a modified k-means algorithm. The ordinal measures are provided in the next section.

3.2. The Elasticity

The concept of elasticity was introduced to the cloud computing environment in early 2012 when the load management concept was introduced to the scheduling architecture of cloud computing. The proposed work continues the effort in the same area, particularly in minimal energy consumption and CO2 emissions. To do so, the proposed work utilizes the concept of a Q-learning algorithm architecture by creating the following aspects [16,17,18,19,20].

• Environment;
• States;
• Actions.

To create the environment, the proposed algorithm enhances the existing k-means algorithm architecture in which the data is clustered based on the Euclidean distance. The ordinal measures of k-means are as follows.

Updated K-Means

To update the existing k-means algorithm, two major changes were made as follows.

• Adding cosine similarity as another measure to be utilized in the calculation of the distance to maintain the records in any cluster. Cosine similarity is quite commonly being used in parameter distance evaluations [21,22,23,24,25]. It is the cosine of the angular difference between two supplied vector values with the same number of attributes. In the case of the proposed work, the cosine similarity is evaluated between two vector values containing four attributes as follows:

  $$v=\{EC,\mathrm{C}\mathrm{O}2,d,M\}$$

  where $EC$ is the energy consumption, $\mathrm{C}\mathrm{O}2$ represents the carbon dioxide emission, ‘$d$’ represents the distance between the user and the location of the datacenter, and $M$ is the total supplied instruction set from users.
• Removing the possibilities of random centroid selection for the first convergence. To reduce the convergence rate, the proposed algorithm architecture utilizes the mean value of every attribute as the first centroid and performs some variation in using a 20% margin value to create the other two centroids at first glance.

The updated k-means algorithm aims to improve performance and accuracy. The first update adds cosine similarity to the distance computation to maintain records in any cluster. Cosine similarity helps the computer locate groups with comparable properties. Second, random centroid selection is removed for the initial convergence. This is because random centroid selection might reduce convergence rates and produce poor clustering solutions. The proposed algorithm design uses the mean value of every characteristic as the first centroid and a 20% margin value to build the other two centroids at first glance. This helps the algorithm start with a better clustering solution and converge faster to a better one.

The rest of the process runs in a similar fashion in which the data is placed in its respective clusters with the nearest hybrid cluster distance. The process continues until the data is assigned into one cluster.

The clustered elements are now considered as an environment for the Q-learning algorithm architecture. The Q-learning is propagated by the weight method to generate actions against each state of the environment. The proposed algorithm architecture considers three states for one cloud and the environment is considered for multiple cloud environments to support load and resource sharing [26,27,28,29,30,31,32]. The proposed process can be understood using the workflow shown in Figure 3.

The reward repository is created by updating Bellman’s equation for reward and penalty generation by integrating the propagation-based reward mechanism. The illustration of the proposed algorithm for reward generation is as follows. The proposed algorithm is different in terms of propagation, reward, and penalty generation. This paper proposes a new algorithm for generating rewards and penalties based on the concept of propagation. Specifically, Bellman’s equation for reward and penalty generation is updated to include the propagation-based reward mechanism. This helps to improve the efficiency and effectiveness of the algorithm in decision-making related to migration and allocation of users in a cloud computing environment.

The proposed algorithm differs from existing algorithms in terms of propagation, reward, and penalty generation. It uses two time slots for weight generation, where t represents the current state value and t − 1 represents the previous state value, number represents the flow and Y for yes and N for no is used. This allows for a more accurate assessment of the current and past states, which in turn helps to make better decisions regarding migration and allocation.

The reward and penalty generated by the algorithm are used to decide whether a user should be migrated from the current physical machine (PM) or not. Additionally, the reward and penalty are also used for the allocation of a user to a PM. If the reward of a PM is higher than the penalties, the user will be allocated to that PM. However, if the reward is lower than the penalties, the user will remain on the existing PM.

The proposed algorithm has been evaluated for several Quality of Service (QoS) parameters and compared with existing algorithm architectures from the literature. The results show that the proposed algorithm outperformed existing algorithms in terms of efficiency, effectiveness, and accuracy. The proposed algorithm uses two time slots for weight generation in which t is the current state value and t − 1 is the previous state value. The algorithmic description is provided as in the following algorithm.

Q-learning: Learn function Q: X × A→ R

Require:

States X = {1,2,3} proposed work has 3 states

Actions A= {1, 2, 3} quad A: X ⇒ A proposed work has 3 actions

Reward function R: X × A→R

Gradient based (weighted sum) transition function T: X: ax + b × A→ X

Learning rate α ϵ [0, 1], typically α = 0.1

Discounting factor ϒ ϵ [0, 1]

procedure Q LEARNING(X, A, R, T, α, ϒ)

Initialize Q: X ax + b: a→0.01b = 0.02x = norm (PC, CO2, d, M) × A→ R arbitrarily

while Q is not converged do

Start in state s ϵ X

while s is not terminal do

Calculate π according to w = ow+nw where nw = ax + b and ow = nw (t − 1) t is the time state of current value and exploration strategy (e.g., π (x) ←arg maxa Q(x, a))

A ← π(s)

r ← R(s, a) P(s, a) ▷ Receive the reward or penalty

s’← T(s, a) ▷ Receive the new state

Q(s’, a) ← (1 − α)·Q(s, a) + α·(r + ϒ·maxa’ Q(s’, a’))

s ← s’

return Q

The reward and the penalty are used to decide whether the user should be migrated from the PM or not or even for the allocation part if the user has to be allocated to the PM, if the reward of the PM is more than that of the served penalties, the user will be allocated to the PM or the user will remain at the existing PM only. The proposed work has been evaluated for several QoS parameters comparing the current state of the art with different algorithm architectures that have been proposed in this context earlier.

In our proposed approach, the preliminary and elastic phase work follow each other. To understand it better, consider the flow of execution shown in Figure 3. When a user sends a request to the cloud controller (CC), the CC transfers the request to the Infrastructure as a Service (IaaS) layer. The IaaS layer then broadcasts the requirement to the job manager, which sorts the jobs based on CPU utilization in descending order. At this stage, we use the updated k-means algorithm to cluster the jobs based on their CPU requirements and geographical locations. This helps to reduce the overall energy consumption by allocating jobs to the nearest data centers, which reduces the distance for data transmission.

Next, the IaaS layer broadcasts the sorted requirements to the data centers, and the data centers compute the overall expected cost based on CPU utilization. At this stage, we use the Q-learning algorithm to optimize the energy consumption of data centers. Q-learning algorithm helps to determine the optimal energy consumption of a data center by considering the tradeoff between energy consumption and performance. This helps to minimize the energy consumption of data centers while still maintaining optimal performance. After the data centers compute the overall expected cost, the IaaS layer passes the responses to the job manager. The job manager views the minimum cost offered by the data centers and also evaluates the location compatibility of the user and the data center using the updated k-means algorithm. Finally, with the help of the Software as a Service (SaaS) allocation architecture, the job is allocated to the data center with the minimum cost.

4. Results and Discussion

The proposed work model minimizes total data center energy consumption and carbon emissions by employing a Q-learning algorithm for job allocation and migration among processing machines (PMs). The proposed algorithm was evaluated based on QoS parameters using the following formulas.

• Energy Consumption: The energy consumption of the proposed model is calculated based on the total amount of energy used in a given interval of time. The total consumed energy can be defined as the sum of idle energy consumption and processing energy consumption. This is shown in Equation (9):

  $$\mathrm{E}\mathrm{C}=\mathrm{E}{\mathrm{C}}_{\mathrm{i}\mathrm{d}\mathrm{o}\mathrm{l}}+\mathrm{E}{\mathrm{C}}_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}}$$

  (9)

  where EC_idol is the energy consumed by the data center in the idle state and EC_processing is the energy consumed during the processing of jobs. The processing energy consumption can be further defined using Equation (10):

  $$\mathrm{E}{\mathrm{C}}_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}}=\mathrm{E}{\mathrm{C}}_{{\mathrm{i}}_{\mathrm{u}}}\times \mathrm{M}\mathrm{I}\mathrm{P}\mathrm{S}+\mathrm{E}{\mathrm{C}}_{{\mathrm{i}}_{\mathrm{u}\mathrm{s}}}\times \mathrm{M}\mathrm{I}\mathrm{P}\mathrm{S}$$

  (10)

  where $\mathrm{E}{\mathrm{C}}_{\mathrm{i}\mathrm{u}}$ is the unit cost of execution of one MIPS, $\mathrm{E}{\mathrm{C}}_{{\mathrm{i}}_{\mathrm{u}\mathrm{s}}}$ is the unit cost of storage of the executed instruction set, and MIPS is the total amount of processing done in the given time interval.
• CO2 emission: The CO2 emission is calculated using Equation (11).

  $$\mathrm{C}\mathrm{O}{2}_{\mathrm{e}}=\mathrm{C}\mathrm{O}{2}_{{\mathrm{e}}_{\mathrm{i}\mathrm{d}\mathrm{o}\mathrm{l}}}+{\mathrm{C}\mathrm{O}{2}_{\mathrm{e}}}_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}}$$

  (11)

  where $\mathrm{C}\mathrm{O}{2}_{\mathrm{eidol}}$ is the amount of CO2 emission released in the idle state and ${\mathrm{C}\mathrm{O}{2}_{\mathrm{e}}}_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}}$ is the amount of CO2 emission released during the processing of jobs. The amount of CO2 emission released during job processing can be defined as

  $${\mathrm{C}\mathrm{O}{2}_{\mathrm{e}}}_{\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g}}={\mathrm{C}02}_{{\mathrm{i}}_{\mathrm{u}}}\times \mathrm{M}\mathrm{I}\mathrm{P}\mathrm{S}+{\mathrm{C}\mathrm{O}2}_{{\mathrm{i}}_{\mathrm{u}\mathrm{s}}}\times \mathrm{M}\mathrm{I}\mathrm{P}\mathrm{S}$$

  (12)

  where ${\mathrm{C}\mathrm{O}2}_{{\mathrm{i}}_{\mathrm{u}}}$ is the unit cost of emitting one unit of CO2 during the execution of one MIPS and ${\mathrm{C}\mathrm{O}2}_{{\mathrm{i}}_{\mathrm{u}\mathrm{s}}}$ is the unit cost of emitting one unit of CO2 during the storage of the executed instruction set.
• Overall cost: This is evaluated based on the total cost of implementation and overall cost of storage.
  The overall cost is calculated using Equation (13):

  $$\mathrm{O}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{l}\mathrm{l} \mathrm{C}\mathrm{o}\mathrm{s}\mathrm{t}=\mathrm{E}\mathrm{C}\times \mathsf{\alpha }+\mathsf{\beta }$$

  (13)

  where $\mathsf{\alpha }$ is the unit cost to produce EC amount of energy to be consumed and $\mathsf{\beta }$ is the malicious processing cost.

In the context of comparing our proposed work model to other architectures using Q-learning and Neural Networks, it should be noted that the use of Neural Networks for this kind of problem is a common approach. However, the typical feedforward Neural Network architecture used for supervised learning in our comparison did not perform as well as expected. This is largely due to its static allocation mechanism, which did not adapt well to changes in user demand.

Our proposed work model utilizing Q-learning is intended to allocate resources dynamically based on current user demand and migration trends. This method enables more efficient and effective resource usage, resulting in superior overall performance compared to static allocation processes such as those utilized in Neural Network designs.

Furthermore, our proposed work model was assessed in two distinct scenarios. The first scenario involves adding users incrementally, with a minimum of 50 and a maximum of 1000. In this case, 10 PMs were maintained for 50 users, and for every 10 users, two additional PMs were deployed to accommodate network demand. The proposed work model outperformed the Neural Network architecture by a margin of 7 to 9 percent in every area due to the updated allocation and reward mechanism that determines the most suitable PM for the allocation of users and the migration of users from one PM to another.

In the second scenario, users are added incrementally, beginning with fifty and ending with one thousand. In all scenarios, the load is maintained at 10,000 MIPS.

Consequently, we believe that our proposed work model utilizing Q-learning provides a more efficient and effective solution for resource allocation in cloud computing environments, particularly when compared to traditional static allocation mechanisms such as those utilized in typical feedforward Neural Network architectures.

The results shown in Table 2 depict the energy consumption and CO2 emissions of a cloud data center as the number of users increases. There may be several reasons why the energy consumption is lower for 50 users than for 60 users, such as resource heterogeneity, workload types, and resource allocation strategies. Table 2 was generated through simulations or experiments designed to measure the energy consumption and CO2 emissions of a cloud data center as the number of users increases. These experiments may involve executing a set of workloads on the cloud data center and using power meters and carbon footprint calculators to measure energy consumption and CO2 emissions. These experiments are typically repeated several times to obtain an average value for each data point in the table.

Furthermore, the experiments may consider various scenarios, such as different resource allocation strategies, workload characteristics, and server utilization rates, to obtain a comprehensive understanding of the energy consumption and CO2 emissions of the cloud data center. The results from these experiments presented in Table 2 demonstrate the relationship between the number of users and the energy consumption and CO2 emissions of the cloud data center.

For 1000 users, the Q-learning algorithm consumed 0.02191633 mj, whereas the Neural Network algorithm consumed 0.02204761 mj which is [($0.02284044-0.02745898)/0.02745898]\times 100=16\%$. When it comes to CO2 emissions, a $\frac{2.72712565-2.80097843}{2.80097843}\times 100=2.6\%$ improvement was observed.

The second section provides significance to the comparison with other state of the art techniques and is illustrated as follows.

As shown in

Table 3. Energy consumption of proposed and state of the art techniques in mj.

Total Work Load in MIPS	EC Proposed	EC MET	EC MOS	ECMOIA
10,000	0.08643221	0.11053343	0.09705079	0.08644106
20,000	0.14009598	0.17507796	0.16331992	0.18034036
30,000	0.21644788	0.21808101	0.24335856	0.26394136
40,000	0.27240836	0.348205	0.27660185	0.32084257
50,000	0.34129012	0.37545341	0.36479844	0.43713668
60,000	0.41864937	0.51451122	0.52244715	0.49067343
70,000	0.48390781	0.57315099	0.62780087	0.51347926
80,000	0.54976218	0.68848141	0.60823041	0.62069295
90,000	0.61699444	0.6681455	0.7321855	0.72589921
100,000	0.68230112	0.75269047	0.70917145	0.73564827

, as the load variation increased in MIPS, the energy consumption increased with the increase in load. For this comparison and all the other comparisons made here, the user count was kept constant at 150. The proposed work was compared with two state-of-the-art techniques namely MOS (Multi-Objective Scheduling) [1] and MOIA (Many Objective Intelligent Algorithm) [3] and MET. Starting from the proposed algorithm and ending at MOIA, the average PC is shown in Figure 4.

Similarly, CO2 emission was also been calculated and compared to existing state of the art techniques and is illustrated in

Table 4. CO2 emissions of proposed vs. state of the art techniques.

Total Workload in MIPS	CO2 Emission Proposed	CO2 Emission MET	CO2 Emission MOS	CO2 Emission MOIA
10,000	15.4968466	16.4625975	16.5163079	17.0115375
20,000	30.2379999	33.4719869	35.1946301	33.3828888
30,000	43.125496	47.8773122	52.7623347	54.6693355
40,000	53.9849289	67.5633117	56.6986584	62.1183083
50,000	69.3994218	89.4304641	78.5179474	88.9770678
60,000	83.1703267	91.3802658	88.7459483	90.9647089
70,000	96.6421697	116.23719	103.859875	110.436456
80,000	108.955624	131.863262	133.2245	133.728346
90,000	123.853874	127.045688	142.450492	143.234144
100,000	135.115958	169.327299	143.069083	147.432519

.

CO2 emission is dependent on energy consumption and the supplied number of jobs. The average energy consumption for the proposed and state of the art techniques is shown in Figure 5. The proposed algorithm outperformed the state of the art techniques by a significant margin. For the same load scenario, the average energy consumption for the proposed algorithm was 75.99 units whereas MET demonstrated an overall consumption of 89.05 units. Compared to MOS [1], the average improvement was $\frac{85.10-75.99}{85.10}=10.70\%$, whereas compared to MOIA [3], the proposed algorithm demonstrated an overall improvement of 13%.

The improvement in the case of the proposed algorithm was observed due to the dynamic nature of load adaption and adjustment of the jobs in terms of higher moderate and lower utilization. Due to the presence of the Q-table in the system, the processes after pre-allocation are updated in terms of policy selection, and the proposed algorithm selects the best PMs for the job when the user has to be migrated from one PM to another.

In a similar context, the overall cost was also evaluated using Equation (7) and is illustrated in

Table 5. Overall cost of proposed vs. state of the art techniques.

Total Workload in MIPS	Total Cost Proposed	Total Cost MET	Total Cost MOS	Total Cost MOIA
10,000	21.9474096	26.8693378	26.8757594	25.5246142
20,000	46.6526045	57.8019894	57.0917066	48.4095822
30,000	65.5526617	68.1012897	69.9788013	72.4359649
40,000	91.729042	101.509389	93.02456	97.6087062
50,000	106.704006	126.240668	132.347404	114.162025
60,000	131.74095	164.541054	143.220692	147.65082
70,000	151.887415	179.201553	188.364215	156.675002
80,000	173.157091	195.458122	222.836448	202.921307
90,000	192.577062	244.842466	243.67097	217.938865
100,000	216.426419	238.359854	268.270447	280.541938

.

Due to the effective management of energy efficiency, the proposed algorithm consumed less energy compared to other architectures and hence the overall cost was also low as the cost is dependent on energy consumption. The average energy consumption in all scenarios is shown in Figure 6.

For all the considered scenarios, the average cost of the proposed algorithm was just below 120 units whereas other than MOIA, all the other algorithm architectures consumed more than 140 units. To be precise in the comparison analysis, the proposed algorithm was $\frac{140.29-119.83}{119.83}\times 100=\frac{20.46}{119.83}\times 100=17.04\%$ improved compared to MET, whereas in comparison to MOS, the proposed algorithm was $\frac{140.56-119.83}{119.83}\times 100=\frac{24.73}{119.83}\times 100=20.63\%$ more efficient in terms of average cost. When compared to MOIA, the proposed algorithm was $\frac{136.38-119.83}{119.83}\times 100=\frac{16.55}{119.83}\times 100=13.81\%$ more cost-effective.

5. Conclusions

This research article presented a simulation architecture with multiple clouds sharing resources and loads among each other. The data centers of the cloud are equipped with physical machines that handle the users in terms of their passed instruction set. The proposed algorithm architecture is divided into two phases namely preliminary allocation and support for elasticity. In the preliminary phase, the user is allocated to a PM based on the distance of the user from the PM and the resource capabilities of the PM with the lowest proposed cost. Once the users are allocated to a PM, a threshold is set for simulations and Q-learning intakes the aggregated data of the utilized simulation until now. The Q-learning algorithm architecture creates a Q-table that updates the reward based on the presented propagation architecture. Q-learning is supplied with three states to be considered and the Q-table is updated by Bellman’s reward policy. The proposed solution to the job scheduling problem reduced the overall energy consumption and CO2 emission compared to the state of the art techniques. The proposed algorithm outperformed a MOS architecture by 10.75% in terms of energy consumption whereas an improvement of 13% was observed compared to MOIA. The proposed work also calculated the overall cost of the system to execute the jobs. The proposed work showed an improvement of more than 13% and just less than 21% in every compared scenario. The proposed work has a lot of future possibilities. The listed work also demonstrates that there are possibilities for the merger of modern Q-learning algorithms with swarm-based algorithms like PSO.