Queuing-Based Federation and Optimization for Cloud Resource Sharing

Abstract

Resource sharing can gain economies of scale and increase utilization of cloud infrastructure, a critical challenge of which is how to design efficient resource sharing solutions among self-interested cloud providers. Cloud federation can realize resource sharing, but the existing methods of forming federation need complex computation to guarantee the stability of federation. To address this shortcoming, after analyzing an optimal allocation approach of service requests among clouds, we propose a pareto optimal resource sharing solution named Cloud Light-Federation Sharing (CLFS), in which each cloud can choose its own optimal strategies individually and federation can be formed without complex computation for allocation of service requests and profits. In addition, an optimal resource sharing solution named Cloud Cooperative-Federation Sharing (CCFS) was also designed, in which cloud providers are fully cooperative and have fair profit allocation. The experimental results show that the two federation methods can significantly improve the total utility and decrease the number of dropped jobs. Although the federation rules of Cloud Light-Federation Sharing are simple, its performance is still very close to that of Cloud Cooperative-Federation Sharing.

1. Introduction

Cloud computing has matured into a promising model for using computational resources and software resources on-demand [1,2]. Nowadays, the excitement surrounding cloud computing has attracted many cloud providers and cloud users, and led to independent cloud systems. In each cloud system, the cloud provider provisions the cloud services, and the users request the services randomly. With the increases in the numbers of cloud providers and cloud users, one of the major problems that faces each cloud system is the uncertainty of the workloads from its users, as a result of which, a cloud might be in an under-provisioning state or over-provisioning state [3]. An under-provisioning state means that the demand of the users exceeds the available resources, which may result in higher service rejection rate and delays for users due to congested resources. An over-provision state means that the cloud over-provisions the available resources for its users. Both of two states result in low efficiency of the cloud system. The former state results in low efficiency in service completion rate, and the latter results in low efficiency in resource utilization.

The uncertainty of the workloads can be observed in Figure 1. The workload logs are from two parallel computing centers: the Gaia cluster at the University of Luxemburg (UniLu Gaia) and the national grid of the Czech republic (MetaCentrum) (http://www.cs.huji.ac.il/labs/parallel/workload/logs.html, accessed on 20 March 2022). According to the workload logs, the job arrival rates (the average number of arrival jobs per hour) of each day from May to August 2014 are shown in Figure 1. The red line shows the distribution of job arrival rate from UniLu Gaia, and the blue line shows that during the same time interval from MetaCentrum. We can see that finding an optimal, fixed resource capacity is infeasible. Given any resource capacity, each computing center may be in over-provision state or under-provisioning state at some time, which is caused by the constant resource constraint of a single cloud. Moreover, a computing center might be in an under-provisioning state while the other one is in an over-provisioning state at certain points, such as days 17, 18, and 54.

A straightforward solution to overcoming this problem is to federate the two computing centers to improve the total resource efficiency and service completion rate. In fact, federation among clouds is becoming an effective solution which approaches the resource-constraint problem of a single cloud. To maximize profits and improve the quality of service (QoS) of users, it allows clouds in the federation to share their spare capacities during over-provisioning periods and borrow spare capacity from other cloud systems during under-provisioning periods. Recently, some cloud federation institutions appeared. For example, to serve the needs of the scientific community, the European Grid Infrastructure Federated cloud (https://www.egi.eu/infrastructure/cloud, accessed on 20 March 2022), a private cloud, was constructed. The OnApp Federation (http://onapp.com/federation, accessed on 20 March 2022) constitutes a network of infrastructure as a service (IaaS) among cloud service providers and connects them through the OnApp, and federation is a future of the cloud. In academia, researchers also investigate this topic from different perspectives; in particular, economic methods and theories have been applied to federations of multiple clouds [4,5,6,7].

The literature and applications have addressed the uncertainty problem of workloads via federation method [8,9], and most of those can be divided into two classes according to the different objectives. Some researches aim to improve the profit of the cloud provider, and the others aim to improve the quality of service of users. However, the above solutions are not suitable to the federation of the private clouds who not only take into account the service valuation brought by cloud resource, but also are concerned about the cost for inner users of waiting time. In our work, we aimed to improve the net value of a whole cloud network by designing a federation framework for cloud systems, the key problem of which is how to incentivize private clouds to share their resources and service requests with others. To address this problem, we designed two cloud federation solutions, the objective of which was to maximize the net utility of cloud federation. To the best of our knowledge, the work presented in this paper is the first to address the problem of maximizing the net utility of cloud federations based on queuing theory where the cost of users’ time is considered. To this end, the contributions of our work can be summarized as follows:

• We modeled the cloud system based on queuing theory, and constructed the utility function for cloud federation, which consists of the value of service and the cost of users’ time.
• Based on the queuing model, we obtained an optimal request allocation rule among the over-provision cloud providers. Hence, we derived a Pareto-optimal light-federation sharing scheme that could improve the net utility of cloud federation through a simplifying running mechanism.
• Furthermore, to obtain optimal net utility of cloud federation, we design a cloud cooperative federation sharing solution with Banzhaf value-based payoff division, and derived a fair cloud federation.

In the rest of the paper, we discuss related literature in Section 2 and present the system model in Section 3. In Section 4, we propose Light-Federation Sharing (CLFS) and Cloud Cooperative-Federation Sharing (CCFS). Section 5 describes the performance evaluation of CLFS and CCFS, and the conclusions are given in Section 6.

2. Related Work

Cloud federation refers to a mesh of clouds that are interconnected based on open standards to provide a universal decentralized computing environment where everything is driven by constraints and agreements in a ubiquitous, multi-provider infrastructure [10].

Early studies on the architecture of a cloud federation can be found in [11,12,13]. Buyya et al. [11] introduced a market-oriented VM exchange model among clouds, and suggested a federation oriented, just in time, opportunistic and scalable application services provisioning environment called InterCloud. Celesti et al. [12] proposed an approach for the federation establishment considering generic cloud architectures according to a three-phase model, representing an architectural solution for federation by means of a cross-cloud federation manager. Bernstein et al. [13] offered interoperability solutions only for low-level functionality of the clouds that focused on IaaS system operators without considering user demands. In the Federated Cloud Management solution [14], interoperability was achieved by high-level brokering instead of bilateral resource renting. Such a federation can be enabled without applying an additional software stack for providing low-level management interfaces. Yuan et al. [15] took into account the cost minimization problem for private cloud data (CDC) center in hybrid clouds, and proposed a temporal task scheduling algorithm (TTSA) to effectively dispatch all arriving tasks to private CDC and public clouds.

Recently, cloud federation was used to solve the difficulty brought by the uncertainty of the workload. Due to the high variability of the workload over time, the providers owning a restricted amount of resources can serve most of the requests, but they will reject users during peak hours or obtain under-utilization of the resources during valley hours. Hadji et al. [16] proposed a method using a linear integer program to optimize the partitioning of the incoming workload across the federation members. A pricing model was suggested to enable providers to set their offers dynamically and achieve the highest revenues. Gouasmi et al. [17] proposed a distributed heuristic algorithm—FDMR (Federated Distributed MapReduce)—which takes advantage of data locality, inter-cloud data transfer, and the high availability of capacities offered by the federation to reduce job costs while respecting deadline constraints. In addition, Ahmed et al. [18] proposed a QoS-aware trust evaluation mechanism which allowed selection of the most useful providers based on a utility value in a federation system. Rosa et al. [19] proposed a provisioning service called CRCPs—Computational Resource and Cost Prediction service—that measures user resources and reports the runtime financial cost before starting the workflow execution.

Economic theories have been applied to resource allocation in cloud federation environments. Shi et al. [20] proposed a framework for a decentralized cloud marketplace including a customizable auction model, an incentive witness mechanism, and a social behavior-based simulator. Goiri et al. [21] used federation to overcome these limitations, in which providers are allowed to dynamically outsource resources to others in response to demand variations. The work studied how to implement federation in cloud providers to increase their profitability by saving capital and operational costs. Samaan [22] proposed an economic model to regulate capacity sharing in a federation of hybrid cloud providers (CPs). The proposed work modeled the interactions among the CPs as a repeated game among selfish players who aimed at maximizing their profits by selling their unused capacity in the spot market while considering the uncertainty of the future workload. Hassan et al. [23] proposed a cooperative game-theoretic solution that was mutually beneficial to the CPs. The work studied two cooperative resource allocation games in an HDCF environment, used a price-based resource allocation strategy, and designed both centralized and distributed algorithms to find optimal solutions. Mashayekhy et al. [5] designed a cloud federation formation mechanism that enabled the cloud providers to dynamically form a cloud federation. Khandelwal et al. [24] used cooperative game theory to model the circumstances under which a cloud provider prefers to join a cloud federation vis-a-vis considering taking a price offer made by an oligopolist. Additionally, using a game theoretic approach to model the federation formation of clouds as a coalition game with externalities, Chen et al. [25] analyzed the interrelated workload factoring and federation formation among self-interesting clouds, providing scalable algorithms to help them to optimally select partners and outsource workload. The federation methods considered in the above work need complex algorithms to form a stable coalition.

Based on the work in [5,23,25], we also aim to improve the efficiency of cloud resources using cloud federation, but pay attention to the following two points:

• In our work, we address the federation of private clouds while taking into account both the service valuation brought by cloud resources and the cost of inner users for waiting time. Thus, the objective of our work was to improve the net utility of cloud federation.
• To avoid complex computation for request and profit allocation in a fully cooperative federation, we designed a simplified federation method which does not require complex computation, and the performance is still close to that of the fully cooperative federation scheme.

3. System Model

In this section, we construct the system framework for our analysis. We view the operation of the cloud service in a single cloud as follows. In each single cloud system, the users arrive into the system and request service randomly, and they depart after spending a random amount of time in the system. The time spent in the system we named congestion delay consists of actual service processing time and waiting delays. We use a queuing model [26,27] to evaluate how service rate changes due to the volume of requests and processing capacity of a cloud system.

Following prior studies [4,28,29] on cloud services, the operation of a cloud service can be modeled to follow an M/M/1 queuing, with $\mu$ being the capacity of the cloud and $\lambda$ being the Poisson arrival rate of service requests. Both parameters are expressed in service counts per unit of time.

The expected congestion delay of each service can be expressed as $w=\frac{1}{\mu -\lambda }$. We assume that the system satisfies Little’s law, $L=\lambda w$, where L expresses the expected number of services in the system. The expected delay cost per unit of time is given by $c\lambda w$, where c is the cost of delay per service per unit of time. The value function $V\left(\lambda \right)$ represents the total value obtained by the cloud system per unit time which is associated with service rate $\lambda$.

For the management and control of computing resources, Mendelson et al. [30,31] offered a methodology based on a queuing model for setting price, utilization, and capacity. Taking into account the value of users’ time, they model the system net value with $V\left(\lambda \right)-cL$ in the short run and $V\left(\lambda \right)-cL-C\left(\mu \right)$ in the long run, where $C\left(\mu \right)$ is the cost of capacity. This methodology has been applied to the service in cloud computing in some works [29]. In our work, following these studies, the net value of a cloud system can be expressed as

$$u=V\left(\lambda \right)-c\lambda w.$$

(1)

The first term in Equation (1) is the value of services per unit of time, and the second term is the expected aggregate delay cost per unit of time. Assume that the value function of service is linear, i.e., $V\left(\lambda \right)=k\lambda$, where k denotes the expect value brought by one service and $k>0$. We can see that, given any fixed capacity $\mu$, u is a concave function in $\lambda$. That means that to obtain maximal net value, some service requests have to be dropped when it is in an under-provisioning state.

Next, we describe the model of cloud federation consisting of multiple cloud systems and a broker as a mediator. Each cloud system makes a decision for the cloud provider (CP) to maximize its net utility. Therefore, the set of cloud systems is also expressed by the set of cloud providers $\mathcal{N}=\{C{P}_1,C{P}_2,\dots ,C{P}_n\}$. The broker is a trusted third party responsible for handling the federation tasks in federation environment, such as job scheduling and revenue allocation among participating cloud providers.

According to the description of a single cloud system, each $C{P}_i$ is characterized by the capacity of the cloud denoted by ${\mu }_i$ and the Poisson arrival rate of service requests denoted by ${\lambda }_i$. The broker performs the federation mechanism to maximize the sum of net utility of cloud federation $\mathcal{N}$. The optimization problem is

$$max\sum _{C{P}_i\in \mathcal{N}}[V\left({\lambda }_i^a\right)-c{\lambda }_i^a{w}_i]$$

(2)

$$s.t.\sum _{C{P}_i\in \mathcal{N}}{\lambda }_i^a\le \sum _{C{P}_i\in \mathcal{N}}{\lambda }_i$$

(3)

$${\lambda }_i^a<{\mu }_i$$

(4)

The vector ${\lambda }^a=({\lambda }_1^a,{\lambda }_2^a,\dots ,{\lambda }_n^a)$ represents the allocation of service requests, and ${\lambda }_i^a$ represents the Poisson arrive rate of service requests allocated to $C{P}_i$. We need to find an allocation ${\lambda }^a$ to maximize the objective function ${\sum }_{C{P}_i\in \mathcal{N}}[V\left({\lambda }_i^a\right)-c{\lambda }_i^a{w}_i]$, which is the total net utility of cloud federation $\mathcal{N}$. Constraint (3) guarantees the total allocated service request rate is no more than actual arrival service request rate. Constraint (4) guarantees the stability in the queues of each CP, so that the rate of incoming requests cannot exceed the service rate of the CP.

4. Cloud Resource Sharing Solutions

In this section, we design two federation solutions: Cloud Light-Federation Sharing (CLFS) and Cloud Cooperative-Federation Sharing (CCFS). The former is easy to be performed but does not obtain an optimal federation utility; the latter is complex for the profit allocation rule but can obtain an optimal federation utility.

4.1. Cloud-Light Federation: From the Non-Cooperative Game Perspective

4.1.1. Light-Federation Scenario

As mentioned above, each CP in cloud federation is self-interested, whose objective is to maximize its net value. Let all users have same expect value function per unit of time $V_i=k{\lambda }_i$. As described in Section 3, the net value of $C{P}_i$ can be expressed by

$$u_i=k{\lambda }_i-c{\lambda }_i\frac{1}{{\mu }_i-{\lambda }_i}.$$

(5)

Given the capacity ${\mu }_i$, the optimization rate of service requests can be obtained by solving the problem ${\lambda }_i^{*}=argmax{u}_i$.

We have that

$${\lambda }_i^{*}={\mu }_i-\sqrt{\frac{c{\mu }_i}{k}}.$$

(6)

If ${\lambda }_i^{*}\le 0$, it means that, for any ${\lambda }_i>0$, the net value of $C{P}_i$$k{\lambda }_i-c{\lambda }_i\frac{1}{{\mu }_i-{\lambda }_i}<0$ because of the existence of the cost of users time. We assume ${\lambda }_i^{*}>0$ for each $C{P}_i$. Let ${\lambda }_i^d$ denote the rate of dropped requests of $C{P}_i$. If ${\lambda }_i^{*}<{\lambda }_i$, $C{P}_i$ is in under-provisioning state, and the optimal strategy of $C{P}_i$ is to drop the requests with the rate ${\lambda }_i^d={\lambda }_i-{\lambda }_i^{*}$. If ${\lambda }_i^{*}\ge {\lambda }_i$, $C{P}_i$ is in over-provisioning state, and ${\lambda }_i^d=0$.

In this section, we present a broker-based light federation sharing scheme. We assume that the broker knows the capacity and the arrival rate of the requests for each CP.

The operation of the broker is described as follows.

• Step 1: The broker receives the dropped service requests with rate ${\lambda }^d={\sum }_{C{P}_i\in \mathcal{N}}{\lambda }_i^d$ submitted by all under-provisioning CPs.
• Step 2: The broker allocates the received requests to over-provisioning CPs.
• Step 3: The broker allocates the profits to over-provisioning CPs according to their added requests which come from under-provisioning CPs and are processed by over-provisioning CPs.

In the light-federation sharing solution, the optimal strategy for under-provisioning CPs is to submit the service requests to the broker with the rate ${\lambda }_i^d={\lambda }_i-{\lambda }_i^{*}$. The over-provisioning CPs can accept or reject a proportion of service requests submitted by the broker. Therefore, the problem that faces the broker is how to allocate the service requests to over-provisioning CPs when it receives the service requests with rate ${\lambda }^d={\sum }_{C{P}_i\in \mathcal{N}}{\lambda }_i^d$.

Lemma 1.

Let ${\lambda }^a=({\lambda }_1^a,{\lambda }_2^a,\dots ,{\lambda }_n^a)$ be the optimal request allocation. For any $i,j$, it satisfies $\frac{w_i}{w_j}=\frac{\sqrt{{\mu }_j}}{\sqrt{{\mu }_i}}$ if ${\lambda }_i^a>0,{\lambda }_j^a>0$.

Proof. 

Let $\lambda$ be the total arrival rate of service requests which need to be allocated to $C{P}_i$ and $C{P}_j$. The optimal allocation ${\lambda }_i^a$ can be expressed as follows:

$${\lambda }_i^a=arg\underset{{\lambda }_i}{max}V\left({\lambda }_i\right)-c{\lambda }_i{w}_i+V(\lambda -{\lambda }_i)-c(\lambda -{\lambda }_i)w_j.$$

(7)

Since $w_i=\frac{1}{{\mu }_i-{\lambda }_i}$ and $w_i=\frac{1}{{\mu }_j-{\lambda }_j}=\frac{1}{{\mu }_j-(\lambda -{\lambda }_i)}$, we have that

$${\lambda }_i^a=arg\underset{{\lambda }_i}{max}V\left({\lambda }_i\right)-\frac{c{\lambda }_i}{{\mu }_i-{\lambda }_i}+V(\lambda -{\lambda }_i)-\frac{c(\lambda -{\lambda }_i)}{{\mu }_i-(\lambda -{\lambda }_i)}.$$

(8)

Since all CPs have same value function $V=k\lambda$, the problem can be expressed as

$${\lambda }_i^a=arg\underset{{\lambda }_i}{min}\frac{{\lambda }_i}{{\mu }_i-{\lambda }_i}+\frac{\lambda -{\lambda }_i}{{\mu }_i-(\lambda -{\lambda }_i)}.$$

(9)

Let $h\left({\lambda }_i\right)={\displaystyle \frac{{\lambda }_i}{{\mu }_i-{\lambda }_i}}+{\displaystyle \frac{\lambda -{\lambda }_i}{{\mu }_i-(\lambda -{\lambda }_i)}}$. Then,

$$h^{\prime }\left({\lambda }_i\right)=\frac{{\mu }_i}{{({\mu }_i-{\lambda }_i)}^2}-\frac{{\mu }_j}{{({\mu }_i-\lambda +{\lambda }_i)}^2}.$$

$$h^{″}\left({\lambda }_i\right)=\frac{{\mu }_i}{{({\mu }_i-{\lambda }_i)}^3}+\frac{{\mu }_j}{{({\mu }_i-\lambda +{\lambda }_i)}^3}>0.$$

We can have that $h\left({\lambda }_i\right)$ is convex in ${\lambda }_i$, and the optimal allocation ${\lambda }_i^a$ satisfies $h^{\prime }\left({\lambda }_i\right)=0$. Then, we have

$$\frac{{\mu }_i}{{({\mu }_i-{\lambda }_i^a)}^2}=\frac{{\mu }_j}{{({\mu }_i-\lambda +{\lambda }_i^a)}^2}$$

$$\Rightarrow \frac{{\mu }_i}{{({\mu }_i-{\lambda }_i^a)}^2}=\frac{{\mu }_j}{{({\mu }_j-{\lambda }_j^a)}^2}$$

$$\Rightarrow {\mu }_i{w}_i^2={\mu }_j{w}_j^2$$

$$\Rightarrow \frac{w_i}{w_j}=\frac{\sqrt{{\mu }_j}}{\sqrt{{\mu }_i}}.$$

□

In addition, if it satisfies $\frac{w_i}{w_j}=\frac{\sqrt{{\mu }_j}}{\sqrt{{\mu }_i}}$ but ${\lambda }_i^a<0,{\lambda }_j^a>0$, this indicates that $C{P}_i$ cannot help to improve the total utility by providing services with its resources because the resource capacity of $C{P}_i$ is far less than that of $C{P}_j$. As $h\left({\lambda }_i\right)$ is convex in ${\lambda }_i$ and it should satisfy ${\lambda }_i\ge 0$ for any i, the optimal request allocation is ${\lambda }_i=0,{\lambda }_j=\lambda$. Thus, we can obtain Corollary 1.

Corollary 1.

Let λ be the service request rate which needs to be allocated to $C{P}_i$ and $C{P}_j$, $\{{\lambda }_i,{\lambda }_j\}$ satisfies $\frac{w_i}{w_j}=\frac{\sqrt{{\mu }_j}}{\sqrt{{\mu }_i}}$, and $\lambda ={\lambda }_i+{\lambda }_j$. The optimal allocation is ${\lambda }_i^a=0,{\lambda }_j^a=\lambda$ if ${\lambda }_i<0$.

In addition, if ${\lambda }_1^a\ge {\lambda }_2^a\dots \ge {\lambda }_k^a>0$, it satisfies

$$\frac{1}{w_1}:\frac{1}{w_2}:\dots :\frac{1}{w_k}=\sqrt{{\mu }_1}:\sqrt{{\mu }_2}:\dots :\sqrt{{\mu }_k}.$$

(10)

This conclusion can be deduced directly by Lemma 1.

4.1.2. Cloud Light-Federation Sharing (CLFS)

In the cloud light-federation environment, the light-federation means that all the CPs only need to consider their own strategies to improve their net utility individually, instead of abiding by some strict federation rules. As mentioned above, the optimal strategy of a under-provisioning CP is to drop its excessive service requests with rate ${\lambda }_i^d$. Therefore, in the light-federation scenario, the under-provisioning CPs authorize the other CPs to process their dropped requests and transfer all utility of those services to others. Let ${\mathcal{N}}_o$ and ${\mathcal{N}}_p$ denote the sets of CPs in an over-provisioning state and an under-provisioning state, respectively. Let $\overline{\lambda }=({\overline{\lambda }}_1,{\overline{\lambda }}_2,\dots ,{\overline{\lambda }}_n)$ denote the vector of added service requests rates of CPs in set $N_o$. The decision of the broker in step 2 denoted by ${\overline{\lambda }}^a=({\overline{\lambda }}_1^a,{\overline{\lambda }}_2^a,\dots ,{\overline{\lambda }}_n^a)$ is the optimal solution of the following problem:

$$max\sum _{C{P}_i\in {\mathcal{N}}_o}V({\lambda }_i+{\overline{\lambda }}_i)-c({\lambda }_i+{\overline{\lambda }}_i)\frac{1}{{\mu }_i-({\lambda }_i+{\overline{\lambda }}_i)}$$

(11)

$$s.t.{\mu }_i>{\lambda }_i+{\overline{\lambda }}_i,C{P}_i\in {\mathcal{N}}_o$$

(12)

$${\overline{\lambda }}_i>0$$

(13)

$$\sum _{C{P}_i\in {\mathcal{N}}_o}{\overline{\lambda }}_i\le \sum _{C{P}_i\in {\mathcal{N}}_p}{\lambda }_i^d$$

(14)

$$\sum _{C{P}_i\in \mathcal{N}}{\lambda }_i<\sum _{C{P}_i\in \mathcal{N}}{\mu }_i$$

(15)

Assume that ${\overline{\lambda }}_1^a\ge {\overline{\lambda }}_2^a\dots \ge {\overline{\lambda }}_k^a>0$. It is worth noting that, because each $C{P}_i$ has own service requests with rate ${\lambda }_i$, the total service request rate ${\lambda }_i^a$ is ${\overline{\lambda }}_i^a+{\lambda }_i$; it satisfies

$$\frac{1}{{\mu }_1-({\overline{\lambda }}_1^a+{\lambda }_1)}:\frac{1}{{\mu }_2-({\overline{\lambda }}_2^a+{\lambda }_2)}:\dots :\frac{1}{{\mu }_k-({\overline{\lambda }}_k^a+{\lambda }_k)}=\sqrt{{\mu }_1}:\sqrt{{\mu }_2}:\dots :\sqrt{{\mu }_k}.$$

(16)

Algorithm 1 describes how the service requests dropped by under-provisioning CPs are allocated to over-provisioning CPs. In line 2, $N_p$ is obtained, which is the set of under-provisioning CPs. Each CP in $N_p$ satisfies ${\lambda }_i^d\ne 0$. In line 3, the algorithm checks whether there exit under-provisioning CPs. If $N_p\ne \varphi$, it should reallocate the dropped jobs processed by lines 5–9. In line 7, according to Equation (16), we compute an optimal solution to maximize the net utility of over-provisioning CPs expressed by Formula (11). In line 8, the algorithm removes cloud providers such that each one’s reallocation satisfies ${\overline{\lambda }}_i^a<0$. In lines 7–9, each iteration removes those CPs until each CP in set $\mathcal{F}$ satisfies ${\overline{\lambda }}_i^a>0$.

Algorithm 1: Service request allocation algorithm in CLFS.

In Algorithm 1, the broker can get ${\lambda }_i^d$ (the dropping service rate of each $C{P}_i$) according to its received service requests from $C{P}_i$. We assume that the broker knows the service process rate ${\mu }_i$ and request arrival rate ${\lambda }_i$ of each CP. In reality, we can obtain ${\mu }_i,{\lambda }_i$ of over-provisioning CPs by submitting probe requests, so that CLFS can be performed in the case that the broker does not know ${\{{\mu }_i,{\lambda }_i\}}_{C{P}_i\in \mathcal{N}}$.

When $|{\mathcal{N}}_p|\ne 0$, the maximal number of iterations is $n-|{\mathcal{N}}_p|$. Computing ${\overline{\lambda }}_i^a$ for each $C{P}_i$ in set $\mathcal{F}$ according to Equation (16) (necessary condition of optimal solution) needs $O\left(\right|F\left|\right)$ computation complexity, $\left|F\right|=n-|{\mathcal{N}}_p|$. Therefore, the computation complexity of Algorithm 1 is $O\left(n^2\right)$.

Theorem 1.

If ${\lambda }_d+{\sum }_{C{P}_i\in {\mathcal{N}}_o}{\lambda }_i<{\sum }_{C{P}_i\in {\mathcal{N}}_o}{\mu }_i$, Algorithm 1 obtains an optimal request allocation of dropped service requests among over-provisioning CPs.

Proof. 

The theorem can be derived from Lemma 1.

Firstly, we show the convergence of Algorithm 1. Let $\mathcal{F}$ denote the current allocation set of over-provisioning CPs. Each iteration decreases the number of CPs in set $\mathcal{F}$, which is lower bounded by 0. Thus, Algorithm 1 is convergent.

Then, there are two cases after allocation in each iteration: (1) there exists at least one $C{P}_i\in \mathcal{F}$ satisfied ${\overline{\lambda }}_i^a<0$; (2) all CPs obtain non-negative allocation, i.e., ${\overline{\lambda }}_i^a>0$ for any $C{P}_i\in \mathcal{F}$. In the first case, to maximize the utility of $\mathcal{F}$, $C{P}_i$ needs to outsource the services. However, according to the light-federation rule, that will not happen because outsourcing the services for an over-provisioning CP will decrease its utility. In this case, the optimal allocation ${\overline{\lambda }}_i^a$ for $C{P}_i$ is zero. Hence, $C{P}_i$ will be removed from $\mathcal{F}$ in current iteration, and the allocation should be continued in the next iteration. In the second case, according to Lemma 1, it has obtained an optimal allocation for all dropped requests among CPs in $\mathcal{F}$ if ${\lambda }_d+{\sum }_{C{P}_i\in {\mathcal{N}}_o}{\lambda }_i<{\sum }_{C{P}_i\in {\mathcal{N}}_o}{\mu }_i$.   □

It is worth noting that, after the request allocation of the broker, each CP in over-provisioning state might be faced with two cases: ${\overline{\lambda }}_i^a+{\lambda }_i>{\lambda }_i^{*}$ and ${\overline{\lambda }}_i^a+{\lambda }_i\le {\lambda }_i^{*}$. Although all the requests dropped by under-provisioning CPs seem to be allocated, it cannot guarantee to be processed if ${\overline{\lambda }}_i^a+{\lambda }_i$ exceeds the optimal process service rate ${\lambda }_i^{*}$, and the rate of dropped requests is ${\overline{\lambda }}_i^a+{\lambda }_i-{\lambda }_i^{*}$.

According to the assumption of transferable utility, the utility of $C{P}_i$ who completes the service requests from others is as follows:

$$u_i=\left\{\begin{array}{cc}k({\lambda }_i+{\overline{\lambda }}_i^a)-c\frac{{\lambda }_i+{\overline{\lambda }}_i^a}{{\mu }_i-({\lambda }_i+{\overline{\lambda }}_i^a)}\hfill & if {\lambda }_i+{\overline{\lambda }}_i^a<{\lambda }_i^{*}\hfill \\ k{\lambda }_i^{*}-c\frac{{\lambda }_i^{*}}{{\mu }_i-{\lambda }_i^{*}}\hfill & if {\lambda }_i+{\overline{\lambda }}_i^a\ge {\lambda }_i^{*}\hfill \end{array}\right.$$

(17)

Proposition 1.

Light-federation sharing is individually rational.

A mechanism is individually rational if the utility of each player is non-negative [32]. According to Equation (17), each $C{P}_i$ obtains utility in the federation that is no less than the utility obtained in operating individually, which means that the cloud providers can obtain non-negative utility if participating the cloud federation, so the light-federation sharing is individually rational.

Proposition 2.

The service request allocation of CLFS is Pareto-optimal allocation.

Proof. 

Although Algorithm 1 obtains optimal request allocation among over-provisioning CPs, it does not provide optimal request allocation among all CPs. This means that there exit $C{P}_i$, $C{P}_j$, ${\lambda }_i>0,{\lambda }_j>0$ but $\frac{w_i}{w_j}\ne \frac{\sqrt{{\mu }_j}}{\sqrt{{\mu }_i}}$. This is because a $C{P}_i$ in low demand state does not drop its requests, even if there exits a $C{P}_j$ in over-provisioning state whose process rate is far higher than that of $C{P}_i$.

Next, we explain why CLFS is Pareto optimal by analyzing all three cases during the allocation:

• Case 1: ${\mathcal{N}}_P=\varphi$ (${\lambda }^d=0$), that is, all the CPs are in over-provisioning state. In this case, any reallocation for service requests will decrease the utility of CPs who lose service requests.
• Case 2: ${\mathcal{N}}_P\ne \varphi$, and there exits $C{P}_i$ in over-provisioning without obtaining request allocation after allocation ( ${\overline{\lambda }}_i^a=0,C{P}_i\in \mathcal{N}\backslash {\mathcal{N}}_p$). In this case, all the CPs with ${\overline{\lambda }}_j^a>0$ must still be in over-provisioning state, and the CPs with ${\lambda }_i^d\ne 0$ are in an optimal state, since ${\lambda }_i^{*}={\lambda }_i-{\lambda }_i^d$. Any reallocation among the over-provisioning state will decrease the utility of CPs who lose service requests, and increasing or decreasing the service requests for CPs in an optimal state will decrease their utility too.
• Case 3: ${\mathcal{N}}_P\ne \varphi$, and after the request allocation of the broker, all the CPs in over-provisioning state obtain request allocation. That is, all cloud providers in $N_p$ will either still be in an over-provisioning state or change to an under-provisioning state. In the former case, each CP will have decreased utility if there is a decrease the service request allocation. In the later case, these CPs will be in an optimal state by dropping overfull service requests, to obtain optimal utility, and no further allocation could improve utility.

Therefore, service request allocation in CLFS is Pareto optimal.   □

4.2. Cloud Cooperative-Federation: From the Perspective of a Coalition Game

In this section, we assume that the CPs participating in the federation are fully cooperative. That is, each CP abides by cooperation rules that have been agreed in advance among the CPs in the federation. The profit shared by all participants in federation equals the total profits of the federation. Therefore, the broker decides on the best possible request transfer policy to the CPs, and each CP always serves requests allocated by the broker. The determination of the optimal request allocation strategy is converted into solving an optimization problem defined by Problem (2). Let $U\left(\mathcal{N}\right)=max{\sum }_{C{P}_i\in \mathcal{N}}[V\left({\lambda }_i^a\right)-c{\lambda }_i^a{w}_i]$.

4.2.1. The Properties of Cloud Cooperative-Federation

The cloud cooperative-federation game is defined by a pair $(\mathcal{N},v)$, where $v:2^N\to R$ is the value function that assigns a real value to each possible coalition $S\subseteq \mathcal{N}$. The value of the coalition S is defined as $v\left(S\right)$. In Cloud Cooperative-Federation Sharing (CCFS), we define the value of coalition S as $v\left(S\right)=U\left(S\right)-{\sum }_{C{P}_i\in S}U\left(\left\{C{P}_i\right\}\right)$, which presents the surplus profit obtained by coalition S.

Firstly, we introduce the optimal request allocation in CCFS. According to the algorithm designed in Section 4.1, we designed a modified CCFS request allocation algorithm which is shown in Algorithm 2. Line 2 obtains the sum of service request rates $\lambda$ from all CPs. Lines 4–8 completes optimal request allocation for $\lambda$, which is similar to Algorithm 1 based on Equation (10) and Corollary 1. Lines 9–11 compute the request rate actually allocated to each $C{P}_i$ if the allocated request rate of $C{P}_i$ exceeds its optimal arrival rate. In Algorithm 2, service requests from all CPs get the optimal allocation to obtain optimal federation utility.

Algorithm 2: Service request allocation algorithm in CCFS.

A coalitional game $(\mathcal{N},v)$ with transferable utility is said to be superadditive if for any two disjoint coalitions $S_1,S_2\subset \mathcal{N}$, $v(S_1\cup S_2)\ge v\left(S_1\right)+v\left(S_2\right)$.

Theorem 2.

The coalitional game $(\mathcal{N},v)$ for a cloud federation is superadditive.

Proof. 

Let $S_1$ and $S_2$ denote two disjoint coalitions, $S_1,S_2\subset \mathcal{N}$, and the utility and the value of coalitions $S_1$ and $S_2$ are as following:

$$U\left(S_1\right)=max\sum _{C{P}_i\in S_1}[V\left({\lambda }_i^a\right)-c{\lambda }_i^a{w}_i],$$

$$v\left(S_1\right)=U\left(S_1\right)-\sum _{C{P}_i\in S_1}U\left(\left\{C{P}_i\right\}\right),$$

$$U\left(S_2\right)=max\sum _{C{P}_i\in S_2}[V\left({\lambda }_i^a\right)-c{\lambda }_i^a{w}_i],$$

$$v\left(S_2\right)=U\left(S_2\right)-\sum _{C{P}_i\in S_2}U\left(\left\{C{P}_i\right\}\right).$$

Since

$$\begin{array}{cc}\hfill U\left(S_1\right)+U\left(S_2\right)=& max{\sum }_{C{P}_i\in S_1}[V\left({\lambda }_i^a\right)-c{\lambda }_i^a{w}_i]\\ \hfill & +max{\sum }_{C{P}_i\in S_2}[V\left({\lambda }_i^a\right)-c{\lambda }_i^a{w}_i]\\ \hfill \le & max{\sum }_{C{P}_i\in \{S_1\cup S_2\}}V\left({\lambda }_i^a\right)-c{\lambda }_i^a{w}_i\\ \hfill =& U(S_1+S_2),\end{array}$$

we have that

$$\begin{array}{cc}\hfill v(S_1+S_2)=& U(S_1+S_2)-{\sum }_{C{P}_i\in \{S_1\cup S_2\}}U\left(\left\{C{P}_i\right\}\right)\\ \hfill \ge & U\left(S_1\right)+U\left(S_2\right)-{\sum }_{C{P}_i\in \{S_1\cup S_2\}}U\left(\left\{C{P}_i\right\}\right).\end{array}$$

Since ${\sum }_{C{P}_i\in \{S_1\cup S_2\}}U(\left\{C{P}_i\right\}={\sum }_{C{P}_i\in S_1}U\left(\left\{C{P}_i\right\}\right)+{\sum }_{C{P}_i\in S_2}U\left(\left\{C{P}_i\right\}\right)$, we have that $v(S_1+S_2)\ge v\left(S_1\right)+v\left(S_2\right)$. □

The superadditivity of the game $(N,v)$ guarantees that federation will never detract from the members’ expected profits in aggregate. Theorem 2 guarantees that cloud providers can improve their expected profits by forming federations with other providers and jointly abiding by cooperative rules. Moreover, the larger the federation, the greater the improvement in the federation’s expected profit—indicating that the most profitable federation is called the grand federation $\mathcal{N}$.

4.2.2. Profit Sharing in CCF

Now, one of the fundamental questions in cooperative game theory is how to allocate the surplus achieved by federation among the agents. Generally, a federation mechanism considers one main property—fairness. Fairness means the profit obtained by a federation should be divided fairly according to the contributions of the members.

To this end, Shapley (1953) proposed to remunerate agents with payoffs that correspond to their individual marginal contributions to the game. The Banzhaf value is a division of payoffs for the grand federation that takes into account the power of the players, which does not consider the order of the players entering the federation. In our case, we assume that each CP is equally likely to join any federation; the Banzhaf value is more reasonable for the payoff division.

The Banzhaf value of cloud provider $C{P}_i$ in cloud federation $\mathcal{N}$ is defined as follows:

$${\beta }_i\left(\mathcal{N}\right)=\frac{1}{2^{n-1}}\sum _{S\subseteq \mathcal{N}\backslash \left\{C{P}_i\right\}}[v\left(\mathcal{S}\cup C{P}_i\right)-v\left(\mathcal{S}\right)].$$

(18)

The value of ${\beta }_i\left(\mathcal{N}\right)$ expresses the marginal contribution of $C{P}_i$ over all possible federations containing $C{P}_i$. The marginal contribution of $C{P}_i$ in federation S is the difference between the value of the federation with and without $C{P}_i$, expressed by $v\left(\mathcal{S}\right)-v\left(\mathcal{S}\backslash C{P}_i\right)$. The normalized Banzhaf index of $C{P}_i$ is defined as

$${\mathcal{B}}_i\left(\mathcal{N}\right)=\frac{{\beta }_i\left(\mathcal{N}\right)}{{\sum }_{C{P}_i\in \mathcal{N}}{\beta }_i\left(\mathcal{N}\right)}.$$

(19)

${\beta }_i\left(\mathcal{N}\right)$ gives a fair division method of grand federation profits among its participants. The payoff of each CP is calculated as follows:

$$p_i\left(\mathcal{N}\right)={\mathcal{B}}_i\left(\mathcal{N}\right)v\left(\mathcal{N}\right).$$

(20)

The utility of $C{P}_i$ obtains the utility $u_i$ expressed by

$$u_i=U\left(\left\{C{P}_i\right\}\right)+p_i\left(\mathcal{N}\right).$$

(21)

Cloud Cooperative-Federation Sharing can obtain higher net value for a cloud network than Cloud Light-Federation Sharing, but the computing of the payoff is a complex problem when the number of CPs is very large. According to the proof in the literature [33], computing the Banzhaf value for a game with a large number of players is NP-hard. Therefore, the computation of profit allocation in CCF is NP-hard.

5. Performance Evaluation

We performed a set of experiments to investigate the efficiency of proposed federation sharing solutions.

5.1. Experiment Setup

To evaluate the performance of proposed federation solutions, we constructed two scenarios.

Scenario 1: Two-player federation

Firstly, we considered that the federation consists of two agents. As mentioned in Section 1, one of the datasets was from the Gaia cluster at the University of Luxemburg, and the other was from the national grid of the Czech republic. We chose the workload logs both from May to August 2014. The data fields of each workload log contain job number, submit time, wait time, run time, number of allocated processors, and average CPU time, from which we can obtain the key parameter ${\lambda }_i$, the number of submitted jobs per hour. In our experiments, the number of allocated processors and average CPU time can be considered as the number of allocated VMs and average VM time.

In our experiments, we set the value of a single job $k=1$ and user’s waiting cost of unit time $c=20$. First, we selected capacity $C_i$ which can make job process rate satisfy ${\mu }_i={\gamma }_i\frac{Numberoftotaljobs}{Timeofthreemonths\left(Hours\right)}$, ${\gamma }_i>1$. This indicates that the average job process rate ${\mu }_i$ is larger than the average job arrival rate ${\lambda }_i$ for each $C{P}_i$. Figure 1 shows the curves of job arrival rates and process rates. If the job arrival rate is higher than the optimal arrival rate, it is in under-provisioning state; otherwise, it is in over-provisioning state. In the two-player scenario, we observe the results by changing parameter ${\gamma }_i$.

Table 1. Experimental settings.

	Job Process Rate ${\mathit{\mu }}_{\mathit{i}}$	Job Arrival Rate ${\mathit{\lambda }}_{\mathit{i}}$
Two-player (1)	${\mu }_1=83\times 1.25$, ${\mu }_2=123\times 1.25$	Obtained by workload log
		of MetaCentrum (${\lambda }_1$) and UniLu Gaia (${\lambda }_2$)
Two-player (2)	${\mu }_1=83\times 200$, ${\mu }_2=123\times 1.25$	Obtained by workload log
		of MetaCentrum (${\lambda }_1$) and UniLu Gaia (${\lambda }_2$)
Three-player	${\mu }_1=40$, ${\mu }_2=80$, ${\mu }_3=120$,	Generated randomly with ${\lambda }_1=20$, ${\lambda }_2=40$, ${\lambda }_3=60$
Multi-player	Generated randomly with ${\mu }_i\in [50,100]$	Generated randomly with ${\lambda }_i\in [50,120]$

lists the parameters of two cases of this scenario, the results of which will be analyzed later.

Scenario 2: Multi-player federation

In this scenario, we extended the experiments to cloud federation with multiple players.

First, we performed experiments in a 3-player federation. We generated the service requests randomly with average arrival rates ${\lambda }_1=20,{\lambda }_2=40,{\lambda }_3=60$, and $\mu =2{\lambda }_i,$ $i=1,2,3$. To evaluate the performance with more players, we also performed experiments with 1000 players in the system. Table 1 lists the parameters in different scenarios.

Next, to demonstrate the queuing model used in our work, we used the analysis similar to [4]. The size of a job is expressed by the product of the number of allocated VMs and average VM time. We assume that the job size follows the exponential distribution with mean value L. Hence, let $C_i$ denote the capacity of cloud system $C{P}_i$; the average service rate (in jobs/sec) for $C{P}_i$ is ${\mu }_i=C_i/L$; and the service time of a job is exponentially distributed with mean $1/{\mu }_i$.

To evaluate the efficiency of the proposed federation we compared three approaches: running individually without federation sharing (WFS), Cloud Light-Federation Sharing (CLFS), and Cloud Cooperative-Federation Sharing (CCFS). In running individually without federation, all CPs operated in terms of optimal policy formula (6). In Cloud Light-Federation Sharing, all CPs only need to consider the benefit of themselves. The objective of Cloud Cooperative-Federation Sharing is to maximize the utility of whole federation, while the property of the profit allocation—fairness—and stability should be considered.

In our experiments we compared two metrics: the expected number of total dropped jobs and the total utility of all CPs.

5.2. Result Analysis

In a two-player scenario, first we chose ${\gamma }_1={\gamma }_2=1.25$. Figure 2a shows the total numbers of dropped jobs under the three solutions. As expected, the numbers of dropped jobs in CLFS and CCFS are obviously less than in WFS. That is because the dropped jobs in WFS have a chance to be reallocated in federation environment. Furthermore, we found that the expected numbers of dropped jobs in CLFS were equal to that in CCFS. That is because the job dropping strategy is taken in CLFS and CCFS only when the job arrival rate is higher than the optimal arrival rate. Therefore, the total expected numbers of dropped job must be the same in CLFS and CCFS.

The total expected numbers of dropped jobs were the same, which means that the total completed job rates in CLFS and CCFS were also the same. However, since the expected delay costs in different CPs were different, the jobs processed resulted in different utility. Figure 2b shows the total utility levels of all CPs in three scenarios. We see that the total utility in WFS was less than the total utility under CLFS and CCFS; and the utility under CLFS was less than that under CCFS. This is because, when the job arrival rate is less than the optimal arrival rate, the CP under CLFS will not submit the jobs to other CPs even if there are lower costs in other CPs.

Next, we demonstrate in what case the CCFS is obviously better than CLFS. When all the CPs in the federation are in an over-provisioning state, no CP will submit its jobs to other CPs in CLFS because it will decrease its own utility. However, if the capacity of one CP is much higher than those of the others, submitting the jobs to this CP will increase the total utility of the federation. Figure 3a,b shows the results of the number of dropped jobs and total utility with the ${\gamma }_1=200,{\gamma }_2=1.25$. The setting constructs a scenario that an over-provisioning CP has much more capacity than the other CP which is in an over-provisioning state too. We can see that the total utility of CLFS is obviously less than that of CCFS during some periods, such as the day intervals $[16,22]$ and $[60,66]$, even if the two CPs are both in an over-provisioning state or an under-provisioning state. This is because CCFS performs a global optimal allocation for all service requests, whereas CLFS only performs an optimal allocation for the over-provisioning CPs for all overfull service requests.

The significant results are more likely to be reflected in a multi-player federation. The results with three CPs are shown in Figure 4a,b. We randomly generated the service requests for three cloud systems with mean rates ${\lambda }_1=20$, ${\lambda }_2=40$, and ${\lambda }_1=60$, respectively. Although the number of dropped jobs under solutions of CLFS and CCFS shown in Figure 4a was zero, CCFS was significantly better than CLFS in the total utility of all CPs, which is shown in Figure 4b.

We can ignore the computation complexity if few CPs participate the federation, such as in two-player and three-player scenarios, and using CCFS can obtain optimal utility. However, according to the description in Section 4.2, when we increase the number of cloud providers to n, to sharing the profit fairly, CCFS needs to consider $2^{n-1}$ cases for each $C{P}_i$.

We increased the number of CPs to 1000 and repeated the experiments. ${\lambda }_i$ was generated in interval [50, 120], and ${\mu }_i$ was generated in interval [50, 100] randomly, which made the total cloud resources in the cloud network not sufficient. The results are shown in Figure 5a,b. We can see that the numbers of dropped jobs in CLFS and CCFS were significantly lower than in WFS, and the total utility obtained in CLFS and CCFS was significantly higher than that in WFS. The performances of CLFS and CCFS were very similar, and the numbers of dropped jobs were same. When the whole cloud network is in overload, CLFS and CCFS can obtain the same total utility. The experimental results show that CLFS can still obtain good performance with simplified federation rules in a federation with a large number of CPs.

6. Conclusions

In this paper, to improve the net utility of cloud network, we proposed two resource sharing solutions: Cloud Light-Federation Sharing (CLFS) and Cloud Cooperative-Federation Sharing (CCFS), based on a queuing model. CLFS obtains Pareto optimal allocation for service request allocation, and does not require complex request and profit computation. In CCFS, to obtain optimal total utility, CPs in the federation are fully cooperative, and there is a fair profit allocation rule. Experimental results showed that the two federation methods can significantly improve the total utility and decrease the number of dropped jobs, compared to the allocation without federation sharing. Although Cloud Light-Federation Sharing is a simplified method with simple federation rules, its performance is very similar to that of Cloud Cooperative-Federation Sharing.

This study was a first attempt to address the problem of maximizing the net utility of cloud federation based on queuing theory. In this work, we did not consider the differences in service requests, and all the service requests were dealt with by the same allocation rules. In reality, of course, this is not quite the case. For example, some tasks may need to obey strict delay bounds, and some are very important. Our future work will be to study the resource sharing mechanism by classifying the tasks into different levels according to their characteristics.