Enhanced Scheduling of AI Applications in Multi-Tenant Cloud Using Genetic Optimizations

Abstract

The artificial intelligence (AI) industry is increasingly integrating with diverse sectors such as smart logistics, FinTech, entertainment, and cloud computing. This expansion has led to the coexistence of heterogeneous applications within multi-tenant systems, presenting significant scheduling challenges. This paper addresses these challenges by exploring the scheduling of various machine learning workloads in large-scale, multi-tenant cloud systems that utilize heterogeneous GPUs. Traditional scheduling strategies often struggle to achieve satisfactory results due to low GPU utilization in these complex environments. To address this issue, we propose a novel scheduling approach that employs a genetic optimization technique, implemented within a process-oriented discrete-event simulation framework, to effectively orchestrate various machine learning tasks. We evaluate our approach using workload traces from Alibaba’s MLaaS cluster with over 6000 heterogeneous GPUs. The results show that our scheduling improves GPU utilization by 12.8% compared to Round-Robin scheduling, demonstrating the effectiveness of the solution in optimizing cloud-based GPU scheduling.

1. Introduction

With the rapid advancement in artificial intelligence (AI) industries and the explosion of large datasets, heterogeneous GPUs within MLaaS (Machine Learning as a Service) frameworks are being swiftly deployed to execute various AI applications [1]. These extensive GPU clusters are crucial for handling a huge number of machine learning (ML) tasks [2,3], such as training and inference, significantly speeding up processes across numerous AI applications [4,5].

However, the diversity of ML tasks coupled with varied GPU configurations, poses significant challenges in resource management and scheduling. Task scheduling has been widely studied in a variety of systems, ranging from low-end mobile devices to high-end industrial systems [6,7,8]. However, ML workloads have significantly different resource usage patterns from traditional applications, making it difficult to efficiently manage system resources [9,10,11]. In particular, GPUs in large-scale production clusters often exhibit low utilizations of less than 50% when executing ML workloads [12,13,14]. This underutilization of advanced GPU features is primarily due to the increasing number of small tasks, such as inference, and this trend is expected to continue in the near future [15].

In response to these challenges, the Multi Instance-GPU (MIG) feature within NVIDIA’s Ampere architecture has been designed to enable GPU sharing across tasks [16]. Nevertheless, this solution does not ensure high GPU utilization, as fragmentation issues often arise when GPUs are only partially allocated. This fragmentation is a recurring problem in shared-GPU environments.

Traditional solutions such as packing strategies have been utilized to mitigate fragmentation issues [17]. Conceptually, task scheduling in GPU environments can be likened to a multi-dimensional bin-packing problem, where tasks and nodes are comparable to spheres and boxes, aiming to minimize the number of boxes used. However, traditional bin-packing methods often fall short in environments with multiple GPUs, due to intrinsic differences in the scheduling needs of GPU sharing.

To better tackle these issues, this paper explores two bin-packing models tailored for GPU scheduling. The first model seeks to consolidate multiple GPUs into a single machine by aggregating their capacities. Although theoretically appealing, this model falls short in practical scenarios due to its failure to consider individual GPU fragmentation. The second model treats each GPU as a separate resource dimension, acknowledging that GPUs, unlike CPUs and memory, are interdependent, with tasks on one GPU often relying on others, rendering traditional bin-packing strategies insufficient for real-world GPU scheduling challenges.

To address these complexities, we propose a novel scheduling method that incorporates a genetic optimization technique within the Simpy 4.1.1 discrete-event simulation framework [18]. This approach specifically targets the issue of GPU underutilization due to fragmentation and aims to enhance overall GPU efficiency. By conducting large-scale simulations on Alibaba’s PAI (Platform for AI) cluster, which includes over 1800 nodes and more than 6000 GPUs [19,20], we demonstrate that our scheduling policy improves GPU utilization by 12.8% compared to Round-Robin scheduling and also performs better than the established bin-packing-based scheduling method, Tetris [21]. This substantial improvement highlights the effectiveness of our approach in optimizing resource allocation within cloud-based ML environments.

The remainder of this paper is organized as follows. In Section 2, we will briefly discuss the problem definition and the basic model for our scheduling. Section 3 describes the genetic optimization of our task scheduling in detail. Section 4 validates the effectiveness of the proposed scheduling based on simulation experiments. Finally, Section 5 concludes this paper.

2. Problem Definition and the Basic Model

2.1. Overview of GPU Utilization in Multi-Tenant Environments

GPUs are costly resources when compared to CPUs and other computing resources. In multi-tenant environments executing ML workloads, it is common to encounter significant underutilization of GPUs. Reports from various industries indicate that the average GPU utilization often remains below 50%, which poses substantial challenges in resource planning within large-scale AI server clusters [12,13]. Notably, in Alibaba’s cloud machine learning platform, PAI, approximately 76% of all GPUs show an average utilization of less than 10% [22], highlighting a systemic issue where many training tasks do not fully leverage the capabilities of modern GPUs [23,24].

2.2. Challenges of GPU-Sharing Technology

To address underutilization, GPU-sharing technology has been implemented, which allows the partial allocation of a single GPU to perform multiple tasks. This technology aims to reduce the number of GPUs required per workload, thereby enabling more ML tasks to be executed within the same resource framework. However, as the volume of ML workloads increases, the cluster can experience extended waiting times, task timeouts, and even scheduling failures. Particularly, when GPU allocation ratios exceed 80%, the system begins to suffer from severe GPU fragmentation, making it impossible to accommodate new tasks despite available GPUs, thus leading to resource wastage [25].

2.3. GPU Fragmentation and Allocation Challenges

Unlike traditional resource fragmentation, GPU fragmentation is exacerbated by the need for the continuous allocation of GPU segments. This requirement is fundamentally different from other resource allocations, such as disk space, where discontinuous allocation can resolve fragmentation [26]. Also, traditional task-scheduling approaches, akin to multi-dimensional bin-packing problems, fail to address the unique challenges posed by GPU-sharing technology [27,28]. These approaches typically model tasks and nodes as spheres and boxes, aiming to minimize the number of boxes. However, in environments where GPUs are shared, traditional bin-packing strategies become ineffective, because they do not account for the unique allocation boundaries and interchangeable nature of partial GPU allocations.

2.4. Scheduling System Architecture

As depicted in Figure 1, our proposed scheduling system architecture accommodates various types of ML jobs developed in frameworks like TensorFlow 2.1.0, PyTorch 1.11.0, and Graph-Learn 0.0.3. These jobs are parsed into tasks based on their roles, with each task comprising multiple task instances executed across several cluster machines. We monitor the CPU, memory, and GPU resource requirements from the user-submitted jobs and search for optimized pairs of machines and task instances that can satisfy these demands. Our scheduling method prioritizes pairs that significantly enhance GPU utilization within the cluster.

2.5. Optimization of GPU Cluster Utilization

Our primary goal of scheduling is to optimize GPU utilization by efficiently assigning $n$ task instances $[t_1,\dots ,t_n]$ to the appropriate machines $[m_1,\dots ,m_N]$, considering the resource requirements of each task instance and the available resources of each machine. The hierarchical structure of jobs, tasks, and task instances is depicted in Figure 2. Our scheduling algorithm first merges the list of task instances with available machines in the cluster, to create a candidate-scheduling list $V$ containing all feasible machine and task instance pairs ($m_j,t_i$). Only pairs where a machine $m_j$ can accommodate a task instance $t_i$ based on resource availability, are included in the candidate list. For a machine $m_j$ to host a task instance $t_i$, the available CPU, memory, and GPU resources $[P_j$, $Q_j$, $R_j]$ of the machine $m_j$ must meet the CPU, memory, and GPU resource demands $[p_i$, $q_i$, $r_i]$ of the task instance $t_i$. The key objective is to find the pair of task instance and machine with the highest GPU utilization, denoted as $[m^{*},t^{*}]$. By assigning task instance $t^{*}$ to machine $m^{*}$, the scheduling of a single task instance is completed. This process is repeated for all task instances submitted to the cluster. The details of this algorithm are outlined in Algorithm 1, which provides a step-by-step description of the process.

Algorithm 1 Overall Scheduling Algorithm

1: $machine M$   2: $task instance T$   3: $candidates V$   4: while True do   5:         $M: [m_1,\dots ,m_N]$ $\leftarrow$ $get\_machines\_of\_cluster\left(\right)$   6:         $T$: $[t_1,\dots ,t_n]$ $\leftarrow$ $get\_task\_instances\_waiting\_to\_be\_scheduled\left(\right)$   7:                    for $m$ in $M$ do   8:                         for $t$ in $T$ do   9:                                    if $m$ $can accommodate$ $t$ then 10:                                            $V$.append ($[m, t]$) 11:                                    end if 12:                         end for 13:                    end for 14:                    $best\_solution$ $\leftarrow$ $Genetic\_algorithm\_for\_scheduling\left(V\right)$ 15:                    $m^{*}$ $\leftarrow$ $best\_solution\left[0\right]$ 16:                   ${ t}^{*}$ $\leftarrow$ $best\_solution\left[1\right]$ 17:                    $t^{*}$ $assign to$ $m^{*}$ 18:                    if $all scheduling complete for jobs submitted to the cluster$ then 19:                               break 20:                    end if 21: end while

3. Scheduling Optimization by Genetic Algorithms

The genetic algorithm (GA) is a search heuristic inspired by the natural evolutionary processes of genetics [29]. In this paper, the search domain encompasses the available resource capacities of GPU within each machine in the cluster. The primary objective of the genetic algorithm discussed in this paper is to discover a scheduling solution that optimizes the utilization of GPUs across the entire cluster. To accomplish this goal, our genetic algorithm manages a specific number of solutions, referred to as a population, and evolves this population to find the best solution.

Our genetic algorithm initiates by generating an initial population randomly. It then chooses a subset of solutions from this population to produce new offspring solutions. As the algorithm progresses through generations, it sustains the population size by replacing some solutions with newly created ones to evolve the population. Ultimately, the solution with the highest fitness value, which will be defined later, within the final population is chosen as the resolution to the scheduling issue addressed by the genetic algorithm. Algorithm 2 provides a concise overview of our GA-based scheduling approach.

Our genetic algorithm operates by taking a list V containing pairs of <machine, task instance> as its input. Initially, V is designated as the starting set of solutions, and its fitness value is computed. The algorithm then proceeds with iterative generations to find the most superior solution. Parents are chosen to produce offspring in the mating pool, followed by the execution of a crossover operation. Subsequently, an adaptive mutation is introduced to generate the succeeding set of solutions. Then, the fitness values for the newly generated solutions are evaluated. This iterative process is repeated across multiple generations until the population converges. Finally, the best solution from the generated population is selected to determine which machine within the cluster each task instance will be assigned to. Figure 3 briefly illustrates the overview of our genetic algorithm.

Algorithm 2 Genetic Algorithm for Scheduling

Input: $valid pair list V$ Output: $optimized solution S$   1: $population P$$\leftarrow$ $V$   2: $calculate\_population\_fitness\left(P\right)$   3: while $num \le NUM\_OF\_GENERATIONS$ do   4:     $selects\_parents\_from\_mating\_pool\left(\right)$   5:     $do\_singlepoint\_crossover\left(\right)$   6:             $do\_adaptive\_mutation\left(\right)$   7:             $P$ $\leftarrow$ $generate\_next\_population P^{\prime }$   8:             $calculate\_population\_fitness\left(P\right)$   9: end while 10: $picked\_machine, picked\_taskInstance = get\_best\_solution\left(\right) from P$

3.1. Fitness Function

The genetic algorithm employs a fitness function to assess the effectiveness of solutions. The fitness function outlined in Algorithm 3 evaluates the quality of a solution, denoted as s, consisting of a machine id m and a task instance id t. The fitness function verifies whether the machine associated with m can accommodate the task instance linked to t. In order for task instance t to be accommodated by machine m, the available resources of machine m—denoted as P, Q, and R—must meet the required resources of task instance t, denoted as p, q, and r. If this condition is met, the function yields the increased GPU utilization as the fitness value when task instance t is scheduled on machine m. Conversely, if the accommodation is not feasible, the fitness value returned is −1.

Algorithm 3 Fitness Function

Input: $One of soultion s:[m_j, t_i]$ Output: $Fitness value f$  1: $j th machine m_j$$\leftarrow$ $s\left[0\right]$  2: $i th task instance t_i$ $\leftarrow$ $s\left[1\right]$  3: if $m_j can accommodate t_i$ then  4:     $f \leftarrow ∆ The increase of GPU utilization$  5: else  6:     $f \leftarrow -1$  7: end if  8: return $f$

3.2. Problem Encoding and Population Size

In our scheduling, a single solution is characterized by the pair ${[m}_j,t_i]$, where $m_j$ denotes the identifier of one of the $N$ machines in the cluster and $t_i$ represents the identifier of one of the $n$ task instances awaiting scheduling. This solution entails the allocation of task instance $t_i$ to machine $m_j$. By combining the $n$ task instances awaiting scheduling and the $N$ machines within the cluster, we generate a solution. Consequently, the entire population comprises all possible $N^n$ permutations of solutions that can be derived from these $n$ task instances and $N$ machines.

Our solution encodes the machine id to which each task instance should be assigned, using an array of integers as shown in Figure 4. Each element of the solution represents the machine id where the corresponding indexed task instance will be placed. Therefore, the length of the solution is equal to the number of task instances to be scheduled.

To determine the population size, we investigate the relationship between the population size and fitness function ranges. Figure 5 shows the fitness value of the final selected solutions when the population size is set to 10, 100, and 1000, respectively. As shown in the figure, a population size of 10 does not produce competitive results in terms of fitness value, but the fitness values are very similar when the population size is 100 and 1000. We choose a population size of 100, as it incurs a low overhead but exhibits sufficiently good results. Based on this, we randomly generate 100 solutions as the initial population.

3.3. Selection

The process of selecting parents for generating offspring in the next generation, known as the selection operation, is typically guided by the fitness function. This function prioritizes solutions with superior fitness levels, increasing their likelihood of being chosen as parents. However, an overemphasis on high-quality solutions during selection may result in premature convergence to a local optimum. To mitigate this issue, a tournament selection approach is implemented, wherein 20% of the population’s solutions are randomly sampled, and the top solution with the highest fitness value among these sampled solutions is designated as the parent solution. This method facilitates the selection of parents for producing the subsequent generation’s offspring solutions within the population, as illustrated in Figure 6.

3.4. Crossover and Mutation

After selection, crossover procedures are applied to the chosen parent solutions to produce offspring solutions for subsequent generations. This process involves combining parent solutions to yield new solutions, known as children, thereby facilitating the enhancement of the population over time. In our genetic algorithm, a widely recognized single-point crossover technique is employed to generate fresh offspring solutions. In a single-point crossover, each parent possesses a single cutting point, and the offspring are derived by replicating distinct segments of each parent’s solution based on this point. In the context of our scheduling, offspring solutions are created by interchanging combinations of machines and task instances through the single-point crossover operation, as illustrated in Figure 7a.

In genetic algorithms, a crossover operation can result in invalid solutions. In our case, as solutions are composed of items [m_j, t_i], where m_j is a machine and t_i is a task instance, some task instances might be duplicated, while others might disappear after crossover. This issue can be resolved by imposing a constraint on the encoding process to ensure that task numbers in a solution are in increasing order. Specifically, instead of task instances being placed in any order within the solution, we assume that they appear sequentially from t₁ to t_n. This assumption ensures that after a crossover, every task instance appears exactly once in the solution.

To assess the effectiveness of our single-point crossover operation, we investigate its fitness value in comparison with two other popular crossover operations: uniform crossover and scattered crossover. As shown in Figure 8, the three crossover methods exhibit very similar results. We chose the single-point crossover as it is the simplest among the three considered, yet it still exhibits competitive results.

The mutation operation is employed to alter particular components of solutions produced through crossover operations, in order to explore the problem space more extensively. These mutation operations are recognized for their role in preserving diversity among the population and preventing premature convergence towards a local optimum. Within our genetic algorithm framework, an adaptive mutation operation is implemented. This strategy involves initially computing the average fitness value across the population and establishing this value as a threshold. Solutions surpassing this threshold are subjected to a reduced mutation probability of 0.25, while those with lower fitness values are assigned a higher mutation probability of 0.1, thereby facilitating mutation operations. This methodology is illustrated in Figure 7b.

3.5. Replacement

After the completion of crossover and mutation procedures, the resultant offspring solutions are utilized to supplant the existing solutions within the population, thereby constituting a fresh population. In the context of our study, 40% of the solutions present in the population are substituted with newly derived offspring solutions. Generally, solutions possessing inferior fitness values are replaced by the newly generated solutions during each iteration. This mechanism functions by replacing incumbent generations, as opposed to merely appending newly mutated entities to the succeeding generation. Consequently, upon the creation of new entities after the selection, crossover, and mutation stages, an existing entity is systematically replaced by a novel one [30].

4. Performance Evaluation

In this section, we perform simulation experiments to validate the effectiveness of the proposed scheduling in terms of GPU utilization within clusters. For comparison, we simulate other scheduling algorithms utilized for machine learning tasks, along with the proposed scheduling. The results suggest that the proposed algorithm can function as a proficient scheduling strategy in cluster settings where GPUs are shared, leading to enhanced GPU utilization without performance penalties.

We implement a scheduling simulation framework to evaluate the performance of machine learning scheduling. The framework is based on Simpy, a process-based discrete-event simulation framework, and utilizes PyGAD 3.3.1, an open-source Python library for genetic algorithms [31]. This paper constructs the input data for scheduling based on ML workload traces collected over two months from the Alibaba PAI cluster. As shown in

Table 1. The experimental configuration of the proposed scheduling.

System	Number of CPUs	Memory (GB)	Number of GPUs	GPU Type	Number of Nodes
PAI	64	512	2	P100	798
	96	512	2	T4	497
	96	512	8	Misc	280
	96	384	8	V100M32	135
	96	512/384	8	V100	104
	96	512	0	N/A	83

, this cluster includes over 1800 nodes and more than 6000 GPUs, and the machines are heterogeneous in terms of hardware and resource configuration. The workloads submitted to the cluster are a mix of training and inference tasks and include a variety of machine learning algorithms such as CNN, RNN, transformer-based language models, and reinforcement learning. A total of 1200 tasks, comprising 7500 task instances, were submitted, and the trace data include submission time, cluster usage time, and requests for CPU, memory, and GPU quantities per instance. This paper uses a sampling of the entire workload trace to generate input data for scheduling.

As jobs are submitted at different times in real scenarios, our scheduling should be updated periodically, implying that the genetic algorithms are run using jobs submitted during the current period, as well as those remaining in the queue from the previous period that have not yet been dispatched. It is important to note that this periodic scheduling does not affect jobs that have already been dispatched and are currently being executed. In our performance evaluations, which are conducted through simulations that incorporate complete job histories, we recognize the submission times of all jobs from the beginning of our scheduling process. Nevertheless, we perform our scheduling periodically, assuming knowledge of only those job submission times that occur up to the current simulation point.

4.1. Convergence Test

To ensure the convergence of the genetic algorithm we implement, we assess its performance by comparing the fitness metrics of individual solutions within the existing population over time. Figure 9 illustrates the trend of fitness values observed throughout the genetic algorithm’s evolution, where the fitness values steadily increase towards the maximum possible value across successive generations within a specific scheduling unit. This graphical representation clearly delineates the evolutionary trajectory of the population as it seeks to pinpoint the optimized solution by applying the genetic algorithm to each scheduling unit. Additionally, this visual aids in monitoring the gradual convergence of fitness values as generations progress within each scheduling unit.

4.2. GPU Utilization

In this subsection, we present experimental results for a comparison between the proposed scheduling and other existing scheduling policies in terms of cluster GPU utilization. The scheduling policies compared to our algorithm include First-Fit, Round-Robin, and the Tetris algorithm, which schedules tasks based on a bin-packing approach. Figure 10 displays a graph showing the GPU utilization of the cluster over the course of a day, with jobs scheduled based on time of day. The x-axis represents the number of jobs processed up to that point, and the y-axis shows the GPU utilization of the cluster after scheduling the corresponding number of jobs.

From the graph, it is evident that the Round-Robin scheduling algorithm, which is widely used in task scheduling, is not suitable for maximizing GPU utilization when considering the diversity of ML workloads and the heterogeneity of GPU resources in the cluster. Note that Round-Robin scheduling assigns tasks in the queue to each GPU machine in turn without considering the load of the tasks and the capacity of GPUs, resulting in some GPUs being overloaded, while others are underutilized.

Tetris and First-Fit exhibit similar GPU utilization patterns, with each algorithm alternately gaining and losing advantage over time. Note that the Tetris algorithm schedules tasks by projecting each task’s resource demands into Euclidean space and selecting the task–machine pair with the highest dot product, proving suboptimal for maximizing GPU utilization. As shown in the figure, however, the proposed algorithm consistently achieves the best results in terms of GPU utilization across the cluster. This is due to the optimized selection of task–machine pairs in terms of GPU utilization, leading to the minimized fragmentation of GPU resources across clusters. The improvement of the proposed algorithm against Round-Robin, First-Fit, and Tetris in terms of GPU utilization is 12.8%, 6.5%, and 6.6%, respectively.

4.3. Completion Time

Even if GPU utilization increases, a performance penalty may occur from the user’s perspective. To validate this, we measure the completion time of jobs for each algorithm and compare the results. Figure 11 depicts the completion time of jobs when scheduled using our proposed algorithm and the comparison algorithms, First-Fit, Round-Robin, and Tetris. The completion time of each job is calculated by subtracting the start time from the finish time of the job. The graph illustrates that our algorithm achieves completion times for scheduled jobs like those of the other three algorithms, implicating that it can perform scheduling without a performance overhead. This demonstrates the efficiency of our algorithm in handling job scheduling, while maintaining a competitive performance from a user perspective.

4.4. Average Slowdown

In our study, the aim of the genetic optimization is to improve GPU utilization, and thus the fitness function was defined accordingly. So, our optimization does not explicitly consider the waiting time of each task when evaluating solutions. However, as GPU utilization improves, the overall waiting time for tasks is also reduced, by maximizing the activation of idle GPUs. To demonstrate this, we show the average slowdown of tasks for each algorithm in Figure 12. As shown in the figure, our scheduling performs significantly better than Round-Robin scheduling with respect to the average slowdown of tasks. Of course, our scheduling does not address fairness issues related to the waiting times of individual tasks, which may lead to some tasks experiencing longer waits. However, the fairness issue is not explicitly considered by other algorithms either, and it is not significantly problematic in our algorithm, as demonstrated by our experimental results. Nevertheless, if addressing this issue is indeed necessary, it is possible to define a fitness function that imposes penalties when there is a significant variance in the waiting times of tasks.

5. Conclusions

Modern MLaaS clusters utilize GPU-sharing technology to manage ever-growing machine learning workloads across heterogeneous GPU resources. However, this approach can cause resource fragmentation, which in turn decreases overall GPU utilization. In this paper, we addressed the challenge of suboptimal GPU utilization within large-scale clusters equipped with heterogeneous GPUs. We introduced a novel scheduling method that employs genetic optimization techniques and compared it with existing scheduling algorithms. Our genetic optimization process consists of machine and task instance pair encoding, single-point crossover, adaptive mutation, and replacement. Our scheduling is performed periodically based on newly submitted job information during the current period, as well as the remaining jobs in the queue from the previous period that have not yet been dispatched. We ensured that our scheduling can be applied to a variety of ML workloads including CNNs, RNNs, transformer-based language models, and reinforcement learning tasks. This encompasses over 1200 tasks, comprising 7500 task instances, on large-scale heterogeneous GPU clusters with over 1800 nodes and 6000 GPUs. Our findings demonstrated that the proposed method enhances GPU utilization in the cluster by 12.8% compared to the widely used Round-Robin algorithm, without incurring performance trade-offs, when compared to conventional scheduling strategies.