Energy Idle Aware Stochastic Lexicographic Local Searches for Precedence-Constraint Task List Scheduling on Heterogeneous Systems

Abstract

The use of parallel applications in High-Performance Computing (HPC) demands high computing times and energy resources. Inadequate scheduling produces longer computing times which, in turn, increases energy consumption and monetary cost. Task scheduling is an NP-Hard problem; thus, several heuristics methods appear in the literature. The main approaches can be grouped into the following categories: fast heuristics, metaheuristics, and local search. Fast heuristics and metaheuristics are used when pre-scheduling times are short and long, respectively. The third is commonly used when pre-scheduling time is limited by CPU seconds or by objective function evaluations. This paper focuses on optimizing the scheduling of parallel applications, considering the energy consumption during the idle time while no tasks are executing. Additionally, we detail a comparative literature study of the performance of lexicographic variants with local searches adapted to be stochastic and aware of idle energy consumption.

1. Introduction

According to the website www.top500.org, accessed on 10 June 2021, in November 2018, the top rank of High-Performance Computing (HPC) system Summit from the Oak Ridge National Laboratory is composed of 2,397,824 CPU. This HPC consumes 9783 kW and achieves a performance of 14.668 GFlops/watt under testing conditions. The final energy consumption is directly related to the quality of the scheduling of the tasks in the HPC system. It is hard to imagine a single scheduling algorithm that solves every kind of task load, which relates to the no-free-lunch theorem [1,2]. Therefore, several scheduling methods have been developed in the literature; in our particular case, we review the approaches when HPC administrators have a restricted time to optimize the final scheduling. To this end, we use the local search approach, which fits for the above statement as in [3,4,5], where the stopping criterion is set to a small amount of fixed objective function evaluations or a small amount of time. Unlike constructive heuristics, which construct the solution, adding one decision variable at a time, or metaheuristics, which require thousands of objective function evaluations (a long-run) [6].

An HPC system is composed of a network of processing units as CPU cores or machines to provide high parallel computing power. These systems are, in most cases, heterogeneous in their processing capabilities and power consumption. To build HPC with significant computing capabilities is common to combine numerous processing units to the final system. However, adding more processing units or machines to the system increases energy consumption in every aspect of the HPC (network devices, ram, hard drives, etc.) [7,8,9]. From the general reader’s perspective, the HPC systems estimate that data centers (a particular case of HPC system focused on data storage) will consume 1/5 of earth’s energy consumption by 2025.

This work tackles the precedence-constraint task scheduling of parallel programs over HPC systems formed by heterogeneous machines, minimizing the energy consumption and makespan, both being NP-Hard [10,11]. We pay special attention to the case when machines are not computing any task, but still are powered on (idle time); in our study model of energy for dynamic voltage frequency scaling (DVFS) CPUs [12], we assume that machines consume a minimum fixed amount of energy while idle. Though the nature of this optimization problem is bi-objective, according to [13] the energy consumption during idle times has a strong effect on the energy consumption. Therefore, we present the first study on different stochastic lexicographic local searches for precedence-constraint task scheduling of parallel programs to our knowledge, giving priority to the makespan (tasks computing time) to reduce idle times in machines.

The paper organization is: Section 2 details our studied problem in heterogeneous systems. Section 3 describes the list scheduling principle. A review of the literature on scheduling using local searches appears in Section 4. The experimental settings and our stochastic lexicographic local search variants are presented in Section 5 and Section 6, respectively. Section 7 analyzes the experimental results according to the makespan and energy objectives. Finally, Section 9 contains the conclusions and future work in local search and scheduling precedence-constraint tasks on heterogeneous systems.

2. Problem Description

The HPC system studied in this work, consist of a set of heterogeneous processing units M completely interconnected. Each processing unit $m_j\in M$ is DVFS capable. Thus, every machine $m_j$ operates on a set of multiple voltages. When the voltage is lower than the maximum, machines operate at a fraction of their top speed $r{s}_k$, the cardinality of the set S of relative speeds is equal to the cardinality of the set V of possible voltages $v_k$ inside the machine. The next table shows the set of voltage configurations with their respective relative speed used in our experimentation.

Every machine $m_j$ has assigned a DVFS configuration from

Table 1. Machine  settings.

	Configuration 0		Configuration 1		Configuration 2
k	$v_k$	$r{s}_k$	$v_k$	$r{s}_k$	$v_k$	$r{s}_k$
0	1.75	100%	1.50	100%	2.20	100%
1	1.40	80%	1.40	90%	1.90	85%
2	1.20	60%	1.30	80%	1.60	65%
3	0.90	40%	1.20	70%	1.30	50%
4	-	-	1.10	60%	1.00	35%
5	-	-	1.00	50%	-	-
6	-	-	0.90	40%	-	-

. For instances with more than three machines, the remaining use of these configurations are assigned in order, first the number zero, followed by the number one, and so on, in a circular structure. Without loss of generality, we consider the assumptions presented in [14,15,16,17,18,19], and the following:

• When a machine is not executing any task (idle time), it uses the lowest voltage possible.

2.1. Instances

An instance of the scheduling problem is composed of: (i) directed acyclic graph (DAG), and (ii) the computation time of the tasks in every $m_j\in M$. The set of tasks of the parallel program with their precedencies is a DAG. Thus, the program is represented as $G=(T,C)$, where T is the set of tasks and C is the set of communication costs (see Figure 1). A complete instance is formed by G and the computational times $P_{i,j}$ of each task $t_i$ in every processing unit $m_j$ at maximum capacity, when $k=0$ (see

Table 2. Computational times of the tasks at maximum capacity $P_{i,j}$.

Task	${\mathit{m}}_0$	${\mathit{m}}_1$	${\mathit{m}}_2$
$t_0$	11	13	9
$t_1$	10	15	11
$t_2$	9	12	14
$t_3$	12	16	10
$t_4$	15	11	19
$t_5$	13	9	5
$t_6$	11	15	13
$t_7$	11	15	10

).

Each task $t_i\in T$ cannot be initiated until all their precedence tasks $t_j\in T$ and communication have been finalized. For any pair of tasks executed in the same $m_j$, their communication cost is equal to zero.

2.2. Solution Representation

Our solution representation consists of two data structures. The first data structure is a matrix of size 2xn (see

Table 3. Configuration machine/voltage.

Task	${\mathit{t}}_0$	${\mathit{t}}_1$	${\mathit{t}}_2$	${\mathit{t}}_3$	${\mathit{t}}_4$	${\mathit{t}}_5$	${\mathit{t}}_6$	${\mathit{t}}_7$
machine	$m_0$	$m_0$	$m_1$	$m_0$	$m_0$	$m_2$	$m_0$	$m_2$
k	1	4	2	0	0	0	0	0

). n is the cardinality of T, the first row of the matrix stores the assigned processing unit, while the second stores the value of the k selected energy configuration. The machine configuration map to their respective voltage $v_k$ and relative speed $r{s}_k$ from Table 1.

The second data structure is an array of size n with the priority execution order of the tasks (see

Table 4. Tasks order of execution.

Task

t0

t4

t3

t1

t2

t5

t6

t7

). The execution order is a topological order from the task graph without violated precedence constraints.

The second data structure is not indispensable for scheduling but simplifies the problem’s search space because it does not necessarily verify the earliest start and finish tasks’ times to compute the makespan. However, this representation has the deficiency that the optimal value of execution time, also known as optimal makespan, may not be in the particular given execution order.

With the representation mentioned above, we detail the objective function to compute the makespan in Algorithm 1 to compute the makespan of complexity $O\left(\right|T\left|\right|C\left|\right)$, similar as in [18,20,21]. The computation times of the tasks are calculated as the original computation time at maximum voltage $P_{i,j}$ divided by their relative speed in Table 1. We show the consider computation times $P_{i,j}^{{}^{\prime }}$ on the following Table.

Using the computation times $P_{i,j}^{{}^{\prime }}$ from

Table 5. Computation times of the tasks with relative speed $P_{i,j}^{{}^{\prime }}$.

Task	${\mathit{t}}_0$	${\mathit{t}}_1$	${\mathit{t}}_2$	${\mathit{t}}_3$	${\mathit{t}}_4$	${\mathit{t}}_5$	${\mathit{t}}_6$	${\mathit{t}}_7$
Time	13.75	31.42	15	12	15	5	11	10

and a counter time in each machine for the last executed task in the machine when it finishes its execution $Tim{e}_j$. Algorithm 1 defines the objective function called makespan.

The function uses the variables: $t{s}_i$ (the starting time of task i), $t{f}_j$ (the finish time of task j), $C_{i,j}^{\prime }$ the communication cost between $t_i$ and $t_j$, which is equal to zero in the case the tasks are executed in the same machine. The function is computed in sequence from the first task of the feasible execution order to the last one in the list. Finally, the parallel program makespan is the maximum completion time between the set of tasks.

Algorithm 1 Makespan objective function

Require:  $G=(T,C)$, with an order execution of the tasks $O=\{o_i,\cdots ,o_{\left|T\right|}\}$, computational costs at relative speeds $P_{i,j}^{\prime }$, and the consider communication costs $C_{i,j}^{\prime }$. Ensure:  O respect the precedence constraints. 1: $Tim{e}_j\leftarrow 0$, $\forall m_j\in M$ 2: for$x=1$ to $\left|O\right|$ do 3:     $t_{\mathrm{current}}$ = $o_x$ 4:     $j\leftarrow$ the index of the machine $m_j$ assign to $t_{current}$ 5:     if $\forall u=\{1,...,|T\left|\right\}\nexists (t_u,t_{\mathrm{current}})\in C$ then 6:         $t{s}_{\mathrm{current}}\leftarrow \mathrm{Tim}{e}_j$ 7:         $t{f}_{\mathrm{current}}\leftarrow t{s}_{\mathrm{current}}+P_{t_{\mathrm{current}},j}^{\prime }$ 8:          $\mathrm{Tim}{e}_j\leftarrow t{f}_{\mathrm{current}}$ 9:     else 10:         $t_{u^{*}}\leftarrow {argmax}_{\left\{t_u\right|(t_u,t_{\mathrm{current}})\in C\}}$$(t{f}_u+C_{t_u,t_{\mathrm{current}}}^{\prime })$ 11:         $t{s}_{\mathrm{current}}\leftarrow max(t{f}_{u^{*}}+C_{t_{u^{*}},t_{\mathrm{current}}}^{\prime },Tim{e}_j)$ 12:         $t{f}_{\mathrm{current}}\leftarrow t{s}_{\mathrm{current}}$ + $P_{t_{\mathrm{current}},j}$ 13:          $Tim{e}_j\leftarrow t{f}_{\mathrm{current}}$ 14:     end if 15: end for 16: return$makespan\leftarrow max\left\{t{f}_i\right\}\forall t_i\in T$

For the energy consumption objective is not necessary the DAG representation, only to know the final makespan. The energy consumption of the tasks is the square power of the selected voltage multiplied by the task completion time in the machine $P_{i,j}^{\prime }$, to add the machines respective idle energy consumption, it is necessary to compute the machine time during idle, which is equal to the makespan subtracting the tasks computation times. Algorithm 2 shows the pseudocode to compute the energy objective. The energy model derives from the complementary metal-oxide-semiconductor (CMOS) circuits in Equation (1). Where capacitive power $P_c$ is computed with the number switches per clock cycle A, the total capacitance load $C_{ef}$, the supply voltage V, and the frequency f.

$$P_c=A{C}_{ef}{V}^{2}f$$

(1)

$$E_c=\sum _{i=1}^{n}A{C}_{ef}{V}_i^{2}f·p_i^{*}=\sum _{i=1}^{n}K{V}_i^2·p_i^{*}$$

(2)

The Equation (2) is our energy model consumption $E_c$. Which is simplified version grouping the constants A, $C_{ef}$ and f in a single constant K. For practical purposes, the constant K is equal to one in the computed results. Finally, $p_i^{*}$ represents the computing task time.

Algorithm 2 Energy objective function

1: $energy\leftarrow 0$ 2: for each$t_i\in T$do 3:     $energy+=v_k^2·P_{i,j}^{\prime }$ 4: end for 5: for each$m_j\in M$do 6:     $idle=makespan$ 7:     for each $t_i$ assigned to $m_j$ do 8:         $idle-=P_{i,j}^{\prime }$ 9:     end for 10:     $energy+=idle·v_{min}^2$ 11: end for

3. List Scheduling Algorithms

Algorithm 3 shows the pseudocode of the general framework for list scheduling algorithms.

Algorithm 3 List scheduling framework

1: Calculate the priority of each task $t_i\in T$ 2: Sort the tasks $t_i$ into a list $L=\{t_1,t_2,......,t_n\}$ by priority order 3: whileL is not empty do 4:     Remove the first task from L and assign it to the machine with the best objective value 5: end while

In the list scheduling principle, tasks are assigned according to priorities and placed in an ordered decreasing list. First, the tasks are sequenced to be scheduled in accordance with the DAG, respecting their precedences regarding a topological order. Then, each task of the list is assigned successively to a machine. Usually, the machine yields the minimum makespan.

Local Searches Related to the List Scheduling Framework

In [22,23] Arabnejad surveyed the most relevant algorithms using the framework for list scheduling in the literature. Among the most popular is the heterogeneous earliest finish time (HEFT) [24], a deterministic reference algorithm in many studies. It assigns to every task the machine, which allows its most immediate finish time. However, HEFT can modify the task’s priority order when detecting available idle time on a machine without precedence-constraint violation. It is out of the scope of the present study to modify the priority order of the tasks. In this paper, we assume the given priority of the tasks is fix. The above makes our objective function in Algorithm 1 feasible in all our studied cases. Recently, several papers regarding energy optimization have used the DVFS technique in several contexts, mathematical programming [25], metaheuristics [26,27,28], heuristics [29,30,31,32,33,34,35,36,37], parallel algorithms [38], among others. However, mainly constructive heuristic methods appear in the literature to schedule tasks in heterogeneous machines. Our perspective is that it could be because of the heavy computational cost of the objective function when no fixed task order execution is given (priority modification as in HEFT). Using the objective function described in Algorithm 1 the complexity is reduced to $O\left(\right|T\left|\right|C\left|\right)$ in the worst-case when every task has an equal number of precedences. However, the real complexity is considerably lower because DAGs do not have cycles.

Local searches are heuristic approximation methods that move a current solution to its nearest local optima solution. However, local searches cannot escape local optima as other more advanced approximation methods as metaheuristics.

Inside the machine tasks scheduling literature, local search is generally a part of a metaheuristic method; for example, we found: iterated local search [39,40,41,42,43], particle swarm optimization [44,45], ant colony optimization [46,47,48], memetic algorithm [49,50,51,52,53], GRASP [18], and variable neighborhood search [54], among others. In the previous examples, local search plays a crucial role in their performances, so we can infer a straightforward improvement is through new local search designs and studies to enlighten ways of improving the final performance of these methods.

Some studies investigate the use of isolating local search improvements; we found examples in [6,13,55,56]. To produce a complete experimental study, we choose to compare and adapt the more relevant ones from the above-mentioned local search works. To be lexicographic and stochastic.

4. Literature Review

In this section, we describe three relevant local search works in the state-of-the-art. A deterministic local search using an aggregation objective function to minimize the makespan and energy [19]. Two stochastic local searches using the best improvement pivot method with lexicographic importance of the objectives [13]. A stochastic local search using two neighborhood operators [6]. The selected works represent the broad ideas on the state-of-the-art scheduling precedence-tasks using local searches.

4.1. Energy Conscious Scheduling (ECS)

The energy conscious scheduling ($ECS$) is a heuristic using a special objective function formulation called relative superiority (RS) [19]; it consists of two phases. The first phase optimizes RS’s value in a given topological order ($b_{level}$ used in the original paper) the possible machines $m_j\in M$ with their respective voltages. The second phase uses the makespan-conservative energy reduction (MCER) technique [19], which visits the tasks in the same topological order. Which is considering the energy consumption in idle times, the variant is called $EC{S}_{+idle}$, the one used for the experimental comparison. The RS computation for $EC{S}_{+idle}$ is shown in Equation (3).

$$\begin{array}{c}\hfill R{S}_{EC{S}_{+idle}}(t_i,m_j,v_k,m^{\prime },v^{\prime })=\\ \hfill \mathrm{If}t{f}_i(m_j,v_k)<t{f}_i(m^{\prime },v^{\prime })\mathrm{Then}\\ \hfill -\left(\frac{E_{1\le i}(m_j,v_k)-E_{1\le i}(m^{\prime },v^{\prime })}{E_{1\le i}(m_j,v_k)}+\frac{t{f}_i(m_j,v_k)-t{f}_i(m^{\prime },v^{\prime })}{t{f}_i(m_j,v_k)-t{s}_i(m_j,v_k)}\right)\\ \hfill \mathrm{Otherwise}\\ \hfill \left(-\left(\frac{E_{1\le i}(m_j,v_k)-E_{1\le i}(t_i,m^{\prime },v^{\prime })}{E_{1\le i}(t_i,m_j,v_k)}\right)+\frac{t{f}_i(m^{\prime },v^{\prime })-t{f}_i(m_j,v_k)}{t{f}_i(m^{\prime },v^{\prime })-t{s}_i(m^{\prime },v^{\prime })}\right)\end{array}$$

(3)

In the original paper, the RS computation for $EC{S}_{+idle}$ is a little bit different. The negative sign of the case when $t{f}_i(m_j,v_k)\ge t{f}_i(m^{\prime },v^{\prime })$ its outside the entire equation, this has been corrected in Equation (3) to compute the correct results.

The Algorithm 4 shows the pseudocode for the $EC{S}_{+idle}$ heuristic. At the constructive first phase the objective functions are partially evaluated until the current evaluated task $t_i$, $t{f}_i$ for makespan, and $E_{1\le i}$ for energy. At the MCER second phase, the objective functions evaluate the complete scheduling and visit neighbor solutions with different machine and voltage configurations. Neighbor solutions that do not increase the makespan and improve energy consumption become the current schedule.

4.2. Random Problem Aware Local Search (rPALS)

rPALS [6] is a stochastic local search for makespan minimization on heterogeneous machines. It is a stochastic version of the deterministic local search PALS [57] for DNA fragment assembly. rPALS achieves the best performance against other list-based heuristics, namely: sufferage, min–min, and p$\mu$-CHC. rPALS uses two neighborhood operators $swaps$ and $moves$, similar to the principle of the variable neighborhood search (VNS) [58], without the stop criterion of finding a local optimal.

The Algorithm 5 shows the pseudocode for the rPALS local search. It starts with an initial $Schedule$ constructed by the minimum completion time (MCT) heuristic [59]. Considering the tasks in any order, MCT assigns each task to the processing unit, which minimizes the finish time of the task.

Main loop iterates until it reach a maximum number of steps $MAXSTEPS$; at each iteration, a machine $m_j$ and a neighborhood $operator$ between $swap$ and $move$ are select randomly.

In the case where the operator $swap$ is selected, it starts a loop selecting a random task $t_i$ from the ones assigned to $m_j$, and a random machine $m_{swap}\ne m_j$ until $MAXSWAPS$ is reached. Then a second inner loop iterates selecting random tasks $t_{swap}$ assigned to $m_{swap}$ for swapping with $t_i$ and $m_j$, if the neighbor solution $Schedul{e}^{\prime }$ improves the overall makespan, it is assigned at the current best solution $Schedule$.

In the case the operator $move$ is selected it starts a loop, selecting a random machine $m_{move}\ne m_j$ until the stop condition $MAXMOVES$ is reached. This loop iterates selecting a random task $t_{move}$ assigned to $m_{move}$, producing a neighbor solution $Schedul{e}^{\prime }$ by assigning the task $t_{move}$ to the machine $m_j$. If the neighbor solution improves the makespan, the solution $Schedul{e}^{\prime }$ is assigned as the current best solution $Schedule$.

Algorithm 4 $EC{S}_{+idle}$

1: Calculate the $b_{level}$ of each task $t_i\in T$ 2: Sort the tasks $t_i$ into a list $L=\{t_1,t_2,......,t_n\}$ by $b_{level}$ 3: First Phase ECS 4: foreach $t_i\in L$ do 5:      $m^{\prime }\leftarrow m_0$ 6:      $v^{\prime }\leftarrow v_0$ 7:      for each $m_j\in M$ do 8:          for each $v_k\in m_j$ do 9:             if RS($t_i$,$m_j$,$v_k$,$m^{\prime }$,$v^{\prime }$)>RS($t_i$,$m^{\prime }$,$v^{\prime }$,$m_j$,$v_k$) then 10:                 $m^{\prime }\leftarrow m_j$ 11:                 $v^{\prime }\leftarrow v_k$ 12:             end if 13:          end for 14:      end for 15:      Assign $t_i$ on $m^{\prime }$ with $v^{\prime }$ 16:  end for 17:  Second Phase MCER 18:  for each $t_i\in L$ do 19:      $m^{\prime }\leftarrow$ The current assigned machine to $t_i$ 20:      $v^{\prime }\leftarrow$ The current assigned voltage to $t_i$ 21:      for each $m_j\in M$ do 22:          for each $v_k\in m_j$ do 23:             if $t{f}_n(m^{\prime },v^{\prime })\le t{f}_n(m_j,v_k)$ then 24:                 if $E(m^{\prime },v^{\prime })<E(m_j,v_k)$ then 25:                     $m^{\prime }\leftarrow m_j$ 26:                     $v^{\prime }\leftarrow v_k$ 27:                 end if 28:             end if 29:          end for 30:      end for 31:      Assign $t_i$ on $m^{\prime }$ with $v^{\prime }$ 32:  end for

Algorithm 5 rPALS

1: $Schedule\leftarrow MCT$ 2: $step\leftarrow 0$ 3: while$step<MAXSTEPS$do 4:     Select a random machine $m_j$ 5:     Select a random neighborhood operator $swap$ or $move$ 6:     if $operator==swap$ then 7:         $tasks\leftarrow 0$ 8:         while $tasks<MAXTASKS$ do 9:            Select a random task $t_i$ assigned to $m_j$ 10:            Select a random machine $m_{swap}$ ($m_j\ne m_{swap}$) 11:            $swaps\leftarrow 0$ 12:            while $swaps<MAXSWAPS$ do 13:                Select a random task $t_{swap}\in m_{swap}$ 14:                $Schedul{e}^{\prime }\leftarrow Schedule$ 15:                $Schedul{e}^{\prime }$ assigns $t_i$ on $m_{swap}$ and $t_{swap}$ on $m_j$ 16:                if $mkspan\left(Schedul{e}^{\prime }\right)<mkspan\left(Schedule\right)$ then 17:                    $Schedule\leftarrow Schedul{e}^{\prime }$ 18:                end if 19:                $swaps++$ 20:            end while 21:            $tasks++$ 22:         end while 23:     end if 24:     if $operator==move$ then 25:         $moves\leftarrow 0$ 26:         Select a random machine $m_{move}$ ($m_j\ne m_{move}$) 27:         while $moves<MAXMOVES$ do 28:            Select a random task $t_{move}$ assigned to $m_{move}$ 29:            $Schedul{e}^{\prime }\leftarrow Schedule$ 30:            $Schedul{e}^{\prime }$ assigns $t_{move}$ to $m_j$ 31:            if $mkspan\left(Schedul{e}^{\prime }\right)<mkspan\left(Schedule\right)$ then 32:                $Schedule\leftarrow Schedul{e}^{\prime }$ 33:            end if 34:            $moves++$ 35:         end while 36:     end if 37:     $step++$ 38: end while

4.3. BEST_RT_MVk and BEST_RMVk_T

Our last consideration is of work from the literature is [13], where the authors propose two best improvement stochastic local searches: BEST_RT_MVk and BEST_RMVk_T. Both local search methods start with a solution constructed by the fast heuristic HEFT [24]. The main difference between the proposed local search methods is in their stochastic selections.

BEST_RT_MVk randomly selects a task to be evaluated within all the possible machines and voltages configurations. Although BEST_RMVk_T randomly selects a machine and voltage configurations to be considered within all the possible tasks. The Algorithms 6 and 7 shows the pseudocode for BEST_RT_MVk and BEST_RMVk_T, respectively.

The original methods in [13] are stochastic local searches (SLS) for makespan and energy optimization. However, every time the methods found a local improvement, the algorithm restarts the stop condition. The above could produce significant computation times when the algorithm does not search a local optimum, an issue presented in our preliminary experimentations.

Algorithm 6 BEST_RT_MVk

1: $Schedule\leftarrow HEFT\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}$▹ Line removed for the experimental comparison 2: function BEST_RT_MVk($Schedule$) 3:     $step\leftarrow 0$ 4:     while $step<MAXSTEPS$ do 5:         $step++$ 6:         Select a random task $t_i$ 7:         for each $m_j\in M$ do 8:            for each $v_k\in m_j$ do 9:                $Schedul{e}^{\prime }\leftarrow Schedule$ 10:                $Schedul{e}^{\prime }$ assigns $t_i$ on $m_j$ with $v_k$ 11:                if $mkspan\left(Schedul{e}^{\prime }\right)\le mkspan\left(Schedule\right)$ then 12:                    if $E\left(Schedul{e}^{\prime }\right)<E\left(Schedule\right)$ then 13:                        $Schedule\leftarrow Schedul{e}^{\prime }$ 14:                        $step\leftarrow 0\hspace{1em}\hspace{1em}\hspace{1em}$▹ Line removed for the experimental comparison 15:                    end if 16:                end if 17:            end for 18:         end for 19:     end while 20: end function

Algorithm 7 BEST_RMVk_T

1: $Schedule\leftarrow HEFT\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}$▹ Line removed for the experimentation 2: function BEST_RMVk_T($Schedule$) 3:     $step\leftarrow 0$ 4:     while $step<MAXSTEPS$ do 5:         $step++$ 6:         Select a random machine $m_j$ 7:         Select a random voltage $v_k\in m_j$ 8:         for each $t_i\in T$ do 9:            $Schedul{e}^{\prime }\leftarrow Schedule$ 10:            $Schedule$ assigns $t_i$ on $m_j$ with $v_k$ 11:            if $mkspan\left(Schedul{e}^{\prime }\right)\le mkspan\left(Schedule\right)$ then 12:                if $E\left(Schedul{e}^{\prime }\right)<E\left(Schedule\right)$ then 13:                    $Schedule\leftarrow Schedul{e}^{\prime }$ 14:                    $step\leftarrow 0\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}$▹ Line removed for the experimentation 15:                end if 16:            end if 17:         end for 18:     end while 19: end function

5. Algorithms in Comparison

In the experimental comparison, two mandatory restrictions must be satisfied:

• The use of a fixed maximum number of neighbor solutions visited;
• Providing only one random initial solution.

The algorithms in Section 4 are adjusting in the following manner.

5.1. $EC{S}_{+idle}$ Stochastic Local Search

The original proposed $EC{S}_{+idle}$ is a deterministic heuristic. To investigate its effects as a stochastic lexicographic local improvement method, we proposed two new stochastic local search (SLS) using the RS objective function in Equation (3). We follow the best improvement pivot rule in [13], and the two SLS variants in the paper: random selection of task (ECS_RT_MVk) and random selection of machine and voltage (ECS_RMVk_T).

Algorithm 8 shows the procedure for the SLS ECS_RT_MVk. The function’s input is a feasible $Schedule$, the search select a random task $t_i$ to be evaluated in all the possible machines in M with their respective voltage configurations if the stopping condition has not been reached. The task $t_i$ is assigned to the best machine $m^{\prime }$ and voltage $v^{\prime }$ according to the RS objective function. The main loop iterates over random tasks until the counter $step$ reaches the maximum number of visited neighbors $MAXSTEPS$.

Algorithm 8$EC{S}_{+idle}$ random $t_i$

1: function ECS_RT_MVk($Schedule$) 2:     $step\leftarrow 0$ 3:     while $step<MAXSTEPS$ do 4:         select a random task $t_i$ 5:         $m^{\prime }\leftarrow$ assigned machine to $t_i$ in $Schedule$ 6:         $v^{\prime }\leftarrow$ assigned voltage to $t_i$ in $Schedule$ 7:         for each $m_j\in M$ do 8:            for each $v_k\in m_j$ do 9:                if $step<MAXSTEPS$ then 10:                    if RS($t_i$,$m_j$,$v_k$,$m^{\prime }$,$v^{\prime }$)>RS($t_i$,$m^{\prime }$,$v^{\prime }$,$m_j$,$v_k$) then 11:                        $m^{\prime }\leftarrow m_j$ 12:                        $v^{\prime }\leftarrow v_k$ 13:                    end if 14:                    $step\leftarrow step+1$ 15:                end if 16:            end for 17:         end for 18:         Assign $t_i$ on $m^{\prime }$ with $v^{\prime }$ in $Schedule$ 19:     end while 20: end function

Algorithm 9 shows the procedure for the SLS ECS_RMVk_T. The function’s input is a feasible $Schedule$, the search select a random machine $m_j$ with a feasible random voltage $v_k$ inside the range of the machine $m_j$ to be evaluated in all the possible tasks in T if the stopping condition has not been reaching. The task $t_i$ is assigned to the best machine $m^{\prime }$ and voltage $v^{\prime }$ according to the RS objective function. The main loop iterates over random tasks until the counter $step$ reaches the maximum number of visited neighbor solutions $MAXSTEPS$.

5.2. rPALS Lexicographic Local Search

The original rPALS was proposed only for the improvement of the final makespan. To improve both objectives (energy and makespan), we follow the same lexicographic criteria to accept variable changes in the solutions as in [13]. If the makespan is not worsening and there is an energy improvement, the neighbor solution becomes the current best $Schedule$. The inner loops from rPALS are removing to control the visit neighbors. The following pseudocode shows the procedure for our proposed LS rPALS Lexicographic Algorithm 10 (rPALS_Lex).

Algorithm 9$EC{S}_{+idle}$ random $m_j/v_k$

1: function ECS_RMVk_T($Schedule$) 2:     while step<MAXSTEPS do 3:         select a random machine $m_j$ 4:         select a random voltage $v_k\in m_j$ 5:         for each $t_i\in T$ do 6:             $m^{\prime }\leftarrow$ assigned machine to $t_i$ in $Schedule$ 7:             $v^{\prime }\leftarrow$ assigned voltage to $t_i$ in $Schedule$ 8:             if $steps<MAXSTEPS$ then 9:                 if RS($t_i$,$m_j$,$v_k$,$m^{\prime }$,$v^{\prime }$)>RS($t_i$,$m^{\prime }$,$v^{\prime }$,$m_j$,$v_k$) then 10:                     $m^{\prime }\leftarrow m_j$ 11:                     $v^{\prime }\leftarrow v_k$ 12:                     Assign $t_i$ on $m^{\prime }$ with $v^{\prime }$ in in $Schedule$ 13:                 end if 14:                 $steps\leftarrow steps+1$ 15:             end if 16:          end for 17:      end while 18:  end function

Algorithm 10 rPALS Lexicographic

1: function rPALS_LEX_LS($Schedule$) 2:     $step\leftarrow 0$ 3:     while $step<MAXSTEPS$ do 4:         Select a random machine $m_j$ 5:         Select a random neighborhood operator $swap$ or $move$ 6:         if $operator==swap$ then 7:            Select a random task $t_i$ assigned to $m_j$ 8:            $v_i\leftarrow$ assigned voltage to $t_i$ on $m_j$ 9:            Select a random machine $m_{swap}$ ($m_j\ne m_{swap}$) 10:            Select a random task $t_{swap}\in m_{swap}$ 11:            $v_{swap}\leftarrow$ assigned voltage to $t_{swap}$ on $m_{swap}$ 12:            $Schedul{e}^{\prime }\leftarrow Schedule$ 13:            $Schedul{e}^{\prime }$ assigns $t_i$ on $m_{swap}$ with $v_{swap}$ 14:            $Schedul{e}^{\prime }$ assigns $t_{swap}$ on $m_j$ with $v_i$ 15:            if $mkspan\left(Schedul{e}^{\prime }\right)\le mkspan\left(Schedule\right)$ then 16:                if $E\left(Schedul{e}^{\prime }\right)<E\left(Schedule\right)$ then 17:                    $Schedule\leftarrow Schedul{e}^{\prime }$ 18:                end if 19:            end if 20:         end if 21:         if $operator==move$ then 22:            Select a random machine $m_{move}$ ($m_j\ne m_{move}$) 23:            Select a random task $t_{move}$ assigned to $m_{move}$ 24:            $Schedul{e}^{\prime }\leftarrow Schedule$ 25:            $Schedul{e}^{\prime }$ assigns $t_{move}$ to $m_j$ with random voltage $v_k$ 26:            if $mkspan\left(Schedul{e}^{\prime }\right)\le mkspan\left(Schedule\right)$ then 27:                if $E\left(Schedul{e}^{\prime }\right)<E\left(Schedule\right)$ then 28:                    $Schedule\leftarrow Schedul{e}^{\prime }$ 29:                end if 30:            end if 31:         end if 32:         $step++$ 33:     end while 34: end function

5.3. Fixed iterations BEST_RT_MVk and BEST_RMVk_T

For a fair experimental comparison, the commented lines in Algorithms 6 and 7 with the legend Line removed for the experimental comparison are not considered. Therefore the LSs are not initialized with the HEFT heuristic and visits the same number of neighbor solutions, fixed by their main loops stop criterion ($MAXSTEPS$).

5.4. Table of Notations Used in the Described Literature

The

Table 6. Notation used in the described literature.

Notation	Meaning
L	List of tasks
$b_{level}$	b-level ordering of L
$t_i$	Task i
$t_{move}$	Task $move$
$t_{swap}$	Task $swap$
$t^{\prime }$	Temporal Task
$t_{s_i}$	Staring time of the task i
$t_{f_i}$	Finish time of the task i
$m_j$	Processing machine j
$m_{swap}$	Processing machine $swap$
$m_{move}$	Processing machine $move$
$m^{\prime }$	Temporal processing machine
$v_i$	Voltage i
$v_k$	Voltage k
$v_{swap}$	Voltage $swap$
$v_{move}$	Voltage $move$
$v^{\prime }$	Temporal voltage
$Schedule$	Candidate solution
$Schedul{e}^{\prime }$	Temporal candidate solution
$mkspan\left(\right)$	Makespan objective function
$E\left(\right)$	Energy objective function
$E_{1\le i}\left(\right)$	Partial energy objective function
$step$	Integer counter
$MAXSTEPS$	Integer constant

shows the common mathematical notations in the mentioned literature.

6. Experimental Setup

The reviewed works from Section 4 differ in their stop criteria. In the original publication of BEST_RMVk_T and BEST_RT_MVk, the maximum number of neighbor solutions visited is variable, a maximum of 100 continuous neighbor visited without improvements. In the work of rPALS, the maximum number of visited neighbors is 4,000,000. Although, in the deterministic search, $EC{S}_{idle}$ the number is $\left|T\right|·\left|M\right|·\left|V\right|$.

On the one hand, few visited neighbors could increase the variance at the final computed results. On the other hand, a high number of visited neighbors increase computational times. We suggest a reasonable fixed number of visited neighbors; we use the ratio between the maximum number of visited neighbors in [6] and the maximum number of visited neighbors without improvement in [13]. Giving a total number of maximum visited neighbors of $\frac{4,000,000}{100}$ = 40,000.

The set of machine and voltage configurations used for the experimentation has a cardinality of six, the ones in [13]. When a particular scheduling problem considers more than six machines, the first configuration is assigned to the next machine, later the second, and so on (round-robin rule) until every considered machine has a valid configuration. Finally, we perform 60 independent executions for every considered scheduling problem.

6.1. Parallel Instance Set

The applications in the experimental set are:

• Fpppp [60];
• LIGO [61];
• Robot [62];
• Sparse [62].

We used the set of instances from [18], consisting of 400 different scheduling problems.

6.2. Friedman Statistical Test

In order to assess the performance of this algorithms it is necessary to validate their outcomes through a statistical test. There are specific statistical tests for comparing two samples and others for more than two samples. A widely accepted statistical test to find significant differences between more than two samples is the analysis of variance (ANOVA) test. However, to use ANOVA it is mandatory that the data follow a normal distribution. There are other statistical tests that do not need to follow such an assumption, they are non-parametric tests. Among those tests, Friedman [63] is a non-parametric test used to compare the performance of several algorithms’ performance. The Friedman statistical test is computed using the following equation when no ties:

$$F_r=\left[\frac{12}{nk(k+1)}\sum _{i=1}^k{R}_i^2\right]-3n(k+1)$$

(4)

Where the number of independent executions is n, the number of algorithms is k, and the rank of each i algorithm is $R_i$. Once the statistical $F_r$ is computed, a reference table is consulted for the achieved p-value [63]. A direct performance evaluation metric in many algorithms’ studies is the Friedman rankings [64] ($R_i$). The original data of the independent executions is transformed into a table of places (ranks) according to the performance of each algorithm (see

Table 7. Example of ranks for three algorithms.

Execution	Algorithm 1	Algorithm 2	Algorithm 3
1	3	2	1
2	3	2	1
3	1	2	3
4	3	2	1
5	3	1	2
6	3	1	2
7	3	1	2

).

With Table 7, the Friedman ranking for the Algorithm 1 is compute as the sum of their ranks: $R_{Algorithm1}=3+3+1+3+3+3+3$, $R_{Algorithm1}=19$. The presented ranks in this work are in terms of the average of makespan and energy consumption. For our computational example, $R_{Algorithm1}=2.71$. This technique is a straightforward way to compare the performance of several algorithms over benchmarks. Using the tool in [65] the process is as simple as input the original data in CSV format, using the command line java Friedman data.csv > output.tex, the output.tex file has to be compiled with a compiler for .

6.3. Task Priority Methods

We evaluate two widely used methods to generate a priority task list in our experimentation: the bottom level (b-level), and the top-level (t-level).

The b-level computes the critical path in the DAG from a current task node $t_{current}$ to the final task in the parallel program $t_n$, taking into consideration the mean of the computing times $\overline{p_i}$ in the machines (See

Table 8. Mean values of the tasks computing times.

Task	${\mathit{m}}_0$	${\mathit{m}}_1$	${\mathit{m}}_2$	$\overline{{\mathit{p}}_{\mathit{i}}}$
0	11	13	9	11.0
1	10	15	11	12.0
2	09	12	14	11.6
3	12	16	10	12.6
4	15	11	19	15.0
5	13	09	05	9.0
6	11	15	13	13.0
7	11	15	10	12.0

) and the communication costs (edges of the DAG in Figure 1).

For the DAG in Figure 1 the b-level values of the tasks are shown in

Table 9. b-level values for the DAG in Figure 1.

Task	0	1	2	3	4	5	6	7
b-level	102.00	67.00	63.66	73.66	80.00	42.00	38.00	12.00

. The tasks’ priority is according to their b-level values in descending order. In a very similar manner, the algorithm calculates the t-level, finding a path from an actual task node $t_{current}$ to the initial task in the parallel program $t_0$ (See

Table 10. t-level values for the DAG in Figure 1.

Task	0	1	2	3	4	5	6	7
t-level	11.00	34.00	39.66	37.66	37.00	65.66	77.00	102.00

). The assigned priorities are according to their t-level values in ascending order.

The computation of the b-level and t-level needs to verify every possible path in the DAG; a straightforward way to do it is to use recursive functions as shown in the Algorithms 11 and 12.

Algorithm 11 b-level recursive function

1: $temp\leftarrow \overline{p_i}\left[t_{current}\right]$ 2: functionb-level($t_{current}$, $temp$) 3:     $max\leftarrow temp$ 4:     for $(t_{\mathrm{current}},t_u)\in E$ do 5:         $temp\leftarrow$B-LEVEL$(t_u,(temp+(t_{current},t_u)+\overline{pi}\left[t_u\right]))$ 6:         if $max<temp$ then 7:            $max\leftarrow temp$ 8:         end if 9:     end for 10:     $\mathbf{return}max$ 11: end function

Algorithm 12 t-level recursive function

1: $temp\leftarrow \overline{p_i}\left[t_{current}\right]$ 2: functiont-level($t_{current}$, $temp$) 3:     $max\leftarrow temp$ 4:     for $(t_u,t_{\mathrm{current}})\in E$ do 5:         $temp\leftarrow$T-LEVEL$(t_u,(temp+(t_u,t_{current})+\overline{pi}\left[t_u\right]))$ 6:         if $max<temp$ then 7:            $max\leftarrow temp$ 8:         end if 9:     end for 10:     $\mathbf{return}max$ 11: end function

7. Results

This section analyzes the computed results with the experimental settings in Section 6. Due to the considerable number of tables needed to present results, we decide not to include them in the final paper.

A more acceptable and brief way to examine the results is to produce Friedman’s average rankings [64] from the non-parametric Friedman statistical test (see Section 6.2). The ranking gives an insight into the algorithm’s performance (the lower the ranking, the better). We compute the Friedman ranking using a free tool presented in [65]; the website has extensive information on the Friedman test and rankings.

We found at every presented comparison in this work that there is a statistical difference after analyzing all the computed p-values by the Friedman statistical test, which satisfies the condition p-value $\le 0.05$, giving a statistical significance of 95% [63]. In the following subsections, we present the results by goal objective.

7.1. Makespan Results

First, we start by analyzing the computed results according to the makespan objective.

Table 11. Makespan Friedman ranking on the 400 scheduling problems (b-level).

Local Search	Ranking
ECS_RMVk_T	2.12
rPALS_Lex	2.36
BEST_RMVk_T	2.55
ECS_RT_MVk	3.26
BEST_RT_MVk	4.70

shows the Friedman ranking when considering the whole scheduling cases and the b-level priority execution.

Table 11 shows that ECS_RMVk_T gets the best performance when using the b-level priority execution while rPALS_Lex is the second best. Notice that both Local Searches, when selecting a random task to verify their machine and voltage possible configurations (ECS_RT_MVk, BEST_RT_MVk), perform the worst. The same algorithms produce low performance when using the t-level priority, as shown in the ranking from

Table 12. Makespan Friedman ranking on the 400 scheduling problems (t-level).

Local Search	Ranking
rPALS_Lex	1.69
BEST_RMVk_T	1.79
ECS_RMVk_T	3.39
ECS_RT_MVk	3.91
BEST_RT_MVk	4.22

. However, in this case, rPALS_Lex obtains the best performance, followed by BEST_RMVk_T.

We infer from the results from Table 11 and Table 12 that the worst strategy for makespan optimization is when local searches evaluate from a random task their best neighbor among all their possible machine and voltage configurations. To give more insight into the results, we compute the Friedman average ranking on the individual subsets of instances: Fpppp, LIGO, Robot, and Sparse. The bests rankings in the table are highlighting in gray.

The Friedman rankings in

Table 13. Makespan Friedman rankings (b-level).

Algorithm	Fpppp	LIGO	Robot	Sparse
BEST_RMVk_T	2.80	2.48	2.02	2.93
BEST_RT_MVk	4.75	4.78	4.55	4.73
ECS_RMVk_T	1.75	2.11	2.40	2.22
ECS_RT_MVk	3.65	3.15	3.38	2.86
rPALS_Lex	2.04	2.47	2.65	2.27

confirms the reports from Table 11, regarding the use of b-level, the worst performance is for the ECS_RT_MVk and BEST_RT_MVk local searches. ECS-RMVk-T achieves the best performance with three best average rankings (Fpppp, LIGO, Sparse) and one second-best (Robot). A highly competitive local search when using b-level priority is rPALS-Lex with three second-best average rankings (Fpppp, LIGO, Sparse) and one best (Robot). Considering only the Robot subset of instances, the local search BEST-RMVk-T achieves the best makespan average ranking.

The results in

Table 14. Makespan Friedman rankings (t-level).

Algorithm	Fpppp	LIGO	Robot	Sparse
BEST_RMVk_T	2.06	1.75	1.43	1.91
BEST_RT_MVk	4.44	4.25	3.99	4.22
ECS_RMVk_T	3.27	3.42	3.40	3.48
ECS_RT_MVk	3.83	3.88	4.13	3.80
rPALS_Lex	1.41	1.69	2.06	1.59

also confirms the results from Table 12, according to the use of t-level, rPALS-Lex achieves the best performance with three best average rankings (Fpppp, LIGO, Sparse) and one second-best (Robot) followed by BEST-RMVk-T with one best average ranking (Robot) and three second-best (Fpppp, LIGO, Sparse). In contrast, the worst performance is the b-level case with ECS_RT_MVk and BEST_RT_MVk.

7.2. Energy Results

In the case of the energy objective, the order of the rankings of the algorithms when using b-level (See

Table 15. Energy Friedman ranking on the 400 scheduling problems (b-level).

Local Search	Ranking
rPALS_Lex	1.78
BEST_RMVk_T	1.90
ECS_RMVk_T	2.85
BEST_RT_MVk	3.88
ECS_RT_MVk	4.59

) and t-level (See

Table 16. Energy Friedman ranking on the 400 scheduling problems (t-level).

Local Search	Ranking
rPALS_Lex	1.56
BEST_RMVk_T	1.69
ECS_RMVk_T	3.47
BEST_RT_MVk	3.64
ECS_RT_MVk	4.64

) is the same.

As in the makespan results section, the worst achieved performance was obtained by ECS_RT_MVk and BEST_RT_MVk. For energy optimization, the best achievable performance is by rPALS_Lex followed by BEST_RMVk_T with an equal number of best and second-best rankings in

Table 17. Energy Friedman rankings (b-level).

Algorithm	Fpppp	LIGO	Robot	Sparse
BEST_RMVk_T	2.40	1.75	1.55	1.90
BEST_RT_MVk	3.83	4.10	3.87	3.71
ECS_RMVk_T	2.56	2.84	2.82	3.17
ECS_RT_MVk	4.80	4.36	4.48	4.72
rPALS_Lex	1.41	1.95	2.28	1.49

and

Table 18. Energy Friedman rankings (t-level).

Algorithm	Fpppp	LIGO	Robot	Sparse
BEST_RMVk_T	1.99	1.56	1.40	1.82
BEST_RT_MVk	3.79	3.71	3.58	3.49
ECS_RMVk_T	3.18	3.61	3.49	3.58
ECS_RT_MVk	4.80	4.50	4.56	4.69
rPALS_Lex	1.24	1.61	1.97	1.42

.

8. Research Findings

A relevant result emerges from our empirical experimentation, the priority order technique in list scheduling may significantly change the performance of some algorithms, according to the makespan objective. The above is the particular case of the ECS function, which changes from first place (ECS_RMVk_T) in the b-level priority results to third place in the t-level results. A deeper examination of the ECS function in Equation (3) shows that if the energy consumption magnitude is significantly greater than the makespan magnitude, it will emphasize the energy objective over the makespan. According to the energy objective, different priorities for the tasks do not change the comparative performance of the algorithms, for the Friedman ranking remaining the same for the algorithms. Therefore, we infer that energy optimization is more sensitive to the DVFS configurations than the tasks’ order of execution.

9. Conclusions and Future Work

As far as we know, we present the first study of stochastic lexicographic local searches for precedence-constraint task scheduling on heterogeneous machines. We adapt three local searches from the literature to be stochastic, iterative, and lexicographic bi-objective (makespan and energy). The above produces three new variants: ECS_RMVk_T, ECS_RT_MVk, and rPALS_Lex. The experimental results show rPALS_Lex as the most competitive algorithm for makespan and energy optimization compared with the other local searches in the experimentation.

The relative superiority objective function from the ECS heuristic works slightly better when using the b-level priority of execution in the tasks, denoted by ECS_RMKV_T best performance achieved in makespan.

The worst strategy for the studied local searches is when an individual random task is selected to verify their possible machine and voltage configurations. Therefore, we recommend the approach when a random machine and voltage configuration is selected and verifies for improvements over the whole set of tasks.

Finally, as the experimental comparison shows, the task’s execution order is essential for the final performance of the algorithms, with radical place changes in the Friedman rankings when using the b-level and t-level priority.

As future work, we would like to study the design of new priority task heuristics, which could improve the performance of the proposed local searches for scheduling.