Systematic Literature Review of Optimization Algorithms for P||C_max Problem

Abstract

In the era of open data and open science, it is important that, before announcing their new results, authors consider all previous studies and ensure that they have competitive material worth publishing. To save time, it is popular to replace the exhaustive search of online databases with the utilization of generative Artificial Intelligence (AI). However, especially for problems in niche domains, generative AI results may not be precise enough and sometimes can even be misleading. A typical example is $P\left|\right|C_{max}$, an important scheduling problem studied mainly in a wider context of parallel machine scheduling. As there is an uncovered symmetry between $P\left|\right|C_{max}$ and other similar optimization problems, it is not easy for generative AI tools to include all relevant results into search. Therefore, to provide the necessary background data to support researchers and generative AI learning, we critically discuss comparisons between algorithms for $P\left|\right|C_{max}$ that have been presented in the literature. Thus, we summarize and categorize the “state-of-the-art” methods, benchmark test instances, and compare methodologies, all over a long time period. We aim to establish a framework for fair performance evaluation of algorithms for $P\left|\right|C_{max}$, and according to the presented systematic literature review, we uncovered that it does not exist. We believe that this framework could be of wider importance, as the identified principles apply to a plethora of combinatorial optimization problems.

1. Introduction

Nowadays, artificial Intelligence (AI) and Machine Learning (ML) are an integral part of the intelligent core of almost any business. Disruptive technologies, such as generative AI, Blockchain (BC), the metaverse, etc., are becoming more and more pervasive. Thus, there is a growing need for the involvement of supercomputers for processing huge datasets in real time. Consequently, it is equally important to optimize the utilization of high-performance computing resources, cloud computing, and massively parallel multiprocessor systems. Even on a smaller scale, in mobile computing, there is still a need for efficiently solving scheduling problems. A recently developed systematic difficulty estimation framework [1] could provide an opportunity for a fair comparison of algorithms. To detect the most efficient optimization algorithm and its required execution time, the framework needs a sufficiently large database containing algorithms and the results of their usage. Numerous optimization algorithms exist for scheduling problems, but there are not nearly enough problem instances on which the algorithms were tested. Accurate and exhaustive comparisons among algorithms in the literature could help enrich the database of instances illustrating algorithms’ performance. This database could also be useful for researchers who develop new optimization algorithms to adequately compare algorithms and establish the value of their findings and results.

Nowadays, researchers commonly use generative AI instead of exhaustive searching of online databases. Unfortunately, generative AI results may be imprecise or even misleading, especially for problems in niche domains. As a case study to describe the challenges that arise when comparing different optimization algorithms, we focus on the problem of scheduling a set of independent tasks (jobs) on a set of identical parallel machines, with the goal of minimizing the time required by the last machine to complete its assigned jobs. This time is referred to as makespan, and the problem is denoted by $P\left|\right|C_{max}$ [2,3]. $P\left|\right|C_{max}$ is an important scheduling problem studied mainly in a wider context of parallel machine scheduling. Solving the $P\left|\right|C_{max}$ problem efficiently is crucial in various real-world scenarios, such as manufacturing processes, data processing, and distributed computing. Researchers and practitioners use various algorithms and optimization techniques to address the complexities associated with this scheduling problem.

In the 1970s, several optimization algorithms were developed and several equivalent notations were introduced [2,4]. Although the proper notations exist, there has been inconsistent use of acronyms, like Identical Parallel Machines Scheduling (IPMS) or just shortly PMS. The same holds for naming conventions. For example, identical parallel machines scheduling problems with the objective function of minimizing makespan ($C_{max}$) [5], the problem of static scheduling of independent tasks on homogeneous multiprocessors [6], and static homogeneous multi processor scheduling problem (MPSP) [7].

The naming conventions also differ depending on the community of researchers using the problem. In the Operations Research community literature, the $P\left|\right|C_{max}$ problem is often named IPMS [8]. In the AI community, it is often called the Multi-Way Number Partitioning (MWNP) problem [9]. However, MWNP has a different formulation and is an isomorphic problem to $P\left|\right|C_{max}$. The second isomorphic problem to $P\left|\right|C_{max}$ is Min-Max (one-dimensional) Bin Packing Problem (MMBPP). In this variant of Bin Packing Problem (BPP), the number of bins is fixed and the goal is to find a minimum bin capacity that accommodates all items. The problem can also be found under different names, from only a special variant of BPP [10] to BPP-2 [11]. However, this is exactly the same problem as $P\left|\right|C_{max}$, actually the variant with a fixed number of machines ($P_m\left|\right|C_{max}$), because there is no practical difference between the variants for reasonably big values of m.

By identifying the equivalence between similar optimization problems, $P\left|\right|C_{max}$, MWNP, MMBPP, multiple Knapsack problem, multiple subset sum, etc., [12], we have uncovered a valuable symmetry that has been leveraged in $P\left|\right|C_{max}$ optimization algorithms. Additionally, in combinatorial optimization problems, symmetries can lead to redundant computations as symmetric solutions are essentially equivalent. The search involving symmetric solutions should be avoided by implementing symmetry-breaking constraints in such a way that symmetric solutions are efficiently eliminated from the search space. It is relatively easy to overcome the difficulties related to naming or symmetry and to identify research papers that focus on the $P\left|\right|C_{max}$ problem through the exhaustive time-consuming search of online databases. It is much more difficult is to obtain the non-confounding results using generative AI, to establish the taxonomy of optimization algorithms, and to standardize problem instances that were generated. Our goal is to contribute toward resolving this difficulty.

In the preparation for this paper, we decided to search for exact, heuristic, and metaheuristic algorithms. As it has been proven that the $P\left|\right|C_{max}$ problem is strongly $\mathcal{NP}$-hard [2], it can be assumed that there always exists a problem instance that is hard to solve regardless of the algorithm choice. At the same time, a special case related to this problem is where the machine number that is equal to 2 is “the easiest hard problem” [13]. Therefore, comparisons need to be experimental and to include a wide range of problem instances.

Different optimization algorithms do not have the same performance on all problem instances. One group of instances might be easy for one optimization algorithm, but difficult for the other and vice versa [14]. It could be said that one optimization algorithm is better if it is able to efficiently solve a wider range of problem instances. In a recent study [15], the issues arising in choosing a group of instances for algorithm comparisons and how to generate problem instances were discussed. To provide a fair comparison of algorithms it is very important that experimental settings are identical or that analogies could be established and all the assumptions could be replicated. Thus, it is very important to publicly share the problem instances, or give precise instructions on how to generate them including to provide the values of parameters, like random seed.

Currently, no universal standard exists on how to compare the algorithms working on $P\left|\right|C_{max}$ problem. Various problem instances were proposed over time, but there is no universal standard set of benchmark instances that everyone could use. We performed a Systematic Literature Review of Optimization Algorithms for the $P\left|\right|C_{max}$ problem to illustrate that an in-depth analysis of the performance comparison is actually missing. A good algorithm comparison framework is necessary to uncover their strengths and weaknesses. While we have encountered research papers across various avenues that address the literature surrounding this specified problem, there seems to be a lack of comprehensive synthesis that brings together all the mentioned components (methods, problem instances, comparisons) and builds upon the entirety of existing knowledge. This presents an opportunity to foster a more cohesive understanding of this domain that would allow researchers to assess their contributions easily and in a consistent manner.

The contributions of this paper are as follows.

• Report the results of a systematic literature review (SLR) conducted to identify, extract, evaluate, and synthesize the studies on the $P\left|\right|C_{max}$ optimization algorithms. Summarize and categorize existing methods;
• Standardize the problem instances that the majority of algorithms were tested on;
• Uncover a comparison methodology for a fair algorithm performance evaluation.

Having in mind the cited guidelines about the organization of systematic literature review papers and the above mentioned goals and contributions, the paper roadmap is based on the following. The main goal is to provide a good framework for the evaluation/comparison of various types of optimization algorithms. It is necessary to present the characteristics of all the algorithms and available instances before analyzing the described evaluation mechanisms and providing the recommendations for their upgrading.

Therefore, the remainder of this paper is structured as follows: Section 2 provides a background on the $P\left|\right|C_{max}$ problem, solutions, and conducted SLR. Section 3 presents SLR methodology, research questions (RQs), and a high-level overview of the selected studies. The taxonomy of solution methods for the $P\left|\right|C_{max}$ problem and a summary of findings on the extracted studies are presented in Section 4. It contains the classification of methods, description of main characteristics for each class, brief explanation of each considered algorithm, and the existing similarities between them. In Section 5, the use of problem instances over time is discussed. We review the most commonly used sets, identify their strengths and weaknesses, and perform their classification. In addition, we propose standardization criteria, select a representative set of standardized instances, and generate an open access repository to store these and any newly proposed benchmark instances. Algorithm comparisons are explored in Section 6. We explain the general methodology for comparing optimization algorithms and how it applies to the comparison of algorithms for the $P\left|\right|C_{max}$ problem. We review the comparisons presented in the literature and illustrate the resulting ranking of algorithms. Section 7 provides an in-depth analysis, based on the results and study limitations, which draws a roadmap for future research. Our main findings and avenues for future work are summarized in Section 8.

2. $\mathit{P}\left|\right|{\mathit{C}}_{\mathit{max}}$ Background

Herein we focus on the $P\left|\right|C_{max}$ problem that was first introduced in 1959 [16] as “Scheduling Many Processors Which Are Exactly Alike to Finish All Tasks as Soon as Possible”. A formal definition of the class of scheduling problems was introduced two decades later in [4] using the three-field notation $\alpha \left|\beta \right|\gamma$ to systematically classify scheduling problems. In the three-field notation, $\alpha$ specifies machine environment, $\beta$ indicates job characteristics, and $\gamma$ defines optimality criteria. For the $P\left|\right|C_{max}$, we substitute $\alpha$ with P indicating that machines are identical and work in parallel, $\beta$ is left empty as jobs are independent, and $\gamma$ is substituted with $C_{max}$ as the objective is to minimize the maximum completion time (makespan). In [2], a more suitable and mathematically more formal definition for the $P\left|\right|C_{max}$ problem is introduced. Here, we present combinatorial and mathematical programming formulations of $P\left|\right|C_{max}$, to introduce a notation that is used systematically and consistently throughout the manuscript.

Definition 1 

(Combinatorial formulation of $\mathit{P}\left|\right|{\mathit{C}}_{\mathit{max}}$). For a given set $M=\{1,\dots ,m\}$of $m\in \mathbb{N}$ identical parallel machines, i.e., processors, and a set $J=\{1,\dots ,n\}$ of $n\in \mathbb{N}$ independent jobs, i.e., tasks with positive processing times $p=\left(p_1,\dots ,p_n\right)$, the goal is to assign each job to exactly one machine in such a way that the latest machine completion time $C_{max}={max}_{i\in M}{C}_i$, is minimized, where $C_i$ is the sum of processing times of jobs assigned to machine i.

Definition 2 

(ILP formulation of $\mathit{P}\left|\right|{\mathit{C}}_{\mathit{max}}$ [17]). For a given instance $(J,p,M)P\left|\right|C_{max}$ can be defined in terms of ILP as:

$$\begin{array}{cc}min{C}_{max}\hfill & \hfill \end{array}$$

(1)

$$\begin{array}{ccc}\mathrm{s}.\mathrm{t}.\hfill & \sum _{i\in M}{x}_{ij}=1,\hfill & \hfill j\in J,\end{array}$$

(2)

$$\begin{array}{ccc}\hfill & \sum _{j\in J}{p}_j{x}_{ij}\le C_{max},\hfill & \hfill i\in M,\end{array}$$

(3)

$$\begin{array}{ccc}\hfill & x_{ij}\in \{0,1\},\hfill & \hfill i\in M,j\in J,\end{array}$$

(4)

where binary variable $x_{ij}$ indicates whether job j is assigned to machine i.

A solution of $P\left|\right|C_{max}$ actually represents the m-partition of the set J. Usually, processing times are natural numbers, but in the literature, these values can sometimes be represented as positive real numbers. The optimal value for $C_{max}$ is usually denoted by $C_{max}^{*}$. The proof that $P\left|\right|C_{max}$ is $\mathcal{NP}$-hard can be found in [2]. It is possible to verify that even $P2\left|\right|C_{max}$ ($m=2$) is $\mathcal{NP}$-hard [3,18].

To identify the progress in the development of solution algorithms, as well as to clearly classify the SLR studies, we preliminary divide optimization algorithms that have been applied to the $P\left|\right|C_{max}$ problem, loosely following [19], into three categories:

• Exact (E)—provide guarantees of optimality;
• Heuristic (H)—include constructive, improvement, and approximation algorithms;
• Metaheuristic (MH)—include general solution frameworks, possibly hybrid algorithms.

More details about the methods in each category and the background of their developments are provided in Section 4.

Table 1 shows what period each SLR study covers, whether it was a $P\left|\right|C_{max}$ specific survey, which category of optimization algorithms is in the focus of each SLR study, or if any instances (not available—N/A or partially—P) and comparison methods (not available—N/A or partially—P) were covered. “Partially” means that the focus of the SLR was not on presenting which instances and comparison methods exist in the literature, but just mentioning both as a side note. Bolded characteristics signify that SLR covers what we need.

Neither of the SLR studies in the literature specify a period in which the SLR study was covered. Therefore, the table contains the period from the first $P\left|\right|C_{max}$-related cited reference up to the date the SLR study was published. As expected, not all SLR studies covered all the optimization algorithms. Some of the important methods to solve the $P\left|\right|C_{max}$ problem are found in [8,20], but majority of methods are scattered throughout other SLR studies. Overwhelmingly, the SLR studies covered a larger group of scheduling problems and devoted a section to discuss the methods that were applied to the $P\left|\right|C_{max}$ problem. Several SLR studies that focused on the $P\left|\right|C_{max}$ problem exhaustively covered only a specific group of methods. Currently, there is no SLR study that discusses all methods and all instances that could be found in the literature. Therefore, in Section 5, we explain which groups of instances were widely used and which were unique in the sense that they were used in a few studies and never again. In addition, we collected various sets of test instances, analyze their usability, and made them accessible for the wider research community. The majority of the SLR studies focused on comparing algorithms using Worst Time Complexity (WTC), Worst Space Complexity (WSC), and/or Approximation Ratio (AR). Only a few SLR studies describe algorithm comparison involving results obtained by solving problem instances. To close the gap, Section 6 of our proposed SLR contains not only a detailed description of the existing comparison mechanisms, but also an explanation of a fair performance evaluation framework.

Table 1. Systematic Literature Review Studies.

Paper	Time Period	$\mathit{P}\left|\right|{\mathit{C}}_{\mathit{max}}$ Specific	Methods	Instances Used	Comparisons Explained
[21]	1959–1970	no	H	P	P
[22]	1959–1974	no	E, H	N/A	P
[23]	1959–1977	no	H	N/A	N/A
[4]	1959–1979	no	H	N/A	P
[24]	1966–1981	yes	H	N/A	P
[18]	1966–1981	no	E, H	N/A	P
[25]	1966–1982	no	H	N/A	P
[26]	1966–1987	no	E, H	P	P
[27]	1969–1987	no	H	N/A	P
[28]	1969–1987	no	E	N/A	P
[29]	1966–1993	no	H, MH	N/A	P
[30]	1959–1994	no	E, H	P	P
[31]	1959–1997	no	E, H	N/A	N/A
[32]	1959–1998	no	E, H	N/A	P
[33]	1959–1999	no	E, H	N/A	P
[34]	1966–2001	no	E, H, MH	N/A	P
[7]	1959–2003	no	MH	P	P
[35]	1959–2004	no	H	N/A	N/A
[36]	1966–2004	no	E, H, MH	N/A	P
[20]	1966–2008	no	E, H, MH	P	P
[37]	1959–2009	no	E, H, MH	N/A	P
[38]	1959–2012	no	E, H, MH	N/A	P
[39]	1966–2013	no	E, H, MH	P	P
[40]	1969–2014	yes	E	P	P
[8]	1959–2017	no	E, H, MH	P	P
[41]	1961–2017	yes	H	P	P
[42]	1959–2018	yes	E, H	P	P
[43]	1982–2022	no	E	N/A	N/A
[44]	1959–2022	no	E, H	P	P
Proposed SLR	1959–2024	yes	All	All	All

Table 1. Systematic Literature Review Studies.

Paper	Time Period	$\mathit{P}\left|\right|{\mathit{C}}_{\mathit{max}}$ Specific	Methods	Instances Used	Comparisons Explained
[21]	1959–1970	no	H	P	P
[22]	1959–1974	no	E, H	N/A	P
[23]	1959–1977	no	H	N/A	N/A
[4]	1959–1979	no	H	N/A	P
[24]	1966–1981	yes	H	N/A	P
[18]	1966–1981	no	E, H	N/A	P
[25]	1966–1982	no	H	N/A	P
[26]	1966–1987	no	E, H	P	P
[27]	1969–1987	no	H	N/A	P
[28]	1969–1987	no	E	N/A	P
[29]	1966–1993	no	H, MH	N/A	P
[30]	1959–1994	no	E, H	P	P
[31]	1959–1997	no	E, H	N/A	N/A
[32]	1959–1998	no	E, H	N/A	P
[33]	1959–1999	no	E, H	N/A	P
[34]	1966–2001	no	E, H, MH	N/A	P
[7]	1959–2003	no	MH	P	P
[35]	1959–2004	no	H	N/A	N/A
[36]	1966–2004	no	E, H, MH	N/A	P
[20]	1966–2008	no	E, H, MH	P	P
[37]	1959–2009	no	E, H, MH	N/A	P
[38]	1959–2012	no	E, H, MH	N/A	P
[39]	1966–2013	no	E, H, MH	P	P
[40]	1969–2014	yes	E	P	P
[8]	1959–2017	no	E, H, MH	P	P
[41]	1961–2017	yes	H	P	P
[42]	1959–2018	yes	E, H	P	P
[43]	1982–2022	no	E	N/A	N/A
[44]	1959–2022	no	E, H	P	P
Proposed SLR	1959–2024	yes	All	All	All

The proposed SLR focuses on the $P\left|\right|C_{max}$ problem and all the methods published since the 1950s as there is no SLR that covers them all systematically. Moreover, we include an explanation of all instances and compare methodologies used in the literature. In the next section, we present our SLR methodology.

3. Systematic Literature Review Mapping Methodology

SLRs are the best way to understand the background for developing rigorous research projects. Thus, in many scientific fields there exist standards and guidelines for performing the SLR. The operations research literature provides a taxonomy of SLRs [45] and our SLR fits into the tutorial being selective towards the $P\left|\right|C_{max}$ problem. Additionally, it is an attempt to understand how to compare optimization algorithms, and how it fits into a broad, comprehensive, computational review. As the optimization community does not have an established SLR standard, we used the methodology based on standards established in closely related fields of software engineering [46]. In this section, we show the SLR methodology by explaining the following:

• Objectives and research questions;
• Search strategy;
• Search criteria;
• Inclusion and exclusion criteria;
• Search and selection procedure;
• Data extraction and synthesis;
• Important characteristics of selected primary studies.

3.1. Objectives and Research Questions

We planned the review process by refining the research objectives into a set of research questions. Our objectives to summarize and categorize the “state-of-the-art” methods, benchmark test instances, and comparison methodology could be accomplished through the following questions:

• RQ1. What are the main characteristics of $P\left|\right|C_{max}$ optimization methods?
• RQ2. What are the characteristics of problem instances that methods were tested on?
• RQ3. What are the characteristics of comparison methodologies used for performance evaluation of optimization methods identified in RQ1?
• RQ4: Based on RQ3, could a fair algorithm’s performance evaluation be defined?

In order to facilitate answering every question, a taxonomy-like characterization of the answers has been identified for the first three questions, and the answer to RQ4 is given in the discussion in Section 7.

3.2. Search Strategy

We utilized the PennState LionSearch tool, Copyright © 2021 The Pennsylvania State University, 201 Old Main, University Park, PA 16802, to explore the available manuscripts. LionSearch is an integrated search engine of books, journal papers, conference papers, and other publications integrated from around ${10}^3$ databases. It includes specialized databases and search engines for operation research, optimization, and applied mathematics disciplines. The tool is provided and maintained by the Pennsylvania State University’s Library.

We set up the following parameters for our search strategy: We looked for peer-reviewed articles written in the English language in the following categories: Journal Article, Book Chapter, Conference Proceedings, and Book/eBook. The detailed specification of the search domains included computer science, engineering, mathematics, and general sciences. The search was applied to the full text to ensure that the relevant study’s keywords were not missed in the title or abstract. The time period was not specified to ensure that all publications receive the chance to be selected.

3.3. Search Criteria

The search criteria had to include the keywords that were explained in Section 2. Therefore, our query was composed of two strings in the conjunctive normal form. The first should provide general context, while the second string specifies precise terms of interest. An example of a search conducted in the LionSearch is as follows:

((Parallel machine scheduling) OR (Identical Machine Scheduling) OR (Scheduling Problems) OR (independent tasks identical parallel machines) OR (independent jobs identical parallel machines) OR (independent tasks identical parallel processors) OR (independent jobs identical parallel processors)) AND ((P||Cmax) OR (PCmax)).

3.4. Inclusion and Exclusion Criteria

The initially retrieved studies from the electronic databases were assessed using the inclusion and exclusion criteria, explained in this subsection.

Inclusion criteria:

• IC1: the language is English;
• IC2: it is relevant to the $P\left|\right|C_{max}$ problem;
• IC3: it is an empirical research paper, a technical report, a proof of concept, a journal article, a thesis, or a conference paper;
• IC4: it cites or is cited by any of the recognized research studies.

Exclusion criteria:

• EC1: study’s focus is not explicitly on scheduling problems related to $P\left|\right|C_{max}$;
• EC2: the study does not address the optimization;
• EC3: the study does not meet all the inclusion criteria.

All studies that satisfied at least one exclusion criterion were not taken into consideration.

3.5. Search and Selection Procedure

The search was initialized in August 2023 and finalized in December 2024. In the first step, the search returned 3482 results. The majority of these materials contained just a reference to $P\left|\right|C_{max}$, did not focus on optimization algorithms/instances, or considered a variant that is not relevant. In Step 2, the selection of relevant results was performed and we ended up with a set of 236 documents. The most important component of the selection procedure happened to be the last inclusion criterion. Step 3, was devoted to reading all 236 manuscripts. The full text of each document was reviewed thoroughly by at least two team members. We had weekly meetings to discuss progress and review the findings. During the thorough reading we identified 100 more papers using a snowballing method. That is, we followed which papers cite the relevant papers and identifed papers that are cited by relevant papers. This resulted in 336 papers to assess quality upon.

In the last step, to assess the methodological quality of the primary studies selected for this review, the following quality criteria was adopted:

• QC1: Does the research clearly address any theoretical aspect? (1 or 0);
• QC2: Does the research clearly explain a method? (1 or 0);
• QC3: Are the findings clearly stated? (1 or 0);
• QC4: Based on the findings, is the research valuable? (1 or 0).

A quality score of 2 was set as the threshold to accept the study for this review. After applying the quality criteria, 261 papers were selected as the primary studies.

3.6. Data Extraction and Synthesis

The data extraction process was conducted by analyzing the selected 261 primary studies. We utilized a predefined extraction form to record the full details of the studies under review and to list the specifics which we could leverage to address motivation, background, and research questions.

3.7. Important Characteristics of the Selected Primary Studies

First, we classified the studies used in this paper according to the type of publication shown in Figure 1.

The figure contains (dotted line) the distribution of studies per year, with their number (maximum 10) indicated on the right side. For each type, the number of studies is written in the circle of the corresponding color.

As shown in Figure 1, the publication years of these references are between 1959 and 2024, with a mean year of 2000.40 and a standard deviation of 15.30. This points to a concentration of references that occur between the year 2000 and the present, with the early 2000s marking a significant surge. The interquartile range falls between 1987 and 2012, indicating a recent focus in the literature.

The majority of the studies, 172 of 261, are journal papers published in the journals from the following fields: discrete and applied mathematics, operations research, and industrial and transportation engineering. Other than clearly identifying the fields, we were not able to extract a useful pattern about the most popular venues for these types of publications. There is a considerate, but significantly smaller number of conference papers, 49 of 261, while books, research reports, and theses appear just sporadically. This distribution reflects a heavy reliance on peer-reviewed journals while also incorporating a variety of other publication types to ensure a comprehensive literature review.

Regarding the impact of publication types, citation counts for these references exhibit considerable variability. The mean number of citations is 455.90, but this is heavily skewed due to a few references receiving high citation numbers, as high as 19,154. The median citation count is much lower at 38, and the interquartile range spans from 10 to 170.50. This indicates that while many references have decent citation counts, there are a few highly influential works significantly affecting the average. The variability underscores the impact of a small number of seminal references. In particular, books were cited 3302.53 times on average (maximum 19,155). Journal papers had an average citation count of 288.87 (maximum 7905), and conference papers were cited 33.65 times on average (maximum 441).

In total, 152 studies provide some information related to RQ1, 81 are relevant for RQ2, while for motivation, background, and answering RQ3 and RQ4 all 261 studies were used. We have uncovered that the majority of the recent publications either compare their approaches with the results published more than 25 years ago, or use those algorithms as part of their solution methods. Contrary to all existing SLRs, we systematically cover all the methods, instances, and comparisons. Additionally, we have made all the timeline(s) clearly visible for each research question and identified group, such that any time period or other characteristics of interest can be distinguished and understood.

4. RQ1: Optimization Algorithms

The basic categorization of optimization methods is given in Section 2 and loosely follows the taxonomy defined in [19]. These basic categories explain three fundamentally different approaches to optimization problems. Exact solvers are able to guarantee the optimality of the solution if enough resources are provided. They generally reduce search space as much as they can to overcome the scarcity of resources. Heuristic methods usually provide “good” solutions within a short running time. In general, heuristics are not able to find optimal solutions or to guarantee the quality of the found solution. Contrary to heuristic approaches that are usually specialized, i.e., they use some a priori knowledge about the considered problem, metaheuristics are general-purpose algorithms. They can be applied to various optimization problems and may be viewed as recipes to guide the development of efficient heuristics for a specific optimization problem. Heuristic and metaheuristic optimization algorithms are focused on the fast construction of feasible solutions or on the iterative improvement in the existing solutions. Figure 2 shows the distribution of the selected primary studies that cover each identified optimization algorithm category.

As shown in Figure 2, the heuristic methods dominate the other two categories. This pattern might be occurring for two reasons. Heuristics usually provide feasible solutions fast; most providing some insights into the solutions quality and/or complexity of the underlying algorithms. This provides researchers with standards that they can compete on, i.e., publish improvements with respect to them. Metaheuristics have been explored for only 30 years, and experienced a meteoric rise since 2000s; hence, it is unsurprising that they occupy a slightly bigger area than the exact solvers. The latter are still very popular regardless of their limitations. Advantages of exact solvers include but are not limited to their theoretical importance, their applicability to special, easier yet very important, variants of the problem, and the new paradigms that enable the efficient explorations of solution space.

In this section, we explain the important steps for developing $P\left|\right|C_{max}$ optimization algorithms and divide the categories into groups based on the strategies used to address the $P\left|\right|C_{max}$ problem. For each group of methods, we present a short history, standardization, and a graph showing the distribution of methods publications over time. When presenting standardization, in each group of optimization algorithms the following table template is used: The columns of the table are labeled: Name of the optimization algorithm (Name); Where and when it is introduced (Reference), known selected characteristics for that optimization algorithm (Known Characteristics); Publications where it is compared with other algorithms from the same group (Comp. With). If any of the information is not provided or not applicable, the corresponding cell contains N/A.

4.1. The Main Issues in Developing Optimization Algorithms

There have been many theoretical discussions about the $P\left|\right|C_{max}$ problem, its properties, mathematical analysis, relations with other problems, hardness of instances, etc., [2,3,14,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94]. The majority of the theoretical conclusions are integrated into various optimization algorithms from all groups. In the remainder of this section, we rely on these theoretical results without explicit citations.

For developing any practical optimization algorithm, aiming to reduce its execution time, it is important to conduct the following:

• reduce search space;
• have suitable data structures;
• have efficient rules for the construction/transformation of solutions.

As we want to ensure that our SLR represents a self-contained material, we provide more detail about each of the items mentioned above.

4.1.1. Search Space Reduction

Usually, the first step in reducing the search space is finding good upper and lower bounds. To quickly find a good upper bound, a common practice is to use some constructive heuristic algorithms, which are described in Section 4.3.1. Many lower bound techniques have been developed for $P\left|\right|C_{max}$ since 1959 [16]. There are no theoretical guarantees that some lower bound technique will be better than others [14,95]; therefore, they are often combined for a better outcome.

To increase the efficiency of the lower bounds calculation, it is usually assumed that jobs are sorted in a non-increasing order of their processing times. The most famous simple lower bound is $L_0=⌈{\displaystyle \frac{1}{m}}{\sum }_{i=1}^n{p}_i⌉$ obtained by LP-relaxation rule that allows dividing job processing time among machines [16]. Another simple lower bound denoted by $L_1$ is calculated as $max\{L_0,{max}_i{p}_i\}$ [16]. The next simple lower bound is $L_2=max\{L_1,p_m+p_{m+1}\}$ [57] obtained by considering the relaxed set of $m+1$ as the biggest jobs, and proved ${\displaystyle \frac{L_2}{C_{max}^{*}}}\ge {\displaystyle \frac{2}{3}}$. By exploiting the fact that there must be a machine processing at least $\nu =\lceil n/m\rceil$ jobs, another simple lower bound is $L_{\nu }={\sum }_{i=n-\nu +1}^n{p}_i$ [57].

A whole class of other lower bounds may be obtained by observing a duality between $P\left|\right|C_{max}$ and BPP [54]. Assuming that items correspond to jobs, the capacity of each bin corresponds to the BPP-based lower bound L. The number of used bins corresponds to the number of machines m. Finally, L is a valid lower bound for $P\left|\right|C_{max}$ when the decision procedure of BPP returns yes. The procedures for calculating BPP-based lower bounds $L_{HS}$, $L_3$, and $L_{FS}$ are described in [54,57,96].

In addition to simple and BPP-based lower bounds, there are two other lower bounds: $L_{GB}$ [97] and $L_{\theta }$ [57]. To compute general lower bound $L_{GB}$, an instance of the $P\left|\right|C_{max}$ problem and machine-ready times (times when each machine can begin processing jobs) are required. Actually, it is possible to compute $L_{GB}$ even in the case when machines are already occupied. On the other hand, $L_{\theta }$ computes upper bound $\Theta$ and lower bound $\theta$ on the number of jobs per machine.

Each lower bound $L_{\xi }$ can be lifted using the lifting procedure [95] and lifted lower bound is denoted as ${\tilde{L}}_{\xi }$. There is also the enhanced lifting procedure [14] where the resulting lower bound is denoted as $L_{\xi }$. The former lifting procedure relaxes instance of $P\left|\right|C_{max}$ considering $k<m$ machines and the largest ${\lambda }_k(n)=k\lceil n/m\rceil +min\{k,n-\lceil n/m\rceil m\}$ jobs. The latter lifting procedure uses a solver for the Subset Sum Problem (SSP) to potentially improve the result of the first lifting procedure.

Some lower-bounding procedures and optimization algorithms perform the search over solution space and require a priori known bounds to reduce that space and, consequently, the execution time. Those particular procedures often use simple, computationally inexpensive lower and upper bounding procedures in order to reduce the solution space [54,57,96]. On the other hand, some procedures depend on the theoretical background of the simpler lower and upper bounds [97]. In addition, quickly found good lower bound can terminate the execution. It can also guarantee optimality if it becomes equal to the objective function value of a previously discovered feasible solution.

Lower-bounding procedures can be very computationally intensive, especially with active lifting procedures. Sometimes it is important to make a balance between a good lower bound and the required execution time for it. Finding a good initial solution can reduce and eventually stop the lower-bounding procedure. On the other hand, a good lower bound can reduce the search space for other solution-searching procedures.

In addition to lower bounds related to makespan, there are also bounds that impose constraints on other solution parameters. These lower bounds can additionally streamline the solution–finding process. That is, given a suboptimal solution S, when searching for an improved solution $S^{\prime }$, the condition $C_{\min }^{S^{\prime }}\ge {\sum }_{i\in m}{C}_i^{S^{\prime }}-(C_{\max }^S-1)(m-1)$ must hold [42].

There is no full review of lower-bounding procedures and their comparison in the literature. Similar papers include [14,95], and relying on their experiments; hence, it can be concluded that no single procedure dominates for all instances. There is a need for a more detailed comparison of all known lower-bounding procedures on a wider set of instances. However, it is out of the scope of this study. Other than the reduction in search space, lower bounds are important because the gaps with respect to them are used as the performance evaluation in many cited articles.

4.1.2. Data Structures

For a $P\left|\right|C_{max}$ optimization algorithm efficiency, it is important to define a proper solution representation that ensures quick access to all necessary data. In the studies selected during the SLR process, the following solution representations can be found: Matrix, Permutation, List, and Path.

The usual way to represent solutions in the mathematical programming-based approaches is to use binary assignment matrix ${\mathbf{X}}_{m\times n}=(x_{ij})$. For each $i\in M,j\in J$, element $x_{ij}$ of assignment matrix ${\mathbf{X}}_{m\times n}$ indicates whether job j is assigned to machine i, i.e., $x_{ij}=1$ when job j is assigned to machine i, and $x_{ij}=0$ otherwise. The assignment matrix representing a feasible solution for $P\left|\right|C_{max}$ has exactly one non-zero element in each column, i.e., it belongs to the class of sparse matrices.

Having in mind that each column in the assignment matrix has exactly one non-zero element, storing zeros is not needed. Hence, for each job $j\in J$ one can only store the machine index $\mu (j)\in M$ such that job j is assigned to machine $\mu (j)$, where $\mu :J\mapsto M$ is an assignment function. Solutions can be considered as permutations of jobs and these permutations actually define the order of their scheduling. Jobs are taken one by one from the permutation and assigned to the least occupied machine (list scheduling principle). Such a representation is indirect. Although each permutation uniquely defines the corresponding solution, it is necessary to perform additional computations before discovering the makespan value. This representation is more suitable for constructive optimization algorithms than for iterative improvement-based ones.

A solution of $P\left|\right|C_{max}$ problem could also be represented as an array $Mchn$ of m assignment lists, one for each machine. For machine $i\in M$, $Mchn\left[i\right]$ contains all indices $j\in J$ such that job j is assigned to machine i. The assignment lists can be implemented in various ways: (1) arrays of size n (inactive elements at the end are filled with zeros), (2) singly and (3) doubly linked lists. Each assignment list can be sorted in a non-increasing order by processing times of jobs. Moreover, assignment lists can be implemented as (4) heaps (facilitate accessing jobs with shortest or longest processing times in constant time) or (5) sets (facilitate accessing/inserting/removing jobs in logarithmic time).

The least intuitive solution representation uses flow networks. The scheduling problem is then formulated as the problem of determining m disjoint assignment paths between the source vertex s and the target vertex t such that each job is used exactly once. Each assignment path $p:s⇝t$ represents a machine along which the assigned jobs can be reconstructed.

4.1.3. Construction/Transformation Rules

As already mentioned, optimization algorithms either construct solutions from scratch, or perform transformations on the existing solutions with an aim of improving them. In both cases, some prior knowledge about the problem and the application of learning techniques may increase the efficiency of the underlying procedure. Therefore, special attention in developing $P\left|\right|C_{max}$ optimization algorithms should be paid to construction and transformation rules. More details are provided in the remainder of this section, where each optimization algorithm is described separately, if applicable.

4.2. Exact Algorithms

The first exact algorithm, presented 1962 [98], is based on dynamic programming. Over the years, many other exact solvers have been proposed. Our SLR revealed that 33 out of 167 selected studies consider the development of exact $P\left|\right|C_{max}$ solvers. Their main advantage is the optimality guarantee of provided solution, while the disadvantage relates to the large resource requirements. Nonetheless, exact algorithms are still studied from both theoretical and practical points of view.

We have classified the exact $P\left|\right|C_{max}$ solvers into two groups based on [99,100] and one group of practical significance.

• Exact exponential algorithms (EE);
• Fixed parameter tractable algorithms (FPT);
• Hybrid exact algorithms (HE).

EE algorithms for $\mathcal{NP}$-hard problems are focused on providing as small as possible WTC and WSC. The main characteristic of FPT algorithms is that they have polynomial complexity with respect to the input data, while their exponential dependence on some fixed parameter contributes to WTC. HE algorithms’ focus is often on exploring the results of some other type of optimization algorithms to provide optimal solutions with guaranties of optimality in a reasonable amount of time.

The practical importance of HE solvers stems from their adaptation and the ability to explore the results of other types of optimization algorithms as the upper bounds used to reduce the search space and increase the execution speed.

4.2.1. Exponential Exact Algorithms

To the best of our knowledge, all EE solvers for $P\left|\right|C_{max}$ are presented in

Table 2. Exponential exact algorithms.

Name	Reference	Known Characteristics	Compared With
${\mathrm{DP}}_{62}$	1962 [98]	DP; WTC: $\mathcal{O}(n{2}^n)$; WSC: $\mathcal{O}(2^n)$	N/A
${\mathrm{DP}}_{66}$	1966 [101]	DP; WTC: $\mathcal{O}(\frac{m^n}{n})$	N/A
${\mathrm{DP}}_{76}$	1976 [102]	DP; WTC: $\mathcal{O}(min\{2^n,nC\})$	N/A
${\mathrm{DP}}_{82}$	1982 [103]	DP; WTC: $\mathcal{O}(nC{2}^n)$; WSC: $\mathcal{O}(p_{max})$	N/A
${\mathrm{DP}}_{87}$	1987 [26]	DP; WTC: $\mathcal{O}(n{C}^m)$	N/A
${\mathrm{DP}}_{11}$	2011 [104]	DP; WTC: $2^{\mathcal{O}(m\sqrt{\left|I\right|})}$	N/A
${\mathrm{SSM}}_{13}$	2013 [105]	SSM; WTC: $O^{*}(m^{\frac{m}{2}}{(\frac{n}{2})}^{m+1})$; WSC: $O^{*}(m^{\frac{m}{2}}{(\frac{n}{2})}^{m-2})$	N/A

C—optimal $C_{max}$ value ($C_{max}^{*}$). $O^{*}({\alpha }^n)=\mathcal{O}(poly(n){\alpha }^n)$. $\left|I\right|$—length of the input.

. We include solver names which is based on the technique used and the year of publication.

The first developed exact solver [98] belongs to the class of EE solvers. The focus of researchers working on these solvers is improving WTC and WSC, which are therefore much better than those for other exact solvers. Unfortunately, this fact has no practical significance, since these algorithms are not implemented.

Even though many more techniques exist for EE solvers, it can be seen that the first and mainly used technique for $P\left|\right|C_{max}$ [98] is dynamic programming (DP). Notable EE algorithms later improved DP with techniques like linear waiting costs [101], partition algorithm scheme [102], the principle of inclusion and exclusion [103], recursive formulation [26], and dynamic allocation [104]. In 2013, another EE $P\left|\right|C_{max}$ solver was presented based on the Sort and Search method (SSM) [105]. The table shows that WTC has been improved consistently and WSC occasionally. Some WTC and WSC values depend on the variables that are indirectly correlated with the size of the input (n and m), and thus, do not have a clear exponential function form. Implementations of EE solvers are rarely found. As shown in Table 2, EE solvers are not directly compared to each other, but there is some cases where they are compared with other optimization algorithms.

Some EE solvers need the assumption of a known $C_{max}$, i.e., C to establish WTC and WSC in a pseudo-polynomial manner. Thus, the following theoretical result for the version of $P\left|\right|C_{max}$ with limited machine capacity - $P|C_{max}<C|$ [106] could be relied on. It has been proven the existence of a lower bound on the WTC under the Exponential Time Hypothesis (ETH). Assuming ETH, there is no algorithm that solves this problem with a worst-case time complexity $2^{n^{\delta }}{C}^{\mathcal{O}(1)}$ where $\delta \in [0,1]$, i.e., it cannot be solved in sub-exponential time. Lower bounds for exponential algorithms are studied in [64,107]. From Table 2 it can be seen that EE solvers are approaching this bound.

4.2.2. Fixed Parameter Tractable Algorithms

FPT algorithms focus on solving problems that are computationally hard in general but become tractable when certain parameter k has a small fixed value. In other words, these algorithms aim to provide efficient solutions for specific instances of a problem based on k. Actually, FPT algorithms are exponential only with respect to k, while the size of the input has polynomial influence to the algorithm’s running time. More formal definition is that algorithms with running time $f(k)n^c$, for a constant c independent of both n and k, are called FPT algorithms [100]. In such a way, value of k controls the combinatorial explosion.

In 2015, it was proven that the $P\left|\right|C_{max}$ problem is FPT parameterized by $p_{max}={max}_i{p}_i$ [62]. Since then, several studies focused on generalizing the results to similar problems [108,109]. Unfortunately, these algorithms cannot be applied in general cases, nor compared to other optimization algorithms. Thus, the practical significance of FPT solvers is very small, and more detailed consideration is out of the scope of this study.

4.2.3. Hybrid Exact Algorithms

The main characteristics of HE algorithms is leveraging the results of other types of optimization algorithms to reduce the search space. Providing optimal solutions, including guaranties of optimality, in a reasonable amount of time is what contributes to their practical value. HE algorithms identified in our SLR are summarized in

Table 3. Hybrid exact algorithms.

Name	Reference	Known Characteristics	Compared With
N/A	1973 [110,111]	B&B + Lagrange multipliers	N/A
N/A	1980 [112]	KP + Reduction method	N/A
BIN, DM	1995 [57]	BPP, B&B + MS + New LB	N/A
CGA, CKK	1998 [9]	Small m; B&B + LDM	N/A
${\mathrm{CP}}_{04}$	2004 [113]	B&B + Cutting plane + Polyhedral theory	N/A
HJ	2008 [14]	B&B + Symmetry-breaking + New LBs	[57]
DIMM	2008 [114]	B&B + Branch and price + Bin. search + SS	[57]
${\mathrm{SNP}}_{ie}$, RNP	2009 [115]	Small m; B&B + LDM	[9]
IRNP	2011 [116]	Small m; B&B + LDM + SSP	[115]
MOF	2013 [117]	Small m; B&B + SSP	[116]
BSBCP, ${\mathrm{CKK}}_i$, ${\mathrm{RNP}}_i$	2013 [118]	Small m; BPP; B&B + LDM + SSP	[9,116]
BSIBC	2013 [119]	Bin. Search + BPP	[114,118]
${\mathrm{SNP}}_{ess}$, ${\mathrm{H}}_{14}$	2014 [120]	Small m; B&B + LDM + SSP	[117,118]
CIW	2014 [121]	Small m; Iter. weakening + Caching	[117,119,120]
WL, ${\mathrm{WL}}^{\prime }$	2017 [8,63]	B&B + Path-related dominance criteria	[14,114]
LCS	2018 [42]	Small m; Iter. weakening + Caching + B&B	[9,115,116,117,119,120,121]
KL	2018 [122]	Bin. Search + B&B	[57,114]
AF	2018 [123]	ILP + AF + Reduction criteria + BPP	[8,14]
iAF	2022 [124]	ILP + AF + Graph compress. + BPP + VNS	[123]
DIST	2023 [12]	Decomposition + MKP	[124]

. For the known characteristics field we opted to present what the HE solvers were composed of. As shown in Table 3, HE solvers have been widely used and are still under development. Unlike other exact solvers, they have been compared with other solvers in the same group.

The first HE algorithm for solving $P\left|\right|C_{max}$ was developed in 1973 [110,111], and is based on Lagrangian multipliers and utilizes Branch and bound (B&B) techniques. The majority of HE algorithms use the B&B techniques with different criteria for branching and bounding searches [8,9,14,42,57,63,110,111,113,114,115,116,117,118,120,122]. Before branching, HE algorithms tend to tighten the search space as much as possible. For that purpose, lower bounding techniques, various heuristics (e.g., Multi-Subset (MS)), metaheuristics (e.g., Scatter Search (SS) and Variable Neighborhood Search (VNS)) and LP techniques are engaged [57,63,114,122]. Fine balancing between different phases is very important for the best performance.

Some exact solvers for $P\left|\right|C_{max}$ focus on solving instances with very small m and very big job processing times, which belong to the MWNP formulation. Complete Greedy Algorithm (CGA) [9], besides B&B simply tries all possible greedy ways to construct solution, while Complete Karmarkar–Karp (CKK), presented in the same paper, combines B&B with Largest Differencing Method (LDM) [53] algorithm. Authors used instances with $m\le 3$, $n\le 200$ and job processing times up to ${10}^{10}$. In [115], Sequential Number Partitioning with Inclusion Exclusion (${\mathrm{SNP}}_{ie}$) and Recursive Number Partitioning (RNP) start with LDM solution and follow the CKK algorithm and B&B strategy, are able to solve instances for m up to five and processing times up to ${10}^8$. IRNP [116] is a version of RNP, improved by using SSP. It can solve instances for m up to 10 and processing times up to ${10}^{10}$. MOF [117] also leverages B&B and SSP with new optimality rule—the principle of weakest-link. It can solve instances for m up to ten and processing times up to ${10}^{10}$. ${\mathrm{CKK}}_i$ and ${\mathrm{RNP}}_i$ [118] are improved versions of CKK and IRNP algorithms. ${\mathrm{RNP}}_i$ are able to solve instances with up to $m=7$ and with job completion times up to ${10}^{14}$. A new version of the Sequential Number Partitioning with extended Schroeppel and Shamir (${\mathrm{SNP}}_{ess}$) was developed in [120]. It generates all first subsets with sums within the lower and upper bounds, and then for each, recursively partitions the remaining numbers $k-1$ ways. It can solve instances for m up to ten and processing times up to ${10}^{14}$. ${\mathrm{H}}_{14}$ is a hybrid algorithm that combines CKK, CGA, ${\mathrm{SNP}}_{ess}$, and MOF. Cached Iterative Weakening (CIW) [121] uses iterative weakening instead of classic B&B, caches feasible subsets, and explores subsets in cardinality order. All algorithms in this paragraph before CIW are anytime algorithms that start with an approximate solution and then improve it until the best partition is found and optimality is proved. In contrast, CIW starts with a lower bound and increases it iteratively until an optimal solution is found, and the first complete partition found is optimal. The LCS algorithm [42] combines the ideas of CIW and the MOF, by search caches only low-cardinality subsets and performs in a two phases. In the first phase, it searches for an upper bound in the way similar to CIW. In the second phase, it uses the B&B technique to either prove the upper bound optimal or find a better optimal solution.

Other than B&B, a few solvers leverage binary search and duality with BPP. In [57], the modification of MF CH (see Section 4.3.1) is used for the BIN algorithm. It is realized by replacing the FFD procedure with exact MTP algorithm [125] with some small modifications and limited number of backtracks to $5·{10}^3$. For starting bounds they used $L_2$ and result of LPT CH. In the same paper, authors presented a very similar approach called MF, as part of the initial phrase for their B&B DM algorithm. BSBCP works in a similar way, but it uses the BCP algorithm [126] for solving BPP. The DIMM algorithm also uses a similar principle with more advanced initial bounds, and BSIBC [119] uses BPP solver improved by the authors.

Recently, ILP and graph compression techniques were combined with Arc Flow (AF) formulation [123,124]. The resulting algorithms (AF and iAF) show remarkable performance on some $P\left|\right|C_{max}$ instances, however, they do not provide any feasible solution if they are not able to find the optimal one. Another notable technique [112] reported in 1980 was based on the analogy between $P\left|\right|C_{max}$ and the Knapsack Problem (KP). Recently, the DIST algorithm [12], based on the decomposition and Multiple Knapsack Problem (MKP), can guarantee the optimality of the provided solution when adequate time limits for subproblems are defined. In all other cases it acts as iterative heuristic and can provide good suboptimal solutions very fast.

4.2.4. Summary of Exact Approaches

To conclude the section on exact approaches, in Figure 3, we present the time distribution of studies in which exact solvers were either developed or compared with other optimization algorithms. From the figure, it is easy to distinguish the development of the exact approaches in any decade of interest. EE solvers were the first developed; however, the most recent study was about a decade ago. FTP solvers were only recently developed with limited practical usability and are not represented in this figure as their focus might only be related to some special variants of the $P\left|\right|C_{max}$ problem. Finally, HE solvers have been developed since the introduction of the formal $P\left|\right|C_{max}$ problem definitions in the early 70s. Their development and usage has been intensified recently.

4.3. Heuristic Approaches

Out of all considered literature items, the largest amount of papers introduced or analyzed heuristic approaches to the $P\left|\right|C_{max}$ problem as can be seen in Figure 2. This is expected because the problem is $\mathcal{NP}$-hard and exact solvers require large amounts of resources. As aforementioned, in general, heuristics are not able to provide guarantees of optimality. However, there are some heuristic methods that provide solutions with provable quality and/or provable runtime estimation.

Definition 3 

($\mathit{\alpha }$-Approximation). An algorithm $\mathcal{A}$ is an α-approximation for an optimization problem Π if for each instance I of problem Π algorithm $\mathcal{A}$ computes a feasible solution to I and: $\mathcal{A}(I)\le \alpha OP{T}_{\Pi }(I)$, if Π is a minimization problem and $\alpha >1$; or $\mathcal{A}(I)\ge \alpha OP{T}_{\Pi }(I)$, if Π is a minimization problem and $\alpha <1$.

If $\alpha$ is a constant, then an $\alpha$-approximation is also known as a constant AR. For $P\left|\right|C_{max}$ if $n\to \infty$, it could be said that it has 2-approximation [127], 4/3-approximation [128], 13/11-approximation [129], 72/61-approximation [53], etc. Usually, AR is not a constant, as it depends on the size of the input data.

Heuristic approaches, also referred to in the literature as approximate algorithms, may be divided into three groups. One group contains algorithms that construct a solution, the other is composed of algorithms improving given solution as categorized in [8,130], while the algorithms from the third group provide solutions arbitrarily close to the optimal and are recognized in [19] as follows:

• Constructive Heuristics (CH);
• Improvement Heuristics (IH);
• Polynomial Time Approximation Schemes (PTAS).

The CH algorithms quickly build a feasible solution. Some of those algorithms might provide also the AR. The goal of IH algorithms is to enhance the quality of a given solution and, similar to CH, some of them may provide AR. PTAS algorithms take the solution quality as an input argument and provide a solution of a given quality in polynomial time.

4.3.1. Constructive Heuristics

The CH algorithms are able to construct feasible solutions for a given instance very fast. Their known characteristics may include AR and WTC, which give a solid performance prediction in terms of solution quality and execution time, respectively. CHs are used both independently and integrated in more complex optimization algorithms. With their execution time being often smaller than most lower-bounding procedures, they are usually executed just after the trivial lower bound ($L_2$). That is, CH algorithms are integrated in more complex optimization algorithms with the goal of providing a good upper bound for more complex lower-bounding procedures, improvement heuristics, exact solvers, and to solve easy instances fast (Section 4.1).

The CH algorithms recognized in our SLR study are summarized in

Table 4. Constructive heuristics.

Name	Reference	Known Characteristics	Compared With
SLS	1966 [127]	AR: $2-{\displaystyle \frac{1}{m}}$; WTC: $\mathcal{O}(nlogm)$	N/A
LPT, KK	1969 [128]	AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3m}}$ $({\displaystyle \frac{7}{6}}$ for $m=2)$; WTC: $\mathcal{O}(nlog(nm))$, AR: $1+{\displaystyle \frac{1-1/m}{1+\lfloor r/m\rfloor }}$; Strongly $\mathcal{NP}$-hard	N/A
${\mathrm{MF}}^{\prime }$	1972 [131]	AR: ${\displaystyle \frac{5}{4}}+{\displaystyle \frac{5}{4}}{2}^{1-k}$; WTC: $\mathcal{O}(nlogn+knlogm)$	N/A
MF	1978 [129]	AR: ${\displaystyle \frac{13}{11}}+2^{-k}$; WTC: $\mathcal{O}(nlogn+knlogm)$	[128,131]
MAAAT	1980 [112]	WTC: $\mathcal{O}(nlogn)$	N/A
LDM	1982 [53]	AR: $\left[{\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3(m-1)}},{\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3m}}\right]({\displaystyle \frac{7}{6}}$ for $m=2)$; WTC: $\mathcal{O}(nlogn+nmlogm)$	N/A
${\mathrm{MF}}_i$	1986 [132]	AR: ${\displaystyle \frac{72}{61}}+2^{-k}$ (tight for $m\ge 12$); WTC: $\mathcal{O}(nlogn+knlogm)$	N/A
${\mathrm{MF}}_e$	1988 [133]	AR: ${\displaystyle \frac{10}{9}}$ for $m=2$; WTC: $\mathcal{O}(nlogn+knlogm)$	[128,129]
COMBINE	1988 [134]	AR: ${\displaystyle \frac{13}{11}}+2^{-k}({\displaystyle \frac{10}{9}}$ for $m=2)$; WTC: $\mathcal{O}(nlogn+knlogm)$	[128,129]
${\mathrm{P}}_1$	1994 [135]	WTC: $\mathcal{O}(n)$	N/A
MS	1995 [57]	WTC: $\mathcal{O}(m{n}^2)$	N/A
${\mathrm{RAS}}_2$	1996 [136]	AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3m}}$; WTC: $\mathcal{O}(nlog(nm))$	[128,129]
LISTFIT	2001 [137]	AR: ${\displaystyle \frac{13}{11}}+2^{-k}$; WTC: $\mathcal{O}(n^{2}logn+k{n}^{2}logm)$	[128,129,134]
FGH, DGH	2001 [130]	AR: ${\displaystyle \frac{13}{11}}+2^{-k}$; WTC: $\mathcal{O}(nlogn+kmn)$	[128,129,136]
AP	2002 [138]	WTC: $\mathcal{O}(nlog(nm))$	[128,129,130]
${\mathrm{H}}_1$	2004 [139]	WTC: $\mathcal{O}(nmlog({\displaystyle \frac{1}{\epsilon }}+{\displaystyle \frac{1}{{\epsilon }^2}}))$	N/A
SS, ${\mathrm{LPT}}_R$	2006 [95]	WTC: $\mathcal{O}(nmS)$, AR: $2-{\displaystyle \frac{1}{m}}$; WTC: $\mathcal{O}(nlog(nm))$	[57,128]
MPS	2007 [140]	AR: $1+{\displaystyle \frac{m-1}{mz}}$; WTC: $\mathcal{O}(nlogn+nm)$	[128]
PSC	2008 [141]	WTC: $\mathcal{O}(nlogn)$	[128]
${\mathrm{PSC}}_i$	2009 [142]	WTC: $\mathcal{O}(nlogn)$	[128]
SPS	2010 [143]	AR: $1+{\displaystyle \frac{m-1}{mz}}$; WTC: $\mathcal{O}(nlogn)$	[128]
DJMS	2015 [144]	AR: ${\displaystyle \frac{13}{11}}+2^{-k}$; WTC: $\mathcal{O}(nlogn+mknlogm)$	[128,129,134,137]
PSMF	2015 [145]	AR: ${\displaystyle \frac{13}{11}}+2^{-k}$; WTC: $\mathcal{O}(k{n}^2)$	[128,129,134,137]
SLACK	2020 [146]	AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3(m-1)}}$ $({\displaystyle \frac{9}{8}}$ for $m=2)$; WTC: $\mathcal{O}(nlog(nm)+\lceil {\displaystyle \frac{n}{m}}\rceil log\lceil {\displaystyle \frac{n}{m}}\rceil )$	[53,128,134]

z—number of initial partitions. k—number of steps. r—number of optimally scheduled longest jobs. S—target sum of job durations. ε—target precision.

. They are based on three identified techniques: List Scheduling (LS), Binary Search, and Decomposition, which are explained in the remainder of this section.

LS technique consists of two main steps. The first one is to arrange jobs into a priority list according to some predefined criterion. The second step performs actual assignment/scheduling of one-by-one jobs from the list to the most appropriate machine as it is defined by some scheduling rule. Many CH algorithms use the LS technique with different job ordering criteria and different scheduling rules.

The first $P\left|\right|C_{max}$ optimization algorithm from the CH group was developed in 1966 [127]. In the literature, it is often referred to as LS because it uses the LS technique. However, we call it Simple List Scheduling (SLS) as the ordering criterion of jobs is not specified. The scheduling rule utilized in SLS assumes scheduling the current job on the least loaded machine, i.e., on the machine that is the earliest to start its execution. Therefore, this scheduling rule is often referred to as the Earliest Start (ES) scheduling rule.

The most popular LS-based algorithm is LPT [128], because of its simplicity, speed, and very good AR and WTC. The more appropriate name for this algorithm would be LPT+ES (as it is suggested in [147]). The LPT part of the name refers to the longest processing time job ordering criterion, while ES is used as a scheduling rule. However, the LPT+ES algorithm is most commonly referred to as the LPT algorithm only and we keep this notation. To distinguish between the two LPT meanings, we always write the LPT ordering criterion and LPT algorithm when appropriate. In the same study [128], Kleitman and Knuth’s (KK) algorithm is presented. It starts with a partial schedule in which $r<n$ longest jobs are optimally scheduled and then arbitrarily schedules the remaining jobs. It has been proven that, under the optimality of partial solution and for a large enough r, KK improves AR in comparison to the LPT algorithm.

Several algorithms leverage the LPT job ordering criterion. m-Assignment-at-a-time (MAAAT), proposed in [112], uses the LPT job ordering criterion and schedules jobs to machines in a zig-zag manner. Ordered jobs are partitioned into subsets of size m and jobs from each subset are scheduled to machines alternating between first-to-last and last-to-first order. Randomized LPT (${\mathrm{LPT}}_R$) [95] iteratively selects two unscheduled jobs with the longest processing times and then randomly assigns one out of this pair to the first available machine.

${\mathrm{RAS}}_2$ [136] heuristic is similar to the LPT algorithm, but instead of ES, Best Fit (BF) is applied to schedule the remaining jobs. That is, the unscheduled job with the longest processing time is assigned to the most heavily loaded machine such that $L_1$ bound is not exceeded. If such a machine does not exist, the job is assigned to the least loaded machine. Therefore, AR and WTC are the same as for the LPT algorithm. In [148], AR was proved to be the same as LPT, and we concluded that WTC is the same as LPT, too.

Fixed Gap Heuristics (FGH) and Dynamic Gap Heuristics (DGH) [130] are also based on the LS technique and have parametric and non-parametric variants. They use the LPT job ordering criterion and an alternation between ES and McNaughton Adjustment (McNA) [17,136] scheduling rules. According to McNA, the machine whose load, after the assignment of the current job, will be closest to the makespan lower bound (without exceeding it) is selected. If this is not possible, the job is assigned to the machine where its completion time would be minimal. The presented procedures work in iterations, using the best solution makespan and various lower bounds [130] for switching between the McNA and ES scheduling rules. Both of these procedures have significantly better AR and significantly larger WTC in comparison to the LPT algorithm.

Three constructive heuristics [138] are based on the following algorithm. The setup is to start with a jobs sorted in a non-increasing order of their processing times and machines divided in r subsets ($r\le m$). The algorithm takes the first subset of machines and assigns the jobs according to the LPT rule, until completion time of the included machines is nearest to a fixed limit (e.g., some lower bound), without exceeding it. When there is no machine in the first set to which it is possible to assign a job, respecting the limit, the next subset of machines is added to the first one and the algorithm follows in the same manner. When the last subset of machines was added the algorithm follows the LPT rule, assigning the rest of the jobs without respecting given limit. AP heuristics (${\mathrm{AP}}_{10}$, ${\mathrm{AP}}_{ni}$, and ${\mathrm{AP}}_i$) are derived based on the described algorithm depending on the way of forming the machine subsets. We concluded that heuristics can be implemented in WTC $\mathcal{O}(nlog(nm))$.

SLACK [146] is a recent modification of the LPT algorithm. It organizes jobs into groups (slacks) of m elements (complemented with zeros) and sorts them in a non-increasing order according to the difference between their longest and shortest jobs. In this way, a new priority list of jobs is obtained and jobs are scheduled one-by-one following the ES scheduling rule. The AR of the SLACK algorithm is improved in comparison to the LPT algorithm, without significantly increasing the WTC.

${\mathrm{P}}_1$ from the 3-PHASE [135] algorithm is also based on LS and does not require sorting. It starts with identifying minimum and maximum job processing time and dividing the interval between these two values into a predefined number c of equal or almost equal sub-intervals. In addition, an array of c elements is introduced and initialized with zeros. Each element contains the index of the machine where the previous job from the corresponding sub-interval is scheduled. Then, the algorithm takes one by one job from the unsorted list and schedules it to the machines in a round round-robin manner. That is, if the previous job from the same sub-interval is scheduled to the machine i, the current one is assigned to machine $i(modm)+1$. AR was not provided in the study, but the scheduling can be performed in linear time. With well-chosen parameter c, a balanced schedule can be obtained.

MULTIFIT algorithm (MF), developed in 1978 [129], is based on the binary search technique, firstly used in [131] for similar but weaker and less popular ${\mathrm{MF}}^{\prime }$ algorithm. At the beginning of the MF algorithm lower ($C_L$) and upper ($C_U$) bounds on the search interval have to be set. MF iteratively performs a binary search in the interval $[C_L(i),C_U(i)]$, where, $C_L(1)=L_1$ and $C_U(1)=max\{2L_1,p_{max}\}$. The main step of the binary search involves the execution of the First Fit Decreasing (FFD) algorithm [77,149] with the bin capacity limit $C=(C_L(i)+C_U(i))/2$ to construct the feasible solution for the BPP. The maximum number of iterations (steps) k is an input parameter of the MF algorithm. Recommended value for k is to be not smaller than seven, and seven is the default value if some another value is not mentioned. As AR depends on this input parameter, MF has a better AR for relatively small values of k, while having a higher WTC in comparison to the LPT algorithm. ${\mathrm{MF}}_e$ [133] and ${\mathrm{MF}}_i$ [132] are the modifications of MF.

The MPS [140] algorithm is based on the decomposition technique and it iteratively combines partial solutions (PS) of a given instance. The set of jobs is divided into a given number (z) of partitions. Each partition contains m disjunctive subsets of jobs, representing an initial PS, which are combined until a complete solution is obtained. The authors derived the properties that each PS should satisfy such that the resulting complete solution has AR ${\displaystyle \frac{z+1}{z}}-{\displaystyle \frac{1}{mz}}$. Although MPS’ AR is strongly related to the number of initial partitions, it has been demonstrated that its values are competitive with the LPT’s AR. A similar idea is used in PSC [141], and its improved variant ${\mathrm{PSC}}_i$ [142]; however, the AR for these two algorithms cannot be easily estimated. The newest algorithm in this group, SPS proposed in [143], has the same AR and lower WTC with respect to MPS.

There are several CH algorithms that are based on a combination of the identified techniques. The multi-Subset (MS) algorithm [57] uses the lower bound of $L_3$ and an efficient $G^2$ heuristic [150] for SSP. In the first phase, using $G^2$ it construcs at schedule the closest possible to $L_3$ for machines one by one, using currently unassigned jobs. In the second phase, if some jobs are still unscheduled, it is proved (assuming that $G^2$ has found an optimal solution) that their number is smaller then m. The optimal way to schedule them is to assign the longest unscheduled job to the less busy machine and so on. AR for this heuristic has not been derived. A similar CH algorithm to MS was developed for HI IH [139] and it is named ${\mathrm{H}}_1$. In this variant the longest yet unassigned task is assigned to the current processor. Then, a subset of the yet unassigned tasks such that the sum of their processing times is as close as possible to a given limit to the makespan will be assigned to the same processor. The MTSS(3) PTAS [151] is used in the second step. The remaining unassigned tasks are considered one by one in non-increasing order of their processing times and assigned to the processor with the smallest load. SS algorithm starts by finding a subset of jobs whose total processing time is minimal but not less than the value of a $L_2$, using the SSP solver. These jobs will be assigned to a first machine. Then, the algorithm computes a lower bound of the instance, which is defined by one machine less and the rest of the jobs. Again, a SSP is solved in order to determine an optimal subset of jobs that will be assigned to a second machine, and so on. In their implementation, authors used pseudo-polynomial time using the dynamic programming algorithm developed by Pisinger [152]. The COMBINE algorithm [134] is a combination of LPT and MF algorithms. It first applies the LPT algorithm and then invokes the MF algorithm with $C_L(1)=max\{L_1,C_{max}^{LPT}/A{R}^{LPT}\}$ and $C_U(1)=C_{max}^{LPT}$. The COMBINE algorithm has the same AR as MF and provides competitive results that correspond to the better among LPT and MF algorithms.

The Different Job and Machine Sets (DJMS) algorithm [144] applies to most m times LPT algorithm followed by MF algorithm on various subproblems of the considered $P\left|\right|C_{max}$. Unlike other LS-based algorithms, DJMS decomposes both job and machine lists.

The LISTFIT [137] algorithm combines MF with two LS techniques involving LPT and Shortest Processing Time (SPT) ordering criteria. The main idea of the LISTFIT algorithm is to create a number of job lists to be explored by the MULTFIT algorithm. In more detail, the set of jobs is iteratively partitioned into two subsets that are ordered according to all combinations of SPT and LPT criteria and used for generating various job lists. In such a way, $4n$ different lists are explored in the MULTFIT algorithm. The best obtained solution is returned as the final one. LISTFIT’s AR corresponds to the MF algorithm, while WTC is significantly larger.

The PSMF algorithm [145] is a recent combination of modified MPS and MF algorithms that has the same AR as MF but worse WTC than both MPS and MF.

The largest Differencing Method (LDM) also known as the Karmarkar–Karp algorithm [53] is based on the idea of how to construct a solution hierarchically in n iterations. In the beginning, jobs are sorted in the non-decreasing order of their processing times, and n partial solutions are created such that j-th partial solution consists of $m-1$ empty machines, while machine m is assigned job j only. In each of the remaining $n-1$ iterations, two partial solutions are replaced with the result of their combination performed in the following way: First, two partial solutions with the maximum difference between the loads of maximally and minimally loaded machines are identified. These two solutions are combined into a new partial solution by joining the load of the minimally loaded machine from the first with the load of the maximally loaded machine in the second partial solution. The second smallest load of the first partial solution is joined with the second largest load of the second partial solution, and so on. In the end, a unique complete solution is obtained, whose AR value is estimated to be between ARs of SLACK and LPT. However, it has a larger WTC than SLACK.

4.3.2. Improvement Heuristics

The IH algorithms start from an existing feasible solution (however it is provided) and iteratively apply different modifications (transformations) trying to improve its makespan. The most frequently applied modifications are move and swap (exchange). The move modification, takes one or more jobs from one machine and schedules it/them to some other machine. In its most general case, swap modification selects two machines, takes subsets of jobs assigned to each of them and exchanges their positions. The simplest and most common swap variant is the binary swap that exchanges positions (machines) for a pair of jobs scheduled on different machines. We use term interchange to denote any modification of some feasible solution. One of the important properties of the IH algorithms is that they can be provided by a stopping criterion to limit their execution. The best found solution is returned as the final result. Table 5 presents all improvement heuristics known to us, with information about other improvement heuristics they were compared with.

IH, proposed in 1978 [153], involves human help (interaction) in some steps of execution. Such IH approach is not practical, especially for large instances, and therefore, it was not further developed.

IC (0/1-INTERCHANGE), proposed in [154], is one of the simplest IH algorithms. It starts from a randomly generated feasible solution and iteratively performs the following steps. It sorts machines in the non-increasing order of their loads. It then calculates the difference between the most and the least heavily loaded machines. On the most heavily loaded machine, it finds the job with the processing time smaller than the calculated difference and moves it to the least heavily loaded machine. The above-mentioned steps are repeated until no such job can be found. Two improved versions of the IC algorithm (ICI and ICII) were developed in [155]. The final version of ICII differs from the original algorithm in the initial solution generation procedure. The longest $2m$ jobs are scheduled using the LPT algorithm, while the remaining jobs are assigned to the machines randomly. In this way, the same AR as the LPT algorithm is obtained without increasing WTC.

A more sophisticated approach, named Knapsack-on-multiple-processors (KOMP) algorithm, was proposed by [112]. The initial solution could be generated by any CH and the authors proposed to use LPT, MF, or their own MAAAT constructive procedure. In the improvement phase, the initial solution is first modified by iterative single-job moves and then by binary swaps (two jobs exchange their machines). Next, the resulting solution is additionally improved by iteratively solving the Knapsack problem for a pair of machines containing the most heavily loaded one using KP solver presented in [156].

The EX (EXCHANGE) algorithm [157] combines the interchange principle with the decomposition. The authors analyzed in detail the case when $m=2$ and proposed to swap and/or move subsets of jobs such that the current makespan is reduced and made closer to $L_0$. In the more general case, a pair of machines (one of them being the most heavily loaded) is selected and 2-machine transformation rules are applied until no improvement can be made, iteratively. In [158], this algorithm is named EXCHANGE, and its AR is calculated.

The 3-PHASE algorithm [135] is composed of three procedures (phases). The first procedure is fast CH ${\mathrm{P}}_1$. The second procedure repeatedly tries to move a job from the most heavily loaded machine to the least loaded machine in order to reduce the makespan. The third procedure tries to improve makespan by exchanging one job between the maximally loaded machine and some other machine.

Table 5. Improvement heuristics.

Name	Reference	Known Characteristics	Compared With
N/A	1978 [153]	Interactive	N/A
IC	1979 [154]	Interchange; AR: $2-{\displaystyle \frac{2}{m+1}}$; WTC: $\mathcal{O}(nlogm)$	N/A
KOMP	1980 [112]	KP + Interchange; WTC: $\mathcal{O}(nmS)$	N/A
EX	1981 [157]	Interchange + Decomposition; AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{2}{3m}}$; WTC: $\mathcal{O}(n^{2}mS)$	N/A
ICI, ICII	1982 [155]	Interchange; AR: ${\displaystyle \frac{3}{2}}-{\displaystyle \frac{1}{2m}}$; WTC: $\mathcal{O}(nlogm)$, Interchange; AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3m}}$; WTC: $\mathcal{O}(nlogm)$	N/A
3-PHASE	1994 [135]	Interchange + Decomposition; AR: 2	[154,155]
X-TMO	1995 [159]	SSP; AR: ${\displaystyle \frac{13}{11}}+2^{-k}$; WTC: $\mathcal{NP}$	N/A
${\mathrm{RAS}}_1$	1996 [136]	Interchange; WTC: $\mathcal{O}(nlog(nm))$	N/A
PI	1998 [160]	Interchange; WTC: $\mathcal{O}(nlog(nm))$	N/A
${\mathrm{CP}}_{99}$	1999 [17]	Cutting plane + ILP + Polyhedral theory	N/A
LPT+, MF+	2002 [161]	Interchange; AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3m}}$; WTC: $\mathcal{O}(nmSlogn)$, Interchange; AR: ${\displaystyle \frac{13}{11}}+2^{-k}$; WTC: $\mathcal{O}(nmSlogn)$	N/A
ME	2004 [162,163]	Interchange + Graph	[135]
HI	2004 [139]	BPP+TS; AR: ${\displaystyle \frac{5}{4}}+2^{-k}$	[135,163]
MSS	2006 [95]	SSP; AR: $2-{\displaystyle \frac{1}{m}}$; WTC: $\mathcal{O}(nmk{S}^2)$	[57,139]
MSK	2008 [14]	KP; AR: $2-{\displaystyle \frac{1}{m}}$; WTC: $\mathcal{O}(n^{2}mk{S}^2)$	[57,95,139]
CA	2011 [164]	Partial	[139,163]
PSMF+	2015 [145]	Interchange + Partial; AR: ${\displaystyle \frac{13}{11}}+2^{-k}$;	[139,163]
${\mathrm{KL}}_h$	2018 [122]	Interchange + BPP	[139]
MMIPMH	2019 [165]	Hopfild; AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3m}}$	N/A
SLACK+	2020 [146]	Interchange; AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3(m-1)}}$	N/A
N/A	2022 [166]	Interchange + KP	N/A
DIST	2023 [12] 2023	Decomposition + MKP; AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3(m-1)}}$; WTC: $\mathcal{NP}$	N/A

Table 5. Improvement heuristics.

Name	Reference	Known Characteristics	Compared With
N/A	1978 [153]	Interactive	N/A
IC	1979 [154]	Interchange; AR: $2-{\displaystyle \frac{2}{m+1}}$; WTC: $\mathcal{O}(nlogm)$	N/A
KOMP	1980 [112]	KP + Interchange; WTC: $\mathcal{O}(nmS)$	N/A
EX	1981 [157]	Interchange + Decomposition; AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{2}{3m}}$; WTC: $\mathcal{O}(n^{2}mS)$	N/A
ICI, ICII	1982 [155]	Interchange; AR: ${\displaystyle \frac{3}{2}}-{\displaystyle \frac{1}{2m}}$; WTC: $\mathcal{O}(nlogm)$, Interchange; AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3m}}$; WTC: $\mathcal{O}(nlogm)$	N/A
3-PHASE	1994 [135]	Interchange + Decomposition; AR: 2	[154,155]
X-TMO	1995 [159]	SSP; AR: ${\displaystyle \frac{13}{11}}+2^{-k}$; WTC: $\mathcal{NP}$	N/A
${\mathrm{RAS}}_1$	1996 [136]	Interchange; WTC: $\mathcal{O}(nlog(nm))$	N/A
PI	1998 [160]	Interchange; WTC: $\mathcal{O}(nlog(nm))$	N/A
${\mathrm{CP}}_{99}$	1999 [17]	Cutting plane + ILP + Polyhedral theory	N/A
LPT+, MF+	2002 [161]	Interchange; AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3m}}$; WTC: $\mathcal{O}(nmSlogn)$, Interchange; AR: ${\displaystyle \frac{13}{11}}+2^{-k}$; WTC: $\mathcal{O}(nmSlogn)$	N/A
ME	2004 [162,163]	Interchange + Graph	[135]
HI	2004 [139]	BPP+TS; AR: ${\displaystyle \frac{5}{4}}+2^{-k}$	[135,163]
MSS	2006 [95]	SSP; AR: $2-{\displaystyle \frac{1}{m}}$; WTC: $\mathcal{O}(nmk{S}^2)$	[57,139]
MSK	2008 [14]	KP; AR: $2-{\displaystyle \frac{1}{m}}$; WTC: $\mathcal{O}(n^{2}mk{S}^2)$	[57,95,139]
CA	2011 [164]	Partial	[139,163]
PSMF+	2015 [145]	Interchange + Partial; AR: ${\displaystyle \frac{13}{11}}+2^{-k}$;	[139,163]
${\mathrm{KL}}_h$	2018 [122]	Interchange + BPP	[139]
MMIPMH	2019 [165]	Hopfild; AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3m}}$	N/A
SLACK+	2020 [146]	Interchange; AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3(m-1)}}$	N/A
N/A	2022 [166]	Interchange + KP	N/A
DIST	2023 [12] 2023	Decomposition + MKP; AR: ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3(m-1)}}$; WTC: $\mathcal{NP}$	N/A

X-TMO [159] uses the new TMO SPP solver to optimally solve a case that includes the most heavily loaded and the least loaded machine. When the solution cannot be improved anymore using these two machines, the procedure uses the second least loaded machine. Then, it continues choosing the next least loaded machine while there exists a chance for improvement.

${\mathrm{CP}}_{99}$ [17] is an approximation algorithm that leverages linear programming formulations combined with a cutting planes method. This approach involves iterative calculations of lower bounds through successive linear programming relaxations.

${\mathrm{RAS}}_1$ [136] and PI [160] algorithms are similar to X-TMO, but they do not solve two-machine problems exactly. Instead, they try to find the best two jobs on two machines for exchange. The presented WTC for ${\mathrm{RAS}}_1$ in [136] is $\mathcal{O}(nlogn+nmlogm)$; however, we concluded that (using efficient data structures) both algorithms can be implemented in WTC $\mathcal{O}(nlog(nm))$.

LPT+ and MF+ [161] start with schedules constructed with LPT and MF CHs. A local search heuristic is applied to refine the solutions generated by constructive heuristics. The neighborhood is created by identifying pairs of jobs assigned to processors with varying loads, ensuring that one of the processors is the most heavily loaded. The local search involves evaluating these job pairs and accepting changes that decrease the load disparity between the two processors. After each adjustment, the neighborhood is redefined, and the search process begins again. The search ends when all job pairs in the neighborhood have been explored without achieving any improvement, indicating that a local optimum has been reached.

A wide family of IH solvers for $P\left|\right|C_{max}$ has been proposed in [162,163]. The authors showed that, among them, 1-SPT, K-SPT, 1-BPT, and K-BPT exhibit the best performance. They are based on a graph representation of the problem and multiple exchanges of jobs between machines. The best result of the mentioned four heuristic is denoted as Multi-Exchange (ME).

Hybrid Improvement (HI) [139] starts with LPT and ${\mathrm{H}}_1$ CHs, and $\epsilon$-DUAL PTAS for $\epsilon ={\displaystyle \frac{1}{5}}$ which also produces $L_{HS}$ and gives AR of ${\displaystyle \frac{5}{4}}+2^{-k}$. Additionally, lower bounds $L3$ and $L_{\theta }$ are used. Then, HI applies the binary search that explores duality between $P\left|\right|C_{max}$ and BPP where for every new target value of makespan new starting solutions are constructed and improved with TS.

Multistart Subset Sum (MSS) [95] is an IH algorithm which tries to iteratively improve the solution by equalizing the load of all pairs of machines with the most heavily loaded one and using an exact SSP solver. The Multistart Knapsack (MSK) algorithm [14] works in the same way, but always puts a minimal number of jobs on a machine with a smaller load. This strategy increases the chance for future improvements. To achieve this, instead of an exact SSP solver, it uses an exact KP solver.

The Composite Algorithm (CA) [164] starts from a family of initial partial solutions and combines these partial solutions (similarly to MPS [140]) until a feasible solution is generated. In the second phase, the CA applies local search procedures on partial solutions trying to generate a feasible solution with an improved makespan compared to the initial feasible solution. In total, five local search procedures are defined, one exploring the single-job moves, while four of them are based on the job swaps. In two of the swap-based neighborhoods, it is allowed to exchange a single job with a subset of jobs (referred to as composite jobs) from another machine. In addition, two neighborhoods allow accepting the solutions of the same quality as the initial one, as long as these moves increase the chances of improving the succeeding neighborhoods (similarly to MPS).

The PSMF+ algorithm [145], is obtained by inserting a 2-exchange procedure in PSMF CH after CUPS procedure and after MTMF procedure. It swaps two jobs from different machines until an advantageous exchange has been identified.

The ${\mathrm{KL}}_h$ algorithm [122] is an initial phase of the KL HE algorithm. It consists of the constructive approximation algorithms, the local searches for improvements for every constructed solution, the lower bounds, and the primal heuristic at the root node of the B&P tree for the makespan value is equal to the best lower bound. The main component is the primal heuristic with task to find a feasible solution for a BPP instance at the root node of HE algorithm.

The MMIPMH method [165] is inspired by balancing the Hopfild neural network with binary decision variables from the assignment ILP model for $P\left|\right|C_{max}$. In practice, it iteratively applies the LPT algorithm on selected partial solutions.

SLACK+ [146] applies a one-job swap neighbor search (NS) on a solution obtained with SLACK constructive heuristic. Swaps are performed between the most heavily loaded and some other machine.

In [166], a wide set of solution modification procedures that perform local improvements of an initial $P\left|\right|C_{max}$ solution is presented. In total, 35 solution modifications are proposed and, when combined with two CH algorithms to obtain an initial solution, they produce 70 simple IHs. Solution modifications are composed of job moves and swaps and are divided into two classes depending on the selection strategies for both jobs and machines. The main goal of these transformations is to obtain a makespan of the schedule as close as possible to $L_0$ lower bound. Transformations from the first class are composed of the following steps: The most heavily loaded machine (called source) is identified, a job from it is selected, as well as the destination machine where this job should be moved. Optionally, the load imbalance may be improved by moving some small-sized jobs from the destination machine to the source one in a deterministic way. There are three different ways to select the job to be moved and five possibilities to select the destination machine. Transformations from the second class combine (mix) all jobs from the source and destination machines (identified in the same ways as for the first class transformations) and schedule them by the KP algorithm such that the load imbalance is minimized. The complexity of all transformations is analyzed having in mind the efficient data structures that have been used.

Decomposition-based Iterative Stochastic Transformation (DIST) [12] is IH based on binary searching for optimal makespan and exact solving subproblems of increasing size using the MKP exact solver. DIST explores the relationship between $P\left|\right|C_{max}$, BPP, and Multiple Subset Sum Problem (MSSP) end, examining efficient algorithms for BPP and MSSP to provide high-quality solutions of $P\left|\right|C_{max}$. The main step of DIST explores a stochastic partitioning strategy to create subproblems of the original problem that are treated as instances of MSSP and solved by the efficient exact solver based on MKP within the predefined time limit. At the beginning of its execution, DIST starts by random partitions that create smaller subproblems with a larger potential to yield improvements of the current solution. In the case were the improvement has not been achieved, the algorithm creates and solves larger subproblems. In such a way, DIST performs nondeterministic transformations of the current solution, and therefore, it could be considered as a stochastic search algorithm. Due to the fact that it applies the time-limited exact solvers to the created subproblems, it belongs to the class of heuristic algorithms. However, DIST exhibits a very good performance within a short execution time, and becomes an exact solver for given big enough time limits.

4.3.3. Polynomial Time Approximation Schemes

The majority of constructive heuristics provide good approximations for the optimization of $P\left|\right|C_{max}$. However, $P\left|\right|C_{max}$ and many other optimization problems can be approximated to any precision. More precisely, there exists a special group of algorithms providing nearly optimal solutions that may be possible in polynomial time. They are known as Polynomial Time Approximation Schemes or PTAS algorithms [167]. In this section, we explain the characteristics of PTAS and review studies that introduced various PTAS algorithms.

Definition 4 

(Approximation Scheme). An approximation scheme for an optimization problem Π is a family of algorithms ${\mathcal{A}}_{\epsilon }$ defined as follows. For a given approximation accuracy parameter $\epsilon >0$ and an instance I of problem Π, algorithm ${\mathcal{A}}_{\epsilon }$ computes a feasible solution of I whose objective value differs from $OP{T}_{\Pi }(I)$ by at most $\epsilon OP{T}_{\Pi }$, i.e., $|{\mathcal{A}}_{\epsilon }(I)-OP{T}_{\Pi }(I)|\le \epsilon OP{T}_{\Pi }(I)$.

Actually, PTAS is defined as an algorithm that, for any given $\epsilon >0$, produces a feasible solution within a factor of $(1+\epsilon )$ of the optimal solution. The running time of the algorithm should be polynomial with respect to the input size and inversely proportional to $\epsilon$. For example, if an algorithm is a $(1+\epsilon )$-approximation PTAS for a certain problem, it means that the solution it provides is guaranteed to be within a factor of $(1+\epsilon )$ times the optimal solution. As $\epsilon$ approaches zero, the approximation factor approaches one, indicating a solution that is arbitrarily close to the optimal.

The parameter $\epsilon$ controls the distance to the optimal objective value, i.e., smaller values of $\epsilon$ result in better solutions (closer to optimum). It is important not to confuse approximation schemes with simple heuristic algorithms, as the later may not provide certain approximation guarantees. Selecting the appropriate value for $\epsilon$ involves a trade-off. A smaller $\epsilon$ generally leads to a more accurate approximation but may result in longer running times. The choice of $\epsilon$ depends on the specific requirements of the problem at hand and the balance between solution quality and computational efficiency that is acceptable for the application. Researchers and practitioners in general cases often experiment with different values of $\epsilon$ to find a suitable compromise for their particular use case.

Let us denote by $\left|I\right|$ the size of instance I, and let $f:\mathbb{R}\to \mathbb{R}$ be a polynomial function. Depending on the running time, the following types of approximation schemes are summarized in

Table 6. Approximation schemes.

Approximation Scheme	QPTAS	PTAS	EPTAS	FPTAS
WTC	${\left|I\right|}^{f(1/\epsilon )lo{g}^{\mathcal{O}(1)}\left|I\right|}$	${\left|I\right|}^{f(1/\epsilon )}$	${f(1/\epsilon )\left|I\right|}^{\mathcal{O}(1)}$	${(|I|/\epsilon )}^{\mathcal{O}(1)}$

. A quasipolynomial-time approximation scheme (QPTAS) if the corresponding algorithms run in time ${\left|I\right|}^{f(1/\epsilon )lo{g}^{\mathcal{O}(1)}\left|I\right|}$. A polynomial-time approximation scheme (PTAS) if they run in time ${\left|I\right|}^{f(1/\epsilon )}$. An efficient polynomial-time approximation scheme (EPTAS) if they run in time ${f(1/\epsilon )\left|I\right|}^{\mathcal{O}(1)}$. A fully polynomial-time approximation scheme (FPTAS) if they run in time ${(|I|/\epsilon )}^{\mathcal{O}(1)}$, which is polynomial in $\left|I\right|$ as well as in $1/\epsilon$.

In [102], the authors proposed an FPTAS for the $P_m\left|\right|C_{max}$ problem, i.e., when the number of machines m is fixed. However, in general, the problem is strongly $\mathcal{NP}$-hard, so there is no FPTAS for $P\left|\right|C_{max}$ unless $\mathcal{P}=\mathcal{NP}$. PTAS approach is not practical for $P\left|\right|C_{max}$ because, even for large values of $\epsilon$, the execution time is unacceptably long for large-size instances. Therefore, PTAS algorithms are usually studied from a theoretical aspect. WTC that includes $\epsilon$ are common comparison measures for this type of optimization algorithms.

The first PTAS for $P\left|\right|C_{max}$ is $\epsilon$-DUAL and it was presented in 1985 [54,168], and improved by [58,169,170,171]. For $(1+\epsilon )$-approximate solutions, the fastest known PTAS for $P\left|\right|C_{max}$ has a running time of $2^{\mathcal{O}((1/\epsilon )log(1/\epsilon )loglog(1/\epsilon ))}+\mathcal{O}(n)$ [172]. Ref. [107] shows that, assuming the exponential time hypothesis (ETH), for $\epsilon >0$ it cannot exist PTAS that yields $(1+\epsilon )$-approximate solutions with running time $2^{{(1/\epsilon )}^{1-\delta }}+poly(n)$ for any $\delta >0$.

Table 7. Polynomial time approximation schemes.

Name	Reference	Known Characteristics	Compared With
$\epsilon$-DUAL	1985 [54,168]	WTC: $n^{\mathcal{O}({(1/\epsilon )}^{2}log(1/\epsilon ))}$	N/A
N/A	1989 [169]	WTC: $n^{\mathcal{O}((1/\epsilon ){log}^2(1/\epsilon ))}$	N/A
N/A	1998 [58,170]	WTC: $2^{\mathcal{O}({(1/\epsilon )}^{poly(1/\epsilon )})}+\mathcal{O}(nlogn)$	N/A
N/A	2010 [171]	WTC: $2^{\mathcal{O}((1/{\epsilon }^2){log}^3(1/\epsilon ))}+\mathcal{O}(nlogn)$	N/A
JR	2019 [174]	WTC: $2^{\mathcal{O}((1/\epsilon ){log}^2(1/\epsilon ))}+\mathcal{O}(n)$	N/A
N/A	2020 [175]	WTC: $2^{\mathcal{O}((1/\epsilon ){log}^4(1/\epsilon ))}+poly(n)$	N/A
BDJR	2022 [173]	WTC: $2^{\mathcal{O}((1/\epsilon )log(1/\epsilon )loglog(1/\epsilon ))}logn+\mathcal{O}(n)$	N/A
N/A	2023 [172]	WTC: $2^{\mathcal{O}((1/\epsilon )log(1/\epsilon )loglog(1/\epsilon ))}+\mathcal{O}(n)$	N/A

shows the WTC of all known PTAS optimization algorithms for $P\left|\right|C_{max}$. Among them, only $\epsilon$-DUAL and BDJR [173] have been implemented. Lower bounds for approximation schemes were studied by [64].

4.3.4. Summary of Heuristic Approaches

One of the oldest constructive techniques, LPT proposed in 1966 [127], still presents the most popular heuristic for the $P\left|\right|C_{max}$ problem due to its simplicity and effectiveness. Ever since its development, it has been used in other approaches to generate the initial solution. The different heuristic approaches developed from 1966 to today are summarized in Figure 4. The figure shows that it is easy to distinguish the development of the heuristic approaches in any decade and for any group of interest. There has been a steady development of improvement heuristics and that PTAS were introduced in 1980s and there has been renewed interest in that type of algorithms lately.

4.4. Metaheuristics

Metaheuristic (MH) algorithms [19] are high-level search strategies designed to find near-optimal solutions for optimization and search problems. These algorithms are not problem-specific but can be applied to a wide range of problems. They are particularly useful when dealing with complex problems for which finding an exact solution is impractical or impossible within a reasonable amount of time.

MH algorithms became popular in the late 1980s and their application to $P\left|\right|C_{max}$ started at that time. We will use [19] classification and divide MH algorithms into Population-Based Metaheuristics (P-class) and Single Solution-Based Metaheuristics (S-class). P-class metaheuristics maintain a population of potential solutions throughout the search process. The algorithms maintain a population though its evolution over time, typically through mechanisms inspired by natural processes such as selection, mutation, and recombination. S-class metaheuristics focus on iteratively improving a single candidate solution. They generally perform local search around the current solution and move towards better solutions through various strategies. We are primarily interested in enumerating the approaches that were successfully applied to $P\left|\right|C_{max}$.

All MH approaches are summarized in

Table 8. Metaheuristics.

Name	Reference	Known Characteristics	Compared With
${\mathrm{SA}}_{88}$	1988 [10]	SA	N/A
HG	1994 [178]	TS	[10]
${\mathrm{SA}}_{97}$	1997 [179]	SA	N/A
TGR	1998 [180]	TS	[178]
${\mathrm{GA}}_{99}$, ${\mathrm{SA}}_{99}$	1999 [181]	GA, SA	N/A
AIS, ${\mathrm{SA}}_{02}$	2002 [161]	IBA	N/A
AntMPS	2003 [7]	ACO	N/A
${\mathrm{SA}}_{06}$	2006 [182]	SA	N/A
${\mathrm{ILS}}_{06}$	2006 [183]	ILS	N/A
HDNN	2006 [184]	ANN	N/A
${\mathrm{HDNN}}_i$	2007 [185]	ANN	N/A
TCNN, ${\mathrm{TCNN}}_i$	2008 [186]	ANN	[184]
${\mathrm{DIMM}}_{SS}$	2008 [114]	SS	N/A
${\mathrm{MC}}_{09}$	2009 [187]	MC	N/A
MVNS, ${\mathrm{GA}}_{09}$	2009 [5]	VNS, GA	N/A
${\mathrm{BCO}}_{09}$	2009 [6]	BCO	N/A
${\mathrm{VNS}}_{09}$	2009 [188]	VNS	[187]
${\mathrm{DPSO}}_{09}$, HDPSO	2009 [189]	PSO	[182]
DSHS	2010 [190]	HS	[182,189]
${\mathrm{DHS}}_{11}$, HDHS	2011 [191]	HS	[5,181,182]
RIVNS, HIVNS	2012 [192]	VNS	[5,181,182]
${\mathrm{SA}}_{12}$	2012 [193]	SA	[182]
SPPSO, ${\mathrm{DPSO}}_{12}$, ${\mathrm{PSO}}_{spv}$	2012 [194]	PSO	N/A
${\mathrm{BCO}}_{12}$	2012 [177]	BCO	N/A
${\mathrm{DHS}}_{12}$, BHS, ${\mathrm{DHS}}_{LS}$	2012 [195]	HS, HS, HS+VNS	[182,189]
CSA	2015 [196]	CS	[182,189]
GES, GES+	2018 [197]	GES	[182]
ICSA	2018 [198]	CS	[182,189,195,196]
${\mathrm{RIVNS}}_1$, ${\mathrm{HIVNS}}_1$, ${\mathrm{RIVNS}}_2$, ${\mathrm{HIVNS}}_2$	2018 [199]	VNS	[5,181,182,192]
${\mathrm{GA}}_{19}$, ${\mathrm{GWO}}_{19}$	2019 [200]	GA, GWO	N/A

. A short review of studies of HS metaheuristics for $P\left|\right|C_{max}$, without practical comparisons, is conducted in [176]. It could be seen that the S-class and P-class MH algorithms are equally represented. The majority of metaheuristics are performing transformations on the existing solutions, while only some sporadic approaches are based on the construction of new and potentially better solutions [177].

Summary of Metaheuristic Approaches

Figure 5 presents the distribution over time of papers describing the applications of different underlying metaheuristics. From the figure, it is easy to distinguish the development of the metaheuristic approaches in any decade and for any algorithm of interest. Up to ten years ago Simulated Annealing (SA) was the most popular metaheuristics [10,161,179,181,182,193]. The first metaheuristic algorithm for $P\left|\right|C_{max}$ problem (MMBPP formulation) from 1988 [10] used SA. The Tabu Search (TS) was applied by [178,180]. The Genetic Algorithm (GA) was applied in [5,181,200]. The Harmony Search (HS) was applied most recently [190,191,195]. Other notable metaheuristics include the Iterated Local Search (ILS) [183], Genetics Algorithms (GA) [181], Gray Wolf Optimization (GWO) [200], Cuckoo Search (CS) [196,198], Immune-Based Approach (IBA) [161], Ant Colony Optimisation (ACO) [7], Scatter Search (SS) [114], Particle Swarm Optimization (PSO) [189,194], Monte Carlo (MC) [187], Bee Colony Optimization (BCO) [6,177], Grouping Evolutionary Strategy (GES) [197], and Variable Neighborhood Search (VNS) [5,188,192,199]. Deep learning methods, being successful in many different areas, might have a potential in applications to $P\left|\right|C_{max}$. However, so far only methods that use Artificial Neural Networks (ANN) have been developed [184,185,186].

4.5. Parallel Optimization Algorithms

Parallelization of optimization methods is a very popular research avenue: however, its description is out of the scope of this study. Optimization algorithms from all three main categories can be parallelized. However, in our SLR, we found only four parallel algorithms for $P\left|\right|C_{max}$. The parallel version of $\epsilon$-DUAL PTAS proposed by [54] is published in [201], while [202] contains parallel versions of ${\mathrm{BCO}}_{12}$ MH [177], and in [203] parallelization of IRNP HE algorithm [116] is presented. BDJR PTAS is presented in [173] along with its parallel implementation.

4.6. Taxonomy of $P\left|\right|C_{max}$ Optimization Algorithms

Figure 6 shows the taxonomy of $P\left|\right|C_{max}$ optimization algorithms. Relationships between different types of algorithms are represented by the connecting lines and similar colors. HE solvers often use some heuristics and/or metaheuristic before B&B phase. Metaheuristics often use some heuristics for initial phase. IHs starts with some CH solution, and tries to improve it. Any type of algorithm can be more or less effectively parallelized.

5. RQ2: Standardization of the Problem Instances

To evaluate the performance of any optimization algorithm, a representative set of benchmark test instances is needed. Unfortunately, our SLR shows that such a representative set does not exist for $P\left|\right|C_{max}$. We found a number of various instance sets, but not all are available to wider audience. Therefore, our aim was to review the most commonly used sets, identify their strengths and weaknesses, classify them into various categories, and select a representative set for each category. Finally, we perform the standardization of identified groups of instances according to the criteria presented in the reminder of this section.

The most commonly used are the four groups of instances that we classify as follows:

• E instances;
• F instances;
• C instances;
• B instances.

Please note that the names for groups of instances do not systematically appear in the corresponding papers, and therefore, we provide detailed explanations. The description of each group of widely used instances includes a short history, main characteristics, and the list of papers exploring these instances in the experimental evaluation. At the end of this section, we provide a graph illustrating the utilization frequency of various instance groups in the relevant publications over time. From that graph it will be easy to distinguish the use of instance groups in any decade and for any group of interest. All standardized instances are now made available at the publicly accessible website https://gitlab.com/pcmax-problem/pcmax-instances, accessed on 20 January 2025. This repository contains either the generation code with the guidance how to use it or the set of instances ready for use.

In addition, we propose a uniform notation ${{}_M{}^{N}I}_p^{{\displaystyle \frac{n}{m}}}$. N and M representing maximal values for n and m, respectively, I denotes related set of instances, and D describes the distribution for generating job durations. $\tilde{I}$ represents set of instance similar to I, and $\overline{I}$ represents subset of I. If different values for ${\displaystyle \frac{n}{m}}$ and/or D are exploited, they are all listed separated by comma. The correspondence between our notation and the instance names from the literature is described in the reminder of this section.

The main characteristics of instances from various groups are summarized in Table 9, Table 10, Table 11 and Table 12. These tables are organized in the following way: The first column contains names of the subgroup of the presented group of instances ($class$), followed by the corresponding notation (that we introduced) in the second column ($notation$). The ratio between the number of jobs and the number of machines (${\displaystyle \frac{n}{m}}$) is presented in the third column. The fourth column shows the number of machines (m), while the type of distribution and its corresponding parameters used to obtain values of processing times are given in the last column (D). For uniform distribution we are using notation ${\mathcal{U}}_{[a,b]}$, where $[a,b]$ represents interval for sampling values. For normal distribution, notation ${\mathcal{N}}_{\mu ,\sigma }$, where $\mu$ represents mean and $\sigma$ denotes standard deviation, is utilized. Each combination of values presented in columns three to five corresponds to one generation rule for instances (some authors also use the term instance size). An instance that has a solution for which $C_{max}^{*}=L_0$ is known as a perfect instance, and the corresponding solution is known as a perfect schedule, i.e., a perfect partition. In these types of schedules, the difference in the loads of any two machines is not greater than one.

At the end of this section, we also cover other instances that do not match any criteria defining the above mentioned groups. However, the introduced notation applies to those instances as well.

5.1. E Instances

The first widely used group of instances for the $P\left|\right|C_{max}$ problem are what we will call E instances. E instances cover a huge range of parameter ${\displaystyle \frac{n}{m}}$. The first part of those instances, $E_1$ instances, was proposed in 1971 for testing of the developed heuristic in [204], the study that was never published. These instances were used also in [21,26]. Extension $E_2$ proposed in 1988 was used in [133,134] where normal distribution is letter replaced with uniform distribution, while $E_3$ and $E_4$ extensions appeared in 2001 [137]. The final version of the complete E group is shown in Table 9. The main idea behind E instances was to develop a framework that would serve for testing various algorithms used for $P\left|\right|C_{max}$ problem.

This group has a variety of different values for ${\displaystyle \frac{n}{m}}$ parameter, which is generally preferred characteristic. All job’s processing times are generated in a uniform way, with the smallest range $[10,20]$ and the largest range being $[100,800]$. However, values for m are too small to make these instances hard in any sense. Another big problem is the absence of information about random seeds used for their generation, as well as corresponding download repositories.

Today, E instances are obsolete for testing state-of-the-art optimization algorithms, although they are still in use for testing MHs [182,189,190,193,194,195,196,197,198,200], CHs [21,26,30,134,137,144,145], and measuring the performance of PTAS [173]. In all extracted studies, the authors generated E instances on their own with different random seeds. Thus, there is no guarantee that the obtained results could be compared fairly. To address this problem, in our repository we provide an open source generator for E instances and we propose to generate 50 instances per generation rule, using 1 as the seed value.

Table 9. E instances with 120 different generation rules.

$\mathit{Class}$	$\mathit{Notation}$	$\frac{\mathit{n}}{\mathit{m}}$	m	D
$E_1$	${{}_5{}^{25}{E}_1}_{{\mathcal{U}}_{[10,20]},{\mathcal{U}}_{[20,50]}}^{2,3,5}$	2, 3, 5	3, 4, 5	${\mathcal{U}}_{[10,20]}$, ${\mathcal{U}}_{[20,50]}$
$E_2$	${{}_2{}^{100}\overline{E_2}}_{{\mathcal{U}}_{[100,800]}}^{5,15,25,50}$	5, 15, 25, 50	2	${\mathcal{U}}_{[100,800]}$
	${{}_3{}^{100}\overline{E_2}}_{{\mathcal{U}}_{[100,800]}}^{3.33,10,16.67,33.33}$	3.33, 10, 16.67, 33.33	3
	${{}_4{}^{100}\overline{E_2}}_{{\mathcal{U}}_{[100,800]}}^{7.5,12.5,25}$	7.5, 12.5, 25	4
	${{}_6{}^{100}\overline{E_2}}_{{\mathcal{U}}_{[100,800]}}^{5,8.33,16.67}$	5, 8.33, 16.67	6
	${{}_8{}^{100}\overline{E_2}}_{{\mathcal{U}}_{[100,800]}}^{3.75,6.25,12.5}$	3.75, 6.25, 12.5	8
	${{}_{10}{}^{100}\overline{E_2}}_{{\mathcal{U}}_{[100,800]}}^{3,5,10}$	3, 5, 10	10
$E_3$	${{}_3{}^{17}\overline{E_3}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[100,200]},{\mathcal{U}}_{[100,800]}}^{3.33,3.67,4.33,4.67,5.33,5.67}$	3.33, 3.67, 4.33, 4.67, 5.33, 5.67	3	${\mathcal{U}}_{[1,100]}$, ${\mathcal{U}}_{[100,200]}$, ${\mathcal{U}}_{[100,800]}$
	${{}_5{}^{27}\overline{E_3}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[100,200]},{\mathcal{U}}_{[100,800]}}^{3.2,3.4,4.2,4.4,5.2,5.4}$	3.2, 3.4, 4.2, 4.4, 5.2, 5.4	5
	${{}_8{}^{42}\overline{E_3}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[100,200]},{\mathcal{U}}_{[100,800]}}^{3.12,3.25,4.12,4.25,5.12,5.25}$	3.12, 3.25, 4.12, 4.25, 5.12, 5.25	8
	${{}_{10}{}^{52}\overline{E_3}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[100,200]},{\mathcal{U}}_{[100,800]}}^{3.1,3.2,4.1,4.2,5.1,5.2}$	3.1, 3.2, 4.1, 4.2, 5.1, 5.2	10
$E_4$	${{}_2{}^9\overline{E_4}}_{{\mathcal{U}}_{[1,20]},{\mathcal{U}}_{[20,50]},{\mathcal{U}}_{[50,100]},{\mathcal{U}}_{[100,200]},{\mathcal{U}}_{[100,800]}}^{4.5}$	4.5	2	${\mathcal{U}}_{[1,20]}$, ${\mathcal{U}}_{[20,50]}$, ${\mathcal{U}}_{[50,100]}$, ${\mathcal{U}}_{[100,200]}$, ${\mathcal{U}}_{[100,800]}$
	${{}_3{}^{10}\overline{E_4}}_{{\mathcal{U}}_{[1,20]},{\mathcal{U}}_{[20,50]},{\mathcal{U}}_{[50,100]},{\mathcal{U}}_{[100,200]},{\mathcal{U}}_{[100,800]}}^{3.33}$	3.33	3

Table 9. E instances with 120 different generation rules.

$\mathit{Class}$	$\mathit{Notation}$	$\frac{\mathit{n}}{\mathit{m}}$	m	D
$E_1$	${{}_5{}^{25}{E}_1}_{{\mathcal{U}}_{[10,20]},{\mathcal{U}}_{[20,50]}}^{2,3,5}$	2, 3, 5	3, 4, 5	${\mathcal{U}}_{[10,20]}$, ${\mathcal{U}}_{[20,50]}$
$E_2$	${{}_2{}^{100}\overline{E_2}}_{{\mathcal{U}}_{[100,800]}}^{5,15,25,50}$	5, 15, 25, 50	2	${\mathcal{U}}_{[100,800]}$
	${{}_3{}^{100}\overline{E_2}}_{{\mathcal{U}}_{[100,800]}}^{3.33,10,16.67,33.33}$	3.33, 10, 16.67, 33.33	3
	${{}_4{}^{100}\overline{E_2}}_{{\mathcal{U}}_{[100,800]}}^{7.5,12.5,25}$	7.5, 12.5, 25	4
	${{}_6{}^{100}\overline{E_2}}_{{\mathcal{U}}_{[100,800]}}^{5,8.33,16.67}$	5, 8.33, 16.67	6
	${{}_8{}^{100}\overline{E_2}}_{{\mathcal{U}}_{[100,800]}}^{3.75,6.25,12.5}$	3.75, 6.25, 12.5	8
	${{}_{10}{}^{100}\overline{E_2}}_{{\mathcal{U}}_{[100,800]}}^{3,5,10}$	3, 5, 10	10
$E_3$	${{}_3{}^{17}\overline{E_3}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[100,200]},{\mathcal{U}}_{[100,800]}}^{3.33,3.67,4.33,4.67,5.33,5.67}$	3.33, 3.67, 4.33, 4.67, 5.33, 5.67	3	${\mathcal{U}}_{[1,100]}$, ${\mathcal{U}}_{[100,200]}$, ${\mathcal{U}}_{[100,800]}$
	${{}_5{}^{27}\overline{E_3}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[100,200]},{\mathcal{U}}_{[100,800]}}^{3.2,3.4,4.2,4.4,5.2,5.4}$	3.2, 3.4, 4.2, 4.4, 5.2, 5.4	5
	${{}_8{}^{42}\overline{E_3}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[100,200]},{\mathcal{U}}_{[100,800]}}^{3.12,3.25,4.12,4.25,5.12,5.25}$	3.12, 3.25, 4.12, 4.25, 5.12, 5.25	8
	${{}_{10}{}^{52}\overline{E_3}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[100,200]},{\mathcal{U}}_{[100,800]}}^{3.1,3.2,4.1,4.2,5.1,5.2}$	3.1, 3.2, 4.1, 4.2, 5.1, 5.2	10
$E_4$	${{}_2{}^9\overline{E_4}}_{{\mathcal{U}}_{[1,20]},{\mathcal{U}}_{[20,50]},{\mathcal{U}}_{[50,100]},{\mathcal{U}}_{[100,200]},{\mathcal{U}}_{[100,800]}}^{4.5}$	4.5	2	${\mathcal{U}}_{[1,20]}$, ${\mathcal{U}}_{[20,50]}$, ${\mathcal{U}}_{[50,100]}$, ${\mathcal{U}}_{[100,200]}$, ${\mathcal{U}}_{[100,800]}$
	${{}_3{}^{10}\overline{E_4}}_{{\mathcal{U}}_{[1,20]},{\mathcal{U}}_{[20,50]},{\mathcal{U}}_{[50,100]},{\mathcal{U}}_{[100,200]},{\mathcal{U}}_{[100,800]}}^{3.33}$	3.33	3

5.2. F Instances

The next identified group of widely used instances is F, in relevant papers often called benchmark instances. In the first part of those instances (uniform), $F_U$, introduced in 1994 [135] for the purpose of testing 3-PHASE IH, is proposed by the authors. Ten years later, in 2004, the (non-uniform) extension, $F_{NU}$, of this group of instances is proposed by [163] with the purpose of providing harder instances required by their ME IHs. F instances are presented in Table 10.

Table 10. F instances with 78 different generation rules.

$\mathit{Class}$	$\mathit{Notation}$	$\frac{\mathit{n}}{\mathit{m}}$	m	D
$F_U$	${{}_5{}^{{10}^3}\overline{F_U}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[1,{10}^3]},{\mathcal{U}}_{[1,{10}^4]}}^{2,10,20,100,200}$	2, 10, 20, 100, 200	5	${\mathcal{U}}_{[1,100]}$, ${\mathcal{U}}_{[1,{10}^3]}$, ${\mathcal{U}}_{[1,{10}^4]}$
	${{}_{10}{}^{{10}^3}\overline{F_U}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[1,{10}^3]},{\mathcal{U}}_{[1,{10}^4]}}^{5,10,50,100}$	5, 10, 50, 100	10
	${{}_{25}{}^{{10}^3}\overline{F_U}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[1,{10}^3]},{\mathcal{U}}_{[1,{10}^4]}}^{2,4,20,40}$	2, 4, 20, 40	25
$F_{NU}$	${{}_5{}^{{10}^3}\overline{F_{NU}}}_{{\xi }_1,{\xi }_2,{\xi }_3}^{2,10,20,100,200}$	2, 10, 20, 100, 200	5	${\xi }_1,{\xi }_2,{\xi }_3$
	${{}_{10}{}^{{10}^3}\overline{F_{NU}}}_{{\xi }_1,{\xi }_2,{\xi }_3}^{5,10,50,100}$	5, 10, 50, 100	10
	${{}_{25}{}^{{10}^3}\overline{F_{NU}}}_{{\xi }_1,{\xi }_2,{\xi }_3}^{2,4,20,40}$	2, 4, 20, 40	25

${\xi }_1\sim \left\{\begin{array}{cc}{\mathcal{U}}_{[89,100]},\hfill & 98\%\mathrm{cases}\hfill \\ 1,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$. ${\xi }_2\sim \left\{\begin{array}{cc}{\mathcal{U}}_{[899,{10}^3]},\hfill & 98\%\mathrm{cases}\hfill \\ {\mathcal{U}}_{[1,19]},\hfill & \mathrm{otherwise}\hfill \end{array}\right.$. ${\xi }_3\sim \left\{\begin{array}{cc}{\mathcal{U}}_{[8999,{10}^4]},\hfill & 98\%\mathrm{cases}\hfill \\ {\mathcal{U}}_{[1,199]},\hfill & \mathrm{otherwise}\hfill \end{array}\right.$.

Table 10. F instances with 78 different generation rules.

$\mathit{Class}$	$\mathit{Notation}$	$\frac{\mathit{n}}{\mathit{m}}$	m	D
$F_U$	${{}_5{}^{{10}^3}\overline{F_U}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[1,{10}^3]},{\mathcal{U}}_{[1,{10}^4]}}^{2,10,20,100,200}$	2, 10, 20, 100, 200	5	${\mathcal{U}}_{[1,100]}$, ${\mathcal{U}}_{[1,{10}^3]}$, ${\mathcal{U}}_{[1,{10}^4]}$
	${{}_{10}{}^{{10}^3}\overline{F_U}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[1,{10}^3]},{\mathcal{U}}_{[1,{10}^4]}}^{5,10,50,100}$	5, 10, 50, 100	10
	${{}_{25}{}^{{10}^3}\overline{F_U}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[1,{10}^3]},{\mathcal{U}}_{[1,{10}^4]}}^{2,4,20,40}$	2, 4, 20, 40	25
$F_{NU}$	${{}_5{}^{{10}^3}\overline{F_{NU}}}_{{\xi }_1,{\xi }_2,{\xi }_3}^{2,10,20,100,200}$	2, 10, 20, 100, 200	5	${\xi }_1,{\xi }_2,{\xi }_3$
	${{}_{10}{}^{{10}^3}\overline{F_{NU}}}_{{\xi }_1,{\xi }_2,{\xi }_3}^{5,10,50,100}$	5, 10, 50, 100	10
	${{}_{25}{}^{{10}^3}\overline{F_{NU}}}_{{\xi }_1,{\xi }_2,{\xi }_3}^{2,4,20,40}$	2, 4, 20, 40	25

${\xi }_1\sim \left\{\begin{array}{cc}{\mathcal{U}}_{[89,100]},\hfill & 98\%\mathrm{cases}\hfill \\ 1,\hfill & \mathrm{otherwise}\hfill \end{array}\right.$. ${\xi }_2\sim \left\{\begin{array}{cc}{\mathcal{U}}_{[899,{10}^3]},\hfill & 98\%\mathrm{cases}\hfill \\ {\mathcal{U}}_{[1,19]},\hfill & \mathrm{otherwise}\hfill \end{array}\right.$. ${\xi }_3\sim \left\{\begin{array}{cc}{\mathcal{U}}_{[8999,{10}^4]},\hfill & 98\%\mathrm{cases}\hfill \\ {\mathcal{U}}_{[1,199]},\hfill & \mathrm{otherwise}\hfill \end{array}\right.$.

F instances are used for testing all types of algorithms for $P\left|\right|C_{max}$ problem. For HE solvers, this group of instances was used in [8,14,57,63,114,119,122], where the authors in [14,95,122,135,139,145,146,161,162,163,164] utilized them for IHs; CHs have been tested on these instances by [139,141,142,143,146], while [114,161] explored F instances to evaluate the performance of MHs. Even though links to the instances provided in some of the papers are not active anymore, all methods have been tested on exactly the same instances that we are providing in our repository. For each generation rule, there are ten generated instances.

With respect to standardization issues, F instances satisfy the majority of criteria and are accompanied with optimal solutions that can be retrieved from both the above mentioned papers and our repository.

5.3. C Instances

In numerous extracted studies, we found a few sets of instances that are generated with similar rules and thus, having similar characteristics. We refer to all these sets as C instances. The first set of C instances, based on the instance set for BPP problem [125], was proposed in 1995 for comparing exact solvers [57]. Since then, they have been used sporadically, while lately, these instances become very popular and are constantly upgraded. The C instances were divided into five distinctive classes with respect to the distributions used to generate the length of jobs. In these classes, D values follow distributions: ${\mathcal{U}}_{[1,100]}$, ${\mathcal{U}}_{[20,100]}$, ${\mathcal{U}}_{[50,100]}$, ${\mathcal{N}}_{100,20}$, and ${\mathcal{N}}_{100,50}$, respectively. These instances are initially generated with a relatively big ${\displaystyle \frac{n}{m}}$ ratio, where m takes values between 2 and 15 and values for n belong to the $[10,{10}^4]$ interval. In [95], there are similarly generated instances having a smaller ${\displaystyle \frac{n}{m}}$ factor with an aim to produce a harder set of instances. More precisely, the authors used ${\displaystyle \frac{n}{m}}=2.5$ and $D\in {\mathcal{U}}_{[{\displaystyle \frac{n}{5}},{\displaystyle \frac{n}{2}}]}$, for each $m\in \{8,12,16,20,24,28,32,36,40,60,80\}$. This was the first time that parameters of D did not take constant values, instead they were dependent on the input data defining each particular instance. Inspired by [95], authors in [8] added two new distribution for generating lengths of jobs (${\mathcal{U}}_{[n,4n]}$ and ${\mathcal{N}}_{4n,n}$), with additional values for ${\displaystyle \frac{n}{m}}\in \{2,2.25,2.75\}$. Later, ${\displaystyle \frac{n}{m}}=3$ is added in [123] where values of m have been used from similar range as in [95]. The classes of C instances used in [123], with included $(m,n)$ pairs $\{(8,18),(6,18)\}$, and ${\displaystyle \frac{n}{m}}$ values $\{4,4.5,5,5.5,6\}$, are summarized in Table 11 and considered as the final version. For all instances, ten times generated per generation rule, from this final version are provided in our repository. Different versions of C instances were used for testing HE solvers [8,12,14,57,63,95,123,124].

Table 11. C instances with 728 different generation rules.

$\mathit{Class}$	$\mathit{Notation}$	$\frac{\mathit{n}}{\mathit{m}}$	m	D
$C_p$	${{}_{100}{}^{200}\overline{C}}_D^2$	2	10, 20, 30, 40, 50, 60, 70, 80, 90, 100	${\mathcal{U}}_{[1,100]}$, ${\mathcal{U}}_{[20,100]}$, ${\mathcal{U}}_{[50,100]}$, ${\mathcal{N}}_{100,20}$, ${\mathcal{N}}_{100,50}$, ${\mathcal{U}}_{[n,4n]}$, ${\mathcal{N}}_{4n,n}$
	${{}_{88}{}^{198}\overline{C}}_D^{2.25}$	2.25	8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88
	${{}_{80}{}^{200}\overline{C}}_D^{2.5}$	2.5	8, 16, 24, 32, 40, 48, 56, 64, 72, 80
	${{}_{80}{}^{220}\overline{C}}_D^{2.75}$	2.75	8, 16, 24, 32, 40, 48, 56, 64, 72, 80
	${{}_{66}{}^{198}\overline{C}}_D^3$	3	6, 12, 18, 24, 30, 36, 42, 48, 54, 60, 66
	${{}_{100}{}^{400}\overline{C}}_D^4$	4	10, 20, 30, 40, 50, 60, 70, 80, 90, 100
	${{}_{88}{}^{396}\overline{C}}_D^{4.5}$	4.5	8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88
	${{}_{80}{}^{400}\overline{C}}_D^5$	5	8, 16, 24, 32, 40, 48, 56, 64, 72, 80
	${{}_{80}{}^{440}\overline{C}}_D^{5.5}$	5.5	8, 16, 24, 32, 40, 48, 56, 64, 72, 80
	${{}_{66}{}^{396}\overline{C}}_D^6$	6	6, 12, 18, 24, 30, 36, 42, 48, 54, 60, 66

Table 11. C instances with 728 different generation rules.

$\mathit{Class}$	$\mathit{Notation}$	$\frac{\mathit{n}}{\mathit{m}}$	m	D
$C_p$	${{}_{100}{}^{200}\overline{C}}_D^2$	2	10, 20, 30, 40, 50, 60, 70, 80, 90, 100	${\mathcal{U}}_{[1,100]}$, ${\mathcal{U}}_{[20,100]}$, ${\mathcal{U}}_{[50,100]}$, ${\mathcal{N}}_{100,20}$, ${\mathcal{N}}_{100,50}$, ${\mathcal{U}}_{[n,4n]}$, ${\mathcal{N}}_{4n,n}$
	${{}_{88}{}^{198}\overline{C}}_D^{2.25}$	2.25	8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88
	${{}_{80}{}^{200}\overline{C}}_D^{2.5}$	2.5	8, 16, 24, 32, 40, 48, 56, 64, 72, 80
	${{}_{80}{}^{220}\overline{C}}_D^{2.75}$	2.75	8, 16, 24, 32, 40, 48, 56, 64, 72, 80
	${{}_{66}{}^{198}\overline{C}}_D^3$	3	6, 12, 18, 24, 30, 36, 42, 48, 54, 60, 66
	${{}_{100}{}^{400}\overline{C}}_D^4$	4	10, 20, 30, 40, 50, 60, 70, 80, 90, 100
	${{}_{88}{}^{396}\overline{C}}_D^{4.5}$	4.5	8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88
	${{}_{80}{}^{400}\overline{C}}_D^5$	5	8, 16, 24, 32, 40, 48, 56, 64, 72, 80
	${{}_{80}{}^{440}\overline{C}}_D^{5.5}$	5.5	8, 16, 24, 32, 40, 48, 56, 64, 72, 80
	${{}_{66}{}^{396}\overline{C}}_D^6$	6	6, 12, 18, 24, 30, 36, 42, 48, 54, 60, 66

To simplify and shorten the notation of C instances for $C_{{\mathcal{U}}_{[1,100]}}$ we will use the notation $C_1$, for $C_{{\mathcal{U}}_{[50,100]}}$ we will use $C_2$, and so on.

With respect to standardization, C instances satisfy the majority of criteria. Optimal solutions and the state-of-the-art solver results can be retrieved from our repository.

5.4. B Instances

B instances are the well-recognized group of instances used for testing MWNP solvers. Originally proposed in 1998 [9], the B instances were defined over a configuration of three machines ($m=3$), up to 100 jobs ($n\le 100$), and with job processing times (precision) $p_i\in [1,{10}^5]$. They are characterized by a small machine count m, and large, uniformly distributed processing times. Over time, these instances were extended to include variations in the number of machines, job counts, and processing time bounds. Therefore, B instances form the most extensive dataset with m extending up to 12, n taking values up to 60, and processing times up to $2^{48}-1$ [9,40,42,115,116,117,118,119,120,121]. To ensure the inclusion of all notable B-like instances from the literature, our proposed standardized dataset includes all distributions ${\mathcal{U}}_{[1,2^{\mathit{class}}-1]}$ for $\mathit{class}\in \{1,\dots ,48\}$, and all coefficients ${\bigcup }_{n=m+2}^{60}{\displaystyle \frac{n}{m}}$ for $m\in \{2,\dots ,12\}$, following the formulation in [205].

Considering the generation of 100 instances per rule, as adopted in prior studies, this results in a comprehensive dataset comprising approximately 2,745,600 instances. This large volume presents significant challenges. The first challenge is the computational demand—the total time required for testing this scale of instances is significant, especially for exact solvers. The second challenge is lightness—many instances may be relatively easy for most solvers, diluting the focus on challenging cases. The third challenge is the growth in dataset size—the number of instances escalates rapidly as the maximal values for n, m, or $\mathit{class}$ increase. The challenges are likely to remain relevant despite constant improvements in the solver performance and computational power.

Our objective is to reduce this set to include only challenging instances, thereby addressing all three problems. Previous research in OR community has indicated that instances tend to become more difficult when ${\displaystyle \frac{n}{m}}\in [2,3]$ [95]. In the context of the MWNP formulation, AI researchers have similarly noted the importance of selecting hard instances. For instance, in [9], the authors experimentally demonstrated that for $m\in \{2,3\}$ and job duration bounded by fixed precision ($\overline{p}$), increasing n initially raises the complexity until a peak, after which complexity begins to decrease. This trend aligns with the phase transition phenomenon in $\mathcal{NP}$-hard problems. It highlights a “critical state” where approximately half of the instances are solvable to perfection and half are not, marking a peak in the computational effort. An instance is solvable to perfection (is perfect) if it has a solution in which difference between each two processor loads is maximally one. In [118], the authors extended this methodology. For fixed m and $\overline{p}$, as n grows, the number of possible schedules ($m^n$) expands exponentially, whereas the number of possible distinct machine loads grows linearly ($\mathcal{O}(\overline{p}n)$) leading to a critical threshold where computational resources peak. Determining whether an instance is perfect requires an extensive exploration within this critical region.

While many open questions remain, this methodology provides a practical framework for identifying generation rules for “harder” instances. Following this methodology for $m\in [2,7]$ and $D\in {\mathcal{U}}_{[1,{10}^{\mathit{class}}-1]}$, where $\mathit{class}\in [1,12]$, the authors in [118] generated 100 instances for each generation rule, increasing n until at least 50 instances were identified as perfect. Due to computational constraints, results for larger values of $class$ and m are limited.

Using a third degree polynomial regression [206] trained on data provided in [118], the authors derived the following predictive model for values of n depending on $\mathit{class}$ and m stored in matrix $T_{class,m}$:

$$\begin{array}{cc}\hfill T_{class,m}=& -0.002clas{s}^3+0.014clas{s}^{2}m-0.016clas{s}^2-0.032class{m}^2\hfill \\ & +0.812classm+2.032class+0.036m^3-0.459m^2+3.388m-4.207\hfill \end{array}$$

(5)

where $R^2>0.999$ predicts known values. This model provides a basis for predicting unknown values, supporting our generation rules Table 12.

Table 12. B instances with 270 different generation rules.

$\mathit{Class}$	$\mathit{Notation}$	$\frac{\mathit{n}}{\mathit{m}}$	m	D
$\{1,\dots ,18\}$	${}_m{}^{T_{class,m}}{\overline{B}}_{{10}^{class}}$	${\displaystyle \frac{T_{class,m}}{m}}$	$\{2,\dots ,16\}$	${\mathcal{U}}_{[1,{10}^{class}-1]}$

Table 12. B instances with 270 different generation rules.

$\mathit{Class}$	$\mathit{Notation}$	$\frac{\mathit{n}}{\mathit{m}}$	m	D
$\{1,\dots ,18\}$	${}_m{}^{T_{class,m}}{\overline{B}}_{{10}^{class}}$	${\displaystyle \frac{T_{class,m}}{m}}$	$\{2,\dots ,16\}$	${\mathcal{U}}_{[1,{10}^{class}-1]}$

Using it we have extended their results up to $class=18$ and $m=16$. The results are presented in

Table 13. Extended experimentally calculated matrix $T_{class,m}$ up to $m=16$ and $class=18$. Original values are bolded.

$\mathit{Class}\setminus \mathit{m}$	2	3	4	5	6	7	8	9	10	11	12	13	14	15	16
1	4	7	9	11	13	16	18	22	27	33	40	48	59	72	86
2	9	11	14	17	19	22	25	29	34	40	47	56	67	79	94
3	12	16	18	22	25	28	32	36	42	48	55	64	75	87	102
4	15	20	25	28	31	35	39	44	49	56	63	72	83	95	110
5	18	24	29	33	38	42	46	52	57	64	72	81	92	104	118
6	22	28	34	39	44	49	54	59	65	72	80	90	100	113	127
7	26	33	39	45	50	56	61	67	74	81	89	99	110	122	137
8	29	37	44	51	57	63	69	75	82	90	99	108	119	132	146
9	33	41	49	57	63	70	77	84	91	99	108	118	129	142	156
10	36	45	55	62	70	77	85	92	100	108	118	128	139	152	167
11	39	50	59	68	77	85	93	101	109	118	127	138	150	163	177
12	42	54	64	74	83	92	101	109	118	128	138	148	160	174	188
13	45	58	69	80	90	100	109	118	128	137	148	159	171	185	200
14	48	62	74	86	97	107	117	127	137	147	158	170	182	196	211
15	51	66	79	92	104	115	125	136	147	157	169	181	194	208	223
16	54	70	84	98	110	122	134	145	156	168	180	192	205	220	236
17	56	73	89	103	117	130	142	154	166	178	190	203	217	232	248
18	59	77	94	109	124	137	150	163	176	189	202	215	229	244	261

, where the bold values were originally reported in [118], and the others are predicted by Equation (5).

Neither of the mentioned papers which used some B-like instance set, provided instances or discussed the characteristics of the random seed values used. We propose generating 100 instances per generation rule, starting with random seed 1 as the standard. In our repository, we have already placed the generated B instances, organized into folders by class, as well as we provided a generator for their creation. If the researchers prefer generating instances rather than using the pre-generated ones, but only need a subset, we strongly recommend the following procedure: Generate the entire set first and then select the preferred subset. This way errors related to random seed handling should be avoided.

The described method of generating instances is both effective and well-suited for practical applications is explored in more detail in [78,92,93]. Recent theoretical advances have additionally analyzed the hardness of instances in the MWNP problem formulation [91], specifically for the $m=2$ case [94]. Practical guidance on generating robust benchmarking tests is provided in [15].

5.5. Other Instances

In this section, we provide some additional sets of instances that do not satisfy our criteria for standardization. However, they appear in $P\left|\right|C_{max}$ literature and should be described for the sake of completeness.

The first group of instances for $P\left|\right|C_{max}$ problem, containing only five examples, appeared in [128]. It was used to evaluate the performance of the LPT algorithm and generated to contain hard instances for that particular algorithm.

$G78$ instance, based on the BPP and proposed in 1978 [71], is characterized with $m=10$ and $n=100$. The job processing times fall within the interval $[{10}^{10},{10}^{11}]$, which made these instance particularly challenging at the time they were introduced. For the $P\left|\right|C_{max}$ problem it was used in [10,178,180]. The authors of [178] used this instance to evaluate the TS algorithm and managed to improve the result obtained by SA in [10]. Additionally, in [180], this instance is used to generate a similar set of instances with $n=2·{10}^3$ and $m=50$ for testing his variant of TS.

Perfect packing or $PP$ instances were proposed in 1995 [57] as hard instances for the purpose of testing a new HE. In this set, for every instance, the optimal schedule has equal competition time on each machine. The set covers instances with $n\le {10}^4$ and $m\in \{3,5,10,15\}$. These instances were also used in [14].

$MK$ instances were proposed 2004 [113] for testing HE algorithm. They are a union of a set of instances ${{}_5{}^{15}\overline{MK}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[10,100]},{\mathcal{U}}_{[50,100]}}$ and set of instances ${{}_{100}{}^{{10}^3}\overline{MK}}_{{\mathcal{U}}_{[1,100]}}$. These instances were also used in [17,207].

BINPACK or $BP$ instances were adopted to $P\left|\right|C_{max}$ in 2004 [163]. They are characterized by the uniform distribution of job duration from the range $(20,100)$. These instances were also used in [137,140,141,142,143,177,202].

TRIPLET or $TR$ were originally designed for the BPP problem [208] These instances for $P\left|\right|C_{max}$ have been used for the first time in [7]. The $TR$ instances are generated in such a way that the optimal solution has exactly three jobs per machine, while the job processing times belong to the $(25,50)$ interval. They were later used in [140,163,164].

Iogra ($IO$), It ($IT$), and Rand ($RN$) instances, adapted from Multiprocessor Scheduling Problem with Communication Delays [209] and from [210] for $P\left|\right|C_{max}$. Although these instances were considered easy, they are used in [166,177,188,202,211].

All the sets of instances in this section are either reused from other problems, small, or too easy in comparison with other described instances. In addition, we do not consider them as a special group because they are less explained, less used, or simply do not provide any novelty compared with B, C, E, and F instances.

5.6. Summary of Instance Groups

The key results of our SLR are the identification of the main differences between various sets of problem instances and the definition of criteria for producing standardized groups of instances for testing $P\left|\right|C_{max}$ optimization algorithms. Figure 7 illustrates the usage of various types of instances over the years. As none of the “Other” group instances satisfies all standardization criteria and they are used ad hoc only, we do not present them here. The figure shows that E instances were the first in use, other types of instances were used from the 1990s. However, all types are still in use.

6. RQ3: Comparisons

Comparing two or more algorithms, also known as optimization benchmarking, is conducted in the literature with the goal of determining which algorithm has performed best on a given representative set of problem instances [212,213]. In some cases, approaches that rely only on formal analysis of the algorithms have been applied assuming that analysis will explain the quality of algorithms on all problem instances. However, for these approaches, one must presume only problem instances with a suitable structure which might not be representative of all problem instances. This is one of the shortcomings of applying formal analysis exclusively [214]. Additionally, the worst-case or average-case analysis of the algorithm’s complexity does not provide complete information about how a given heuristic will perform on any given set of instances. Finally, the stochastic analysis of the algorithm’s performance might require the knowledge of underlying instance distribution. This distribution, in practice might not be easily identifiable. These are the main motivations why empirical (experimental) work is necessary in order to ensure adequate comparison [214,215].

In the remainder of this section, the general methodology for comparing optimization algorithms is explained. It is then followed up with the specific details of comparing algorithms applied on $P\left|\right|C_{max}$ problem. For each step of the methodology, its importance and relevance are explained, and studies in which the methodology is applied are cited. Finally, the comparisons made in the literature are presented.

6.1. General Methodology

The choice of appropriate performance metrics, along with adequately selecting parameter values and working with proper instances, form the bedrock for a robust performance analysis of the developed algorithm. In more detail, the experimental procedure for optimization benchmarking consists of the following basic steps [212,213,216]:

• Select performance metrics;
• Design the experiment;
• Select test instances;
• Perform an experiment;
• Analyze and present results.

Performance metrics are categorized into two essential groups: those impacting effectiveness and those influencing efficiency [217]. Effectiveness implies collecting data on solutions’ quality within a limited amount of time or other computation resources (e.g., maximal number of functions evaluations). Here, the algorithm’s performance is described via statistics, such as a measure of central tendencies and variability. Known measures of effectiveness are the arithmetic mean and the corresponding standard deviation or median and range. On the other hand, the efficiency of an algorithm is an estimate of runtime (or computational steps) to reach a solution of predetermined quality.

It is generally assumed that the proposed algorithm should be compared against the state-of-the-art methods for the considered problem. However, note that generating a state-of-the-art method is an iterative process, and identifying the current state-of-the-art methods could be performed only after benchmarking. Usually, the development of new algorithms is a product of fusion between old and new ideas that positively affect runtime and solution’s quality. As it is hard to determine what the state-of-the-art method is, the comparison is usually conducted against some other well-known algorithms. In the absence of such methods, comparisons are conducted against the more general methods (greedy approach, integer programming). Sometimes, the comparisons are made even against a simple random restart procedure, as that may offer some reference points [216].

As mentioned above, optimization benchmarking is conducted in the literature with the goal to determine which algorithm has performed best on a given representative set of problem instances. Representative sets assume the exhaustive coverage of different cases for any important characteristic of the instances.

Evaluating algorithm performance can be performed through graphical analysis or statistical testing. During the comparison process, the authors should always aim to achieve fairness. As there are various aspects of the study that may influence the fair comparison, we list a few recommendations provided in [213]:

• Avoid comparing tuned versus untuned algorithms. Namely, the best scenario is if algorithms are compared at their peak performance, i.e., after parameter values are determined via some tuning procedure.
• Whenever possible, conduct a comparison on the same machine, with respect to both hardware and software characteristics. If this is not possible, use a machine with similar characteristics and perform the appropriate scaling of the results.
• The analysis must adhere to the 3Rs of Data Science: repeatability, reproducibility, and replicability all pointing towards getting the same results. Repeatability involves the same researcher and environment. Reproducibility engages another researcher on the same computer system. Replicability means that the estimated performance could be achievable by anyone in any computing environment.

6.2. $P\left|\right|C_{max}$ Problem Specific Methodology

The most important part of any computational evaluation of an algorithm’s performance is to address the correct measure of performance. A very important moment for practical comparison of any proposed algorithm is the quality of its implementation. Additional information related to MH algorithms can be found in [218]. The usual performance metrics for $P\left|\right|C_{max}$ problem-specific comparisons are the quality of the solution and the running time, where the time metric could refer to the following:

• time to complete the execution (until the stopping criterion is fulfilled) in the majority of the reviewed literature and explained in [123];
• time required to find the best solution (time-to-best) in several reviewed papers and explained in [12].

Sometimes the time of execution of some parts of the optimization algorithm can be relevant, for example, the time can be measured for lower bound values [95], initial solution construction [122], etc.

Measurement of the solution quality may be more difficult, especially in cases when the optimal solution of the given instance is not known. The quality of solution metrics used in the literature for $P\left|\right|C_{max}$ are as follows:

• $makespan$—the objective function value [6,110,111,148,188]. This is the simplest solution quality measure, used mostly in the cases when the optimum or some other relevant data are not known. The makespan values obtained by analyzed algorithms are compared and the lowest among them is declared as the best. The main disadvantage of this metric is that it does not give any information about the distance of compared solutions from the best possible (optimum) solution.
• $gap$ (error)—relative distance between the compared solutions [5,8,14,20,21,26,95,113,114,133,134,135,136,137,138,139,141,142,145,148,153,157,159,163,164,173,177,182,183,184,189,190,193,194,195,196,197,198,199,200,204,219,220]. The $gap$ is calculated as ${\displaystyle \frac{makespan-bound}{bound}}$, where bound is usually some estimation for the makespan of the best possible solution $(LB)$; however, it can also be the makespan of the solution provided by some other algorithm $(UB)$. Other than having the same drawback as the previous metric, the main disadvantage of using the $gap$ is that it can be calculated in different ways. When calculated differently, there is no consistent way of establishing a fair comparison.
• $opt$—achieved optimality flag [5,12,20,95,114,139,141,142,144,145,148,161,163,164,166,183,194,220]. When the optimal makespan value is known, this flag indicates whether the considered algorithm(s) managed to provide the optimal solutions or not.
• $OPT$—proved optimality [12,57,63,113,118,123,124,137,139,144,164,207]. This metric differs from the previous one in the fact that the evaluated algorithm is able to prove the optimality of the provided solution.

There are some additional algorithm performance measures used in the relevant literature. For example, some exact solvers are characterized by the number of solved LP relaxations or the number of branches (nodes) required to find optimal solutions and prove their optimality [8,14,17,63,110,111,113]. Iterative algorithms perform better when they require small number of iterations to provide high-quality solutions [6,12,166,178,180,202]. Stochastic algorithms’ performance should be evaluated also by means of statistical tools illustrating their stability with respect to the value of seed used for initialization of random number generator. This requires a sufficient number of executions (runs, restarts) to have the corresponding values of mean, standard deviation, and other statistical data meaningful for the derived conclusions [213,217].

Other than the statistical data related to the repeated execution of (stochastic) the algorithm on a single instance, the majority of papers usual present summary (average) performance measures for a set of instances from the same class e.g., [123,207]. These kinds of measurements can testify to the robustness of the analyzed algorithm with respect to the characteristics of different instances.

For parallel algorithms, it is important to take into account the number of processing elements involved in the algorithm execution [202,221,222]. To ensure fairness in comparison, the stopping criterion is usually reduced appropriately. However, sometimes, the goal is to examine if the enriched hardware resources can contribute toward the improvement in the final solution quality. In addition, it could be very informative to examine the contribution of each part of the algorithm (executed in parallel) and to modify them if necessary.

In order to conduct a comparison between two or more optimization algorithms for $P\left|\right|C_{max}$ the next step involves the selection of benchmark instances in such a way as to allow some sort of generalization. As the answer to Research Question 2 (Standardization of the problem instances), discussed in Section 5, we recommend using any of the four instance types B, C, E, and F, preferably all four of them. Finally, in the remainder of the section, we provide comparisons of specific algorithms for $P\left|\right|C_{max}$ problem.

6.3. Comparisons of Specific Algorithms $P\left|\right|C_{max}$ Problem

In this section, we provide information about the existing comparisons between various algorithms. Usually, algorithms from the same group are compared, although sometimes we also found the comparison between algorithms belonging to different groups. The remainder of this section follows the categorization and grouping from Section 4, with these non-typical cases pointed out when appropriate. For each group of methods, the types of comparisons are presented first. Then, we try to establish a relationship graph among the algorithms based on the available comparisons. The nodes in that graph represent the algorithms, while the direct edges source represents the “better” algorithm in that comparison.

The experimental results on the same group of standardized instances are used to evaluate the performance of one algorithm relative to others. The performance measures used to build a graph are $gap$ and the number of instances for which the optimum is reached/optimality proven. Having in mind the drawbacks of $gap$ calculations, we have also included the elapsed time and the number of iterations, if needed and provided in the literature. When elapsed time is taken into account, it was normalized based on known characteristics of the resources. In extreme cases, when it is not possible to conclude which algorithm among those examined is better, we did not connect them by the directed edge. Transitive relationships were not shown in the graphs. Therefore, the position in the graph (the higher the better) is only an estimate of the quality of the algorithm.

6.3.1. Comparisons of Exact Algorithms

In the reviewed papers, Exponential exact algorithms have pure theoretical importance and usually are not implemented at all. In addition, due to their complexity, it is not rewarding to compare their performance on (benchmark) problem instances. For their theoretical comparison, standard metrics (WTC and WSC) are utilized. However, there are some exceptions.

The ${DP}_{87}$ algorithm was compared with the HE algorithms BIN and DM in [57]. It was significantly slower than the other two algorithms for instances ${{}_3{}^{50}\tilde{C}}_{1-5}$. It is also compared with the LPT and ${\mathrm{RAS}}_2$ CHs, as well as ${\mathrm{RAS}}_1$ IH on instances ${{}_5{}^{50}I}_{{\mathcal{U}}_{[10,100]}}$ in [136]. As expected, the other algorithms demonstrated superiority with respect to solving times. The second practical problem with this algorithm mentioned in [136], relates to its memory requirements for instances with $m=5$ and $n\ge 30$. More precisely, the machine used for experimental evaluation did not have enough memory to complete the experiment.

The practical importance of Fixed parameter tractable algorithms is also very small, and therefore, they are compared mainly using the theoretical metrics WTC and WSC. However, theoretical metrics cannot estimate the performance of any particular algorithm on any tested instance. Despite bad theoretical indicators, an algorithm may provide an optimal solution very quickly if the instance turns out to be “easy” for that algorithm. On the other hand, having experimental results for different sets of instances enables the application of machine learning techniques to predict the difficulty of any new instance that has not been tested yet as it is conducted in [1].

Hybrid exact algorithms are, contrary to the first two groups, focused on solving some particular instances. Performance evaluation of these algorithms is conducted by numerical experiments involving selected sets of benchmark instances. Therefore, general performance could not be judged outside the tested set of instances. In addition, the published results cannot be repeated or reproduced easily, because the algorithms’ implementations, as well as the utilized instances, are not publicly available. Consequently, there are not many papers that report on comparing these algorithms, especially those that involve more than two HE algorithms.

BIN and DM algorithms, presented in paper [57], have been compared to each other and with ${\mathrm{DP}}_{87}$ EE. The first comparison is conducted on a subset of an early version of C instances, ${{}_3{}^{50}\tilde{C}}_{1-5}$. As the best-performing algorithm, DM was identified, while BIN sometimes was an order of magnitude slower. ${\mathrm{DP}}_{87}$ performed the worst and it was practically unusable for instances with $m\ge 3$. In the second experiment, the authors used ${{}_{15}{}^{{10}^4}\tilde{C}}_{1-5}$ instances. Although not being able to solve all instances, DM clearly outperformed BIN, except for some very small instances. In the third experiment, the authors used $PP$ instances and made the same conclusion.

CGA and CKK [9] algorithms were compared with each other on instances ${{}_2{}^{200}\tilde{B}}_{{10}^{10}}$ and ${{}_3{}^{100}\tilde{B}}_{{10}^5}$ where always $m=3$. In both cases, CKK outperforms CGA. Especially on the perfect instances, the difference in performance is significant.

${CP}_{04}$ algorithm [113] is compared with simple ILP solver. For the experimental evaluation, memory was limited to 50 Mb, while the running maximum time was set to $2·{10}^3$ s for both algorithms. Experiments were performed on $MK$ instances. The first experiment involved ${{}_5{}^{15}\overline{MK}}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[10,100]},{\mathcal{U}}_{[50,100]}}$ instances. The obtained results show that both algorithms were always able to converge to the optimal solution, with ${\mathrm{CP}}_{04}$ requiring less computational time for the largest instances. On the other hand, the ILP solver was faster for the smallest instances. The second experiment was performed on the ${{}_{100}{}^{{10}^3}\overline{MK}}_{{\mathcal{U}}_{[1,100]}}$ instances. In this case, the ILP solver was not able to find optimal solutions in the majority of cases, while ${\mathrm{CP}}_{04}$ provided optimal solutions for almost all instances with drastically smaller computation effort. The authors of [207] critically reviewed ${\mathrm{CP}}_{04}$ and showed that the algorithm can be significantly outperformed by DM on the same sets of instances.

The HJ algorithm [14] is compared with DM. As the starting bounds, MSK CH and $L_{FS}$ were used. The $PP$ instances were used in the first experiment. Due to the very good starting bounds, HJ solved to optimality in all of the instances in a very short time, and branching was required only for three instances. On the other hand, DM failed to solve 13 instances. The second experiment involved ${{}_{15}{}^{{10}^4}\tilde{C}}_{1-5}$ instances from [57]. Both algorithms exhibited similar behavior; however, five instances remained unsolved by HJ, which is again better comparing to eight instances unsolved by DM. In the third experiment, HJ showed very good performance on F instances, although not being directly compared with any other algorithm. The authors concluded that it solved to optimality 13 instances being open problems for some time. The algorithm was not able to solve 3 not-uniform and 18 uniform instances. In the fourth experiment, HJ algorithm was tested on ${{}_{80}{}^{200}\tilde{C}}_{{\mathcal{U}}_{[{\displaystyle \frac{n}{5}},{\displaystyle \frac{n}{2}}]}}^{2.5}$ instances. The authors were not satisfied with $35\%$ of unsolved instances in this case and concluded that an alternative approach should be developed.

DIMM, proposed in [114], is compared with DM on F instances. The main ingredients of DIMM include ${\mathrm{DIMM}}_{SS}$ MH and MT1 [151] Knapsack solver. The obtained results show that DIMM solved all instances to optimality within small computation times, clearly outperforming DM.

${\mathit{SNP}}_{ie}$ and RNP [115] algorithms were compared with each other and with CKK and CGA algorithms. In the first experiment, the instances ${{}_3{}^{40}\tilde{B}}_{{10}^7}$, with $m=3$ and $n\ge 25$, were used. According to the provided results, ${\mathrm{SNP}}_{ie}$ significantly outperforms CKK with respect to running time. Authors mentioned (without providing profs) that for $m=4$ and $m=5$ CGA is faster than CKK, and in the second experiment, they compared ${\mathrm{SNP}}_{ie}$ and RNP only with CGA on instances ${{}_4{}^{33}\tilde{B}}_{{10}^5}$, with $m=4$ and $n\ge 20$. CGA is outperformed by several orders of magnitude by the other two algorithms. RNP showed better performance than ${\mathrm{SNP}}_{ie}$. The third experiment was performed for the same algorithms on instances ${{}_5{}^{30}\tilde{B}}_{{10}^4}$, with $m=5$ and $n\ge 20$. The obtained results are similar to those in the previous experiment. Finally, an additional experiment is carried out involving instances ${{}_5{}^{40}\tilde{B}}_{{10}^8}$, with $m>2$ and $n\ge 20$. Again RNP was the best algorithm, followed by ${\mathrm{SNP}}_{ie}$.

IRNP [116] was compared with RNP on instance sets ${{}_6{}^{52}\tilde{B}}_{2^{31}}$ and ${{}_{10}{}^{40}\tilde{B}}_{2^{31}}$. On all instances, IRNP was a few orders of magnitude faster than RNP.

MOF [117] was compared with IRNP and demonstrated its superiority on ${{}_{10}{}^{40}\tilde{B}}_{2^{31}}$ instances.

BSBCP, ${\mathit{CKK}}_i$, ${\mathit{RNP}}_i$, IRNP, and CGA algorithms are compared in [118]. For $m=2$, the authors showed that CKK dominates CGA. Without providing experimental results, the authors mentioned that BSBCP performed better than IRNP for $m>7$. Finally, on instances ${{}_7{}^{50}\tilde{B}}_{2^{48}}$, with $m>2$ and $n\ge 25$, ${\mathrm{RNP}}_i$ clearly outperformed IRNP.

BSIBC [119] was compared with BSBCP and DIMM. The first experiment was conducted for BSIBS and BSBCP algorithms on ${{}_{20}{}^{45}\tilde{B}}_{{10}^{15}}$ instances. Both algorithms performed well, with BSIBC solving the majority of the instances faster. In the second experiment, the authors compared different variants of the BSIBC algorithm against DIMM on a subset of $F_U$ recognized to contain hard instances in [114]. The conclusion was that BSIBC with limited discrepancy search is able to outperform DIMM by up to three orders of magnitude.

${\mathit{SNP}}_{ess}$ and HI14 [120] were compared with each other and with MOF on ${{}_{10}{}^{50}\tilde{B}}_{2^{48}}$ instances. ${\mathrm{SNP}}_{ess}$ is always faster than MOF, except for instances with small n. As expected, the authors reported smaller computation times for HI14 in comparison to other algorithms for all instances.

CIW [121] was compared with ${\mathrm{SNP}}_{ess}$, MOF, and BSBCP, and clearly outperformed all other algorithms: ${\mathrm{SNP}}_{ess}$ on ${{}_7{}^{60}\tilde{B}}_{2^{48}}$ instances, MOF on ${{}_{10}{}^{60}\tilde{B}}_{2^{48}}$ instances, and BSBCP on ${{}_{12}{}^{60}\tilde{B}}_{2^{48}}$ instances.

WL [8] was compared with DIMM and HJ solvers. First, the comparison was conducted on ${{}_{15}{}^{50}\tilde{C}}_{1-5}$ instances. According to the authors, this subset of the first version of C instances [57] contains difficult test examples. In this experiment, WL was compared with HJ and was faster on all instances except the biggest ones. However, in total, HJ solved more instances within less running time. In the second experiment, HJ outperformed WL on F instances, although both algorithms had plenty of unsolved instances. In the third experiment, WL was compared with DIMM on F instances and it was shown that WL can solve three instances faster than DIMM. However, DIMM was generally a superior approach. The fourth experiment involved comparison of WL with HJ on ${{}_{80}{}^{200}\tilde{C}}_{{\mathcal{U}}_{[{\displaystyle \frac{n}{5}},{\displaystyle \frac{n}{2}}]}}^{2.5}$ instances presented in [14]. It revealed that WL has comparable performances with HJ. The fifth experiment shows clear superiority of the WL algorithm in comparison with its variant ${\mathrm{WL}}^{\prime }$ on an extended set of ${{}_{15}{}^{30}\tilde{C}}_{1-5}$ instances from the first experiment. An additional comparison of WL and ${\mathrm{WL}}^{\prime }$ on a more expanded ${{}_{25}{}^{100}\tilde{C}}_{1-5}^{2,2.5,3,4,5}$ instances was performed in [63]. The authors made the same conclusion: WL outperforms ${\mathrm{WL}}^{\prime }$.

The LCS algorithm [42] in comprehensive practical experiments was compared with CGA, RNP, IRNP, ${\mathrm{SNP}}_{ie}$, BSIBC, BSBCP, ${\mathrm{SNP}}_{ess}$, MOF, and CIW on $\tilde{B}$ instances with $n\ge 20$. The obtained results can be summarized as follows: ${\mathrm{SNP}}_{ess}$ is dominant for $m=3$ and for small values of n when $m\in \{4,5\}$. CIW dominates for large values of n when $m\in [4,12]$. ${\mathrm{SNP}}_{ie}$ performs the best for $m\in [8,10]$ if the values of n are very small. BSIBC and BSBCP outperform others for $m\in \{11,12\}$ and small values of n. For $m\in [6,12]$ and $n\le 35$, MOF often performs the best. From the presented results, it is clear that CIW dominates other algorithms in the majority of cases. LCS performs better on instances with $m>5$ and requires less memory for $m>8$.

KL [122] was compared with DM and DIMM on F instances. As expected, DM was outperformed by DIMM which managed to solve all instances in a much shorter time. KL also solved all instances in significantly less time than DIMM. In addition, it did not require branching: the initial heuristic ${\mathrm{KL}}_h$ was enough.

AF [123] was compared with HJ and WL solvers. Comparison with HJ was performed on instances similar to a subset of C instances with ${\displaystyle \frac{n}{m}}=2.5$ presented in [14]. AF dominated HJ by solving all instances to optimality. Comparison with WL involved the whole set of C instances. In this case, AF outperformed WL, it successfully solved 77 out of 3500 instances.

iAF [124] was compared with the AF solver on C instances and clearly improved AF results by solving all but one instance within a shorter execution time.

DIST [12] was compared with iAF solver on C instances with ${\displaystyle \frac{n}{m}}=2$. Within the subproblem time limit of $0.06s$ and 30 runs, DIST optimally solved all instances in a much shorter execution time. It is important to note that in some cases DIST was not able to guarantee the optimality of the provided solutions.

Based on the described empirical results given in papers included in our SLR, the relationship between all exact algorithms is illustrated in Figure 8. Blue color represents HE algorithms, and light blue represents EE algorithms. The first observation is related to a separation of the graph into two parts (left and right), connected just with the ${\overline{F}}_U$ branch. The right part of the graph is characterized by algorithms oriented to solving instances with small m and large job processing times. They were compared just on sets of $\tilde{B}$ instances. The left part of the graph is focused on algorithms for solving instances with larger m, and smaller job processing times. On the left side, a bigger variety of instances can be seen, primarily F and variants of C. These two types of algorithms had been developed separately, the first for the purpose of solving $P\left|\right|C_{max}$ and the second for solving MWNP. Only one comparison between these two types of solvers is performed, between BSIBC and DIMM algorithms on ${\overline{F}}_U$ instances, with BSIBC being superior. Due to the lack of detailed comparisons in the literature, we did not have enough results to make clear relations between algorithms. Therefore, we made a compromise by keeping several algorithms on the same level of the comparison graph in Figure 8. This is specially related to the right part of the graph.

Final remark about the experimental evaluation of exact algorithms, is the importance to utilize standard benchmark instances and report various metrics for quality of solution under the same conditions. This will ensure fair and comprehensive comparison.

6.3.2. Comparisons of Heuristic Algorithms

The comparison of Constructive heuristics is usually very easy as there are many proposed comparison metrics. Both theoretical and experimental comparisons are relevant. For theoretical comparison, WTC and AR metrics are utilized, while practical comparison involves similar metrics as for HE algorithms.

If $n>m$ is assumed, WTC can be expressed as dependent only on n, which enables sorting of the CH algorithms in the non-decreasing order as follows. ${\mathrm{P}}_1$: $\mathcal{O}(n)$; SLS, LPT, MAAAT, ${\mathrm{RAS}}_2$, AP, ${\mathrm{LPT}}_R$, PSC, ${\mathrm{PSC}}_i$, SPS, and SLACK: $\mathcal{O}(nlogn)$; ${\mathrm{MF}}^{\prime }$, MF, ${\mathrm{MF}}_e$, ${\mathrm{MF}}_i$, and COMBINE: $\mathcal{O}(knlogn)$; MPS: $\mathcal{O}(n^2)$; LDM: $\mathcal{O}(n^{2}logn)$; FGH, DGH and PSMF: $\mathcal{O}(k{n}^2)$; LISTFIT and DJMS: $\mathcal{O}(k{n}^{2}logn)$; MS: $\mathcal{O}(n^3)$; SS: $\mathcal{O}(n^{2}S)$; ${\mathrm{H}}_1$: $\mathcal{O}(n^{2}log({\displaystyle \frac{1}{\epsilon }}+{\displaystyle \frac{1}{{\epsilon }^2}}))$; and finally, KK has $\mathcal{NP}$ WTC. The meaning of parameters k, S, and $\epsilon$ is explained in Table 4.

Comparing CH algorithms with respect to AR is not always straightforward. The AR of LPT equals ${\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3m}}$, which is presented when the algorithm was introduced [128] and again proved in [223]. However, for some of the CH algorithms, AR is not easy to calculate. The AR for MF has been improved from $1.22+2^{-k}$ [129], to $1.2+2^{-k}$ and ${\displaystyle \frac{13}{11}}+2^{-k}$ for $m\ge 13$ [224], and finally to ${\displaystyle \frac{13}{11}}+2^{-k}$ for arbitrary m [225]. AR for LDM was an open problem for a long time, and first result was given for $m=2$ [226], while the most recent analysis of general case was provided in [227], with the conclusion that AR should belong to the interval $\left[{\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3(m-1)}},{\displaystyle \frac{4}{3}}-{\displaystyle \frac{1}{3m}}\right]$. Additional results can be found in [228,229,230,231].

If it is assumed that m and k are approaching infinity, AR can be expressed as constant, and the CH algorithms can be sorted in the non-increasing order of AR values as follows. SLS, KK, ${\mathrm{LPT}}_R$: 2; Specific case for MPS and SPS: $1+{\displaystyle \frac{1}{z}}$ where $z>1$, so upper bound of ${\displaystyle \frac{3}{2}}$ can be used. LPT, LDM, ${\mathrm{RAS}}_2$ and SLACK: ${\displaystyle \frac{4}{3}}$; ${\mathrm{MF}}^{\prime }$: ${\displaystyle \frac{5}{4}}$; MF, COMBINE, LISTFIT, FGH, DGH, DJMS and PSMF: ${\displaystyle \frac{13}{11}}$; ${\mathrm{MF}}_i$: ${\displaystyle \frac{72}{61}}$; Specific values are provided for $m=2$ for some CHs: LPT, LDM: ${\displaystyle \frac{7}{6}}$; SLACK: ${\displaystyle \frac{9}{8}}$ and finally ${\mathrm{MF}}_e$ and COMBINE: ${\displaystyle \frac{10}{9}}$.

There are also some additional asymptotic performance indicators for LPT [232,233,234,235,236,237], and for MPS [238]. The statistical asymptotic performance is analyzed for SLACK [239], for LPT [86,240,241,242,243,244,245,246], for some other LS variants [243,246,247,248,249], and in general [250]. However, the obtained results involve some parameters that are not always available. Therefore, these results are not utilized in this SLR.

Theoretical performance indicators illustrate the expected solution quality and required running time of each particular CH. However, the practical behavior of these algorithms may differ significantly. For experimental evaluation, achieved solution quality (the makespan value) and the running time spent for each problem instance are usually the most relevant performance indicators. The majority of the reviewed papers report on experimental evaluation. However, usually, only a few CHs are compared simultaneously, on a very specific set of problem instances, which makes the conclusions non-adequate in many situations. Some of the conclusions drawn by the authors of reviewed papers and related to the experimental comparison of CH algorithms are presented here.

The variant of MF algorithm [129] where the parameter k takes value 7 (${\mathrm{MF}}_7$) was compared with LPT and ${\mathrm{MF}}_7^{\prime }$. The quality measure was defined as the average gap with respect to the lower bound. The comparison was made on a small instance set ${}_{10}{}^{30}I$. Based on the presented results, the authors concluded that ${\mathrm{MF}}_7$ was clearly the best and ${\mathrm{MF}}_7^{\prime }$ performed better than LPT in the majority of cases.

The ${\mathit{MF}}_e$ algorithm [133] was compared with MF, LPT; and the $\epsilon$-DUAL PTAS algorithm with $\epsilon ={\displaystyle \frac{1}{5}}$. The first version of $E_2$ instances with mean 450, where variance is not explicitly provided, but limits for values are in interval $[100,800]$ (denoted as ${\widetilde{E_2}}_{{\mathcal{N}}_{450}}$) was used. The experimental results showed that ${\mathrm{MF}}_e$ performed generally the best, MF outperformed the other two algorithms, and LPT was better than $\epsilon$-DUAL.

The COMBINE algorithm [134] was compared with MF and LPT algorithms, on the same variant of $E_2$ instances as in [133]. It was shown that COMBINE provides better results on average, which is expected considering that it represents a combination of the other two mentioned algorithms. In addition, it was confirmed that MF outperforms LPT.

The performance of the ${\mathit{RAS}}_2$ algorithm [136] was compared with LPT and MF; ${\mathrm{RAS}}_1$ IH; and ${\mathrm{DP}}_{87}$ EE solver. The first experiment is conducted on instances ${{}_5{}^{50}I}_{{\mathcal{U}}_{[10,100]}}$. In this experiment, it is shown that ${\mathrm{RAS}}_2$ performed better than LPT, slightly worse than ${\mathrm{RAS}}_1$. In addition, its running times are significantly shorter compared to ${\mathrm{DP}}_{87}$, where the exact solver failed on some instances, due to the memory limitations of the used machine. In the second experiment, tests are conducted on instances ${{}_{150}{}^{2·{10}^3}I}_{{\mathcal{U}}_{[10,100]}}$. In comparison with LPT, ${\mathrm{RAS}}_2$ obtained better quality solutions within slightly longer computation times. With respect to ${\mathrm{RAS}}_1$, ${\mathrm{RAS}}_2$ provided marginally worse solutions within significantly shorter computational times. Based on the results of the third experiment on instances ${{}_{50}{}^{250}I}_{{\mathcal{U}}_{[10,100]}}$ and the fourth experiment on instances ${{}_{250}{}^{{10}^4}I}_{{\mathcal{U}}_{[10,100]}}$, it can be seen that ${\mathrm{RAS}}_2$ is similar to MF in terms of the solution quality. However, ${\mathrm{RAS}}_2$ provides a significant reduction in CPU times.

The LISTFIT algorithm [137] shows considerably better average performance compared to LPT, MF, and COMBINE on the E instance set. As expected, COMBINE outperforms MF, and MF outperforms LPT.

Among all variants of FGH and DGH algorithms [130], ${\mathrm{npar-FGH}}_{10}$, ${\mathrm{npar-DGH}}_1$ and ${\mathrm{npar-DGH}}_2$ were evaluated. The listed algorithms have been compared with LPT, MF (${\mathrm{MF}}_7$ and ${\mathrm{MF}}_{16}$), ${\mathrm{RAS}}_2$; and ${\mathrm{RAS}}_1$ IH on their own instance sets, with sizes up to ${{}_{{10}^3}{}^{{10}^4}I}_{{\mathcal{U}}_{[1,{10}^3]}}$. The authors utilized the gap with respect to $L_2$ as a performance measure. As can be seen from the presented experimental results, the listed algorithms dominate for the majority of instances. However, in some cases, MF algorithms or ${\mathrm{RAS}}_1$ performed the best. There was no significant difference in the computational time between the compared heuristics. Upon the proper parameter tuning, no obvious difference between the FGH and DGH algorithms can be found. Although they did not usually produce identical solutions, the computed gap deviations were similar.

${\mathit{AP}}_{10}$, ${\mathit{AP}}_{ni}$, and ${\mathit{AP}}_i$ (AP) algorithms [138] were compared with LPT, FGH, DGH, and MF. Instances set ${{}_{50}{}^{{10}^3}I}_{{\mathcal{U}}_{[1,100]}}$ were used in the first experiment, while the second experiment was conducted on instances ${{}_{50}{}^{{10}^3}I}_{{\mathcal{U}}_{[1,{10}^3]}}$. The authors concluded that the algorithms they proposed require much less time to run than FGH, DGH, and MF. According to the solution quality, AP algorithms showed superiority only in comparison with LPT.

$H_1$ [139] has not been directly compared with other algorithms. Instead, the authors mentioned that it was capable of finding 556 out of 780 optimal solutions on the F instances when using lower bounds $L_3$, $L_{\theta }$, and $L_{HS}$.

The SS algorithm [95] was compared with MS, the better among MF and LPT; and MSS IH with respect to the solution quality. Instance sets ${{}_{80}{}^{200}\tilde{C}}_{{\mathcal{U}}_{[{\displaystyle \frac{n}{5}},{\displaystyle \frac{n}{2}}]}}^{2.5}$ and ${{}_{15}{}^{100}I}_{{\mathcal{U}}_{[50,100]}}$ were utilized. MSS performed the best, followed by SS and MS. In the case when SS was used (instead of ${\mathit{LPT}}_R$) to construct an initial schedule for the MSS IH, even better solutions were achieved; however, at the cost of increased computation time.

The MPS algorithm [140] was compared with LPT. It produced better AR on $BP$, $TR$, and $F_{NU}$ instances than LPT. However, LPT showed slightly better results in terms of execution time. The issue with these experiments is that the authors do not provide insight into makespan values or other directly dependent statistics, making it difficult to compare MPS with other algorithms based on solution quality.

Based on the solution quality, the PSC algorithm [141] was compared with LPT; and the 3-PHASE and 1-SPT IHs. Comparison with LPT and 3-PHASE was conducted on F instances. On the other hand $BP$ instances were utilized for comparison with LPT and 1-SPT. The provided experimental results show that PSC outperforms LPT on both instance sets. With respect to both IHs, PSC exhibits comparable performance.

${\mathit{PSC}}_i$ (the best performing among all presented variants) [142] was compared with LPT; and 3-PHASE and K-SPT IHs under the same conditions as in the previous paper. Regardless of the performed modifications, the conclusion of the experimental evaluation remains unchanged.

A similar experimental evaluation was performed for the SPS algorithm [143], and a similar conclusion was made. SPS performed better than LPT on $BP$ and F instances. It was comparable with 3-PHASE on F instances and with 1-SPT on $BP$ instances.

The DJMS algorithm [144] was compared with LPT, MF, COMBINE, and LISTFIT on E instances. It showed the smallest overall average percentage error compared to LPT, MF, COMBINE, and LISTFIT, although LISTFIT performed slightly better with respect to the percentage of instances solved to optimality.

PSMF algorithm [145] is evaluated against LPT, MF, COMBINE, and LISTFIT on E instances. It can be seen that PSMF performed the best, and it flowed with LISTFIT, COMBINE, MF, and LPT, respectively. In addition, authors used the $F_{NU}$ set to compare PSMF with ME (the best result of 1-SPT, 1-BPT, K-SPT, and K-BPT), PSMF+ and HI IHs; and DM HE. In this experiment, HI always outperforms all other algorithms. PSMF obtains the worst average relative error (the same order of magnitude as ME and DM) and solves the fewest instances to optimality. PSMF+ performs better than PSMF with respect to the average relative error and the number of provided optimal solutions with negligible increase in the running time. Actually, it can be ranked as the second the best algorithm, outperformed only by HI.

The SLACK algorithm [146] was compared with LPT, COMBINE, and LDM on F instances. It produced better overall experimental results than LPT and COMBINE but not better than LDM. COMBINE outperformed LPT. The SLACK+ IH was also compared with LDM, showing generally better results on $F_{NU}$ instances but weaker results on $F_U$ instances, with computational times more than an order of magnitude longer in both cases. The issue with these experiments is that they provide only win/equal/lose results without concrete makespan or gap values, making a more precise comparison with other algorithms impossible.

The studies in [41,219] compare LPT, MF, COMBINE, and LISTFIT on the instance set ${{}_{10}{}^{300}I}_{{\mathcal{U}}_{[1,100]}}$. The results indicate that LISTFIT achieves the best performance, followed by COMBINE and MF, with LPT performing the worst, which is consistent with previous findings.

In [113], the author examined the performance of the ${\mathrm{CP}}_{04}$ HE algorithm exploring various CHs for generating the initial solution: LPT, ${\mathrm{FGH}}_1$, ${\mathrm{MF}}_{17}$, and ${\mathrm{AP}}_{10}$. Instance set ${{}_{15}{}^{{10}^3}I}_{{\mathcal{U}}_{[1,100]}}$ is used for the evaluation. Based on the experimental evaluation, it can be concluded that ${\mathrm{CP}}_{04}$ with ${\mathrm{AP}}_{10}$ produced the best results on hard instances, while better results can be obtained on easy instances starting from ${\mathrm{MF}}_{17}$, the obtained initial solution. However, the provided results cannot be used for direct comparisons of the given constructive heuristics.

WTC and AR are also standard performance measures for theoretical comparison of Improvement Heuristics. Their experimental evaluation involves the quality of the produced solution and the required computational time on the selected test instance. However, WTC may not be known for some IHs, as well as AR for IHs that start from a random initial solution. On the other hand, AR which originates from CH, used to obtain an initial solution for the improvement procedure, may not be improved in a theoretical sense. In those cases, experimental comparison is the only choice.

Assuming $n>m$, enables to express WTC as dependent only on n and sorts the IH algorithms in non-decreasing order with respect to complexity as follows: IC, ICI, ICII: $\mathcal{O}(nlogn)$; KOMP: $\mathcal{O}(n^{2}S)$; LPT+, MF+: $\mathcal{O}(n^{2}Slogn)$; EX: $\mathcal{O}(n^{3}S)$; MSS: $\mathcal{O}(n^{2}k{S}^2)$; MSK: $\mathcal{O}(n^{3}k{S}^2)$; X-TMO and finally DIST with WTC of $\mathcal{NP}$ complexity.

For some of the IHs, WTC estimations have not been provided in the related papers. Therefore, the necessary derivations were performed to complete the above-presented list. For example, in [157] WTC of EX algorithm, was considered only for the 2-machine case and it is estimated to $\mathcal{O}(n^2)$. In [158], precise pseudo-code for the case $m>2$ is provided, allowing us to derive WTC. Assuming that every improvement minimally decreases the load of the most loaded machine (i.e., by 1), the number of iterations in the main loop is related to the sum of job durations (S). In the worst case the improvement will always occur for the last considered pair of machines, implying that WTC cannot be worse than $\mathcal{O}(n^{2}mS)$, i.e., $\mathcal{O}(n^{3}S)$ for $n>m$. Similarly, the WTC for MSS, MSK, LPT+, and MF+ could be calculated. Estimation of the WTC for the 3-PHASE algorithm is more complicated, as mentioned in [135]. The same holds for HI [139], and therefore the corresponding algorithms are missing in the aforementioned list.

If it is assumed that m and k approach infinity, AR can be expressed as a constant. Thus, the IH algorithms can be sorted in the non-increasing order of AR, with their constructive heuristics in parentheses: X-TMO (MF), MF+ (MF), PSMF+ (PSMF): ${\displaystyle \frac{13}{11}}$; HI ($\epsilon$-DUAL): ${\displaystyle \frac{5}{4}}$; EX, ICII, LPT+ (LPT), MMIPMH (LPT), SLACK+ (SLACK), DIST (SLACK): ${\displaystyle \frac{4}{3}}$; ICI: ${\displaystyle \frac{3}{2}}$; IC, 3-PHASE, MSS (${\mathrm{LPT}}_R$), MSK (${\mathrm{LPT}}_R$): 2.

A comparison of CHs and IHs with respect to WTC and AR can be found in

Table 14. Heuristic algorithms comparison by AR and WTC. “?” represents unknown values.

Name	WTC	AR	WTC Points	AR Points	$\Sigma$ Points
${\mathrm{MF}}_i$	$\mathcal{O}(knlogn)$	$72/61$	13	7	20
COMBINE	$\mathcal{O}(knlogn)$	$13/11$	13	6	19
MF	$\mathcal{O}(knlogn)$	$13/11$	13	6	19
SLACK	$\mathcal{O}(nlogn)$	$4/3$	14	4	18
${\mathrm{MF}}^{\prime }$	$\mathcal{O}(knlogn)$	$5/4$	13	5	18
LPT	$\mathcal{O}(nlogn)$	$4/3$	14	4	18
${\mathrm{RAS}}_2$	$\mathcal{O}(nlogn)$	$4/3$	14	4	18
ICII	$\mathcal{O}(knlogn)$	$4/3$	13	4	17
SPS	$\mathcal{O}(nlogn)$	$3/2$	14	3	17
ICI	$\mathcal{O}(knlogn)$	$3/2$	13	3	16
FGH	$\mathcal{O}(k{n}^2)$	$13/11$	10	6	16
DGH	$\mathcal{O}(k{n}^2)$	$13/11$	10	6	16
PSMF	$\mathcal{O}(k{n}^2)$	$13/11$	10	6	16
SLS	$\mathcal{O}(nlogn)$	2	14	2	16
${\mathrm{P}}_1$	$\mathcal{O}(n)$	?	15	0	15
MPS	$\mathcal{O}(n^2)$	$3/2$	12	3	15
LDM	$\mathcal{O}(n^{2}logn)$	$4/3$	11	4	15
LISTFIT	$\mathcal{O}(k{n}^{2}logn)$	$13/11$	9	6	15
DJMS	$\mathcal{O}(k{n}^{2}logn)$	$13/11$	9	6	15
MAAAT	$\mathcal{O}(nlogn)$	?	14	0	14
AP	$\mathcal{O}(nlogn)$	?	14	0	14
${\mathrm{LPT}}_R$	$\mathcal{O}(nlogn)$	?	14	0	14
PSC	$\mathcal{O}(nlogn)$	?	14	0	14
${\mathrm{PSC}}_i$	$\mathcal{O}(nlogn)$	?	14	0	14
IC	$\mathcal{O}(knlogn)$	2	13	1	14
${\mathrm{MF}}_e$	$\mathcal{O}(knlogn)$	?	13	0	13
MF+	$\mathcal{O}(n^{2}Slogn)$	$13/11$	6	6	12
LPT+	$\mathcal{O}(n^{2}Slogn)$	$4/3$	6	4	10
EX	$\mathcal{O}(n^{3}S)$	$4/3$	5	4	9
MS	$\mathcal{O}(n^3)$	?	8	0	8
X-TMO	$\mathcal{NP}$	$13/11$	1	6	7
SS	$\mathcal{O}(n^{2}S)$	?	7	0	7
KOMP	$\mathcal{O}(n^{2}S)$	?	7	0	7
PSMF+	?	$13/11$	0	6	6
MSS	$\mathcal{O}(n^2{S}^2)$	2	4	1	5
HI	?	$5/4$	0	5	5
DIST	$\mathcal{NP}$	$4/3$	1	4	5
MMIPMH	?	$4/3$	0	4	4
SLACK +	?	$4/3$	0	4	4
MSK	$\mathcal{O}(n^3{S}^2)$	2	3	1	4
${\mathrm{H}}_1$	$\mathcal{O}(n^{2}log(1/\epsilon +1/{\epsilon }^2))$	?	2	0	2
3-PHASE	?	2	0	1	1
KK	$\mathcal{NP}$	?	1	0	1

. The question mark (?) in the second or the third column indicates that the information about the corresponding performance indicator is missing. The algorithms are graded with respect to WTC and AR. The corresponding points are presented in the fourth and fifth columns, respectively. A larger number of points indicates a higher grade. If the performance is missing, the grade of the corresponding algorithm is set to 0. Finally, WTC and AR points are summed up (values presented in the last column) defining the overall grade of algorithms. In Table 14, the algorithms are sorted in the non-increasing value with respect to the sum of points. Regarding the experimental evaluation of the IH algorithms, proposed SLR revealed the following information.

The IC algorithm [154] was compared with the LPT CH on instances ${}_{10}{}^{500}I$, using five different distributions. In these experiments, IC showed comparable but generally performed weaker than LPT. The authors also concluded that the IC algorithm is very fast for the considered set of instances.

The KOMP algorithm [112] was compared with LPT and MF CHs on ${}_{10}{}^{100}I$ instances with controlled ${\displaystyle \frac{p_{min}}{p_{max}}}$ ratio. The results clearly indicated KOMP’s dominance over the other algorithms with respect to solution quality. Additionally, MF showed slightly better performance than LPT. However, the reported computational times of KOMP are sometimes up to two orders of magnitude larger than in MF and LPT cases, raising questions about its practical usability.

The EX algorithm [157] was compared with the LPT CH on ${}_5{}^{60}I$ instances for ten different uniform distributions of job durations. EX performed better without a significant increase in computational time.

The 3-PHASE algorithm [135] was compared with IC and ICII; BIN HE; and LPT CH algorithms on the $F_U$ instances (introduced in this work). 3-PHASE showed better performance than IC, ICII, and LPT CH, where LPT CH outperformed IC and ICII. IC and ICII showed identical performance. BIN solved to optimality more instances than 3-PHASE; however, required a larger computational time.

The X-TMO algorithm [159] was compared with LPT, MF, and ${\mathrm{MF}}_e$ CHs on ${}_{12}{}^{100}{I}_{{\mathcal{U}}_{[20,250]}}$ instances. As the initial solution for X-TMO each of CHs’ solutions is provided and it improved all of them with a significant increase in computational time. The results of comparing the CHs on the same instances suggested the following order: ${\mathrm{MF}}_e$, MF, and LPT.

The PI algorithm [160] was compared with LPT and MF CHs on ${}_{10}{}^{30}I$ instances with four different uniform distributions of job processing times. The authors reported the dominance of PI over CHs with respect to solution quality and a negligible increase in computational time. Among CHs, MF performed better.

The ${\mathit{CP}}_{99}$ algorithm [17] was compared to a simple ILP solver on ${{}_5{}^{15}MK}_{{\mathcal{U}}_{[1,100]},{\mathcal{U}}_{[10,100)},{\mathcal{U}}_{[50,100]}}$ instances, within a node limit of $5·{10}^5$. Without providing the resulting table, the authors concluded that ${\mathrm{CP}}_{99}$ performs well on all instances. In the cases when the ILP solver provided optimal solutions, ${\mathrm{CP}}_{99}$ was able to find the same solutions within a significantly smaller running time. When the ILP solver failed, ${\mathrm{CP}}_{99}$ always converged to a feasible solution.

The ME algorithm (representing the best results of 1-SPT, 1-BPT, K-SPT, and K-BPT) [162,163] was compared with the LPT CH; DM HE (with a backtrack limit of $4·{10}^3$); and the 3-PHASE algorithms on $F_{NU}$ instances. DM dominated in this experiment, with ME outperforming the remaining algorithms, and LPT being the weakest. In the subsequent experiments, ME was compared with LPT and 3-PHASE on $BP$ and $TR$ instances. ME performed the best, while LPT was consistently the weakest. In [162], DM, 3-PHASE, and LPT were compared on $F_U$ instances, where DM outperformed 3-PHASE, which in turn was better than LPT.

The HI algorithm [139,220] (with an iteration limit of ${10}^3$ for the TS improvement phase) was compared with LPT CH; DM HE (with a backtrack limit of $4·{10}^3$); and the 3-PHASE algorithms on $F_U$ instances. HI produced the best results, followed by DM and 3-PHASE. On $F_{NU}$ instances, HI was compared with LPT, DM, and ME algorithms, yielding similar results: HI dominated, DM was second, and ME was third. Detailed comparison results are provided in [251].

The MSS algorithm [95] was compared on the ${{}_{15}{}^{{10}^4}\tilde{C}}_{1-5}$ instances with DM and BIN HE solvers. The results provided by MSS and DM were close, while MSS dominated in comparison with BIN. For $PP$ instances, MSS achieved results similar to DM, where DM outperformed BIN. MSS was also compared with HI on F instances, showing comparable results, although sometimes slightly weaker.

The MSK algorithm [14] was compared with MSS and HI on F instances. The authors did not provide detailed comparison results, just the direct comparison by the number of better, equal, and worse solutions. They concluded that MSK performed better than the other two algorithms. MSK also showed a lower average gap compared to DM HE on ${{}_{15}{}^{{10}^4}\tilde{C}}_{1-5}$ instances.

The CA algorithm [164] was compared with DM HE; and HI on $F_U$ instances, where CA dominated and HI was the second. The comparison on $F_{NU}$ instances shows that CA clearly outperformed DM, HI, and ME, solving all instances to optimality, with HI being the second, followed by DM. On $BP$ instances, CA performed similarly to HI, requiring slightly larger computation time, while ME was the third and DM took the last place. On $TR$ instances, CA dominated HI, DM, and ME, solving all instances to optimality. The order of remaining algorithms was: HI, ME, DM.

The ${\mathit{KL}}_h$ algorithm [122] was compared with HI and ${\mathrm{DIMM}}_{SS}$ MH on F instances. ${KL}_h$ solved all instances to optimality within less time than the other two algorithms. The other two algorithms were not able to solve all instances, where ${\mathrm{DIMM}}_{SS}$ was more successful.

The MMIPMH algorithm [165] outperformed the LPT CH on a very small ${}_4{}^{9}I$ instance set based on a direct comparison of makespan values.

In [166], the authors compared 70 introduced IHs with each other on a subset of $Iogra$ instances. The ten best-performing algorithms are identified, as it was not possible to distinguish between them with statistical significance.

The PTAS algorithms for $P\left|\right|C_{max}$, as well as EE and HE algorithms, make the most sense in theoretical discussions. The primary measure for comparing them is WTC. However, some PTAS algorithms are implemented and experimentally compared with CHs.

The ε-DUAL algorithm, with $\epsilon ={\displaystyle \frac{1}{5}}$, was compared with the ${\mathrm{MF}}_e$, MF, and LPT CHs on the ${\widetilde{E_2}}_{{\mathcal{N}}_{450}}$ instances in [133], as it is mentioned earlier in this section. The average performance of the $\epsilon$-DUAL algorithm was close to its worst error bound, generating low-quality solutions compared to other methods. However, the required running time was comparable to others.

The BDJR algorithm [173], parallelized on 16 CPU cores was tested with $\epsilon \approx 17.29\%$. The obtained results were comparable with LPT, MF, and DJMS CHs on E instances; however, the required computational times were significantly longer.

In Figure 9, the relationships between all heuristic algorithms are displayed, based on the available empirical results. The orange color represents IH algorithms, the light orange represents CH algorithms, and the lightest orange (at the very bottom left) represents PTAS algorithms. Additionally, small blue nodes present some exact algorithms, making the connection between heuristic and exact algorithms, i.e., between Figure 8 and Figure 9. Dashed-line edges are used merely to highlight necessary edge intersections. The graph is divided into two major parts: the left part primarily features F edges, while the right side is dominated by E edges. However, these two parts are more connected than in Figure 8. The first connection between these two parts is established by $F_{NU}$ edges, linking PSMF with algorithms from the F-part of the graph. The second connection arises from the $F_U$ edge, connecting PSC and MF+ with the E-part. On the E-part, there is an additional F-path from SLACK+ to LPT. However, this path is problematic because it is based merely on better/worse comparisons, which are not informative enough to support the integration of this path into the F-part. It can be seen that LPT has the largest number of input edges, indicating that it is the most studied and widely used heuristic for this problem.

Once again, due to the lack of empirical comparisons in the literature, several compromises had to be made to distinguish between results across various instance sets. This may lead to some unclear relationships.

For example, considering the results on $BP$ instances, the order of ${\mathrm{PSC}}_i$, PSC, and SPS CHs should be reversed. However, as they belong to the F-part, the presented order is established, although, the other one is indicated too.

Similarly to the case of exact algorithms, no single heuristic method that dominates across all instances could be identified based on the available results. Therefore, it is crucial to perform a comparison of all methods on a wide range of instances under equal conditions to gain a comprehensive understanding of the practical performance of the proposed heuristics. Due to the fact that CH algorithms are usually quite simple to implement and that they represent a good starting point for many other methods, the results of this SLR indicate that they might be extensively used in future comparisons. Additionally, special attention should be paid to selecting appropriate parameters for each particular algorithm.

6.3.3. Metaheuristics Comparisons

Metaheuristics generally provide high-quality solutions, although without a guarantee. No improvement with respect to the AR of the initial solution was certified. An initial solution is usually generated by some CH or IH. Moreover, the performance of these algorithms heavily relies on the values of the parameters, making WTC an unreliable measure. Therefore, the performance of MHs can be estimated only by experimental evaluation.

The ${SA}_{88}$ algorithm [10] was tested on a small instance set ${}_{12}{}^{{10}^3}I$ and the $G78$ instance. The experimental evaluation on ${}_{12}{}^{{10}^3}I$ instances was used to discuss the quality of the obtained solutions. The author concluded that only for $m=2{\mathrm{SA}}_{88}$ can find the optimal solutions. As the number of machines increases, finding optimal solutions becomes harder. For the $G78$ instance, based on private correspondence with its creator, the author concluded that ${\mathrm{SA}}_{88}$ outperformed several tested approximate algorithms. However, no information about the specific approximate algorithms is provided. Considering that Graham created $G78$, one can assume that LPT was certainly among them.

The HG algorithm [178] was tested on the ${}_{50}{}^{2·{10}^3}{I}_{{\mathcal{U}}_{[0,1]}}$ set of instances with known optimal solutions. The provided results show an average relative error of at most ${10}^{-5}$. In addition, HG algorithm is compared with ${\mathrm{SA}}_{88}$ on the $G78$ instance. In this experiment, HG provided a solution of better quality.

The ${SA}_{97}$ algorithm [179] was evaluated on the ${}_{50}{}^{5·{10}^3}{I}_{{\mathcal{U}}_{[{10}^3,5·{10}^4]}}$ instance set. The authors examined the influence of neighborhood types and the number of iterations on the quality of the provided solution. Two types of neighborhoods are combined with two values of iteration limits (100 and ${10}^3$). The reported gap with respect to $L_2$ ranged from 0.2% for small-sized instances up to 3% for larger instances. However, no comparison with other algorithms was provided.

The TGR algorithm [180] was compared with HG on the ${}_{50}{}^{2·{10}^3}{I}_{{\mathcal{U}}_{[0,1]}}$ instance set and, in the majority of the cases, it provided better quality solutions requiring smaller number of iterations.

The ${GA}_{99}$ and ${SA}_{99}$ algorithms [181] were compared with each other and with the LPT CH on instance ${}_3{}^{7}I$. The results show that ${\mathrm{GA}}_{99}$ and ${\mathrm{SA}}_{99}$ are comparable and that they outperform LPT. In the next experiment, ${\mathrm{GA}}_{99}$ showed better performance than ${\mathrm{SA}}_{99}$ on instances ${}_2{}^{10}I$ and ${}_{10}{}^{30}I$.

The AIS and ${SA}_{02}$ algorithms [161] were compared with each other, as well as with the LPT and MF CHs, and the LPT+ and MF+ IHs on $F_U$ instances and five instances from [127] obtained using a stepwise distribution. This job duration distribution is known to produce the worst-case instances for LPT. In the first experiment on $F_U$ instances, AIS performed slightly better than ${\mathrm{SA}}_{02}$, although requiring an order of magnitude larger processing time. The heuristic algorithms were ranked in decreasing order of solution quality as follows: LPT+, MF+, LPT, and MF. In the second experiment, 25 instances ${}_2{}^{95}{I}_{\mathcal{U}[{10}^9,{10}^{10}]}$ from [252] were used to compare only the two metaheuristics (due to the weak performance of other algorithms). AIS was superior with respect to solution quality.

The AntMPS algorithm [7] was tested on ${}^{{10}^3}{I}_{\mathcal{U}[20,100]}$ instances for BPP and on $TR$ instances. Interestingly, AntMPS, as a $P\left|\right|C_{max}$ metaheuristic, was directly compared with different BPP metaheuristics, that were actually solving different problems on the same instances. However, AntMPS was given an optimal number of bins as the input value for m to simulate the $P\left|\right|C_{max}$ problem. The resulting makespan actually represents bin capacity and, if it is not exceeded, the corresponding instance is considered as solved to optimality. With the stopping criterion limited to ${10}^3$ iterations, AntMPS was able to solve the majority of instances to optimality, while for the remaining the resulting gap was minimal.

The ${SA}_{06}$ algorithm [182] was compared with LISTFIT CH and PI IH on E instances. Given a stopping criterion of $30n$ iterations, it outperformed other algorithms with respect to the solution quality: providing the smallest gap with respect to $L_1$ for almost all instances. For $E_4$ class, ${\mathrm{SA}}_{06}$ found the larger number of optimal solutions. The second-best position was taken by PI. The execution times were not reported because ${\mathrm{SA}}_{06}$ turns out to be very fast, i.e., it completed all executions within less than $1s$.

The ${ILS}_{06}$ algorithm [183] was compared with the LPT CH on ${}_{40}{}^{{10}^3}{I}_{{\mathcal{U}}_{[1,100]}}$ instances, generated by the authors, where it demonstrated better performance. They used interesting performance measure of algorithms for a single instance, calculated as the ratio between the makespan produced by the algorithm for that instance and the best-known makespan for all instances of the same size. In general, ${\mathrm{ILS}}_{06}$ outperformed LPT with respect to the solution quality. However, the authors noticed different behavior of ${\mathrm{ILS}}_{06}$ algorithm related to ${\displaystyle \frac{n}{m}}$ radios, ranging from 2 to 333, where the largest improvement of 7.21% over LPT was achieved for ${\displaystyle \frac{n}{m}}\approx 3$. The authors also observed that the largest execution times correspond to the smallest ${\displaystyle \frac{n}{m}}$ ratios. ${\mathrm{ILS}}_{06}$ solved 83% of instances to optimality.

The HDNN algorithm [184] was compared with the LPT CH on ${}_3{}^{10}{I}_{{\mathcal{U}}_{[1,3]}}$ instances, where it outperformed LPT in reasonable time. As a performance measures, the authors used the gaps between the obtained solution and both the LPT solution and the optimal solution.

The ${\mathit{HDNN}}_i$ algorithm [185] was compared with the LPT CH on ${}_5{}^{100}{I}_{{\mathcal{U}}_{[1,3]}}$ instances, also demonstrating superior performance in reasonable time, and using the same measures as for HDNN.

The TCNN and ${\mathit{TCNN}}_i$ algorithms [186] were compared with HDNN on ${}_3{}^{50}{I}_{\mathcal{U}[1,50]}$ instances, where ${\mathrm{TCNN}}_i$ showed the best performance, followed by TCNN. The authors measured the best and average makespan over 100 random instances with the same input parameter values.

The ${\mathit{DIMM}}_{SS}$ algorithm [114] was compared with 1-SPT (one of four ME algorithms) and HI IHs on $F_U$ instances. The presented results show that ${\mathrm{DIMM}}_{SS}$ provided the best results measured as the relative gap with respect to the $max(L_3,L_{HS},L_{\theta })$. The second was HI which, in the majority of cases, required less computational time. In the second experiment, ${\mathrm{DIMM}}_{SS}$ was compared with 3-PHASE, 1-SPT, and HI algorithms on $F_{NU}$ instances, with similar results. 3-PHASE showed the worst performance.

The MVNS and ${GA}_{09}$ algorithms [5] were compared with the LPT CH on ${}_{20}{}^{200}{I}_{\mathcal{U}[1,100]}$ instances, where MVNS showed the best results and ${\mathrm{GA}}_{09}$ performed the worst, with respect to the relative gap between produced makespan and $L_1$. Regarding the execution time, MVNS also outperformed ${\mathrm{GA}}_{09}$.

The ${BCO}_{09}$ algorithm [6] was tested on the $I{T}_{10,20,30,40,50,100}$ set of instances, achieving optimal solutions for all instances within the execution time less than 2 s for a single instance.

The ${\mathit{VNS}}_{09}$ and ${\mathit{MC}}_{09}$ algorithms [187] were compared on the $I{T}_{10,20,30,40,50,100}$ set of instances. Better performance was demonstrated by ${\mathrm{VNS}}_{09}$, providing optimal solutions for all instances. Additionally, ${\mathrm{VNS}}_{09}$ required less CPU time than ${\mathrm{BCO}}_{09}$ in [6].

The ${\mathit{DPSO}}_{09}$ and HDPSO algorithms [189] were compared with ${\mathrm{SA}}_{06}$ on E instances. The results showed that ${\mathrm{DPSO}}_{09}$ slightly outperformed ${\mathrm{SA}}_{06}$, while HDPSO performed significantly better than ${\mathrm{DPSO}}_{09}$, with respect to relative gap between produced makespan and $L_1$. It is important to note that the results for ${\mathrm{SA}}_{06}$ on $E_2$ were not consistent with those reported in [182].

The DSHS algorithm [190], with a stopping criterion of 100 iterations, was compared with HDPSO and ${\mathrm{SA}}_{06}$ on $E_{1,2}$ instances, showing better results than both competitors with respect to both $L_1$-gap and execution time. As expected, ${\mathrm{SA}}_{06}$ performed the worst regarding the both measures.

The ${\mathit{DHS}}_{11}$ and HDHS algorithms [191] were compared with LPT CH, ${\mathrm{GA}}_{99}$, ${\mathrm{SA}}_{06}$, and MVNS on ${}_{20}{}^{200}{I}_{{\mathcal{U}}_{[1,100]}}$ instances. All MHs were limited to 100 iterations. Both introduced algorithms outperformed the other competitors, where HDHS performed the best with respect to the quality of solutions ($L_1$-gap); however, requiring longer execution time than ${\mathrm{DHS}}_{11}$. Unexpectedly, all other competitors performed worse than LPT. This was surprising because MVNS was tested on the same type of instances in [5], where it outperformed LPT. This discrepancy can be explained by the fact that experiments were performed on the newly generated test instances. However, the results for LPT seem to be copied from [5].

The RIVNS and HIVNS algorithms [192] were compared with LPT CH; ${\mathrm{GA}}_{99}$, ${\mathrm{SA}}_{06}$, and MVNS on ${}_{20}{}^{200}{I}_{{\mathcal{U}}_{[1,100]}}$ instances. All MHs were limited to 100 iterations. RIVNS and HIVNS outperformed the other competitors, with HIVNS emerging as the best, with respect to both quality of solution ($L_1$-gap) and execution time. The experiments were conducted using the same version of instances as in [190], and the results of both algorithms were better than those of HDHS. In this experiment, LPT achieved better average solution quality than ${\mathrm{GA}}_{99}$ and MVNS, while ${\mathrm{SA}}_{06}$ was just slightly better than LPT.

The ${SA}_{12}$ algorithm [193] was compared with ${\mathrm{SA}}_{06}$ on $E_2$ and ${}_{20}{}^{500}{I}_{\mathcal{U}[1,100],\mathcal{U}[100,800]}$ instances, where it achieved better results with respect to $L_1$-gap. It is interesting to note that the authors reimplemented the ${\mathrm{SA}}_{06}$ algorithm and regenerated $E_2$ instances, but they were unable to reproduce the ${\mathrm{SA}}_{06}$ results and reported worse performance of ${\mathrm{SA}}_{06}$ with respect to [182].

The SPPSO, ${\mathit{DPSO}}_{12}$ [253], and ${\mathit{PSO}}_{spv}$ [254] algorithms were compared in [194] on ${}_{50}{}^{500}{I}_{\mathcal{U}[1,100],\mathcal{U}[100,800]}$ instances. All algorithms used population size n and were limited to ${\displaystyle \frac{2·{10}^3}{n}}$ iterations. SPPSO dominated over other algorithms with respect to average $L_1$-gap, followed by ${\mathrm{PSO}}_{spv}$.

The ${\mathit{BCO}}_{12}$ algorithm [177] was compared with a BPP solver, and in [211] with DM and DIMM HE algorithms on $IO$, $I{T}_{500}$, and $RN$ instances. Both HE algorithms outperformed ${\mathrm{BCO}}_{12}$ solving all instances to optimality and having much better execution times. DM is faster than DIMM.

The ${\mathit{DHS}}_{12}$, BHS, and ${\mathit{DHS}}_{LS}$ algorithms [195] were compared in two experiments. In the first experiment, BHS, ${\mathrm{DHS}}_{12}$, and ${\mathrm{DHS}}_{LS}$ were compared on $E_2$ instances, where ${\mathrm{DHS}}_{LS}$ dominated, followed by DHS, with respect to average $L_1$ gap, and all algorithms had similar execution times. In the second experiment, ${\mathrm{DHS}}_{12}$, ${\mathrm{DHS}}_{LS}$, ${\mathrm{SA}}_{06}$, ${\mathrm{DPSO}}_{09}$, and HDPSO were compared on E instances. With respect to the average $L_1$ gap, ${\mathrm{DHS}}_{LS}$ dominated, followed by HDPSO provided slightly worse average solution quality, but within a significantly shorter execution time. DHS, ${\mathrm{SA}}_{06}$, and ${\mathrm{DPSO}}_{09}$ ranked third, fourth, and fifth, respectively, and also required significantly longer execution times. It is worth mentioning that the general conclusions, and in particular the relationship between ${\mathrm{DPSO}}_{09}$ and ${\mathrm{SA}}_{06}$, are different from those in [189].

The CSA algorithm [196] limited to ${10}^3$ iterations was compared with ${\mathrm{SA}}_{06}$, ${\mathrm{DPSO}}_{09}$, and HDPSO on E instances, considering average gap with respect to $L_1$. CSA provided solutions similar to HDPSO with significantly lower CPU time consumption. Both of them outperformed ${\mathrm{SA}}_{06}$ and ${\mathrm{DPSO}}_{09}$. Regarding this experiment, again, it is important to mention significant differences in the presented results compared to older studies.

The GES and GES+ algorithms [197] limited to ${10}^3$ iterations were compared with ${\mathrm{SA}}_{06}$ on E instances, with respect to $L_1$, where GES+ emerged as the best and GES ranked as the second. In addition, the new algorithms required shorter average running times. The results for ${\mathrm{SA}}_{06}$, once again, were not consistent with older papers.

The ICSA algorithm [198] with limitation to ${10}^3$ iterations, was compared with ${\mathrm{SA}}_{06}$, ${\mathrm{DPSO}}_{09}$, HDPSO, ${\mathrm{DHS}}_{12}$, ${\mathrm{DHS}}_{LS}$, and CSA on E instances. The authors reported the domination of ICSA in terms of solution quality. In the performed experimental evaluation, the results of the older algorithms were taken from the literature, and the results of ICSA were obtained using newly generated E instances with unspecified parameters used for generation. We have a reasonable doubt about the reliability of the provided results. Based on the analysis of the results from previous experiments, the methodology of using E instances is questionable. As [198] provides the most comprehensive comparisons of different algorithms so far, it is used here for a more detailed analysis.

In the presented experiments related to E instances, for each combination of parameters, the authors randomly generated 50 problem instances without providing details about the random seeds. They presented the average gap with respect to $L_1$. Let us focus on the small group $E_4$, because of the simplicity of its instances and the possibility of solving them to optimality using a commercial linear programming solver. We generated 50 groups of $E_4$ instances (50 generations of 50 instances for each parameter combination), and, in the last two columns of

Table 15. ${\mathrm{OPT}}_B$, ${\mathrm{OPT}}_W$ and results of metaheuristic solvers presented in [198] for $E_4$ instances. Bolded values indicate identified discrepancy.

m, n	D	${\mathrm{SA}}_{06}$	${\mathrm{DPSO}}_{09}$	HDPSO	${\mathrm{DHS}}_{12}$	${\mathrm{DHS}}_{\mathit{LS}}$	CSA	ICSA	${\mathrm{OPT}}_{\mathit{B}}$	${\mathrm{OPT}}_{\mathit{W}}$
2, 9	${\mathcal{U}}_{[1,20]}$	1.001	1.000	1.000	N/A	N/A	1.000	1.000	1.000	1.001
	${\mathcal{U}}_{[20,50]}$	1.001	1.001	1.001	N/A	N/A	1.001	1.001	1.002	1.001
	${\mathcal{U}}_{[50,100]}$	1.004	1.004	1.004	N/A	N/A	1.004	1.003	1.002	1.003
	${\mathcal{U}}_{[100,200]}$	1.004	1.004	1.004	N/A	N/A	1.004	1.004	1.002	1.005
	${\mathcal{U}}_{[100,800]}$	1.002	1.002	1.002	N/A	N/A	1.002	1.001	1.002	1.003
3, 10	${\mathcal{U}}_{[1,20]}$	1.001	1.001	1.001	N/A	N/A	1.001	1.002	1.003	1.006
	${\mathcal{U}}_{[20,50]}$	1.008	1.007	1.007	N/A	N/A	1.007	1.005	1.006	1.009
	${\mathcal{U}}_{[50,100]}$	1.010	1.009	1.009	N/A	N/A	1.010	1.006	1.011	1.010
	${\mathcal{U}}_{[100,200]}$	1.017	1.016	1.016	N/A	N/A	1.016	1.008	1.010	1.020
	${\mathcal{U}}_{[100,800]}$	1.009	1.009	1.009	N/A	N/A	1.009	1.006	1.007	1.008
Average		1.0057	1.0053	1.0053	1.0064	1.0061	1.0054	1.0036	1.0046	1.0067

, we present the best (${\mathrm{OPT}}_B$) and the worst (${\mathrm{OPT}}_W$) average gaps of optimal results among these 50 groups, obtained using the exact solver. The table also contains the characteristics of instances and results of other algorithms provided in [198].

As shown in Table 15, the average results of all metaheuristics, except ICSA, are between ${\mathrm{OPT}}_B$ and ${\mathrm{OPT}}_W$. If we assume that all presented heuristics provided near-optimal solutions for all instances from $E_4$, these results can be considered acceptable. However, the average gap with respect to all instances presented for ICSA is significantly smaller than the best average gap of ${\mathrm{OPT}}_B$, leading us to conclude that the authors of [198] may have been particularly fortunate when generating the E instances.

The ${\mathit{HIVNS}}_1$, ${\mathit{RIVNS}}_1$, ${\mathit{HIVNS}}_2$, and ${\mathit{RIVNS}}_2$ algorithms [199] limited to ${10}^3$ iterations, were compared with ${\mathrm{GA}}_{99}$, ${\mathrm{SA}}_{06}$, MVNS, RIVNS, and HIVNS on ${}_{20}{}^{200}{I}_{{\mathcal{U}}_{[1,100]}}$ instances. The authors generated ten instances for the same combination of parameters and used the gap with respect to $L_1$ as the performance measure. All presented algorithms outperformed the older competitors. ${\mathrm{RIVNS}}_1$ and ${\mathrm{RIVNS}}_2$ achieved the best average results, while ${\mathrm{RIVNS}}_2$ was slightly faster. ${\mathrm{HIVNS}}_1$ performed the worst.

The ${GWO}_{19}$ and ${GA}_{19}$ algorithms [200] were compared with each other on E instances, where ${\mathrm{GWO}}_{19}$ outperformed ${\mathrm{GA}}_{19}$. However, the type of measurement used was not explained, and therefore, the results were unusable for additional comparisons.

Figure 10 illustrates the relationships between the employed metaheuristic algorithms based on the available empirical results. MHs from the P-class are represented in dark green, while light green is used for S-class MHs. Additionally, some exact and heuristic algorithms (in the corresponding colored nodes) are included to establish a connection with the graphs in Figure 8 and Figure 9. As the first observation, it is evident that the graph can be separated into several parts. The largest (central) part is primarily characterized by the utilization of E instances. Based on our analysis of the performances of the ICSA algorithm [198], we demonstrated that this branch should not be considered as a fully accurate representation of the relationships between the corresponding algorithms.

The second (upper right) part of the graph contains algorithms compared on ${}_{20}{}^{200}{I}_{{\mathcal{U}}_{[1,100]}}$ instances and the relationships between the algorithms could clearly be established (algorithms performing better are on top). Additionally, there is a smaller, central right, part of the graph that involves $F_U$ instances, which helps to establish relationships between metaheuristics and the results of exact and heuristic algorithms. ANN-based metaheuristics are presented in the left part of the graph, besides ${\mathrm{ILS}}_{06}$. Finally, in isolated parts of the graph, comparisons of algorithms that have no direct relationship with other algorithms provided. Similarly to heuristic and exact methods, no single metaheuristic method dominates across all instances.

7. RQ4: Discussion, Challenges, and Future Work

Although the $P\left|\right|C_{max}$ problem was introduced more than half a century ago, Figure 11 shows that interest in publishing research related to $P\left|\right|C_{max}$ did not diminish. Several papers have been published on this topic every year. We observe that majority of publications related to metaheuristic approaches were published from the early 2000s to the 2020s. On the contrary the publications on heuristics and exact solvers, which exhibit a consistent developmental trend from the inception of the problem.

The identified limitations of any SLR are related to biases in availability of publications and in the study selection processes, to inaccuracy in the study extraction process, and to misclassification of published results [255]. We have identified the following challenges, which provide the exciting opportunities for future work:

• Isomorphic versions of $P\left|\right|C_{max}$ problem;
• A lot of taxonomies available;
• Standardization of identified groups of instances;
• Instance nomenclature limitations;
• Fair algorithm performance evaluation;
• The state-of-the-art algorithm identification.

Isomorphic versions and multiple taxonomies challenges could be addressed by identifying and presenting various isomorphic versions of $P\left|\right|C_{max}$ problem and selecting the taxonomy that was most appropriate for the corresponding studies. However, there might still be cases of specialized $P\left|\right|C_{max}$ problem versions as well as algorithms that were not covered by our exhaustive search. An example is a set of exact solvers and approximate algorithms for high-multiplicity instances, i.e., instances containing a small number of groups with a lot of identical jobs [256,257,258,259]. Theoretical verification of model equivalence and performance evaluation of corresponding algorithms for each newly discovered isomorphism represent promising a avenue for future research.

Instance groups standardization and instance nomenclature allow us to establish benchmarks and simplify the procedure of checking and referencing published results. To that end, we have constructed a graph of algorithm comparisons for each group of algorithms in the selected studies. This graph from each group of the algorithms shows that it is easy to see the relationships between them in reference to testing on the same instance groups. However, the facts that not all instances can be considered as standardized and not all algorithms were tested on all instances resulted in our inability to provide a comprehensive graphical representation. Our nomenclature covers four characteristics of instances, which to the best of our knowledge is the most efficient way to describe all instances in the instance groups we identified. The comprehensive set of standardized instances still needs to be identified and systematically characterized.

Fair algorithm performance evaluation challenge is rather a group of many inherent issues that include but are not limited to using ad-hoc instances, not providing the full set of characteristics of the results, instances, solvers, controlling seed values for random variables, and the lack of public repositories containing the algorithms, instances and their optimal/best-known solutions. By making at least the standardized groups of instances available, along with all the identified characteristics, we are setting the stage for fair evaluation of algorithm performance. Resolving the remaining issues remains as a potential subject for future work.

In addition, fair algorithm comparison must include the analysis of all involved components. From the selected studies, we learned that the choice of the lower bounds affects both the solution quality and the speed of obtaining the solution. Thus, we attempted to exhaustively explain all available lower-bounding strategies. Moreover, many algorithms and their performance depend on the underlying algorithm that was implemented to help in the solution finding process. We identified and described many such algorithms. Combining various parts from different algorithms should be additionally explored, as it may result in the development of more efficient search procedures.

Another issue for fair algorithm performance evaluation is related to testing stochastic algorithms. Here, it is important to determine the number of repetitions (executions, runs) of the algorithm by combining minimal statistical requirements and a number of runs already utilized in the literature. For example, the authors of [260] suggest that any number of runs between 20 to 100 could be sufficient for a good estimation of the algorithm’s performance, in [137] the authors used 100 repetitions, while in [12], 30 repetitions were performed. Standardization of the number of repetitions, supported by theoretical statistical considerations, still needs to be performed.

For some CH and IH algorithms, only AR and WTC parameters were available. Even though this was not a primary focus of our research, we have tried to establish AR and WTC for the remaining CH and IH algorithms to be able to include all of them into comparison. As a future work we envision the systematic experimental evaluation of all CH and IH algorithms on the standardized set of instances, under the same condition.

The state-of-the-art algorithm identification is directly tied with fair algorithm performance evaluation. Due to the fact that not all algorithms were tested on all instances, under the same conditions, and other related issues, we were unable to identify even the best-performing algorithm for any of the standardized groups of instances. Instead, we have established the standardized framework to characterize the state-of-the-art candidates, i.e., the algorithms that perform the best on the majority of instances from all standardized groups. Another interesting utilization of the obtained results involves the selection of the best-performing algorithm for the unseen instances. More precisely, future work may include building a rich database of experimental results for different algorithms on different sets of instances. Such a database would enable AI/ML techniques to predict the difficulty of any new instance that has not been tested yet [1].

8. Conclusions

We conducted a systematic literature review of optimization algorithms applied to the $P\left|\right|C_{max}$ problem. We summarized and categorized the “state-of-the-art” methods, benchmark test instances, and performance indicators. As a result, we identified a taxonomy of optimization algorithms where the categories represent high-level types of optimization methods (exact, heuristic, metaheuristic) that have been used in the attempts to solve the $P\left|\right|C_{max}$ problem.

Researchers are actively working on developing new methods that need to be evaluated appropriately. We have applied a systematic literature review mapping principles to try to find them all and provided statistics of all the identified literature along with the challenges and their resolutions. One of the problems we faced was the symmetry among the optimization problems that are similar to $P\left|\right|C_{max}$. Therefore, the $P\left|\right|C_{max}$ problem was recognized under different names, not only in various communities that tried to solve it but even within the communities. In was very challenging and time consuming to identify all these symmetric problems as $P\left|\right|C_{max}$. Another difficulty was related to test instances. Several dozens of various sets of instances were used in different papers, thus making difficult to obtain fair comparisons of the used methods. We have standardized four main groups of instances and made them publicly available along with explanations of their characteristics. Various criteria (performance indicators) had been used to compare optimization algorithms. We have identified a considerable number of them that were applied to the $P\left|\right|C_{max}$ problem. We made a proper match between categories of methods and performance indicators, and, for each category, we proposed a suitable subset of indicators that should be enough to objectively evaluate the newly developed optimization method.

The first avenue for future research could be identifying the best algorithms (the-state-of-the-art) for $P\left|\right|C_{max}$ problem. This could be performed by testing best solvers from all categories on all standardized instances under the same conditions. Then, the best-performing algorithms should be evaluated against each other and a set containing the-state-of-the-art methods should be formed. It is not realistic to assume that a single best approach with respect to all instances could be found. Upon an in-depth experimental analysis, potentially supported by the theoretical consideration, the best that can be carried out is to determine the combination of algorithms’ and instances’ characteristics that yield a good performance. As the second avenue of future studies we envision the application of artificial intelligence techniques, machine learning in particular, to predict the performance of a given algorithm on some newly introduced instances. The third avenue could be the hybridization of algorithms from different categories to build a solver capable of efficiently solving a wide variety of instances. The fourth avenue could be the generation of additional sets containing standardized instances for the $P\left|\right|C_{max}$ problem. Moreover, theoretical analysis of the problem and the development of new solvers could be another avenue of future research.

In addition, our considerations could be generalized as a framework to investigate other optimization problems. A fair comparison of optimization methods, as well as rich data for learning and predicting the performance of various algorithms, would be of great importance and applicability in many domains, including rather exotic blockchain maintenance and security evaluation. However, significant work is required to prepare all relevant data and the results for the application of the considered algorithms to each particular optimization problem instances. Automating this process may also be an interesting avenue for future research.