Coordinated Ship Welding with Optimal Lazy Robot Ratio and Energy Consumption via Reinforcement Learning

Abstract

Ship welding is a crucial part of ship building, requiring higher levels of robot coordination and working efficiency than ever before. To this end, this paper studies the coordinated ship-welding task, which involves multi-robot welding of multiple weld lines consisting of synchronous ones to be executed by a pair of robots and normal ones that can be executed by one robot. To evaluate working efficiency, the objectives of optimal lazy robot ratio and energy consumption were considered, which are tackled by the proposed dynamic Kuhn–Munkres-based model-free policy gradient (DKM-MFPG) reinforcement learning algorithm. In DKM-MFPG, a dynamic Kuhn–Munkres (DKM) dispatcher is designed based on weld line and co-welding robot position information obtained by the wireless sensors, such that robots always have dispatched weld lines in real-time and the lazy robot ratio is 0. Simultaneously, a model-free policy gradient (MFPG) based on reinforcement learning is designed to achieve the energy-optimal motion control for all robots. The optimal lazy robot ratio of the DKM dispatcher and the network convergence of MFPG are theoretically analyzed. Furthermore, the performance of DKM-MFPG is simulated with variant settings of welding scenarios and compared with baseline optimization methods. Compared to the four baselines, DKM-MFPG owns a slight performance advantage within 1% on energy consumption and reduces the average lazy robot ratio by 11.30%, 10.99%, 8.27%, and 10.39%.

1. Introduction

In recent decades, the development of the ship-building industry has been greatly stimulated by maritime transportation, which has become a core element of global economic development. The ship-building [1], ship repair, and offshore structures [2] industries highly rely on ship-welding techniques to construct more ships. To enhance welding efficiency, there is a trend to apply intelligent and innovative technologies for employing a group of robots to coordinately accomplish ship-welding tasks, known as the coordinated ship-welding problem.

The early welding task usually considers a single robot to achieve the shortest welding time or distance path. In [3], the spot welding task has been converted into the traveling salesman problem and is solved by the genetic algorithm [4] to realize the minimum time path planning. In [5], the improved lazy PRM algorithm based on online collision-free path planning is proposed for the arc welding robot. On the basis of planning the shortest path from one weld line to another, task allocation methods such as market-based allocation [6] help determine the welding sequence among numerous weld lines, which greatly decreases time consumption. In [7], the welding path planning problem considers the combinational optimization problem by arranging the sequence and directions of welding seams with time and energy optimization, which is tackled by an improved discrete MOEA/D based on a hybrid environment selection. Different from the single-robot welding case, coordinated welding, that is, some welding tasks are executed by a group of robots, has received more attention due to its efficiency and robustness. In [8], the double global optimum genetic algorithm—particle swarm optimization (GA-PSO)—is proposed to solve the welding path planning of two robots with the shortest collision-free path. In [9], the gantry welding with minimal path length and energy consumption is solved by the combination of a genetic algorithm, an adaptive reference direction-guided many-objective evolutionary algorithm, and a lazy probabilistic roadmap (PRM) trajectory planning method. Note that the coordinated welding problem requires both path planning to desired weld lines and multi-objective optimization such as minimal time and lazy robot ratio. Despite GA and MA for single-objective optimization, multi-objective evolutionary algorithms (MOEA) such as MOEA/D [10] and NSGA-II [11] can be effective. To further improve the performances of search and computation efficiency, NSGA-III is proposed in [12], which emphasizes non-dominated population members to get close to a set of supplied reference points. In [13], another MOEA method, AnD, is designed, which consists of an angle-based selection strategy and a shift-based density estimation strategy to delete poor individuals one by one in the environmental selection. To our knowledge, the case of utilizing more than two robots and the coordinated ship-welding process requirement of executing a weld line by a pair of robots together have not been solved. Furthermore, it is challenging when determining the welding sequence to dispatch the group of robots with multiple objectives.

According to the ship-building or ship-welding tasks [14], if two or more workers/robots fail to cooperate well, it will result in a heavy workload for one and idleness for the other. Note that excessive use and workload for ship-welding robots can lead to performance degradation and even functional failure. Therefore, how to prevent robots from being lazy and to enhance their working efficiency are important issues to balance the workload for all robots. In [15], the efficiency of ship-welding robots is significantly enhanced by using the ship’s block panel assembly process. Concerning energy consumption and efficiency, a multi-objective optimization method is proposed in [16] through parameter optimization for arc welding. When coordinated ship-welding tasks are required where a weld line needs to be welded by two robots simultaneously, the robot idleness becomes a more difficult problem, as the insufficient number of robots can not even finish the task, let alone optimize the working efficiency.

Recently, reinforcement learning (RL) [17] has shown the potential to achieve coordinated tasks of multiple agents under optimization objectives. In [18], RL algorithm deep deterministic policy gradient (DDPG) and convolutional neural networks are designed to achieve the dynamic coordinated scheduling task of automated guided vehicles. To quickly adopt the motion based on the resulting feedback of robots, model predictive control and RL controllers [19], intelligent optimization algorithm and DDPG controllers [20], and deeply learning-based recurrent neural network controllers [21] are designed. In [22], a RL-based dueling deep Q-network is used for the optimization of berth allocation with multiple ships. To deal with multiple optimization objectives, RL serves as an end-to-end framework for providing effective solutions. In [23], a deep RL-based multi-objective optimization algorithm is proposed to decompose the multiple objectives and collaboratively optimize them as parameters of neural networks. Considering the dynamic environment and multi-objective optimization, RL is designed in [24] to learn better evolutionary behaviors from dynamic environment information. Nevertheless, current RL methods seldom focus on the coordinated ship-welding problem, especially when synchronous welding requirements and multiple objectives coexist.

Motivated by the above discussion, this paper focuses on the coordinated ship-welding task consisting of synchronous weld lines executed by a pair of robots and normal weld lines executed by one robot, where the objectives for robots are the optimal lazy robot ratio and energy consumption while accomplishing the task. In the literature, most coordinated welding problems focus on the case of two robots [3,5,7,8,9]. This is due to the fact that two robots’ cooperation is the basic and simplest case in coordinated ship welding. As welding tasks become increasingly complex, the cooperation of more than two robots becomes a trend and hence brings some new problems in determining how to improve the working efficiency in scaling when lazy robots occur. To this end, the DKM dispatcher in DKM-MFPG is designed to dynamically dispatch all the unemployed robots. Then, an MFPG motion planner is designed to achieve the energy-optimal motion for robots. The contributions of this paper are listed as follows:

(1)

Different from the welding tasks [3,5,7,8,9] with one or two robots and normal weld lines, the coordinated ship-welding task in this paper requires multiple robots with synchronous weld lines that need to be executed by a pair of robots together. To this end, a novel DKM-MFPG algorithm is proposed to achieve the coordinated ship-welding task composed of normal weld lines and synchronous weld lines, which is a first attempt and effective solution applied in future large-scale unmanned ship-welding scenarios. In this paper, the optimal lazy robot ratio under the DKM dispatcher and the convergence of neural networks under MFPG are analyzed theoretically in detail, which is ignored in [18,22].

(2)

Distinct from the single objective of shortest time/distance path in [3,5,8] and the objectives of shortest time/distance path and energy consumption in [7,9], this paper intends to achieve the minimal lazy robot ratio and task accomplishment time, which are tackled by the DKM dispatcher and the MFPG motion planner, respectively. Comparison experiment results show that our DKM-MFPG algorithm outperforms single- and multi-objective optimization methods on the lazy robot ratio by around 10%, while maintaining close energy consumption performance.

The rest of the article is organized as follows. Section 2 demonstrates the formulation of the problem. Section 3 is the methodology of DKM-MFPG with theoretical analysis. Section 4 numerically simulates DKM-MFPG under variant scenarios and comparison baselines with result analysis. Finally, Section 5 is the conclusion.

2. Problem Formulation

Consider that the coordinated ship-welding task in this paper is constructed on a grid-world Cartesian coordinate system $\mathbb{W}=\left\{oxy\right\}$ with origin O, horizontal direction x, vertical direction y, and grid length $c\in {\mathbb{R}}^{+}$. The coordinated ship-welding task is composed of n robots labeled by $i=1,\cdots ,n$ and m weld lines for ship building labeled by $j=1,\cdots ,m$. In this task, n robots are required to weld all m weld lines with $m_s$ synchronous weld lines (SWLs) labeled by $j=1,\cdots ,m_s$ and $m-m_s$ normal weld lines (NWLs) labeled by $j=m_s+1,\cdots ,m$. Each of the SWLs is required to be synchronously executed by a pair of robots, and each of the NWLs can be finished by one robot. The execution of welding is started from the welding starting point $p_j^{*}={[x_j^{*},y_j^{*}]}^T$ for the jth weld line.

In practice, the lengths of each weld line $L_j\in {\mathbb{R}}^{+}$ are not all the same. Meanwhile, according to the literature [5,9], weld lines in a ship-welding task can be placed in different directions, such as vertical (i.e., welding from the top to the bottom of the line, parallel to the y-axis), horizontal (i.e., welding from the right to the left of the line, parallel to the x-axis), and oblique (i.e., 45° to the x- or y-axis). The ideal execution path varies depending on the direction in which the weld is placed. The ideal velocity at which a weld j is executed is $c_j\in {\mathbb{R}}^2$, where $c_j={\left[-c,0\right]}^T$ for the horizontal, $c_j={\left[0,-c\right]}^T$ for the vertical, and $c_j={\left[-c,c\right]}^T$ or $c_j={\left[c,-c\right]}^T$ for the oblique direction. An illustrative example of the coordinated ship-welding task is shown in Figure 1.

Assumption 1.

Each weld line can only be executed once.

The kinematics of the ith ship-welding robot is given by

$$\begin{array}{cc}\hfill p_i(k+1)& =p_i\left(k\right)+u_i\left(k\right),\hfill \end{array}$$

(1)

where $p_i\left(k\right)={\left[x_i\left(k\right),y_i\left(k\right)\right]}^T\in {\mathbb{R}}^2$ denotes its position at time step k, and $u_i\left(k\right)={\left[u_{x_i}\left(k\right),u_{y_i}\left(k\right)\right]}^T\in {\mathbb{R}}^2$ is the motion input and can be selected as move up ($u_i={[0,c]}^T$), down ($u_i={[0,-c]}^T$), left ($u_i={[-c,0]}^T$), right ($u_i={[c,0]}^T$), still ($u_i={[0,0]}^T$), and oblique ($u_i={[c,c]}^T$, $u_i={[-c,c]}^T$, $u_i={[c,-c]}^T$, $u_i={[-c,-c]}^T$), where the oblique motion can only be chosen for the welding procedure of oblique weld lines.

Remark 1.

In the practical deployment of the real-world ship building environment [9], the motion of welding robots for ship building is always restricted on the gantry welding system. As shown in Figure 2, welding robots can only move up, down, left, or right along with the gantry. To address this, this paper mainly focuses on constructing the coordinated ship-welding environment and the motion of robots on the grid world to simulate this process.

Remark 2.

To focus the study on the following dispatch of the welding sequence in the coordinated ship-welding task and the optimization objectives, the robots’ kinematics take the linear grid motion as shown in (1). Noting that robots with nonlinear dynamics can generally be efficiently controlled by existing mature techniques, such as feedback linearization [25], this paper does not put effort into the robot’s complicated dynamics.

Next, two core issues of the coordinated ship-welding task will be put forward, that is, welding sequence dispatch and coordinated ship-welding motion.

Welding sequence dispatch. Due to the existence of multiple robots with multiple weld lines, how to reasonably assign the welding sequence of each robot to the weld lines is crucial to improving the task accomplishment efficiency. The dispatch of robot i can be expressed as ${\chi }_i:\mathbb{R}\to [0,1,\cdots ,m]$, ${\chi }_i\left(0\right)=0$. ${\chi }_i\left(k\right)=j$ means that the ith robot is dispatched to the jth weld line, while ${\chi }_i\left(k\right)=0$ means that it is not dispatched to any weld line.

Coordinated ship-welding motion. On the basis of the weld lines dispatched to robots, since the initial positions of robots are not always exactly at the ideal weld starting point of weld lines, robots need to first move from their initial position to the weld starting point of each weld line before executing the welding process. Before giving the definition of coordinated ship-welding error, first, the completion of welding ${\overline{L}}_{i,j}\left(k\right)={\left[{\overline{L}}_{i,j}^x\left(k\right),{\overline{L}}_{i,j}^y\left(k\right)\right]}^T\in {\mathbb{R}}^2$ associated with the ith robot and jth dispatched weld line at k time is defined, that is,

$$\begin{array}{cc}\hfill {\overline{L}}_{i,j}\left(k\right)=& \left\{\begin{array}{cccc}\hfill & \left(p_i\left(k\right)-p_j^{*}\right)s_i\left(k\right)-{\displaystyle \frac{L_j}{∥c_j∥}}{c}_j,\hfill & \hfill & \mathrm{for}\mathrm{NWLs};\hfill \\ \hfill & \left(\left(p_i\left(k\right)-p_j^{*}\right)s_i\left(k\right)-{\displaystyle \frac{L_j}{∥c_j∥}}{c}_j\right)+\left(p_i\left(k\right)-p_{i^{\prime }}\left(k\right)\right)s_i\left(k\right),\hfill & \hfill & \mathrm{for}\mathrm{SWLs};\hfill \end{array}\right.\hfill \end{array}$$

(2)

where $s_i\left(k\right)$ reflects whether the ith robot reaches the welding starting point at k, and if so, $s_i\left(k\right)=1$, vice versa $s_i\left(k\right)=0$. ${\displaystyle \frac{L_j}{∥c_j∥}}{c}_j$ is the weld path relative to the starting point that needs to be tracked to complete welding of the jth weld line. For NWLs, this value directly portrays the completion of welding, whereas for synchronous welds, the completion of welding is determined by the real-time position between the ith robot and the other $i^{\prime }$th robot that is co-welding with it. This means that the two robots are required to maintain synchronized welding at all times to complete the SWLs.

Based on the welding completion, the coordinated ship-welding error $e_i\left(k\right)\in {\mathbb{R}}^4$ of the ith robot with the dispatched weld $j={\chi }_i\left(k\right)$ at k is

$$\begin{array}{c}\hfill e_i\left(k\right)={\left[\left(x_i\left(k\right)-x_j^{*}\right)\left(1-s_i\left(k\right)\right),\left(y_i\left(k\right)-y_j^{*}\right)\left(1-s_i\left(k\right)\right),{\overline{L}}_{i,j}^x\left(k\right),{\overline{L}}_{i,j}^y\left(k\right)\right]}^T,\end{array}$$

(3)

where the first two terms correspond to the position error between the ith robot and the welding starting point of the dispatched jth weld line before the start of the welding, and the last two terms correspond to the welding completion in (2).

Based on (1), the coordinated ship-welding error dynamics can be expressed as

$$\begin{array}{cc}\hfill e_i(k+1)=& \left[\left(x_i\left(k\right)+u_{x_i}\left(k\right)-x_j^{*}\right)\left(1-s_i(k+1)\right),\left(y_i\left(k\right)+u_{y_i}\left(k\right)-y_j^{*}\right)\left(1-s_i(k+1)\right),\right.\hfill \\ \hfill & {\left.{\overline{L}}_{i,j}^x(k+1),{\overline{L}}_{i,j}^y(k+1)\right]}^T,\hfill \end{array}$$

(4)

where ${\overline{L}}_{i,j}(k+1)={\left[{\overline{L}}_{i,j}^x(k+1),{\overline{L}}_{i,j}^y(k+1)\right]}^T\in {\mathbb{R}}^2$ satisfies

$$\begin{array}{cc}\hfill {\overline{L}}_{i,j}(k+1)=& \left\{\begin{array}{cccc}\hfill & \left(p_i\left(k\right)+u_i\left(k\right)-p_j^{*}\right)s_i(k+1)-{\displaystyle \frac{L_j}{∥c_j∥}}{c}_j,\hfill & \hfill & \mathrm{for}\mathrm{NWLs};\hfill \\ \hfill & \left(\left(p_i\left(k\right)+u_i\left(k\right)-p_j^{*}\right)s_i(k+1)-{\displaystyle \frac{L_j}{∥c_j∥}}{c}_j\right)\hfill & \hfill & \\ \hfill & +\left(p_i\left(k\right)+u_i\left(k\right)-p_{i^{\prime }}\left(k\right)-u_{i^{\prime }}\left(k\right)\right)s_i\left(k+1\right).\hfill & \hfill & \mathrm{for}\mathrm{SWLs};\hfill \end{array}\right.\hfill \end{array}$$

(5)

This paper considers the coordinated ship-welding task under the following two optimization objectives.

2.1. Lazy Robot Ratio

As shown in Figure 1, if SWLs exist in the coordinated ship-welding tasks, more robots need to be kept active such that the welding of SWLs can be finished successfully. In other words, in the most extreme cases when only one robot is active and the others are lazy, none of the SWLs can be finished and the coordinated ship-welding task has failed. Therefore, to keep as many robots as possible active and to prevent the robots from being lazy, the concept of the lazy robot ratio is introduced in this paper.

The lazy robot ratio is defined as the percentage of the number of lazy robots to all robots, where lazy robots are defined as robots that do not have any weld lines dispatched when there are still not any dispatched weld lines. The formulation of the lazy robot ratio at time step k can be expressed as

$$\begin{array}{cc}\hfill R\left(k\right)& \left\{\begin{array}{cccc}\hfill & 0,\hfill & \hfill & \mathrm{if}\mathrm{all}\mathrm{robots}\mathrm{or}\mathrm{weld}\mathrm{lines}\mathrm{have}\mathrm{been}\mathrm{dispatched};\hfill \\ \hfill & {\displaystyle \frac{{\displaystyle {\displaystyle \sum _{i=1}^n}}\left(1-\mathrm{sgn}\left({\chi }_i\left(k\right)\right)\right)}{n}},\hfill & \hfill & \mathrm{else};\hfill \end{array}\right.\hfill \end{array}$$

(6)

where $\mathrm{sgn}\left(·\right)$ is the sign function and ${\chi }_i\left(k\right)$ is the dispatched weld line index for the ith robot. Note that if the ith robot is not dispatched to a weld line, then ${\chi }_i\left(k\right)=0$; otherwise, ${\chi }_i\left(k\right)$ is greater than 0 corresponding to the weld line label, and then $1-\mathrm{sgn}\left({\chi }_i\left(k\right)\right)=0$.

To measure the lazy robot ratio, the dispatch ${\chi }_i\left(k\right)$ for all robots $i=1,\dots ,n$ at each time step needs to be designed and used, and the real-time states of weld lines need to be known (i.e., whether weld lines exist that have not been dispatched to robots). The measure of the lazy robot ratio helps evaluate the working efficiency of robots. When it equals 0 most of the time, it means all robots are put to work and the working efficiency can be high. Therefore, in coordinated ship-welding tasks with multiple robots (more than two), it is required to employ as many available robots as possible to improve overall efficiency.

Then, the objective for optimizing the lazy robot ratio is defined as

$$\begin{array}{c}\hfill \underset{{\chi }_i}{min}{\displaystyle \frac{{\displaystyle \sum _{k=0}^{N_a}}R\left(k\right)}{(N_a+1)n}},\end{array}$$

(7)

where $k=N_a$ denotes the time step at which all weld lines have been dispatched to robots. The above objective indicates the ratio of lazy robots among all the n robots at each time step from $k=0$ to $k=N_a$. After $k=N_a$, all weld lines have been dispatched and $R\left(k\right)$ will be 0. A smaller value of this objective means a higher ratio of robots put to work, which helps guarantee the accomplishment of the coordinated ship-welding task. Minimizing the objective in (7) not only ensures the working efficiency of robots, but guarantees that the scenario of not having enough robots for SWLs will be much less likely to occur.

2.2. Energy Consumption

Based on the coordinated ship-welding error dynamics (4), the optimal coordinated ship-welding energy for the ith robot is defined as

$$J_i^{*}\left(e_i\left(k\right),{u_i}^{*}\left(k\right)\right)=\underset{u_i\left(k\right)}{min}{\displaystyle \sum _{k=0}^{\infty }}{C}_i\left(e_i\left(k\right),u_i\left(k\right)\right),$$

(8)

where ${u_i}^{*}\left(k\right)$ is the optimal motion planner associated with $u_i\left(k\right)\in {\Omega }_u$, ${\Omega }_u$ is the admissible set,

$$C_i\left(e_i\left(k\right),u_i\left(k\right)\right)=e_i^T\left(k\right)I_e{e}_i\left(k\right)+u_i^T\left(k\right)I_u{u}_i\left(k\right)$$

(9)

is the cost function, and $I_e$, $I_u$ are diagonal matrices with constant diagonal elements. In the above equation, the optimization of the $e_i^T\left(k\right)I_e{e}_i\left(k\right)$ term allows the coordinated ship-welding error to converge to 0, i.e., the robot achieves coordinated ship welding of the weld lines; the $u_i^T\left(k\right)I_u{u}_i\left(k\right)$ term reflects the energy required by the planner. In the energy function (8), if the cost function $C_i$ equals 0, energy consumption $J_i^{*}$ will not increase anymore, and its value remains unchanged. To achieve $C_i=0$, in (9), it is noticed that the non-negative items $e_i^T\left(k\right)I_e{e}_i\left(k\right)$ and $u_i^T\left(k\right)I_u{u}_i\left(k\right)$ should both be 0, which requires reducing the welding error $e_i$ to 0 under the designed control input $u_i$. Note that the $u_i$ under consideration is an admissible control, which means it can make $e_i$ converge to 0, and then $u_i$ will also equal 0 to keep the fixed equilibrium point. Therefore, minimizing energy use requires reducing the welding error to 0 under the designed $u_i$.

Remark 3.

The challenges of achieving optimal energy consumption lie in the fact that (1) the optimal motion planner ${u_i}^{*}\left(k\right)$ needs to be designed to achieve both the convergence of $e_i$ and optimal energy consumption $J_i^{*}$ under the assumption that the planner is in the admissible set ${\Omega }_u$; and (2) the optimal planner should be solved by the corresponding discrete-time Bellman function, where its analytical solutions can be hard to obtain. Note that reinforcement learning can solve the above problem online through neural networks to obtain the optimal policy. Therefore, the reinforcement learning-based motion planner is considered in this paper to solve these problems.

Definition 1.

[Coordinated ship-welding task problem.] Considering the coordinated ship-welding task consisted of m weld lines with $m_s$ SWLs and n robots in (1). Suppose Assumption 1 holds. Design welding sequence dispatch input ${\chi }_i$ and motion planner $u_i$ for each robot such that the optimal lazy robot ratio (7) and energy consumption of coordinated ship-welding motion (8) are achieved.

Remark 4.

The coordinated welding problem in this paper differs from the other welding problems in three aspects: (1) Different from the single-robot cases [3,5,7] and two-robot cases [8,9], more than two robots are considered to be used, which can be challenging for dispatching. (2) The coordinated ship-welding motion requires two robots welding a SWL together, which is not considered in [3,5,7,8,9] and brings difficulty to determining the optimal welding sequence. (3) Distinct from the minimal time and energy included in the multi-objective welding problems [7,9], the lazy robot ratio of more than two robots and energy consumption are to be minimized.

3. Methodology of DKM-MFPG

To solve the problem in Definition 1, the DKM-MFPG (dynamic Kuhn–Munkres model-free policy gradient) algorithm based on dispatching and reinforcement learning is proposed, where the DKM (dynamic Kuhn–Munkres) dispatcher uses the position information obtained by wireless sensors to dynamically design the welding sequence dispatch and ensures a low lazy robot ratio, while the MFPG (model-free policy gradient) optimal motion planner yields an energy-optimal planner for the coordinated ship-welding motion based on the dispatch results of the DKM. The DKM-MFPG framework is illustrated in Figure 3.

3.1. DKM Dispatcher

The design of the DKM dispatcher aims to reduce the lazy robot ratio through a dynamic dispatch mechanism that immediately dispatches a weld line that has not yet been dispatched to any robot, to a robot that has not been dispatched to any weld line in real-time. At the same time, considering that the term $e_i^T\left(k\right)I_e{e}_i\left(k\right)$ of the motion energy cost (9) is affected by the welding sequence dispatch, the design of the DKM dispatcher also prioritizes this term with a smaller value, thus providing the basis for subsequent motion design.

To determine whether the ith robot should be dispatched to the jth weld line, a dispatch degree function is first designed as follows:

$$D_{i,j}\left(k\right)={\displaystyle \frac{e_i^T\left(k\right)I_e{e}_i\left(k\right)}{d_i\left(k\right)+\epsilon }},$$

(10)

when the ith robot is not assigned to any weld line at k, $d_i\left(k\right)=1$, otherwise, $d_i\left(k\right)=0$; $\epsilon$ is a very small positive constant. Based on the above equation, the dynamic dispatch mechanism of DKM can be described as follows: when the ith robot is not assigned to any weld line at time step k, the smaller the value of $e_i^T\left(k\right)I_e{e}_i\left(k\right)$ with the jth weld line, the smaller the value of the dispatch degree $D_{i,j}\left(k\right)$. At this time, the smaller the value of the dispatch degree $D_{i,j}\left(k\right)$, the greater the probability that the ith robot is immediately dispatched to the jth weld line; when the ith robot is dispatched to a weld line at k, the value of $D_{i,j}\left(k\right)$ tends to infinity, and the probability that the ith robot will be dispatched to another weld line before completing welding of the current weld line is nearly zero.

Remark 5.

Note that the dispatch degree function (10) includes $e_i\left(k\right)$, which is determined by real-time positions of the jth dispatched weld line and the $i^{\prime }$th co-welding robot associated with the ith robot. According to [26], this position information can be effectively localized and broadcasted by wireless sensor devices among robots. Therefore, the position information is directly used in the DKM dispatcher.

Based on the dispatch degree function (10), the dispatch degree matrix $D\left(k\right)\in {\mathbb{R}}^{n\times (m+m_s)}$ between all the robots and all the weld lines can be obtained as

$$D\left(k\right)=\left[\begin{array}{cc}{D}_{2m_s}\left(k\right)& D_{m-m_s}\left(k\right)\end{array}\right],$$

(11)

where $D_{2m_s}\left(k\right)\in {\mathbb{R}}^{n\times 2m_s}$ is the dispatch degree matrix between n robots and $m_s$ SWLs defined as

$$\begin{array}{c}\hfill D_{2m_s}\left(k\right)=\left[\begin{array}{ccccc}{D}_{1,1}\left(k\right)& D_{1,1}\left(k\right)& \cdots & D_{1,m_s}\left(k\right)& D_{1,m_s}\left(k\right)\\ ⋮& ⋮& \ddots & ⋮& ⋮\\ D_{n,1}\left(k\right)& D_{n,1}\left(k\right)& \cdots & D_{n,m_s}\left(k\right)& D_{n,m_s}\left(k\right)\end{array}\right],\end{array}$$

(12)

which contains n rows and $2m_s$ columns; this is because when dispatching, each SWL requires a row corresponding to a robot in each of its two corresponding columns. The $D_{m-m_s}\left(k\right)\in {\mathbb{R}}^{n\times (m-m_s)}$ is the dispatch degree matrix between n robots and $m-m_s$ NWLs defined as

$$\begin{array}{cc}\hfill & D_{m-m_s}\left(k\right)=\left[\begin{array}{ccc}{D}_{1,m_s+1}\left(k\right)& \cdots & D_{1,m}\left(k\right)\\ ⋮& \ddots & ⋮\\ D_{n,m_s+1}\left(k\right)& \cdots & D_{n,m}\left(k\right)\end{array}\right].\hfill \end{array}$$

(13)

In order to determine the welding sequence dispatch of n robots to m welds based on the dispatch degree matrix, the following lemma is introduced.

Lemma 1.

(Kuhn–Munkres (KM) algorithm [27].) Consider an arbitrary $n\times m$ matrix G with its element $g_{i,j}$ in row $i\in \{1,...,n\}$ and column $j\in \{1,...,m\}$. A set of elements of matrix G are said to be independent if no two of them lie in the same line (the word “line” applies both to the rows and to the columns of matrix G). Therefore, a set of n independent elements ${[g_1^{*},...,g_n^{*}]}^T$ of matrix G are chosen so that the sum of these elements is minimum. The KM algorithm $KM(·):{\mathbb{R}}^{n\times m}\to {\mathbb{R}}^n$ can be denoted by

$$KM\left(G\right)=G^{*},$$

(14)

where $G^{*}={[g_1^{*},...,g_n^{*}]}^T$ is the minimum dispatch for G.

Based on the dispatch degree function (10), the dispatch degree matrix (11), and the KM algorithm (14) with Lemma 1, the DKM dispatcher is designed as

$${\chi }_i\left(k\right)=\left\{\begin{array}{cc}j,& \mathrm{if}{D}_{i,j}\left(k\right)=KM\left(D\left(k\right)\right){|}_i;\\ 0,& \mathrm{else};\end{array}\right.$$

(15)

which means the ith robot is dispatched to the jth weld line if the value of the dispatch degree equals the minimum value of the ith row assignment in $KM\left(D\left(k\right)\right)$ between the ith robot and the jth weld line. From (10) and (15), the relationship between the DKM dispatcher and the resulting allocation can be summarized as the dynamic dispatch in real-time, which guarantees that lazy robots will not exist. This is reflected in $d_i\left(k\right)$ in (10). When the ith robot has a dispatched weld line, $d_i=0$, which means $D_{i,j}\left(k\right)$ is a very large value for any $j=1,\cdots ,m$. On the other hand, when the ith robot has just finished its previously dispatched weld lines and has not been assigned, $d_i=1$, and $D_{i,j}\left(k\right)$ becomes a smaller value. Note that the Kuhn–Munkres algorithm always selects the minimum value of $D_{i,j}$ for the ith robot. That is to say, when $d_i=0$, $D_{i,j}$ tends to infinity such that the ith robot will not be dispatched to a new weld line j until it finishes its current welding task; when $d_i\left(k\right)=1$ the ith robot can be dispatched with a weld line according to the value of $D_{i,j}$.

Proposition 1.

Consider the coordinated ship-welding task consisting of n robots in (1) and $m_s$ SWLs and $m-m_s$ NWLs with different lengths and placement directions. Assume Assumption 1 holds and all robots will complete the welding after being dispatched to a weld line. Then the welding sequence dispatch ${\chi }_i\left(k\right)$ output by Algorithm 1 can achieve the optimal lazy robot ratio objective (7).

Algorithm 1 DKM dispatcher

Initialization At $k=0$, set the number n and the initial positions $p_i\left(0\right)$ of robots. Set the number $m_s$ of SWLs, the number $m-m_s$ of NWLs. Calculate the dispatch degree matrix $D\left(0\right)$ by (11). Determine ${\chi }_i\left(0\right)$ based on (15) and $D\left(0\right)$. for time step $k\in \left[1,\cdots ,N_a\right]$ do     Let $j={\chi }_i(k-1)$.     Calculate the dispatch degree matrix $D\left(k\right)$ by (11).     Determine ${\chi }_i\left(k\right)$ based on (15) and $D\left(k\right)$.     Until all weld lines have been dispatched, end loop. end for Output: Welding sequence dispatch ${\chi }_i\left(k\right)$ between all robots $i=1,\cdots ,n$ and all weld lines $j=1,\cdots ,m$ at $k=0,\cdots ,N_a$.

Proof. 

When the number of robots is greater than the sum of the number of welding robots required for all weld lines, the weld sequence can be determined by one-step dispatch; otherwise, multi-step dispatch is required. For both cases, the achievement of objective (7) is analyzed here.

Case 1. The number of robots is greater than the sum of the number of welding robots required for all weld lines, that is, $n\ge m+m_s$. In this case, the DKM dispatcher can assign all the weld lines to the robots in one step. According to the definition, lazy robots are defined as robots that do not perform welding tasks when there are still not dispatched and unfinished weld lines. Therefore, $R\left(0\right)=0$ holds.

Case 2. In the case where there are fewer robots than weld lines, there are three kinds of possible states for dispatching: (1) Initial state at time step 0. At this state, all robots are dispatched with a weld line, which means no lazy robots exist and $R\left(0\right)=0$; (2) Intermediate state between 0 and the last time step for dispatching. During this time, once a robot has finished its previously dispatched weld line, the DKM dispatcher instantly dispatches a new weld line for it. This means during this time, robots always have their dispatched weld lines; thus, no lazy robots exist and $R\left(k\right)=0$; and (3) Final state when all weld lines have been dispatched at $k=N_a$. As the last unwelded weld line has been dispatched to a robot, there is no weld line to be welded for robots just as in Case 1. To this end, lazy robots no longer exist from $k=N_a$, and $R\left(k\right)$ will remain 0 from now on. An example of the DKM dispatcher for this procedure can be found in Figure 4, where five robots and nine weld lines including a SWL labeled by 1 are dispatched, and the robots are all dispatched to a weld line until the final state with the lazy robot ratio equal to 0.

To sum up, the lazy robot ratio is always kept as $R\left(k\right)=0$ under the DKM dispatcher, which is the minimal value for the non-negative variable. Therefore, it is concluded that the optimal lazy robot ratio can also be achieved in this case.    □

3.2. MFPG Motion Planner

In this subsection, the robot will be set with state, action, and cost functions based on the MFPG (model-free policy gradient) reinforcement learning method according to the real-time weld sequence dispatch of the DKM dispatcher. Then, the Q-network (action-value network) and the actor network are constructed to obtain the energy-optimal coordinated ship-welding motion planner. Meanwhile, inspired by the literature [28,29], convergence proof of network weights based on the Lyapunov function are given.

State. The state of the ith robot is defined as the coordinated ship-welding error (3), where the jth weld line is dispatched to the ith robot according to the DKM dispatcher. Note that since the state values may have duplicate values under different dispatched weld lines when the same robot has participated in multiple dispatches, the above states are accompanied by a label of the dispatched weld line number to differentiate them during training.

Action. The definition of the action of robot i is consistent with the coordinated ship-welding motion control input $u_i\left(k\right)\in {\mathbb{R}}^2$ in Equation (1).

Cost Function. The cost function for the ith robot is defined as the cost function in the optimal coordinated ship-welding motion energy (8), i.e., Equation (9).

Define $J_i\left(e_i\left(k\right)\right)={\displaystyle \sum _{k=0}^{\infty }}{C}_i\left(e_i\left(k\right),u_i\left(k\right)\right)$ to be the coordinated ship-welding motion energy function corresponding to the cost function. According to Bellman’s principle of optimality [17], the optimal coordinated ship-welding motion energy function should satisfy the following discrete-time Bellman function, that is,

$$J_i^{*}\left(e_i\left(k\right)\right)=\underset{u_i\left(k\right)}{min}\left(C_i\left(e_i\left(k\right),u_i\left(k\right)\right)+J_i^{*}\left(e_i\left(k+1\right)\right)\right).$$

(16)

The above optimal value is achieved by designing the optimal action policy

$$u_i^{*}\left(k\right)=\underset{u_i\left(k\right)}{argmin}\left(C_i\left(e_i\left(k\right),u_i\left(k\right)\right)+J_i^{*}\left(e_i\left(k+1\right)\right)\right).$$

(17)

Notice that the cost function is affected by both the state and action of the robot, so to more accurately reflect the relationship between the cost function and the coordinated ship-welding motion energy function, the optimal Q-function (action-value function) is used in place of the optimal coordinated ship-welding motion energy function in (16), i.e.,

$$Q_i^{*}\left(e_i\left(k\right),u_i\left(k\right)\right)=C_i\left(e_i\left(k\right),u_i\left(k\right)\right)+Q_i^{*}\left(e_i\left(k+1\right),u_i\left(k+1\right)\right).$$

(18)

The optimal action policy to achieve (18) is

$$u_i^{*}\left(k\right)=\underset{u_i\left(k\right)}{argmin}{Q}_i^{*}\left(e_i\left(k\right),u_i\left(k\right)\right).$$

(19)

To approximate the unknown Q in (18) and $u_i^{*}$ in (19), the Q-network and actor network are designed as follows.

Q-network. Let $W_{c_i}^{*}\in {\mathbb{R}}^{h_c}$ be the optimal weight parameter of the Q-network for the ith robot. Then the optimal Q-function can be approximated as

$$Q_i^{*}\left(e_i\left(k\right),u_i\left(k\right)\right)=W_{c_i}^{*T}{\psi }_{c_i}\left({\sigma }_{c_i}^T{z}_i\left(k\right)\right)+{\delta }_{c_i},$$

(20)

where ${\psi }_{c_i}=tanh(·)$ is the activation function of the Q-network, and $h_c$ is the number of nodes in the hidden layer; ${\sigma }_{c_i}\in {\mathbb{R}}^{6\times h_c}$ is the function of the network from the input layer to the hidden layer, $z_i\left(k\right)={\left[e_i^T,u_i^T\right]}^T$, and ${\delta }_{c_i}$ is the network reconstruction error.

Actor network. Let $W_{a_i}^{*}\in {\mathbb{R}}^{h_a\times 2}$ be the optimal weight parameter of the actor network of the ith robot. Then the optimal action policy can be expressed as

$$u_i^{*}\left(e_i\left(k\right)\right)=W_{a_i}^{*T}{\psi }_{a_i}\left({\sigma }_{a_i}^T{e}_i\left(k\right)\right)+{\delta }_{a_i},$$

(21)

where ${\psi }_{a_i}=tanh(·)$ is the activation function of the actor network, and $h_a$ is the number of nodes in the hidden layer; ${\sigma }_{a_i}\in {\mathbb{R}}^{2\times h_a}$ is the function of the actor network’s input layer to the hidden layer, and ${\delta }_{a_i}$ is the reconstruction error.

Let ${\widehat{W}}_{c_i}$, ${\widehat{W}}_{a_i}$ be the estimates of $W_{c_i}^{*}$, $W_{a_i}^{*}$, respectively. Then the estimate network of the optimal Q-function and the optimal action policy is constructed as

$$\begin{array}{cc}\hfill {\widehat{Q}}_i\left(e_i\left(k\right),u_i\left(k\right)\right)& ={\widehat{W}}_{c_i}^T{\psi }_{c_i}\left({\sigma }_{c_i}^T{z}_i\left(k\right)\right),\hfill \end{array}$$

(22)

$$\begin{array}{cc}\hfill {\widehat{u}}_i\left(e_i\left(k\right)\right)& ={\widehat{W}}_{a_i}^T{\psi }_{a_i}\left({\sigma }_{a_i}^T{e}_i\left(k\right)\right).\hfill \end{array}$$

(23)

To train the Q-network, the following training error function is defined:

$$E_{c_i}={\displaystyle \frac{1}{2}}{\epsilon }_{c_i}^2,$$

(24)

where ${\epsilon }_{c_i}=C_i\left(e_i\left(k\right),u_i\left(k\right)\right)+{\widehat{W}}_{c_i}^T\Delta {\psi }_{c_i},\Delta {\psi }_{c_i}={\psi }_{c_i}\left({\sigma }_{c_i}^T{z}_i\left(k+1\right)\right)-{\psi }_{c_i}\left({\sigma }_{c_i}^T{z}_i\left(k\right)\right)$. To minimize the above error function, the following gradient descent update is applied to the weights of the Q-network:

$$\begin{array}{cc}\hfill {\widehat{W}}_{c_i}\left(k+1\right)=& {\widehat{W}}_{c_i}\left(k\right)-{\alpha }_{c_i}{\displaystyle \frac{\partial E_{c_i}}{\partial {\widehat{W}}_{c_i}\left(k\right)}}\hfill \\ \hfill =& {\widehat{W}}_{c_i}\left(k\right)-{\alpha }_{c_i}{\epsilon }_{c_i}\Delta {\psi }_{c_i},\hfill \end{array}$$

(25)

where ${\alpha }_{c_i}$ is a positive learning rate constant.

For the actor network, the weight parameters are updated using the policy gradient method, i.e.,

$$\begin{array}{cc}\hfill {\widehat{W}}_{a_i}\left(k+1\right)=& {\widehat{W}}_{a_i}\left(k\right)-{\alpha }_{a_i}{\displaystyle \frac{\partial {\widehat{Q}}_i\left(e_i\left(k\right),u_i\left(k\right)\right)}{\partial {\widehat{W}}_{a_i}\left(k\right)}}\hfill \\ \hfill =& {\widehat{W}}_{a_i}\left(k\right)-{\alpha }_{a_i}{\widehat{W}}_{c_i}^T\left(k\right)\nabla {\psi }_{c_i}{\sigma }_{c_i}^T{\eta }_i{\psi }_{a_i}\left({\sigma }_{a_i}^T{e}_i\left(k\right)\right),\hfill \end{array}$$

(26)

where $\nabla {\psi }_{c_i}={\displaystyle \frac{\partial {\psi }_{c_i}}{\partial z_i\left(k\right)}}$; ${\eta }_i={\displaystyle \frac{\partial z_i\left(k\right)}{\partial {\widehat{u}}_i\left(k\right)}}$; ${\alpha }_{a_i}$ is a positive learning rate constant.

Proposition 2.

Consider the coordinated ship-welding task consisting of n robots (1), $m_s$ SWLs, and $m-m_s$ NWLs with different lengths and placement directions. Suppose Assumption 1 holds and that all the weld lines can be dispatched to the corresponding robots by Proposition 1, then the optimal coordinated ship-welding motion energy of all robots (8) can be achieved by Algorithm 2 under ${\widehat{u}}_i$. Meanwhile, the weight estimation errors ${\tilde{W}}_{c_i}={\widehat{W}}_{c_i}-W_{c_i}^{*}$ and ${\tilde{W}}_{a_i}={\widehat{W}}_{a_i}-W_{a_i}^{*}$ for the Q-network and the actor network are uniformly ultimately bounded (UUB).

Algorithm 2 MFPG motion planner

Initialize the weight parameters ${\widehat{W}}_{c_i}$, ${\widehat{W}}_{a_i}$ of the Q-network and the actor network, as well as the coordinated ship-welding error $e_i\left(0\right)$, the action $u_i\left(0\right)$, and the cost function $C_i\left(e_i\left(0\right),u_i\left(0\right)\right)$. for $k=0$ to N do     Obtain the welding sequence dispatch ${\chi }_i\left(k\right)$ by using Algorithm 1.     Obtain the coordinated ship-welding error $e_i\left(k\right)$ according to (4), and the cost function value $C_i\left(k\right)$ according to (9).     for $i=1$ to n do         Update the weight parameters ${\widehat{W}}_{c_i}$, ${\widehat{W}}_{a_i}$ of the Q-network and actuator network according to (25) and (26), respectively.     end for     Obtain the position of the next time step $p_i\left(k\right)$ according to (1) and the action $u_i\left(k\right)$. end for Output: Optimal coordinated ship-welding motion planner ${\widehat{u}}_i$ (23) for all robots $i=1,\cdots ,n$.

Proof. 

Consider the candidate Lyapunov function

$$L\left(k\right)=L_1\left(k\right)+L_2\left(k\right),$$

(27)

where $L_1\left(k\right)={\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{{\alpha }_{c_i}}}{\tilde{W}}_{c_i}^T{\tilde{W}}_{c_i}$, $L_2\left(k\right)={\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{{\alpha }_{a_i}}}{\tilde{W}}_{a_i}^T{\tilde{W}}_{a_i}$.

Let ${\varphi }_{c_i}={\tilde{W}}_{c_i}^T\Delta {\psi }_{c_i}$. Take the first difference of $L_1\left(k\right)$ along ${\tilde{W}}_{c_i}$ while using Young’s inequality and the updated law of the Q-network (25) yields

$$\begin{array}{cc}\hfill \Delta L_1\left(k\right)=& {\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{{\alpha }_{c_i}}}{\tilde{W}}_{c_i}^T\left(k+1\right){\tilde{W}}_{c_i}\left(k+1\right)-{\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{{\alpha }_{c_i}}}{\tilde{W}}_{c_i}^T\left(k\right){\tilde{W}}_{c_i}\left(k\right)\hfill \\ \hfill =& {\displaystyle \sum _{i=1}^n}\left(-2{\tilde{W}}_{c_i}^T\Delta {\psi }_{c_i}\left(C_i+{\tilde{W}}_{c_i}^T\Delta {\psi }_{c_i}+W_{c_i}^{*}\Delta {\psi }_{c_i}\right)\right.\hfill \\ \hfill & \left.+{\alpha }_{c_i}{∥\Delta {\psi }_{c_i}∥}^2{∥C_i+{\tilde{W}}_{c_i}^T\Delta {\psi }_{c_i}+W_{c_i}^{*}\Delta {\psi }_{c_i}∥}^2\right)\hfill \\ \hfill \le & {\displaystyle \sum _{i=1}^n}\left(-{∥{\varphi }_{c_i}∥}^2+{∥C_i+W_{c_i}^{*}\Delta {\psi }_{c_i}∥}^2\right),\hfill \end{array}$$

(28)

which holds when $0<{\alpha }_{c_i}\le {\displaystyle \frac{1}{\Delta {\overline{\psi }}_{c_i}^2}}$, where ${∥\Delta {\psi }_{c_i}∥}^2\le \Delta {\overline{\psi }}_{c_i}^2$ is bounded.

Also, let ${\varphi }_{a_i}={\tilde{W}}_{a_i}^T{\psi }_{a_i}$. Take the first difference of $L_2\left(k\right)$ along ${\tilde{W}}_{a_i}$ while using Young’s inequality and substituting the updated law of the actor network (26) yields

$$\begin{array}{cc}\hfill \Delta L_2\left(k\right)=& \sum _{i=1}^n{\displaystyle \frac{1}{{\alpha }_{a_i}}}{\tilde{W}}_{a_i}^T\left(k+1\right){\tilde{W}}_{a_i}\left(k+1\right)-{\displaystyle \sum _{i=1}^n}{\displaystyle \frac{1}{{\alpha }_{a_i}}}{\tilde{W}}_{a_i}^T\left(k\right){\tilde{W}}_{a_i}\left(k\right)\hfill \\ \hfill =& {\displaystyle \sum _{i=1}^n}\left(-2{\varphi }_{a_i}{\widehat{W}}_{c_i}^T\nabla {\psi }_{c_i}{\sigma }_{c_i}^T{\eta }_i+{\alpha }_{a_i}{∥{\widehat{W}}_{c_i}^T\nabla {\psi }_{c_i}{\sigma }_{c_i}^T{\eta }_i{\psi }_{a_i}∥}^2\right)\hfill \\ \hfill \le & {\displaystyle \sum _{i=1}^n}\left(-\left(1-{\alpha }_{a_i}{∥\nabla {\psi }_{c_i}∥}^2{∥{\sigma }_{c_i}∥}^2{∥{\psi }_{a_i}∥}^2\right){∥{\widehat{W}}_{c_i}∥}^2\right.\hfill \\ \hfill & \left.+2{∥{\tilde{W}}_{c_i}∥}^2+2{∥W_{c_i}^{*}∥}^2+{∥\nabla {\psi }_{c_i}{\sigma }_{c_i}^T{\eta }_i{\varphi }_{a_i}∥}^2\right).\hfill \end{array}$$

(29)

Combining (28) with (29) yields

$$\Delta L\left(k\right)\le {\displaystyle \sum _{i=1}^n}\left(-{∥{\varphi }_{c_i}∥}^2-\left(1-{\alpha }_{a_i}{∥\nabla {\psi }_{c_i}∥}^2{∥{\sigma }_{c_i}∥}^2{∥{\psi }_{a_i}∥}^2\right){∥{\widehat{W}}_{c_i}∥}^2+\overline{M}\right),$$

(30)

where $\overline{M}={∥C_i+W_{c_i}^{*}\Delta {\psi }_{c_i}∥}^2+2{∥{\tilde{W}}_{c_i}∥}^2+2{∥W_{c_i}^{*}∥}^2+{∥\nabla {\psi }_{c_i}{\sigma }_{c_i}^T{\eta }_i{\varphi }_{a_i}∥}^2$ is bounded. By selecting $0<{\alpha }_{c_i}\le {\displaystyle \frac{1}{\Delta {\overline{\psi }}_{c_i}^2}}$, $0<{\alpha }_{a_i}\le {\displaystyle \frac{1}{{∥\nabla {\psi }_{c_i}∥}^2{∥{\sigma }_{c_i}∥}^2{∥{\psi }_{a_i}∥}^2}}$, one has $∥{\widehat{W}}_{c_i}∥\ge \sqrt{{\displaystyle \frac{M}{1-{\alpha }_{a_i}{∥\nabla {\psi }_{c_i}∥}^2{∥{\sigma }_{c_i}∥}^2{∥{\psi }_{a_i}∥}^2}}}$, $\Delta L\left(k\right)<0$, which implies the weight estimate errors ${\tilde{W}}_{c_i}$, ${\tilde{W}}_{a_i}$ of the Q-network and the actor network are UUB. Then the optimal motion planner ${\widehat{u}}_i$ of the actor network outputted by Algorithm 2 can approximate the optimal action policy $u_i^{*}$, which in turn achieves the optimal Q-function value corresponding to the optimal coordinated ship-welding motion energy. Therefore, Algorithm 2 can effectively achieve the energy consumption objective of coordinated ship welding (8). □

Remark 6.

The MFPG motion planner is specifically designed for the coordinated ship-welding task problem, where the state and action of each robot are based on the DKM dispatcher-based welding error and the motion in the grid-world environment, and the cost function is defined corresponding to the energy consumption optimization objective. Compared to the other reinforcement learning methods [22,23,24] applied to the robot energy optimization problems, our method has the following advantages: (1) the state of the welding error is novelly defined such that the coordinated ship-welding motion for normal and synchronous weld lines can both be well-controlled and achieved with optimal energy consumption; (2) the model-free update of the Q-network and actor network can be faster and more efficient, as the model dynamics are not calculated for the network updating procedure; and (3) the integration of the Q-network and the policy-gradient-updated actor network can simultaneously evaluate the energy cost and optimize the motion planner, and the network convergence is theoretically guaranteed to improve the learning stability.

Remark 7.

The specific welding process requirement of synchronous weld lines is originated from the practical application, which requires a pair of robots such that one holds a soldering gun, and at the same time, the other holds a soldering tin to cooperate with each other. To deal with this, the coordinated welding error is constructed, and its convergence and energy optimization are achieved by our MFPG motion planner.

From Propositions 1 and 2, we now give the main result as follows.

Theorem 1.

Consider the coordinated ship-welding task consisting of n robots (1), $m_s$ SWLs, and $m-m_s$ NWLs with different lengths and placement directions. Suppose Assumption 1 holds. Then, the optimal lazy robot ratio (7) and coordinated ship-welding motion energy (8) objectives can be solved by the welding sequence dispatch ${\chi }_i\left(k\right)$ of the DKM dispatcher in Proposition 1 and the actor putput ${\widehat{u}}_i$ of the MFPG motion planner in Proposition 2.

Proof. 

According to the conclusions of Propositions 1 and 2, the welding sequence dispatch ${\chi }_i\left(k\right)$ of the DKM dispatcher and the optimal coordinated ship-welding motion planner ${\widehat{u}}_i$ of the MFPG can effectively achieve the optimal lazy robot ratio (7) and energy consumption (8) objectives, respectively. Therefore, the coordinated ship-welding optimal task problem in Definition 1 can be solved successfully. □

Remark 8.

The trade-offs between keeping robots active and minimizing energy use are balanced by first guaranteeing the optimal lazy robot ratio via the DKM dispatcher, and then achieving the optimal energy under this dispatch for all robots via the MFPG motion planner. This is due to more than two robots existing and hence the working efficiency is a prime consideration. The emphasis on the lazy robot ratio ensures as many robots are put to work as possible and hence how to optimize energy consumption becomes meaningful.

4. Numerical Simulation

In this section, two numerical simulations are set up, in which Simulation 1 aims to verify the effectiveness of the DKM-MFPG algorithm in multiple coordinated ship-welding scenarios, and Simulation 2 sets up a comparison between the DKM-MFPG algorithm and multiple single-objective optimization and multi-objective optimization algorithms, so as to demonstrate the advantages of the proposed method in this paper by analyzing the values of the optimization objectives as well as the performance indexes.

4.1. Setting of Test Scenarios

The test scenarios are based on a $20\times 20$ grid world where the length of each grid is $c=0.05$. The execution welding speed is set to $e_{T_j}=0.05$ for vertically and horizontally placed welds, and $e_{T_j}=0.0705$ for obliquely placed welds. The time step of the simulation is capped at $k=100$. Five representative test scenarios corresponding to actual welding tasks are listed in

Table 1. Specifications of 5 coordinated ship welding scenarios.

Scenarios	Robot Number	Weld Line Number	Weld Line Length	Weld Line Placement
S1	4	6	$0.2$	Vertical
S2	5	12	$0.1$, $0.2$, $0.3$	Vertical
S3	5	12	$0.2$	Horizontal
S4	5	13	$0.1$, $0.2$, $0.3$	Vertical and Horizontal
S5	5	13	$0.141$, $0.2$, $0.282$, $0.423$	Vertical and Slope

, where the specific settings vary for different test objectives, i.e., the number of robots (S1 vs. S2–S5) and the number of weld lines (S1 vs. S2, S3 and S4, S5), line length (S1, S3 vs. S2, S4 and S5) and placement directions (S1, S2 vs. S3, S4 and S5). Specifically, the five test scenarios cover the following three situations:

(1)

Variant number of robots and weld lines.The number of robots and weld lines is the most important key point for affecting the performance of the proposed algorithm. Note that in the case of the number of robots larger than that of the weld lines, the dispatch can be easily determined by calculating the task accomplishment time in one step with a deterministic lazy robot ratio, which can be neglected. To comprehensively verify the performance of algorithms, two cases are considered: (I) the small scene case where the numbers of robots and weld lines are both small, e.g., S1; (II) the number of robots is smaller than that of the weld lines, e.g., S2–S5, where more than 10 weld lines are set since 10 weld lines cover at least 20 grids occupying most of the area of the grid world with $20\times 20$. The comparison between S1 and S2–S5 is given to test the performance associated with the variant robot number, and at the same time, the comparisons between S1, S2 and S3, S4 and S5 are given to test the effectiveness under the variant number of weld lines.

(2)

Variant length of weld lines. Note that the finishing time of a weld line executed by one robot is calculated by the robot’s initial position toward the welding starting point and the weld length. Therefore, the variant length of weld lines can lead to numerous different dispatching solutions when other settings remain unchanged. Specifically, S1 and S3 have the same length of weld lines, whereas the different lengths are set for comparison in S2, S4, and S5.

(3)

Variant placement of weld lines. S1 and S2 show the general case of vertical placement, whereas S3 considers the case of horizontal placement. S4 and S5 test the variant placements including the combination of different directions.

The variant settings aim to cover the utmost possible scenarios for the coordinated ship-welding task problem in practice. The settings of variant number and length of weld lines are common in [3,5,7,8,9]. The setting of variant placement of weld lines is considered in [5,9] with different starting and ending positions, which is a more complex dispatching situation. It is noted that the number setting of robots is first under consideration in accordance with the requirement of synchronous welding. It has the potential to discover an efficient way of multiple welding robots with a low lazy robot ratio.

The test of these scenarios validates that our proposed method is applicable to various settings of coordinated ship-welding tasks, including the cases of (1) small and simple scenes with few robots and weld lines; (2) large and simple scenes with more than two robots and more than ten weld lines; (3) small but complex scenes with few robots and weld lines, where the lengths and placement of weld lines are different; and (4) large and complex scenes with numerous robots and different types of weld lines. Except for the above coordinated ship-welding scenarios, DKM-MFPG also has the potential to be applied to problems that are constructed on the grid-world environment and require coordination between robots with optimization objectives.

Remark 9.

In the literature [3,5,7,8,9], the spot welding problem can be regarded similarly to the traveling salesman problem (TSP) [30] as the benchmark problem. However, the TSP only considers the single-salesman case with single-objective optimization, e.g., the time cost or the distance cost, which can only be used to represent single-robot welding. Furthermore, the length and placement settings of different weld lines are not considered in this benchmark. As a result, the methods in the TSP are not applicable to this paper. Another potential benchmark problem, that is, the multi-objective multiple traveling salesman problem (MOmTSP) [31], employs multiple salesmen and optimizes multiple objectives. To our knowledge, complex situations such as synchronous welding (that is, two robots are required for welding one weld line simultaneously) and welding settings of different lengths and placements have rarely been discussed in MOmTSP. Therefore, the five test scenarios are designed to justify the testing effectiveness of the proposed algorithm by taking into account the following specific settings: the variant number of robots and weld lines, variant length, and placement direction of weld lines.

The simulation software used for testing the S1 to S5 scenarios is based on Python and OpenAI Gym environment [32], where the scenarios are constructed on the grid world. The positions of weld lines and robots, and the dynamics of robots are defined and initialized within the grid-world environment. The algorithms are introduced by (1) obtaining the position and related data from the software environment; (2) calculating the welding sequence dispatch of the DKM dispatcher based on the position data; (3) determining the state, action, cost function, and network updates of the MFPG motion planner based on the welding sequence dispatch and positions; and (4) applying the planner for the environment at every step until obtaining all the required resulted data. The outcome form for the resulted data contains (1) arrays of each robot’s trajectory in the environment; (2) arrays of the number of the lazy robots; and (3) arrays of the state, action, cost function, and network weights of the robots at every time step.

4.2. Simulation 1: Effectiveness Validation

The parameters of DKM-MFPG are given as follows. The learning rates are ${\alpha }_{c_i}=0.1$ and ${\alpha }_{a_i}=0.01$. The number of hidden units of Q and actor networks is 6, and their weights are initialized randomly within $\left(0,1\right)$ and $\left(-1,0\right)$, respectively. The constant elements of the diagonal matrices $I_s$, $I_u$ in (9) are set as 1.

The robot motion trajectories for S1–S5 are plotted in Figure 5, and the results show that the robots successfully complete the welding of all weld lines in these scenarios under the DKM-MFPG algorithm. From Figure 6, it can be seen that the number of non-lazy robots equals the initial setting number of robots in each scenario until the last dispatched time step $k=N_a$ in all scenarios, which implies that the lazy robot ratio is always 0 and the optimization objective of the lazy robot ratio in (7) as well as Proposition 1 is validated. The curves of state, action, cumulative energy values, and weights of the network for all scenarios under the DKM-MFPG algorithm are plotted in Figure 7. According to the simulation result in Figure 7, the curves of Q-network weights remain unchanged and the energy cost curves also plateau simultaneously, which validates the accomplishment of the desired training performance. It can be seen that the above curves converge gradually in all the scenarios, which verifies the conclusion of Proposition 2 that the motion energy optimization objective (8) of coordinated ship welding is accomplished. Therefore, combining the above results, as stated in Theorem 1, DKM-MFPG can effectively complete the coordinated ship-welding task problem in Definition 1.

4.3. Simulation 2: Comparison Simulation

Comparison baselines. To analyze the performance of the DKM-MFPG algorithm by comparison, four baseline algorithms are set, namely, GA [4] (genetic algorithm) and MA [6] (market-based allocation) based on single-objective optimization, and NSGA-III [12] and AnD [13] algorithms based on multi-objective optimization. It is noted that there have been many attempts to successfully apply single-objective optimization methods (e.g., some evolutionary algorithms) to multi-objective optimization problems, as indicated in the literature [33]. Single-objective optimization methods can yield solutions with better objective values in a shorter execution time, or perhaps even better solutions among multiple objectives for a given objective value. Therefore, to test the effectiveness of different algorithms more comprehensively, single-objective optimization methods such as GA and MA are also set as baselines in this section.

Consider that there are two optimization objectives in this chapter, i.e., lazy robot ratio and energy consumption for coordinated ship welding. To deal with the problem of balancing the two optimization objectives in a single-objective optimization method, a “weighted sum method” [34] can be used to combine each objective function into a single unified function, i.e.,

$$z=w_1{z}_1+w_2{z}_2,$$

(31)

where z is the composite function, $z_1$ and $z_2$ are the two objectives, and $w_1$ and $w_2$ are the weights selected as ranging from $0.1$ to $0.9$ with interval $0.1$ such that $w_1+w_2=1$ as performed in [35]. For GA, NSGA-III, and AnD, the population size, the crossover rate, the mutation rate, and the number of iterations are set as 200, $r_c=0.6$, $r_m=0.1$, and 200, respectively. The fitness function and cost of bid for GA and MA are selected as the sum of the lazy robot ratio and energy consumption as in (31).

Performance metrics. In addition to the optimization objectives of lazy robot ratio and the evaluation of energy consumption, the performance metrics of the coverage rate (C-metric) and the execution time have been added to better evaluate the performance between different algorithms.

C-metric [36]: C-metric is used to compare the rate of non-dominated solutions between algorithms A and B, which is calculated as

$$C\left(A,B\right)={\displaystyle \frac{\left|\left\{b\in B;\exists a\in A:a\prec b\right\}\right|}{\left|B\right|}},$$

(32)

where $\left|·\right|$ denotes the number of elements in a set; a and b denote the solution obtained by A and B, respectively; $a\prec b$ means a dominates b. From (32), the larger $C\left(A,B\right)$ is, the more solutions of B dominated by at least one solution of A exist. If $C\left(A,B\right)=1$, all solutions of B are dominated by at least one solution in A; if $C\left(A,B\right)=0$, no solution of B is dominated by any solution of A. Note that $C\left(A,B\right)$ is not necessarily equal to $1-C\left(B,A\right)$; thus, both the $C\left(A,B\right)$ and $C\left(B,A\right)$ are calculated. If $C\left(A,B\right)$ is large and $C\left(B,A\right)$ is small, A is better than B, in a sense.

Execution time [37]: The execution time is defined as the time from the start point in producing solutions to the final time when no more solutions are obtained or when the iteration number reaches the limit by executing an algorithm. The shorter the execution time, the stronger the possibility of the competitiveness of the algorithm for real-world applications.

The numerical simulation results of the Pareto front of all methods under S1–S5 are given in Figure 8, and the average objective values calculated by the solutions of the Pareto front are shown in

Table 2. Average objective values of all methods under S1–S5.

Average Objective Values	Scenarios	GA	MA	NSGA-III	AnD	DKM- MFPG
Average Energy Consumption	S1	5.31	5.17	4.96	5.17	5.66
	S2	10.11	10.38	10.27	9.82	9.52
	S3	9.84	10.62	10.21	9.72	10.01
	S4	15.53	15.73	15.09	15.18	14.69
	S5	36.90	35.70	37.20	37.45	37.44
	Average	15.54 (+0.47%)	15.52 (+0.36%)	15.55 (+0.53%)	15.47 (+0.03%)	15.46
Average Lazy Robot Ratio	S1	9.79%	12.50%	23.19%	14.86%	0.00%
	S2	21.88%	21.22%	6.54%	22.02%	0.00%
	S3	4.19%	3.25%	1.62%	3.24%	0.00%
	S4	9.53%	6.86%	7.11%	6.69%	0.00%
	S5	11.09%	11.12%	2.90%	5.17%	0.00%
	Average	11.30% (+11.30%)	10.99% (+10.99%)	8.27% (+8.27%)	10.39% (+10.39%)	0.00%

. It is noted that the diverse solutions of single-objective optimization methods are obtained by tuning different sets of weights for calculating the sum of the two objectives as the objective/fitness function, and the non-dominated solutions can then be picked out just like the Pareto front obtained by the multi-objective optimization methods. DKM-MFPG only has one certain solution due to the fact that the DKM dispatcher has determined the minimal lazy robot ratio dispatch; meanwhile, MFPG directly drives robots to reach the corresponding path with optimal energy. The values of the C-metric and the execution time are given in

Table 3. C-metric values between methods under S1–S5.

Scenarios	$\mathit{C}\left(\mathit{A},\mathit{B}\right)$	GA	MA	NSGA-III	AnD	DKM-MFPG
S1	GA	0.00	0.00	0.00	0.00	0.00
	MA	0.00	0.00	0.00	0.00	0.00
	NSGA-III	0.00	0.00	0.00	0.00	0.00
	AnD	0.00	0.00	0.00	0.00	0.00
	DKM-MFPG	0.00	0.00	0.00	0.00	0.00
S2	GA	0.00	0.00	0.00	0.00	0.00
	MA	0.00	0.00	0.00	0.00	0.00
	NSGA-III	0.33	0.33	0.00	0.00	0.00
	AnD	0.33	0.33	0.00	0.00	0.00
	DKM-MFPG	0.67	1.00	1.00	0.67	0.00
S3	GA	0.00	0.67	0.33	0.25	0.00
	MA	0.33	0.00	0.00	0.00	0.00
	NSGA-III	0.33	0.00	0.00	0.25	0.00
	AnD	0.67	1.00	0.67	0.00	0.00
	DKM-MFPG	0.33	0.67	0.33	0.50	0.00
S4	GA	0.00	0.75	0.00	0.50	0.00
	MA	0.33	0.00	0.00	0.00	0.00
	NSGA-III	1.00	0.75	0.00	0.75	0.00
	AnD	0.33	0.75	0.33	0.00	0.00
	DKM-MFPG	0.67	0.75	0.67	0.75	0.00
S5	GA	0.00	0.00	0.00	0.00	0.00
	MA	0.33	0.00	0.00	0.00	0.00
	NSGA-III	0.33	0.00	0.00	0.33	0.00
	AnD	0.33	0.00	0.00	0.00	0.00
	DKM-MFPG	0.33	0.00	0.33	0.33	0.00

and

Table 4. Execution time between methods under S1–S5 (Unit: second).

	GA	MA	NSGA-III	AnD	DKM-MFPG
S1	8.75	4.02	21.98	80.97	1.46
S2	14.68	4.86	25.38	86.15	2.28
S3	14.10	4.39	24.19	83.71	1.68
S4	16.23	5.01	25.87	87.62	2.43
S5	16.97	5.28	26.85	89.91	3.09
Average	14.15 (+84.53%)	4.71 (+53.57%)	24.85 (+91.20%)	85.67 (+97.45%)	2.19

, respectively.

Comparison of DKM-MFPG with baselines. From Figure 8a, all methods find nearly the same non-dominated solution in S1. This is because S1 only contains four robots and six weld lines, a small scenario with limited exploration space. From Figure 8b–e with more than ten weld lines in S2–S5, more than one solution can be obtained by the other four methods due to the variant number of robots and weld lines, and variant length and placement direction of weld lines. From Figure 8 and Table 2, it is indicated that the solution of DKM-MFPG outperforms most of the solutions of the other methods on the lazy robot ratio, maintaining approximate performance in terms of the energy consumption compared to the other methods. Specifically, DKM-MFPG reduces the average lazy robot ratio by 11.30%, 10.99%, 8.27%, and 10.39% compared to baselines, respectively. The reason for this lies in the fact that the DKM dispatcher can dynamically and immediately dispatch new tasks for robots that have just finished their previous tasks and tend to be lazy; thus, the lazy robot ratio can be decreased.

Moreover, from Table 3, except for S1, more than 80% of the C-metric values of DKM-MFPG with respect to the other four methods are over 0, and nearly all the C-metric values of the other methods with respect to DKM-MFPG are 0. This means that few of the solutions obtained by the other methods dominate DKM-MFPG’s. This is also due to the fact that the DKM-MFPG algorithm’s lazy robot ratio is almost zero, so that a dominated solution exists only if other baseline algorithms reach the same lazy robot ratio and have less energy consumption.

From Table 4, the average execution time of DKM-MFPG is the lowest, saving 84.53%, 53.57%, 91.20%, and 97.45% time by comparison. This is because the pre-training and end-to-end optimization of the reinforcement learning model can effectively secure the execution procedure.

4.4. Discussion

Compared with the baselines, DKM-MFPG achieves close performance on energy consumption and an increase of around 10% on the lazy robot ratio. The reason lies in that the DKM dispatcher prioritizes the lazy robot ratio by its dynamic dispatch mechanism; therefore, the lazy robot ratio is always 0, and thus it has an advantage over the baselines. In addition, the execution time of DKM-MFPG also has an obvious advantage over the baselines by saving more than 50% of the time. This is because reinforcement learning methods train the model first, then directly output the optimal policy according to the trained model and real-time input, which secures the execution procedure. To sum up, our designed DKM-MFPG approach successfully achieves satisfactory results for the coordinated ship-welding task compared with baselines.

There are some known limitations of the proposed DKM-MFPG approach. Compared with heuristic searching methods such as GA, NSGA-III, and AnD, the diversity of solutions of DKM-MFPG is lacking due to the decomposition of the two objectives and the dispatching mechanism of the DKM dispatcher. Furthermore, as a reinforcement learning-based approach with the implementation of neural networks, a certain level of computer power is required to process large amounts of data, which is also a limitation of the heuristic or other learning-based methods.

When implementing DKM-MFPG for real-world ship-building environments, the following steps can be considered: (Step 1) Construct the mapping from the real-world to the simulator. (Step 2) Set the number of robots and the number, lengths, and placement of weld lines in the simulator according to real-world specifications. (Step 3) Apply the DKM-MFPG algorithm and obtain the trajectory data of all robots. (Step 4) Implement the trajectory data for real-world robots to guide their motion. Compared to the simulation, the following uncertainties can be encountered in the experiment: (1) Sensor noise. Due to the limitation of the sensor precision and noise, the position information cannot always be precise. This uncertainty can be tackled by constructing the grid world with a larger grid length, such that the sensing error within a certain range will not affect the choice of a specific grid point. (2) Actuator faults. When executing the control inputs given by DKM-MFPG, actuators cannot always execute the given control commands precisely. To deal with this, actuator faults can be detected and calculated in advance, and design compensation should be implemented for the planner to offset the impact.

5. Conclusions

This paper deals with the coordinated ship-welding task, with objectives of optimal lazy robot ratio and energy consumption. A DKM-MFPG RL algorithm is designed to achieve task accomplishment with minimal lazy robot ratio and energy using the DKM dispatcher and MFPG motion planner, respectively. The theoretical analysis of the DKM dispatching and the MFPG network weight convergence is provided, and the proposed algorithm performed well, validated by comparison of numerical simulations under different scenarios and with different methods.

For the ongoing research of further improving the performance of DKM-MFPG, increasing solution diversity and operating the robots with nonlinear motion trajectories can be potential options. The solution diversity of DKM-MFPG can be possibly improved by integrating the multi-objective optimization methods with the reinforcement learning cost, such that more sets of dispatches can be obtained.

Note that our DKM-MFPG algorithm deals with the optimization of the lazy robot ratio and energy consumption, which focuses more on the upper-level dispatch and motion planning in the above grid-world environment. Operating welding robots in environments with more complex motion dynamics can be an interesting lower-level control problem. This problem can possibly be solved by adding some restriction conditions for optimization in DKM-MFPG.

Moreover, constraints like robot battery life, tool wear, or path overlap between robots are important considerations in the coordinated ship-welding problem and hence, we will further devote ourselves to designing reinforcement learning-based approaches under constraints in future work.