A Framework for Human-Robot-Human Physical Interaction Based on N-Player Game Theory

Abstract

In order to analyze the complex interactive behaviors between the robot and two humans, this paper presents an adaptive optimal control framework for human-robot-human physical interaction. N-player linear quadratic differential game theory is used to describe the system under study. N-player differential game theory can not be used directly in actual scenerie, since the robot cannot know humans’ control objectives in advance. In order to let the robot know humans’ control objectives, the paper presents an online estimation method to identify unknown humans’ control objectives based on the recursive least squares algorithm. The Nash equilibrium solution of human-robot-human interaction is obtained by solving the coupled Riccati equation. Adaptive optimal control can be achieved during the human-robot-human physical interaction. The effectiveness of the proposed method is demonstrated by rigorous theoretical analysis and simulations. The simulation results show that the proposed controller can achieve adaptive optimal control during the interaction between the robot and two humans. Compared with the LQR controller, the proposed controller has more superior performance.

1. Introduction

In the past decade, physical human-robot interaction has attracted the attention of the research community due to the urgent requirement for robot technology in unstructured environment [1,2,3,4]. Physical human-robot interaction combines the advantages of humans and robots, which means that humans are good at reasoning and problem solving with high flexibility, while robots perform well in terms of execution as well as guaranteeing the accuracy of task execution [5,6]. The combination of these advantages has led to the wide application of physical human-robot interaction, such as teleoperation [7,8], collaborative assembly [9,10], and collaborative transportation [11,12,13].

Two types of specific human robot interaction strategies have been widely studied: co-activity type of interaction strategy and master-slave control strategy [14,15]. Co-activity type of interaction strategy is used in typical rehabilitation robots that help limb movement training or intelligent industrial systems that support heavy objects to resist gravity, where robots completely ignore human users’ behaviors [16,17]. In contrast, the master–slave control strategy is used in the teleoperated robots or force extender exoskeletons use where robots completely follow the control of human users [18]. However, these strategies can only be used for specific interactive behaviors, the general framework for analyzing various interactive behaviors between robot and humans is still missing [19,20].

It has been pointed out that game theory can be used as a general framework to analyze complex interactive behaviors between multiple agents because different combinations of individual cost functions and different optimization objectives can be used to describe various interactive behaviors in game theory [21]. In [22], the human and the robot were been regarded as two agents and game theory was used in order to analyze the performance of the two agents. In [23], the optimal control was obtained for a given game with a linear system cost function by solving the coupled Riccati equation. In [24], an optimal control algorithm was developed for human-robot collaboration by solving the Riccati equation in each loop. In [25,26,27,28], policy iteration was used to solve the Nash equilibrium solution in order to improve the calculation speed. In [29], cyber-physical human systems was modeled via an interplay between reinforcement learning and game theory. In [30], haptic shared control for human-robot collaboration was modeled by a game-theoretical approach. In [31], human-like motion planning was studied based on game theoretic decision making. In [32], cooperative game was used for human-robot collaborative manufacturing. In [33], a bayesian framework was proposed for nash equilibrium inference in human-robot parallel play. In [19], non-cooperative differential game theory was used to model human-robot interaction system that results in a variety of interaction strategies. However, the above studies only consider two agents, that is, the interaction between one human and one robot. Therefore, the aforementioned methods are not suitable for human-robot-human physical interaction where more than one human interact with one robot physically. It is worth noting that the physical interaction between one robot and two humans will bring greater advantages such as operating larger loads, improving the flexibility and robustness of the system [28,34,35,36,37]. These greater advantages are brought by the team collaboration between the robot and two humans. To the authors’ acknowledgment, no literature have researched the problem of the physical interaction between one robot and two humans based on game theory.

In the paper, a general adaptive optimal control framework for human-robot-human physical interaction is proposed based on N-player game theory. Accordingly, the robot and two humans can interact with each other optimally by learning each other’s control. N-player differential game theory was used to model the human-robot-human interaction system in order to analyze the complex interactive behaviors between the robot and two humans. In N-player differential game theory, humans’ control objectives are assumed to be knowledge [38,39]. However, N-player differential game theory can not be used directly in actual scenerie since the robot cannot know humans’ control objectives in advance. In order to let the robot know humans’ control objectives, the paper presents an online estimation method to identify unknown humans’ control objectives based on the recursive least squares algorithm. Subsequently, the Nash equilibrium solution of the multi-human robot physical interaction is obtained by solving the coupled Riccati equation to achieve coupled optimization. Finally, the effectiveness of the proposed method is demonstrated by rigorous theoretical analysis and simulation experiments. This paper makes the following four contributions.

(1)

N-player differential game theory is firstly used to model the human-robot-human interaction system.

(2)

An online estimation method to identify unknown humans’ control objectives based on the recursive least squares algorithm is presented.

(3)

A general adaptive optimal control framework for human-robot-human physical interaction is propose based on (1) and (2).

(4)

The effectiveness of the proposed method is demonstrated by rigorous theoretical analysis and simulation experiments.

The remainder of this paper is organized, as follows: Section 2 models the human-robot-human physical interaction system based on N-player differential game theory. Section 3 establishes an adaptive optimal control law, and the control performance of the system is analyzed theoretically. Section 4 verifies the effectiveness of the proposed method through simulation experiments. Finally, Section 5 concludes this work.

2. Problem Formulation

2.1. System Description

The system considered contains two humans and one robot. An example scenario is shown in Figure 1, where the robot and the humans collaborate to perform an object transporting task. In this shared control task, when the control objectives of humans’ change, the robot should recognize the humans’ control objectives and response adaptively and optimally. The forces exerted by the humans on the object are measured by force sensors at the interaction point. It is worth noting that the humans’ control objectives are unknown to the robot.

The forward kinematics of the robot are described as

$$x\left(t\right)=\varphi \left(q\right(t\left)\right)$$

(1)

where $x\left(t\right)\in {\mathbb{R}}^m$ and $q\left(t\right)\in {\mathbb{R}}^n$ are the positions in Cartesian space and joint space respectively, m and n are degrees of freedom. Derivation of Equation (1) with time can be obtained

$$\dot{x}\left(t\right)=J\left(q\left(t\right)\right)\dot{q}\left(t\right)$$

(2)

where $J\left(q\left(t\right)\right)\in {\mathbb{R}}^{m\times n}$ is the Jacobian matrix.

The following impedance model is given in Cartesian space

$$M_d\ddot{x}\left(t\right)+C_d\dot{x}\left(t\right)=u\left(t\right)+f_1\left(t\right)+f_2\left(t\right)$$

(3)

where $M_d\in {\mathbb{R}}^{m\times m}$ is the desired inertial matrix, $C_d\in {\mathbb{R}}^{m\times m}$ is the damping matrix, $u\left(t\right)\in {\mathbb{R}}^m$ is the control input in the Cartesian space [40,41,42], $f_1\left(t\right)\in {\mathbb{R}}^n$ is the contact force between object and human 1, $f_2\left(t\right)\in {\mathbb{R}}^n$ is the contact force between object and human 2.

To track a common and fixed target $x_d\in {\mathbb{R}}^m$ (${\dot{x}}_d\in {\mathbb{R}}^m$) in cooperative object transporting task, Equation (3) can be transformed, as following

$$M_d(\ddot{x}\left(t\right)-{\ddot{x}}_d)+C_d(\dot{x}\left(t\right)-{\dot{x}}_d)=u\left(t\right)+f_1\left(t\right)+f_2\left(t\right).$$

(4)

In order to ease the design of the control, Equation (4) can be rewritten as the following state-space form

$$\begin{array}{c}\hfill \dot{z}=Az+B_{1}u+B_2{f}_1+B_3{f}_2\\ \hfill z=\left[\begin{array}{c}x\left(t\right)-x_d\\ \dot{x}\left(t\right)\end{array}\right],A=\left[\begin{array}{cc}{0}_m& 1_m\\ 0_m& -M_d^{-1}{C}_d\end{array}\right]\\ \hfill B_1=B_2=B_3=B=\left[\begin{array}{c}{0}_m\\ M_d^{-1}\end{array}\right]\end{array}$$

(5)

where $0_m$ and $1_m$ denote $m\times m$ zero and unit matrices, respectively.

2.2. Problem Formulation

According to non-cooperative differential game theory, in the paper, the interaction between the robot and the humans is described as a game between N players (in this paper, $N=3$) [43]. In the game, each player will minimize their respective cost function

$$\begin{array}{cc}\hfill \Gamma & \equiv {\int }_{t_0}^{\infty }{z}^{T}Qz+u^{T}u{d}_t\hfill \\ \hfill {\Gamma }_1& \equiv {\int }_{t_0}^{\infty }{z}^T{Q}_{1}z+f_1^T{f}_1{d}_t\hfill \\ \hfill {\Gamma }_2& \equiv {\int }_{t_0}^{\infty }{z}^T{Q}_{2}z+f_2^T{f}_2{d}_t\hfill \\ \hfill Q& =\left[\begin{array}{cc}{Q}_{01}& 0_{n\times n}\\ 0_{n\times n}& Q_{02}\end{array}\right]\hfill \\ \hfill Q_1& =\left[\begin{array}{cc}{Q}_{11}& 0_{n\times n}\\ 0_{n\times n}& Q_{12}\end{array}\right]\hfill \\ \hfill Q_2& =\left[\begin{array}{cc}{Q}_{21}& 0_{n\times n}\\ 0_{n\times n}& Q_{22}\end{array}\right]\hfill \end{array}$$

(6)

where $\Gamma ,{\Gamma }_1,{\Gamma }_2$ are cost functions of the robot, human 1, and human 2, respectively, $Q,Q_1,Q_2$ are state weights matrices of the robot, human 1 and human 2, respectively. Each player achieves the cooperative object transporting task by minimizing the error to the target while minimizing their own costs. $Q,Q_1,Q_2$ contain two components corresponding to position regulation and velocity, respectively. $Q_{01},Q_{11},Q_{21}$ correspond to position regulation and $Q_{02},Q_{12},Q_{22}$ correspond to velocity.

In [27], the N-player game has been studied if the cost functions are known. However, ${\Gamma }_1,{\Gamma }_2$ are unknown to the robot because they are determined by the humans. Therefore, a method is proposed in the paper to estimate ${\Gamma }_1,{\Gamma }_2$ in order to achieve adaptive optimal control and, thus, the human-robot-human cooperative object transporting task.

2.3. N-Player Differential Game Theory

Based on the differential game theory of linear systems, for N-player game the following linear differential equation [43] is considered:

$$\dot{z}=Az+B_1{u}_1+\cdots +B_N{u}_N,z\left(0\right)=z_0.$$

(7)

Each player has a quadratic cost function that they want to minimize:

$${\Gamma }_i={\int }_0^{\infty }{z}^T{Q}_{i}z+u_i^T{u}_i{d}_t,i=1,\cdots ,N$$

(8)

Different types of multi-agent behaviors are defined in game theory, which can be achieved through different concepts of game equilibrium [44,45]. In this paper, Nash equilibrium is considered. In the sense of Nash equilibrium, each player minimizes their cost function:

$$\begin{array}{cc}\hfill & u_i=-{\eta }_{i}z,{\eta }_i=B_i^T{P}_i\hfill \\ \hfill {(A-\sum _{j\ne i}^N{B}_i{\eta }_i)}^T{P}_i+P_i(A-& \sum _{j\ne i}^N{B}_i{\eta }_i{)}_i+Q_i-P_i{B}_i{B}_i^{T}P=0,i=1,\cdots ,N\hfill \end{array}$$

(9)

where N is equal to 3 in this paper. In the sense of Nash equilibrium, the humans and the robot minimizes their own cost function:

$$\begin{array}{cc}\hfill u& =-\alpha z\hfill \\ \hfill \alpha & =B^T{P}_r\hfill \end{array}$$

(10a)

$$\begin{array}{cc}\hfill f_1& =-\beta z\hfill \\ \hfill \beta & =B^T{P}_1\hfill \end{array}$$

(10b)

$$\begin{array}{cc}\hfill f_2& =-\gamma z\hfill \\ \hfill \gamma & =B^T{P}_1\hfill \end{array}$$

(10c)

$$\begin{array}{c}\hfill A_r^T{P}_r+P_r{A}_r+Q-P_{r}B{B}^T{P}_r=0_{2n},A_r=A-B\beta -B\gamma \end{array}$$

(10d)

$$\begin{array}{c}\hfill A_1^T{P}_1+P_1{A}_1+Q-P_{1}B{B}^T{P}_1=0_{2n},A_1=A-B\alpha -B\gamma \end{array}$$

(10e)

$$\begin{array}{c}\hfill A_2^T{P}_2+P_2{A}_2+Q-P_{2}B{B}^T{P}_2=0_{2n},A_2=A-B\alpha -B\beta \end{array}$$

(10f)

where $\alpha \equiv \left[\begin{array}{c}{\alpha }_e,{\alpha }_v\end{array}\right]$ is the feedback gain of the robot, $\beta \equiv \left[\begin{array}{c}{\beta }_e,{\beta }_v\end{array}\right]$ is the feedback gain of the human 1, $\gamma \equiv \left[\begin{array}{c}{\gamma }_e,{\gamma }_v\end{array}\right]$ is the feedback gain of of the human 2. ${\alpha }_e,{\beta }_e,{\gamma }_e$ are the position error gains, ${\alpha }_v,{\beta }_v,{\gamma }_v$ are the velocity gains, $P_r,P_1,P_2$ are the solutions of the above well-known Riccati equation consisting of Equation (10d–f). The robot and the humans influence each other through $A_r,A_1$, and $A_2$ in order to achieve the interactive control and the coupling optimization.

$\beta ,\gamma$ are unknown to the robot. Therefore, we aim to propose a method to estimate them in the following section.

3. Adaptive Optimal Control

A recursive least squares algorithm with forgetting factors is used in this paper to get the estimate $\widehat{\beta },\widehat{\gamma }$ of $\beta ,\gamma$ in order to estimate the feedback gains of the humans in real time and avoid the data saturation phenomenon caused by the standard least squares algorithm [46]. Subsequently, the estimate ${\widehat{Q}}_1,{\widehat{Q}}_2$ of $Q_1,Q_2$ can be obtained using Equation (10e,f).

Equation (10e) is used as the model for identification. For convenience, we let ${\theta }_1=-{\beta }^T$, $y_1=f_1^T$, $W=z^T$. Subsequently, Equation (10b) can be rewritten as

$$y_1=W{\theta }_1.$$

(11)

The feedback gain of the human 1 are estimated by minimizing the total prediction error

$$J_1={\int }_0^{t}exp(-{\lambda }_{1}t){\parallel y_1\left(s\right)-W\left(s\right){\widehat{\theta }}_1\parallel }^{2}ds$$

(12)

where ${\lambda }_1$ is the constant forgetting factor. The update rule of the parameter ${\theta }_1$ can be obtained as

$$\begin{array}{cc}\hfill {\dot{\widehat{\theta }}}_1& =-P{W}^T{e}_1\hfill \\ \hfill \dot{P}& ={\lambda }_{1}P-P{W}^{T}WP\hfill \\ \hfill e_1& ={\widehat{y}}_1-y_1.\hfill \end{array}$$

(13)

The estimated error of ${\widehat{\theta }}_1$ is

$$e_{{\theta }_1}\left(t\right)=exp(-{\lambda }_{1}t)P\left(t\right)P^{-1}\left(0\right)e_{{\theta }_1}\left(0\right).$$

(14)

Thus, the estimate $\widehat{\beta }$ can be obtained as

$$\widehat{\beta }=-{\widehat{\theta }}_1^T.$$

(15)

Similarly, we let ${\theta }_2=-{\gamma }^T$, $y_2=f_2^T$, $W=z^T$. Afterwards, Equation (10c) can be rewritten as

$$y_2=W{\theta }_2.$$

(16)

The feedback gain of the human 2 are estimated by minimizing the total prediction error

$$J_2={\int }_0^{t}exp(-{\lambda }_{2}t){\parallel y_2\left(s\right)-W\left(s\right){\widehat{\theta }}_2\parallel }^{2}ds$$

(17)

where ${\lambda }_2$ is the constant forgetting factor. The update rule of the parameter ${\theta }_2$ can be obtained as

$$\begin{array}{cc}\hfill {\dot{\widehat{\theta }}}_2& =-P{W}^T{e}_2\hfill \\ \hfill \dot{P}& ={\lambda }_{2}P-P{W}^{T}WP\hfill \\ \hfill e_1& ={\widehat{y}}_2-y_2.\hfill \end{array}$$

(18)

The estimated error of ${\widehat{\theta }}_2$ is

$$e_{{\theta }_2}\left(t\right)=exp(-{\lambda }_{2}t)P\left(t\right)P^{-1}\left(0\right)e_{{\theta }_2}\left(0\right).$$

(19)

Thus, the estimate $\widehat{\gamma }$ can be obtained as

$$\widehat{\gamma }=-{\widehat{\theta }}_2^T.$$

(20)

Equations (13), (15), (18) and (20) are critical, because they enable each agent to recognize their partners’ control objectives and use Equation (10a–f) to adjust their own control.

In order to ensure the performance of cooperative object transporting task, we let

$$Q+Q_1+Q_2\equiv C$$

(21)

where C is the total weight. The cooperative object transporting task fixes the task performance through the total weight C and uses Equation (21) to share the the effort between 2 humans and the robot. Equation (21) makes the proposed controller be able to adjust the contributions between the humans and the robot and makes the humans and the robot take complementary roles as well.

The control architecture is shown in Figure 2.

A pseudo-code summarizes the implementation procedures of the proposed method as Algorithm 1.

Algorithm 1 Adaptive optimal control algorithm based on N-player game

Input: Current state z, target $x_d$.

Output: Robot’s control input u, estimated the humans’ cost function state weight ${\widehat{Q}}_1,{\widehat{Q}}_2$ in Equation (10e,f).

Begin

Define $x_d$, initialize $Q,{\widehat{Q}}_1,{\widehat{Q}}_2,u,f_1,f_2,\widehat{z},\alpha ,\widehat{\beta },\widehat{\gamma },P_r,{\widehat{P}}_1,{\widehat{P}}_2$, set ${\lambda }_1$ in Equation (13), ${\lambda }_2$ in Equation (18), C in   Equation (21), the terminal time $t_f$ of one trial.

While $t<t_f$ do

Measure the position $x\left(t\right)$, velocity $\dot{x}\left(t\right)$, and form z.

Update $\widehat{\beta }$ using Equations (13) and (15), Update $\widehat{\gamma }$ using Equations (18) and (20).

Solve the Riccati equation in Equation (10d) to obtain P, and calculate the robot’s control input u.

Calculate estimated the humans’ cost function state weights ${\widehat{Q}}_1,{\widehat{Q}}_2$ in Equation (10e,f) using the Riccati equation.

Compute robot’s cost function state weight Q according to Equation (21).

Theorem 1. 

Consider the robot dynamics shown in Equation (5). If the robot and the humans estimate the parameters of their partners’ controller and adjust their own control according to Equations (10a–f), (13), (15), (18), (20) and (21), then the following conclusions will be drawn:

• The closed-loop system is stable, and $z,\alpha ,\widehat{\beta },\widehat{\gamma },u$ are bounded.
• ${lim}_{x\to \infty }{\widehat{Q}}_1=Q_1$, ${lim}_{x\to \infty }{\widehat{Q}}_2=Q_2$, which indicate that ${\widehat{Q}}_1,{\widehat{Q}}_2$ converge to the correct values $Q_1,Q_2$, if z is persistently exciting.
• The Nash equilibrium is achieved for th human-robot-human interaction system.

Proof of Theorem 1. 

$\widehat{\beta },\widehat{\gamma }$ influence $u,f_1,f_2,z$ as following:

$$\dot{\widehat{z}}=A\widehat{z}+B\widehat{u}+B{f}_1+B_2.$$

(22)

By subtracting Equation (5) from Equation (22), we have

$${\dot{e}}_z=A{e}_z+B(\widehat{u}-u)+B{e}_{f_1}+B{e}_{f_2}$$

(23)

where $e_z=\widehat{z}-z$. By considering Equation (10a–c), we have

$${\dot{e}}_z=(A-B\alpha )e_z+B{e}_{{\theta }_1}^{T}z+B{e}_{{\theta }_2}^{T}z.$$

(24)

Consider the Lyapunov function candidate as following

$$W=\frac{1}{2}{z}^{T}z+\frac{1}{2}{e}_{{\theta }_1}^T{e}_{{\theta }_1}+\frac{1}{2}{e}_{{\theta }_2}^T{e}_{{\theta }_2}+\frac{\chi }{2}{e}_z^T{e}_z$$

(25)

where $\chi =min(\frac{2({\lambda }_1-\rho )\pi }{{\phi }^2{\parallel B\parallel }^2},\frac{2({\lambda }_2-\rho )\pi }{{\phi }^2{\parallel B\parallel }^2})$, with $\rho$ being the upper bound of the maximum eigenvalue of $\dot{P}{P}^{-1}$, $\pi$ being the lower bound of the minimum eigenvalue of $B\alpha -A$, $\phi$ being the upper bound of $\parallel z\parallel$.

When considering function $V=\frac{1}{2}{z}^{T}z$ and differentiating V with respect to time, we obtain

$$\begin{array}{cc}\hfill \dot{V}& =z^T\dot{z}=-z^T(B\alpha +B\beta +B\gamma -A)z.\hfill \end{array}$$

(26)

According to Equation (10d), $B\alpha +B\beta +B\gamma -A$ is positive definite if Q is positive definite, it follows ${lim}_{t\to \infty }\parallel z\parallel =0$. Therefore, z is bounded and we define $\phi$ as the upper bound of $\parallel z\parallel$. By differentiating Equation (25), with respect to time, and considering Equations (14), (19) and (24), we obtain

$$\begin{array}{cc}\hfill \dot{W}=& z^T\dot{z}+e_{{\theta }_1}^T{\dot{e}}_{{\theta }_1}+e_{{\theta }_2}^T{\dot{e}}_{{\theta }_2}+\chi e_z^T{\dot{e}}_z\hfill \\ \hfill =& -z^T(B\alpha +B\beta +B\gamma -A)z\hfill \\ \hfill & -{\lambda }_1{e}_{{\theta }_1}^T{e}_{{\theta }_1}+e_{{\theta }_1}^T\dot{P}{P}^{-1}{e}_{{\theta }_1}-{\lambda }_2{e}_{{\theta }_2}^T{e}_{{\theta }_2}+e_{{\theta }_2}^T\dot{P}{P}^{-1}{e}_{{\theta }_2}\hfill \\ \hfill & -\chi e_z^T(B\alpha -A)e_z+\chi e_z^{T}B{e}_{{\theta }_1}^{T}z+\chi e_z^{T}B{e}_{{\theta }_2}^{T}z\hfill \\ \hfill \le & -z^T(B\alpha +B\beta +B\gamma -A)z\hfill \\ \hfill & -{\lambda }_1\parallel e_{{\theta }_1}{\parallel }^2+\rho \parallel e_{{\theta }_1}{\parallel }^2-{\lambda }_2\parallel e_{{\theta }_2}{\parallel }^2+\rho {\parallel e_{{\theta }_2}\parallel }^2\hfill \\ \hfill & -\chi \pi \parallel e_z{\parallel }^2+\chi \phi \parallel B\parallel \parallel e_z\parallel \parallel e_{{\theta }_1}\parallel +\chi \phi \parallel B\parallel \parallel e_z\parallel \parallel e_{{\theta }_2}\parallel \hfill \\ \hfill =& -z^T(B\alpha +B\beta +B\gamma -A)z\hfill \\ \hfill & -(\sqrt{{\lambda }_1-\rho }\parallel e_{{\theta }_1}\parallel -\sqrt{\frac{\chi \pi }{2}}\parallel e_z{\parallel )}^2\hfill \\ \hfill & -2\sqrt{{\lambda }_1-\rho }\sqrt{\frac{\chi \pi }{2}}\parallel e_{{\theta }_1}\parallel e_z\parallel \parallel +\chi \phi B\parallel \parallel e_z\parallel \parallel e_{{\theta }_1}\parallel \hfill \\ \hfill & -(\sqrt{{\lambda }_2-\rho }\parallel e_{{\theta }_2}\parallel -\sqrt{\frac{\chi \pi }{2}}\parallel e_z{\parallel )}^2\hfill \\ \hfill & -2\sqrt{{\lambda }_2-\rho }\sqrt{\frac{\chi \pi }{2}}\parallel e_{{\theta }_2}\parallel e_z\parallel \parallel +\chi \phi \parallel B\parallel \parallel e_z\parallel \parallel e_{{\theta }_2}\parallel \hfill \\ \hfill \le & -z^T(B\alpha +B\beta +B\gamma -A)z\hfill \\ \hfill & +(-2\sqrt{{\lambda }_1-\rho }\sqrt{\frac{\chi \pi }{2}}+\chi \phi \parallel B\parallel )\parallel e_z\parallel \parallel e_{{\theta }_1}\parallel \hfill \\ \hfill & +(-2\sqrt{{\lambda }_2-\rho }\sqrt{\frac{\chi \pi }{2}}+\chi \phi \parallel B\parallel )\parallel e_z\parallel \parallel e_{{\theta }_2}\parallel \hfill \\ \hfill \le & 0\hfill \end{array}$$

(27)

According to Equations (26) and (27), we have ${lim}_{t\to \infty }\parallel z\parallel =0$, ${lim}_{t\to \infty }\parallel e_z\parallel =0$. Therefore, $z\left(t\right)$ is bounded and ${lim}_{t\to \infty }\parallel {\dot{e}}_z\parallel =0$. According to Equation (27), we have ${lim}_{t\to \infty }\parallel e_{{\theta }_1}\parallel =0$, ${lim}_{t\to \infty }\parallel e_{{\theta }_2}\parallel =0$. Because of $e_{{\theta }_1}={\widehat{\theta }}_1-{\theta }_1={(-\widehat{\beta })}^T-{(-\beta )}^T={\beta }^T-{\widehat{\beta }}^T$, $e_{{\theta }_2}={\widehat{\theta }}_2-{\theta }_2={(-\widehat{\gamma })}^T-{(-\gamma )}^T={\gamma }^T-{\widehat{\gamma }}^T$, we can obtain ${lim}_{t\to \infty }\parallel {\beta }^T-{\widehat{\beta }}^T\parallel =0$, ${lim}_{t\to \infty }\parallel {\gamma }^T-{\widehat{\gamma }}^T\parallel =0$. $\beta ,\gamma$ are assumed to be bounded, since they are the feedback gains of the humans. Therefore, $\widehat{\beta },\widehat{\gamma }$ are also bounded. According to Equation (10a–c), $P_1,P_2$ are also bounded. According to Equation (10d), $A_r$ is bounded. Therefore, $P,\alpha$ and u are bounded.

According to Equation (10e), we can calculate the estimated errors $e_{Q_1}={\widehat{Q}}_1-Q_1$, $e_{Q_2}={\widehat{Q}}_2-Q_2$. $e_{Q_1},e_{Q_2}$ are due to the errors $e_P,e_{P_1},e_{P_2}$. Because $e_P,e_{P_1},e_{P_2}$ converge to zero, we have ${lim}_{t\to \infty }\parallel e_{Q_1}\parallel =0$, ${lim}_{t\to \infty }\parallel e_{Q_2}\parallel =0$, that is ${lim}_{t\to \infty }{\widehat{Q}}_1=Q_1$, ${lim}_{t\to \infty }{\widehat{Q}}_2=Q_2$.

Multiplying Equation (10d) by ${\widehat{z}}^T$ on the left side and by $\widehat{z}$ on the right side, and considering Equation (13), we have

$$\begin{array}{cc}\hfill 0=& {\widehat{z}}^{T}Q\widehat{z}+{\widehat{z}}^T{P}_{r}B{B}^T{P}_r\widehat{z}+{\widehat{z}}^T{P}_r\dot{\widehat{z}}\hfill \\ \hfill & +\widehat{z}{P}_r{\dot{\widehat{z}}}^T+{\widehat{z}}^T{P}_{r}H{e}_z+\widehat{z}{P}_{r}H{e}_z^T\hfill \\ \hfill \equiv & \widehat{\sigma }.\hfill \end{array}$$

(28)

Considering ${lim}_{t\to \infty }{e}_z=0$, ${lim}_{t\to \infty }{\dot{e}}_z=0$, we can obtain

$$\begin{array}{cc}\hfill \underset{t\to \infty }{lim}\sigma \equiv & \underset{t\to \infty }{lim}(z^{T}QZ+z^T{P}_{r}B{B}^T{P}_{r}z\hfill \\ \hfill & +z^T{P}_r\dot{z}+z{P}_r{\dot{z}}^T)\hfill \\ \hfill =& 0.\hfill \end{array}$$

(29)

Similarly, we can obtain

$$\begin{array}{cc}\hfill \underset{t\to \infty }{lim}{\sigma }_1\equiv & \underset{t\to \infty }{lim}(z^T{Q}_{1}Z+z^T{P}_{1}B{B}^T{P}_{1}z\hfill \\ \hfill & +z^T{P}_1\dot{z}+z{P}_1{\dot{z}}^T)\hfill \\ \hfill =& 0\hfill \\ \hfill \underset{t\to \infty }{lim}{\sigma }_2\equiv & \underset{t\to \infty }{lim}(z^T{Q}_{2}Z+z^T{P}_{2}B{B}^T{P}_{2}z\hfill \\ \hfill & +z^T{P}_2\dot{z}+z{P}_2{\dot{z}}^T)\hfill \\ \hfill =& 0.\hfill \end{array}$$

(30)

${\lim }_{t\to \infty }\sigma =0$, ${lim}_{t\to \infty }{\sigma }_1=0$ and ${lim}_{t\to \infty }{\sigma }_2=0$ indicate that the Nash equilibrium is achieved for the human-robot-human interaction system. ☐

4. Simulations and Results

4.1. Experimental Design and Ssimulation Settings

With the development of the robot technology, in the future, robots will enter our homes and become a member of family in our daily lives. In our daily lives, we often need to carry various objects. Some objects (e.g., objects with smaller size and lower weight) can be successfully carried by one human; some objects (e.g., objects with medium size and medium weight) need to be carried successfully by two humans; some objects (e.g., objects with larger size and higher weight) can be carried successfully by three or more humans. Consider one scenario: In our home, we have an object (such as a table with a relatively larger size and higher weight) that need to be carried by three humans. However, there are only two humans in the home. In this case, we can let the robot help us carry the object together with the two humans. The robot can play the same role as one human. A simulation is conducted with CoppeliaSim in order to verify the control performance of the controller proposed in this paper. The version of CoppeliaSim that we used is CoppeliaSim 4.0.0 (CoppeliaSim Edu, Windows). Figure 3 demonstrates the CoppeliaSim simulation scenario of cooperative object transporting task. The humans cooperate with the robot to transport the object between −10 cm and +10 cm back and forth along the horizontal direction.

The controller that is proposed in this paper implements interactive control because every agent considers the control of other partners. In order to present the advantages of the proposed controller, we compare the proposed controller with the linear quadratic regulators (LQR) optimal controller. The LQR controller can be obtained by setting $A_r=A$, $A_1=A$, $A_2=A$ in Equation (10d–f). The LQR controller allows each agent to form its own control input optimally, but it ignores the controls of other partners. Let $Q=Q_1=Q_2=diag(100,0)$.

The cost functions of the humans usually change during the physical human-robot-human interaction. The robot needs to identify the change and adaptively adjust its own cost function in order to complete the cooperative object transporting task. In order to verify the ability of the robot to adaptively interact with two humans when humans’ cost functions change, we simulated a scenario where the robot cooperated with the humans to perform an object transporting task. The task performance is achieved by setting the value of C in Equation (21). Let $C=diag(300,0)$. The cost functions of the human 1 and the human 2 change randomly according to $Q_1=diag(50,0)+\rho ·diag(50,0),Q_2=diag(50,0)+\rho ·diag(50,0)$ ($\rho$ is a uniformly distributed random number between $[0,1]$).

The human-robot-human cooperative object transporting task can be fulfilled with less effort with the proposed controller. In order to make this affirmation, we made a comparison with a human-robot cooperative object transporting task. In simulation of the human-robot-human cooperative object transporting task, we let $Q=Q_1=Q_2=diag(100,0)$. In simulation of the human-robot cooperative object transporting task, we let $Q=diag(100,0)$, $Q_1=diag(100,0)$, $Q_2=diag(0,0)$.

We assume that the humans and the robot do not have prior knowledge of each other (thus, initially $\widehat{\alpha }\equiv 0,\widehat{\beta }\equiv 0,\widehat{\gamma }\equiv 0$). The control input of the robot are generated by Equations (5), (10a–f), (13), (15), (18) and (20). The simulated interaction forces $f_1,f_2$ of the human 1 and the human 2 are generated by a similar set of equations. The simulation time is 40 s. Let the inertia of the robot $M_d$ = 6 kg, the damping of the robot $C_d=-0.2N·{\mathrm{m}}^{-1}$[19], the real-time least squares algorithm forgetting factor ${\lambda }_1={\lambda }_2=0.95$. Simulation time step is 0.005 s.

4.2. Results

Figure 4 depicts the change in position of the end effector with respect to time. The results plotted in Figure 4 is a smooth curve that looks like a sinusoidal signal. This smooth curve is determined by Equation (3). In Equation (3), $u\left(t\right),f_1\left(t\right),f_2\left(t\right)$ are iteratively calculated by our proposed controller based on game theory. Due to the fact that the humans and the robot do not transport the object at a constant speed using our method, the end effector follows a curve signal rather than a straight line signal. As can be seen from Figure 4, the end effector can reach the target position with the proposed controller which means that the cooperative object transporting task is successfully fulfilled. In contrast, the end effector can not reach the target position with the LQR controller, which means that the cooperative object transporting task is not successfully fulfilled. The reason why the cooperative object transporting task can be successfully fulfilled with the proposed controller rather than with the LQR controller is that the proposed controller considers the interaction with other partners. When one partner decreases effort, the other partners will gradually increase their efforts to ensure the successful fulfillment of the cooperative object transporting task. In contrast, the LQR controller does not consider the interaction with other partners, so the cooperative object transporting task cannot be guaranteed to be successfully fulfilled.

In Figure 5, we can see that the estimated humans’ feedback gains converge to the real values in a few seconds. This means that the humans’ feedback gains can be successfully estimated by the proposed method.

Figure 6 demonstrates that fulfilling the cooperative object transporting task requires larger control gains $\beta ,\gamma$ with the LQR controller compared with the controller proposed in this paper. It means that accomplishing the same task requires less effort using the proposed controller. This is because that the proposed controller considers the interaction with other partners and calculates the minimal effort for the humans and the robot to complete the task. In contrast, the LQR controller doesn’t consider the interaction with other partners, so the humans and the robot only minimize their own cost function and may, therefore, require larger effort.

The feedback gains are affected by the state weights of the cost functions. In order to verify the advantages of the proposed controller when the state weights vary, we let $Q_1$ vary from 0 to 10Q with $Q_2=diag(100,0)$ and let $Q_2$ vary from 0 to $10Q$ with $Q_1=diag(100,0)$ respectively. It can be seen from Figure 7 that accomplishing the same task always requires less effort using the proposed controller. We can also see that the difference between the control gains with our proposed controller and the control gains with LQR controller becoming smaller when $Q_1/Q$ or $Q_2/Q$ increases, this is because that the robot’s relative influence decreases.

From Figure 4, Figure 5, Figure 6 and Figure 7, we can conclude that the human-robot-human cooperative object transporting task can be fulfilled with less effort and the system can be kept stable using the proposed controller.

It can be seen from Figure 8 that, when the cost functions of the human 1 and the human 2 change, the cost function of the robot will also change adaptively. When the sum of the state weights of the human 1 and the human 2 $Q_1+Q_2$ increases, the state weight of the Robot Q decreases accordingly. Conversely, when the sum of the state weights of the human 1 and the human 2 $Q_1+Q_2$ decreases, the state weight of the robot Q increases accordingly. The reason why the robot can change adaptively is that we set the constant C value in Equation (21). Equation (21) makes the proposed controller able to adjust the contributions between the humans and the robot and makes the humans and the robot take complementary roles as well.

Figure 9 shows that, using the proposed controller, the adaptive cooperative object transporting task can be fulfilled and the system can be kept stable.

From Figure 8 and Figure 9, we can conclude that the adaptive cooperative object transporting task can be fulfilled with the proposed controller. During the physical interaction, the robot can successfully identify the change of each human’s cost function, and then adaptively adjust its own cost function to achieve interactive optimal control.

Figure 10 demonstrates that fulfilling the human-robot-human cooperative object transporting task requires smaller control gains ${\beta }_e,{\beta }_v$ as compared with the human-robot cooperative object transporting task. It means that accomplishing the same task requires less effort by means of the human-robot-human physical interaction. This is because the human-robot-human cooperative object transporting task considers the interaction with more partners (two partners) and calculates minimal effort for the humans and the robot to complete the task. In contrast, the human-robot cooperative object transporting task consider the interaction with less partners (only one partner), so the human and the robot may therefore require larger effort.

5. Conclusions

In this paper, the human-robot-human physical interaction problem has been studied. An adaptive optimal control framework for the human-robot-human physical interaction has been proposed based on N-player game theory. The recursive least squares algorithm based on forgetting factor has been used to identify unknown control parameters of the humans online. The performance of the controller proposed in this paper has been verified by simulations of cooperative object transporting task. The simulation results show that the proposed controller can achieve adaptive optimal control during the interaction between the robot and two humans and keep the system stable. Compared with the LQR controller, the proposed controller has more superior performance. Compared with the human-robot physical interaction, accomplishing the same cooperative object transporting task requires less effort by means of the human-robot-human physical interaction based on the approach proposed in the paper. Although this paper only conducts simulations on the physical interaction between one robot and two humans, it is worth mentioning that the framework that is proposed in this paper has the potential to be generalized to the situation where multiple robots physically interact with multiple humans. As future work, we will extend the framework to the interaction between multiple robots and multiple humans.