Generalized Reynolds Operators on Hom-Lie Triple Systems

Abstract

In this paper, we first introduce the notion of generalized Reynolds operators on Hom-Lie triple systems associated to a representation and a 3-cocycle. Then, we develop a cohomology of generalized Reynolds operators on Hom-Lie triple systems. As applications, we use the first cohomology group to classify linear deformations and we study the obstruction class of an extendable order n deformation. Finally, we introduce and investigate Hom-NS-Lie triple system as the underlying structure of generalized Reynolds operators on Hom-Lie triple systems.

1. Introduction

Lie triple systems first appeared in Cartan’s work [1] on Riemannian geometry. Since then, Jacobson [2,3] has studied Lie triple systems from Jordan theory and quantum mechanics. Lie triple systems extend the classical theory of Lie algebras and Lie groups by introducing a trilinear product, capturing the interplay between three elements. Lie triple systems have found applications in diverse fields, such as quantum mechanics, differential geometry and numerical analysis of differential equations. As a Hom-type algebra [4] generalization of Lie triple systems, Hom-Lie triple systems were introduced by Yau in [5]. Furthermore, Ma et al. [6] established the cohomology, central extensions and deformations of Hom-Lie triple systems. Further research on Hom-Lie triple systems has been developed (see [7,8,9,10,11] and references cited therein).

The notion of Rota–Baxter operators on associative algebras was introduced by Baxter [12] in his study of the fluctuation theory. Subsequently, the notion of a relative Rota–Baxter operator (also called an $\mathcal{O}$-operator) on a Lie algebra was independently introduced by Kupershmidt [13], to better understand the classical Yang–Baxter equation and related integrable systems. Recently, relative Rota–Baxter operators have been widely studied (see [14,15,16,17,18,19]). In addition, other operators related to (relative) Rota–Baxter operators are constantly emerging. Among them is the Reynolds operator, motivated by the work of Reynolds [20] on turbulence in fluid dynamics. Kampé de Fériet [21] created the notion of the Reynolds operator as a mathematical subject in general. Inspired by the twisted Poisson structure, Uchino [22] introduced generalized Reynolds operators on associative algebras, also known as twisted Rota–Baxter operators, and studied their relationship with NS-algebras.

In recent years, Das [23] introduced the cohomology of generalized Reynolds operators on associative algebras, and considered NS-algebras as the underlying structure motivated by Uchino’s work. Das also developed the notions of generalized Reynolds operators on Lie algebras and NS-Lie algebras in [24]. Generalized Reynolds operators on other algebraic structures have also been widely studied, including 3-Lie algebras [25,26], 3-Hom-Lie algebras [27], Hom-Lie algebras [28], Lie-Yamaguti algebras [29], Lie triple systems [29,30] and Lie supertriple systems [31].

Inspired by these works, we propose generalized Reynolds operators on Hom-Lie triple systems, we investigate the corresponding cohomology theory, which will be used to describe deformations, and we establish Hom-NS-Lie triple systems as the underlying structure in the present paper.

The paper is organized as follows. In Section 2, we recall some basic notions and facts about Hom-Lie triple systems. In Section 3, we introduce the notion of generalized Reynolds operators on a Lie triple system and we give some examples. In Section 4, we develop the cohomology of generalized Reynolds operators on Hom-Lie triple systems. In Section 5, we study linear deformations and higher-order deformations of generalized Reynolds operators on Hom-Lie triple systems via the cohomology theory. In Section 6, we introduce the notion of Hom-NS-Lie triple systems, which is the underlying algebraic structure of generalized Reynolds operators on Hom-Lie triple systems.

Throughout this paper, $\mathbb{K}$ denotes a field of characteristic zero. All the vector spaces and (multi)linear maps are taken over $\mathbb{K}$.

2. Preliminaires

In this section, we will briefly recall representations and the cohomology of Hom-Lie triple systems from [5,6].

Definition 1 

([6]). (i) A Hom-Lie triple system (Hom-L.t.s.) is a triplet $(\mathfrak{L},[-,-,-],\alpha )$ in which $\mathfrak{L}$ is a vector space together with a trilinear operation $[-,-,-]:\mathfrak{L}\times \mathfrak{L}\times \mathfrak{L}\to \mathfrak{L}$ and a linear transformation $\alpha :\mathfrak{L}\to \mathfrak{L}$, called the twisted map, satisfying $\alpha \left(\right[a,b,c\left]\right)=\left[\alpha \right(a),\alpha (b),\alpha (c\left)\right]$, such that

$$\begin{array}{cc}\hfill & [a,b,c]+[b,a,c]=0,\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill & {\circlearrowright }_{a,b,c}[a,b,c]=0,\hfill \end{array}$$

(2)

$$\begin{array}{cc}\hfill & \left[\alpha \right(a),\alpha (b),[x,y,z\left]\right]=\left[\right[a,b,x],\alpha (y),\alpha (z\left)\right]+\left[\alpha \right(x),[a,b,y],\alpha (z\left)\right]+\left[\alpha \right(x),\alpha (y),[a,b,z\left]\right],\hfill \end{array}$$

(3)

where $x,y,z,a,b\in \mathfrak{L}$ and ${\circlearrowright }_{a,b,c}$ denotes the sum over the cyclic permutation of $a,b,c$—that is, ${\circlearrowright }_{a,b,c}[a,b,c]=[a,b,c]+[c,a,b]+[b,c,a]$. In particular, $(\mathfrak{L},[-,-,-],\alpha )$ is called a regular Hom-Lie triple system if α is an invertible linear map.

(ii) Let $\mathfrak{J}$ be a subspace of a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$. Then, $\mathfrak{J}$ is called a subalgebra of $\mathfrak{L}$ if $\alpha \left(a\right)\in \mathfrak{J}$ and $[a,b,c]\in \mathfrak{J}$, for $a,b,c\in \mathfrak{J}$.

(iii) A homomorphism between two Hom-Lie triple systems $({\mathfrak{L}}_1,{[-,-,-]}_1,{\alpha }_1)$ and $({\mathfrak{L}}_2,[-,-,$${-]}_2,{\alpha }_2)$ is a linear map $\phi :{\mathfrak{L}}_1\to {\mathfrak{L}}_2$ satisfying

$$\phi \left({\alpha }_1\left(x\right)\right)={\alpha }_2\left(\phi \left(x\right)\right),\phi \left({[x,y,z]}_1\right)={[\phi \left(x\right),\phi \left(y\right),\phi \left(z\right)]}_2,\forall x,y,z\in {\mathfrak{L}}_1.$$

Example 1. 

Let $\mathfrak{L}$ be a two-dimensional vector space with a basis ${\epsilon }_1$, ${\epsilon }_2$. If we define a trilinear non-zero operation $[-,-,-]$ and a linear transformation α on $\mathfrak{L}$ as follows:

$$[{\epsilon }_1,{\epsilon }_2,{\epsilon }_2]=-[{\epsilon }_2,{\epsilon }_1,{\epsilon }_2]={\epsilon }_1,\alpha \left({\epsilon }_1\right)={\epsilon }_1,\alpha \left({\epsilon }_2\right)=-{\epsilon }_2,$$

then $(\mathfrak{L},[-,-,-],\alpha )$ is a Hom-Lie triple system.

Example 2. 

A Lie triple system is a Hom-Lie triple system with $\alpha ={\mathrm{id}}_{\mathfrak{L}}$.

Example 3. 

Let $(\mathfrak{L},[-,-],\alpha )$ be a Hom-Lie algebra—that is, it consists of a vector space $\mathfrak{L}$, a skew-symmetric bilinear map $[-,-]:\mathfrak{L}\times \mathfrak{L}\to \mathfrak{L}$ and a linear transformation α on $\mathfrak{L}$ satisfying $\alpha \left(\right[x,y\left]\right)=\left[\alpha \right(x),\alpha (y\left)\right]$ and ${\circlearrowright }_{x,y,z}[\alpha \left(x\right),[y,z]]=0$ for $x,y,z\in \mathfrak{L}$. Then, $(\mathfrak{L},[-,-,-],\alpha )$ is a Hom-Lie triple system, where $[x,y,z]=\left[\alpha \right(x),[y,z\left]\right],\forall x,y,z\in \mathfrak{L}.$

Note that Yamaguti [32] introduced the representation and cohomology theory of Lie triple systems. Furthermore, based on Yamaguti’s work, the authors in [6] developed the representation and cohomology theory of Hom-Lie triple systems, which can be described as follows.

Definition 2 

([6]). A representation of a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ on a Hom-vector space $(V,\beta )$ is a bilinear map $\theta :\mathfrak{L}\times \mathfrak{L}\to \mathrm{End}\left(V\right)$, such that for all $x,y,a,b\in \mathfrak{L}$

$$\begin{array}{cc}\hfill & \theta \left(\alpha \right(x),\alpha (y\left)\right)\circ \beta =\beta \circ \theta (x,y),\hfill \end{array}$$

(4)

$$\begin{array}{cc}\hfill & \theta \left(\alpha \right(a),\alpha (b\left)\right)\theta (x,y)-\theta \left(\alpha \right(y),\alpha (b\left)\right)\theta (x,a)-\theta \left(\alpha \right(x),[y,a,b\left]\right)\circ \beta +D\left(\alpha \right(y),\alpha (a\left)\right)\theta (x,b)=0,\hfill \end{array}$$

(5)

$$\begin{array}{cc}\hfill & \theta \left(\alpha \right(a),\alpha (b\left)\right)D(x,y)-D\left(\alpha \right(x),\alpha (y\left)\right)\theta (a,b)+\theta \left(\right[x,y,a],\alpha (b\left)\right)\circ \beta +\theta \left(\alpha \right(a),[x,y,b\left]\right)\circ \beta =0,\hfill \end{array}$$

(6)

where $D(x,y)=\theta (y,x)-\theta (x,y)$. We also denote a representation of $\mathfrak{L}$ on $(V,\beta )$ by $(V,\beta ;\theta )$. In particular, $(V,\beta ;\theta )$ is called a regular representation of $\mathfrak{L}$ if β is an invertible linear map on the vector space V.

Example 4. 

Let $(\mathfrak{L},[-,-,-],\alpha )$ be a Hom-Lie triple system. Define bilinear map

$$\mathcal{R}:\mathfrak{L}\times \mathfrak{L}\to \mathrm{End}\left(\mathfrak{L}\right),(a_1,a_2)\mapsto (x\mapsto [x,a_1,a_2]),$$

with $\mathcal{L}(a_1,a_2)\left(x\right)=\mathcal{R}(a_2,a_1)x-\mathcal{R}(a_1,a_2)x=[a_1,a_2,x]$. Then, $(\mathfrak{L},\alpha ;\mathcal{R})$ is a representation of the Hom-Lie triple system $\mathfrak{L}$, which is called the adjoint representation of $\mathfrak{L}$.

Let $(V,\beta ;\theta )$ be a representation of a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$. Denote the $(2n+1)$-cochains of $\mathfrak{L}$ with coefficients in representation $(V,\beta ;\theta )$ by

$$\begin{array}{cc}\hfill & {\mathcal{C}}_{\mathrm{HLts}}^{2n+1}(\mathfrak{L},V):=\{f\in \mathrm{Hom}({\mathfrak{L}}^{\otimes 2n+1},V)\left|\beta \right(f(a_1,\cdots ,a_{2n+1})=f(\alpha \left(a_1\right),\cdots ,\alpha \left(a_{2n+1}\right)),\hfill \\ \hfill & f(a_1,\cdots ,a_{2n-2},a,b,c)+f(a_1,\cdots ,a_{2n-2},b,a,c)=0,{\circlearrowright }_{a,b,c}f(a_1,\cdots ,a_{2n-2},a,b,c)=0\}.\hfill \end{array}$$

For $n\ge 1,$ let $\delta :{\mathcal{C}}_{\mathrm{HLts}}^{2n-1}(\mathfrak{L},V)\to {\mathcal{C}}_{\mathrm{HLts}}^{2n+1}(\mathfrak{L},V)$ be the corresponding coboundary operator of the Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ with coefficients in the representation $(V,\beta ;\theta )$, More precisely, for $a_1,\cdots ,a_{2n+1}\in \mathfrak{L}$ and $f\in {\mathcal{C}}_{\mathrm{HLts}}^{2n-1}(\mathfrak{L},V)$ as

$$\begin{array}{cc}\hfill & \delta f(a_1,\cdots ,a_{2n+1})\hfill \\ \hfill =& \theta ({\alpha }^{n-1}\left(a_{2n}\right),{\alpha }^{n-1}\left(a_{2n+1}\right))f(a_1,\cdots ,a_{2n-1})-\theta ({\alpha }^{n-1}\left(a_{2n-1}\right),{\alpha }^{n-1}\left(a_{2n+1}\right))f(a_1,\cdots ,a_{2n-2},a_{2n})\hfill \\ \hfill & +\sum _{i=1}^n{(-1)}^{i+n}D({\alpha }^{n-1}\left(a_{2i-1}\right),{\alpha }^{n-1}\left(a_{2i}\right))f(a_1,\cdots ,a_{2i-2},a_{2i+1},\cdots ,a_{2n+1})\hfill \\ \hfill & +\sum _{i=1}^n\sum _{j=2i+1}^{2n+1}{(-1)}^{i+n+1}f(\alpha \left(a_1\right),\cdots ,\alpha \left(a_{2i-2}\right),\alpha \left(a_{2i+1}\right),\cdots ,[a_{2i-1},a_{2i},a_j],\cdots ,\alpha \left(a_{2n+1}\right)).\hfill \end{array}$$

So, $\delta \circ \delta =0.$ See [6] for more details.

In particular, for $f\in {\mathcal{C}}_{\mathrm{HLts}}^1(\mathfrak{L},V)$, f is a 1-cocycle on $(\mathfrak{L},[-,-,-],\alpha )$ with coefficients in $(V,\beta ;\theta )$ if $\delta f=0,$ i.e.,

$$\theta (a_2,a_3)f\left(a_1\right)-\theta (a_1,a_3)f\left(a_2\right)+D(a_1,a_2)f\left(a_3\right)-f\left([a_1,a_2,a_3]\right)=0.$$

A 3-cochain $\mathfrak{H}\in {\mathcal{C}}_{\mathrm{HLts}}^3(\mathfrak{L},V)$ is a 3-cocycle on $(\mathfrak{L},[-,-,-],\alpha )$ with coefficients in $(V,\beta ;\theta )$ if $\delta \mathfrak{H}=0,$ i.e.,

$$\begin{array}{cc}\hfill & \theta (\alpha \left(a_4\right),\alpha \left(a_5\right))\mathfrak{H}(a_1,a_2,a_3)-\theta (\alpha \left(a_3\right),\alpha \left(a_5\right))\mathfrak{H}(a_1,a_2,a_4)-D(\alpha \left(a_1\right),\alpha \left(a_2\right))\mathfrak{H}(a_3,a_4,a_5)\hfill \\ \hfill & +D(\alpha \left(a_3\right),\alpha \left(a_4\right))\mathfrak{H}(a_1,a_2,a_5)+\mathfrak{H}([a_1,a_2,a_3],\alpha \left(a_4\right),\alpha \left(a_5\right))+\mathfrak{H}(\alpha \left(a_3\right),[a_1,a_2,a_4],\alpha \left(a_5\right))\hfill \\ \hfill & +\mathfrak{H}(\alpha \left(a_3\right),\alpha \left(a_4\right),[a_1,a_2,a_5])-\mathfrak{H}(\alpha \left(a_1\right),\alpha \left(a_2\right),[a_3,a_4,a_5])=0.\hfill \end{array}$$

(7)

3. Generalized Reynolds Operators on Hom-Lie Triple Systems

In this section, we introduce the notion of generalized Reynolds operators on Hom-Lie triple systems, which can be regarded as the generalization of relative Rota–Baxter operators on Hom-Lie triple systems [17,19] and generalized Reynolds operators on Lie triple systems [29,30]. We give its characterization by a graph and provide some examples.

Definition 3. 

(i) Let $(\mathfrak{L},[-,-,-],\alpha )$ be a Hom-Lie triple system and $(V,\beta ;\theta )$ be a representation of $\mathfrak{L}$. A linear operator $R:V\to \mathfrak{L}$ is called a generalized Reynolds operator on a Hom-Lie triple system associated to $(V,\beta ;\theta )$ and 3-cocycle $\mathfrak{H}$ if R satisfies:

$$\begin{array}{cc}\hfill & \alpha \left(Ru\right)=R\beta \left(u\right),\hfill \end{array}$$

(8)

$$\begin{array}{cc}\hfill & [Ru,Rv,Rw]=R\left(\theta \right(Rv,Rw)u+D(Ru,Rv)w-\theta (Ru,Rw)v+\mathfrak{H}(Ru,Rv,Rw\left)\right),\hfill \end{array}$$

(9)

where $u,v,w\in V$.

(ii) A morphism of generalized Reynolds operators from R to $R^{\prime }$ consists of a pair $(\eta ,\zeta )$ of a Hom-Lie triple system morphism $\eta :(\mathfrak{L},[-,-,-],\alpha )\to ({\mathfrak{L}}^{\prime },{[-,-,-]}^{\prime },{\alpha }^{\prime })$ and a linear map $\zeta :V\to V^{\prime }$ satisfying

$$\begin{array}{cc}\hfill & {\alpha }^{\prime }\circ \eta =\eta \circ \alpha ,{\beta }^{\prime }\circ \zeta =\zeta \circ \beta ,\hfill \end{array}$$

(10)

$$\begin{array}{cc}\hfill & \eta \circ R=R^{\prime }\circ \zeta ,\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & \zeta \left(\theta (a,b)u\right)={\theta }^{\prime }(\eta \left(a\right),\eta \left(b\right))\zeta \left(u\right),\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & \zeta \left(\mathfrak{H}(a,b,c)\right)={\mathfrak{H}}^{\prime }(\eta \left(a\right),\eta \left(b\right),\eta \left(c\right)),\hfill \end{array}$$

(13)

for $a,b,c\in \mathfrak{L},u\in V.$

Remark 1. 

(i) A generalized Reynolds operator R on Hom-Lie triple system $(\mathfrak{L},[-,-,-],$$\alpha )$ with $\alpha =\mathrm{id}$ is nothing but a generalized Reynolds operator R on Lie triple system $(\mathfrak{L},[-,-,-\left]\right)$. See [29,30] for more details about generalized Reynolds operators on Lie triple systems.

(ii) Any relative Rota–Baxter operator (in particular, a Rota–Baxter operator of weight 0) on a Hom-Lie triple system is a generalized Reynolds operator with $\mathfrak{H}=0$. See [17,19] for more details about relative Rota–Baxter operators on Hom-Lie triple systems.

Example 5. 

Let $(V,\beta ;\theta )$ be a representation of a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$. Suppose that $f:\mathfrak{L}\to V$ is an invertible linear map and f satisfies $\beta \circ f=f\circ \alpha$, take $\mathfrak{H}=-\delta f$. Then, $R=f^{-1}:V\to \mathfrak{L}$ is a generalized Reynolds operator.

Example 6. 

In [9], Hou, Ma and Chen introduced the notion of the Nijenhuis operator by the 2-order deformation of Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$. More precisely, a linear map $N:\mathfrak{L}\to \mathfrak{L}$ is called a Nijenhuis operator if for all $a,b,c\in \mathfrak{L}$ the following equations hold:

$$\begin{array}{cc}\hfill N\circ \alpha =& \alpha \circ N,\hfill \\ \hfill [Na,Nb,Nc]=& N([a,Nb,Nc]+[Na,b,Nc]+[Na,Nb,c])-N^2([Na,b,c]+[a,Nb,c]\hfill \\ \hfill & +[a,b,Nc])+N^3[a,b,c].\hfill \end{array}$$

In this case, the Hom vector space $(\mathfrak{L},\alpha )$ carries a new Hom-Lie triple-system structure with bracket

$$\begin{array}{cc}\hfill {[a,b,c]}_N=& [a,Nb,Nc]+[Na,b,Nc]+[Na,Nb,c]-N\left(\right[Na,b,c]+[a,Nb,c]+[a,b,Nc\left]\right)\hfill \\ \hfill & +N^2[a,b,c],\forall a,b,c\in \mathfrak{L}.\hfill \end{array}$$

This deformed Hom-Lie triple system ${\mathfrak{L}}_N=(\mathfrak{L},{[-,-,-]}_N,\alpha )$ has a representation on $(\mathfrak{L},\alpha )$ by ${\theta }_N(a,b)c=[c,Na,Nb]$ for $a,b\in {\mathfrak{L}}_N,c\in \mathfrak{L}$. The map $\mathfrak{H}:{\mathfrak{L}}_N\times {\mathfrak{L}}_N\times {\mathfrak{L}}_N\to \mathfrak{L},\mathfrak{H}(a,b,c)=-N([Na,b,c]+[a,Nb,c]+[a,b,Nc])+N^2[a,b,c]$ is a 3-cocycle with coefficients in $(L,\alpha ;{\theta }_N)$. Moreover, the identity map $id:\mathfrak{L}\to {\mathfrak{L}}_N$ is a generalized Reynolds operator.

Example 7. 

Let $(\mathfrak{L},[-,-,-],\alpha )$ be a Hom-Lie triple system and $(\mathfrak{L},\alpha ;\mathcal{R})$ the adjoint representation. Set the 3-cocycle $\mathfrak{H}(a,b,c)=-[a,b,c]$, for $a,b,c\in \mathfrak{L}$; then, a linear operator $T:\mathfrak{L}\to \mathfrak{L}$ defined by Equations (8) and (9) is called a Reynolds operator on $(\mathfrak{L},[-,-,-],\alpha )$. More specifically, T satisfies:

$$\begin{array}{cc}\hfill & \alpha \left(Ta\right)=T\alpha \left(a\right),\hfill \\ \hfill & [Ta,Tb,Tc]=T\left(\right[Ta,Tb,c]+[a,Tb,Tc]+[Ta,b,Tc]-[Ta,Tb,Tc\left]\right),\hfill \end{array}$$

where $a,b,c\in \mathfrak{L}$.

Example 8. 

Let $D:\mathfrak{L}\to \mathfrak{L}$ be a derivation on a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$. If $D+\frac{1}{2}id$ is invertible, then ${(D+\frac{1}{2}id)}^{-1}$ is a Reynolds operator on $(\mathfrak{L},[-,-,-],\alpha )$.

Given a 3-cocycle $\mathfrak{H}$ in the cochain complex of $\mathfrak{L}$ with coefficients in V, one can construct the twisted-semidirect-product Hom-Lie triple system. More precisely, the direct sum $\mathfrak{L}\oplus V$ carries a Hom-Lie triple-system structure with the bracket given by

$$\begin{array}{cc}\hfill & {[a+u,b+v,c+w]}_{{⋉}_{\mathfrak{H}}}=[a,b,c]+D(a,b)w-\theta (a,c)v+\theta (b,c)u+\mathfrak{H}(a,b,c),\hfill \\ \hfill & (\alpha \oplus \beta )(a+u)=\alpha \left(a\right)+\beta \left(u\right),\forall a,b,c\in \mathfrak{L},u,v,w\in V.\hfill \end{array}$$

We denote this twisted-semidirect-product Hom-Lie triple system by $\mathfrak{L}{⋉}_{\mathfrak{H}}V$.

Proposition 1. 

A linear map $R:V\to \mathfrak{L}$ is a generalized Reynolds operator on $\mathfrak{L}$ if and only if the graph of R

$$Gr\left(R\right)=\{Ru+u|u\in V\}$$

is a subalgebra of the twisted-semidirect-product Hom-Lie triple system by $\mathfrak{L}{⋉}_{\mathfrak{H}}V$.

Proof. 

Let $R:V\to \mathfrak{L}$ be a linear map; then, for any $u,v,w\in V$, we have

$$\begin{array}{cc}\hfill \alpha \oplus \beta (Ru+u)=& \alpha \circ R\left(u\right)+\beta \left(u\right),\hfill \\ \hfill {[Ru+u,Rv+Rw+w]}_{{⋉}_{\mathfrak{H}}}=& [Ru,Rv,Rw]+D(Ru,Rv)w-\theta (Ru,Rw)v+\theta (Rv,Rw)u\hfill \\ \hfill & +\mathfrak{H}(Ru,Rv,Rw),\hfill \end{array}$$

which implies that the graph $Gr\left(R\right)$ is a subalgebra of the twisted-semidirect-product Hom-Lie triple system $\mathfrak{L}{⋉}_{\mathfrak{H}}V$ if and only if R satisfies Equations (8) and (9), which means that R is a generalized Reynolds operator. □

$Gr\left(R\right)$ is isomorphic to V as a vector space. Define a trilinear operation on V by

$$\begin{array}{cc}\hfill & {[u,v,w]}_R=D(Ru,Rv)w-\theta (Ru,Rw)v+\theta (Rv,Rw)u+\mathfrak{H}(Ru,Rv,Rw),\hfill \end{array}$$

(14)

for all $u,v,w\in V.$ By Proposition 1, we ascertain that $(V,{[-,-,-]}_R,\beta )$ is a Hom-Lie triple system. Moreover, R is a homomorphism of Hom-Lie triple systems from $(V,{[-,-,-]}_R,\beta )$ to $(\mathfrak{L},[-,-,-],\alpha )$.

4. Cohomology of Generalized Reynolds Operators on Hom-Lie Triple Systems

In this section, first, we construct a representation of the Hom-Lie triple system $(V,{[-,-,-]}_R,\beta )$ on the Hom-vector space $(\mathfrak{L},\alpha )$. Then, we develop a cohomology theory of generalized Reynolds operators on Hom-Lie triple systems.

Lemma 1. 

Let $R:V\to \mathfrak{L}$ be a generalized Reynolds operator on a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ associated to $(V,\beta ;\theta )$ and 3-cocycle $\mathfrak{H}$. For any $u,v\in V$, $a\in \mathfrak{L}$, define ${\theta }_R:V\otimes V\to \mathrm{End}\left(\mathfrak{L}\right)$ by

$$\begin{array}{c}\hfill {\theta }_R(u,v)\left(a\right)=[a,Ru,Rv]+R(\theta (a,Rv)u-D(a,Ru)v-\mathfrak{H}(a,Ru,Rv));\end{array}$$

(15)

then, $(\mathfrak{L},\alpha ;{\theta }_R)$ is a representation of the Hom-Lie triple system $(V,{[-,-,-]}_R,\beta )$.

Proof. 

For any $u,v,s,t\in V,a\in \mathfrak{L}$, note that

$$\begin{array}{cc}\hfill & D_R(u,v)\left(a\right)\hfill \\ \hfill =& {\theta }_R(v,u)\left(a\right)-{\theta }_R(u,v)\left(a\right)\hfill \\ \hfill =& [a,Rv,Ru]+R\left(\theta \right(a,Ru)v-D(a,Rv)u-\mathfrak{H}(a,Rv,Ru\left)\right)-[a,Ru,Rv]\hfill \\ \hfill & -R\left(\theta \right(a,Rv)u-D(a,Ru)v-\mathfrak{H}(a,Ru,Rv\left)\right)\hfill \\ \hfill =& [Ru,Rv,a]+R\left(\theta \right(Ru,a)v-\theta (Rv,a)u-\mathfrak{H}(Ru,Rv,a\left)\right).\hfill \end{array}$$

(16)

Further, we obtain

$$\begin{array}{cc}\hfill & {\theta }_R(\beta \left(u\right),\beta \left(v\right))\alpha \left(a\right)(\mathrm{by}\mathrm{Equations}\left(4\right),\left(8\right)\mathrm{and}\left(15\right))\hfill \\ \hfill =& \left[\alpha \right(a),R\beta (u),R\beta (v\left)\right]+R\left(\theta \right(\alpha \left(a\right),R\beta \left(v\right)\left)\beta \right(u)-D(\alpha \left(a\right),R\beta \left(u\right)\left)\beta \right(v)-\mathfrak{H}(\alpha \left(a\right),R\beta \left(u\right),R\beta \left(v\right)\left)\right)\hfill \\ \hfill =& \left[\alpha \right(a),\alpha (Ru),\alpha (Rv\left)\right]+R\left(\theta \right(\alpha \left(a\right),\alpha \left(Rv\right)\left)\beta \right(u)-D(\alpha \left(a\right),\alpha \left(Ru\right)\left)\beta \right(v)-\mathfrak{H}(\alpha \left(a\right),\alpha \left(Ru\right),\alpha \left(Rv\right)\left)\right)\hfill \\ \hfill =& \alpha \left(\right[a,Ru,Rv]+R(\theta (a,Rv)u-D(a,Ru)v-\mathfrak{H}(a,Ru,Rv)\left)\right)\hfill \\ \hfill =& \alpha \left({\theta }_R(u,v)\left(a\right)\right),\hfill \end{array}$$

$$\begin{array}{cc}\hfill & {\theta }_R(\beta \left(u\right),\beta \left(v\right)){\theta }_R(s,t)a-{\theta }_R(\beta \left(t\right),\beta \left(v\right)){\theta }_R(s,u)a-{\theta }_R(\beta \left(s\right),{[t,u,v]}_R)\alpha \left(a\right)\hfill \\ \hfill & +D_R(\beta \left(t\right),\beta \left(u\right)){\theta }_R(s,v)a(\mathrm{by}\mathrm{Equations}\left(8\right),\left(9\right)\mathrm{and}\left(14\right)–\left(16\right))\hfill \\ \hfill =& {\theta }_R(\beta \left(u\right),\beta \left(v\right))\left([a,Rs,Rt]+R\left(\theta \right(a,Rt)s-D(a,Rs)t-\mathfrak{H}(a,Rs,Rt\left)\right)\right)\hfill \\ \hfill & -{\theta }_R(\beta \left(t\right),\beta \left(v\right))\left([a,Rs,Ru]+R\left(\theta \right(a,Ru)s-D(a,Rs)u-\mathfrak{H}(a,Rs,Ru\left)\right)\right)\hfill \\ \hfill & -[\alpha \left(a\right),R\beta \left(s\right),R{[t,u,v]}_R]-R(\theta (\alpha \left(a\right),R{[t,u,v]}_R)\beta \left(s\right)-D(\alpha \left(a\right),R\beta \left(s\right)){[t,u,v]}_R\hfill \\ \hfill & -\mathfrak{H}(\alpha \left(a\right),R\beta \left(s\right),R{[t,u,v]}_R))+D_R(\beta \left(t\right),\beta \left(u\right))\left([a,Rs,Rv]+R\left(\theta \right(a,Rv)s-D(a,Rs)v-\mathfrak{H}(a,Rs,Rv\left)\right)\right)\hfill \\ \hfill =& \left(\right[[a,Rs,Rt],\alpha \left(Ru\right),\alpha \left(Rv\right)]-[[a,Rs,Ru],\alpha \left(Rt\right),\alpha \left(Rv\right)]+[\alpha \left(Rt\right),\alpha \left(Ru\right),[a,Rs,Rv]]\hfill \\ \hfill & -\left[\alpha \right(a),\alpha (Rs),[Rt,Ru,Rv\left]\right])+R(\theta \left(\alpha \right(Rt),\alpha (Rv\left)\right)\mathfrak{H}(a,Rs,Ru)-\theta \left(\alpha \right(Ru),\alpha (Rv\left)\right)\mathfrak{H}(a,Rs,Rt)\hfill \\ \hfill & +D\left(\alpha \right(Ru),\alpha (Rt\left)\right)\mathfrak{H}(a,Rs,Rv)+D\left(\alpha \right(a),\alpha (Rs\left)\right)\mathfrak{H}(Rt,Ru,Rv)+\mathfrak{H}\left(\alpha \right(a),\alpha (Rs),[Rt,Ru,Rv\left]\right)\hfill \\ \hfill & +\theta \left(\alpha \right(Ru),\alpha (Rv\left)\right)\theta (a,Rt)s-\theta \left(\alpha \right(Ru),\alpha (Rv\left)\right)D(a,Rs)t-\theta \left(\alpha \right(Rt),\alpha (Rv\left)\right)\theta (a,Ru)s\hfill \\ \hfill & +\theta \left(\alpha \right(Rt),\alpha (Rv\left)\right)D(a,Rs)u+D\left(\alpha \right(a),\alpha (Rs\left)\right)\theta (Ru,Rv)t-D\left(\alpha \right(a),\alpha (Rs\left)\right)\theta (Rt,Rv)u\hfill \\ \hfill & +D\left(\alpha \right(a),\alpha (Rs\left)\right)D(Rt,Ru)v-\theta \left(\alpha \right(a),[Rt,Ru,Rv\left]\right)\beta \left(s\right)+D\left(\alpha \right(Rt),\alpha (Ru\left)\right)\theta (a,Rv)s\hfill \\ \hfill & +D\left(\alpha \right(Ru),\alpha (Rv\left)\right)D(a,Rs)t-D\left(\right[a,Rs,Rt],\alpha (Ru\left)\right)\beta \left(v\right)+D\left(\right[a,Rs,Rt],\alpha (Rv\left)\right)\beta \left(u\right)\hfill \\ \hfill & +\theta \left(\right[a,Rs,Rt],\alpha (Rv\left)\right)\beta \left(u\right)-\theta \left(\right[a,Rs,Ru],\alpha (Rv\left)\right)\beta \left(t\right)-D\left(\right[a,Rs,Rv],\alpha (Ru\left)\right)\beta \left(t\right)\hfill \\ \hfill & +\theta \left(\alpha \right(Rt),[a,Rs,Rv\left]\right)\beta \left(u\right)+D\left(\right[a,Rs,Rv],\alpha (Rt\left)\right)\beta \left(u\right)-\theta \left(\alpha \right(Ru),[a,Rs,Rv\left]\right)\beta \left(t\right)\left)(\mathrm{by}\mathrm{Equations}\right(5\left)\mathrm{and}\right(6\left)\right)\hfill \\ \hfill =& \left(\right[[a,Rs,Rt],\alpha \left(Ru\right),\alpha \left(Rv\right)]-[[a,Rs,Ru],\alpha \left(Rt\right),\alpha \left(Rv\right)]+[\alpha \left(Rt\right),\alpha \left(Ru\right),[a,Rs,Rv]]\hfill \\ \hfill & -\left[\alpha \right(a),\alpha (Rs),[Rt,Ru,Rv\left]\right])+R(\theta \left(\alpha \right(Rt),\alpha (Rv\left)\right)\mathfrak{H}(a,Rs,Ru)-\theta \left(\alpha \right(Ru),\alpha (Rv\left)\right)\mathfrak{H}(a,Rs,Rt)\hfill \\ \hfill & +D\left(\alpha \right(Ru),\alpha (Rt\left)\right)\mathfrak{H}(a,Rs,Rv)+D\left(\alpha \right(a),\alpha (Rs\left)\right)\mathfrak{H}(Rt,Ru,Rv)+\mathfrak{H}\left(\alpha \right(a),\alpha (Rs),[Rt,Ru,Rv\left]\right)\hfill \\ \hfill & -\mathfrak{H}\left(\right[a,Rs,Rt],\alpha (Ru),\alpha (Rv\left)\right)-\mathfrak{H}\left(\alpha \right(Rt),[a,Rs,Ru],\alpha (Rv\left)\right)-\mathfrak{H}\left(\alpha \right(Rt),\alpha (Ru),[a,Rs,Rv\left]\right)\left)(\mathrm{by}\mathrm{Equations}\right(3\left)\mathrm{and}\right(7\left)\right)\hfill \\ \hfill =& 0.\hfill \end{array}$$

Similarly, we also have

$$\begin{array}{cc}\hfill & {\theta }_R(\beta \left(u\right),\beta \left(v\right))D_R(s,t)a-D_R(\beta \left(s\right),\beta \left(t\right)){\theta }_R(u,v)a+{\theta }_R({[s,t,u]}_R,\beta \left(v\right))\alpha \left(a\right)\hfill \\ \hfill & +{\theta }_R(\beta \left(u\right),{[s,t,v]}_R)\alpha \left(a\right)=0.\hfill \end{array}$$

Therefore, $(\mathfrak{L},\alpha ;{\theta }_R)$ is a representation of $(V,{[-,-,-]}_R,\beta )$. □

Let $R:V\to \mathfrak{L}$ be a generalized Reynolds operator on a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ associated to $(V,\beta ;\theta )$ and 3-cocycle $\mathfrak{H}$. Recall that Lemma 1 gives a representation $(\mathfrak{L},\alpha ;{\theta }_R)$ of the Hom-Lie triple system $(V,{[-,-,-]}_R,\beta )$. Consider the cochain complex of $(V,{[-,-,-]}_R,\beta )$ with coefficients in $(\mathfrak{L},\alpha ;{\theta }_R)$:

$$({\oplus }_{n=1}^{\infty }{\mathcal{C}}_R^{2n-1}(V,\mathfrak{L}),{\delta }_R).$$

More precisely,

$$\begin{array}{cc}\hfill & {\mathcal{C}}_R^{2n+1}(V,\mathfrak{L}):=\{f\in \mathrm{Hom}(V^{\otimes 2n+1},\mathfrak{L})|\alpha \left(f(v_1,\cdots ,v_{2n+1})\right)=f(\beta \left(v_1\right),\cdots ,\beta \left(v_{2n+1}\right)),\hfill \\ \hfill & f(v_1,\cdots ,v_{2n-2},u,v,w)+f(v_1,\cdots ,v_{2n-2},v,u,w)=0,{\circlearrowright }_{u,v,w}f(v_1,\cdots ,v_{2n-2},u,v,w)=0\},\hfill \end{array}$$

and its coboundary map ${\delta }_R:{\mathcal{C}}_R^{2n-1}(V,\mathfrak{L})\to {\mathcal{C}}_R^{2n+1}(V,\mathfrak{L})$ is given as follows:

$$\begin{array}{cc}\hfill & {\delta }_{R}f(v_1,\cdots ,v_{2n+1})\hfill \\ \hfill =& {\theta }_R({\beta }^{n-1}\left(v_{2n}\right),{\beta }^{n-1}\left(v_{2n+1}\right))f(v_1,\cdots ,v_{2n-1})-{\theta }_R({\beta }^{n-1}\left(v_{2n-1}\right),{\beta }^{n-1}\left(v_{2n+1}\right))f(v_1,\cdots ,v_{2n-2},v_{2n})\hfill \\ \hfill & +\sum _{i=1}^n{(-1)}^{i+n}{D}_R({\beta }^{n-1}\left(v_{2i-1}\right),{\beta }^{n-1}\left(v_{2i}\right))f(v_1,\cdots ,v_{2i-2},v_{2i+1},\cdots ,v_{2n+1})\hfill \\ \hfill & +\sum _{i=1}^n\sum _{j=2i+1}^{2n+1}{(-1)}^{i+n+1}f(\beta \left(v_1\right),\cdots ,\beta \left(v_{2i-2}\right),\beta \left(v_{2i+1}\right),\cdots ,{[v_{2i-1},v_{2i},v_j]}_R,\cdots ,\beta \left(v_{2n+1}\right)).\hfill \end{array}$$

for any $v_1,\cdots ,v_{2n+1}\in V$ and $f\in {\mathcal{C}}_R^{2n-1}(V,\mathfrak{L})$.

In particular, for $f\in {\mathcal{C}}_R^1(V,\mathfrak{L})=\{f\in \mathrm{Hom}(V,\mathfrak{L})|\alpha \circ f=f\circ \beta \}$, $v_1,v_2,v_3\in V$, we have

$${\delta }_{R}f(v_1,v_2,v_3)={\theta }_R(v_2,v_3)f\left(v_1\right)-{\theta }_R(v_1,v_3)f\left(v_2\right)+D_R(v_1,v_2)f\left(v_3\right)-f\left({[v_1,v_2,v_3]}_R\right).$$

Next, when $n=0$, in order to get the first cohomology group of the generalized Reynolds operator R, we need additional conditions; that is, the Hom-Lie triple system is regular, and its representation is also regular. In the next section, we will use the first cohomology group to classify the linear deformation of generalized Reynolds operators; see Proposition 3.

For any $(a,b)\in {\mathcal{C}}_R^0(V,\mathfrak{L}):=\{(a,b)\in {\wedge }^2\mathfrak{L}|\alpha \left(a\right)=a,\alpha \left(b\right)=b\}$, we define ${\delta }_R:{\mathcal{C}}_R^0(V,\mathfrak{L})\to {\mathcal{C}}_R^1(V,\mathfrak{L}),(a,b)\mapsto \wp (a,b)$ by

$$\wp (a,b)u=RD(a,b){\beta }^{-1}\left(u\right)-[a,b,R{\beta }^{-1}\left(u\right)]+R\mathfrak{H}(a,b,R{\beta }^{-1}\left(u\right)),\forall u\in V$$

where $\beta$ is an invertible linear map on the vector space V.

Proposition 2. 

Let $R:V\to \mathfrak{L}$ be a generalized Reynolds operator on a regular Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ associated to regular representation $(V,\beta ;\theta )$ and 3-cocycle $\mathfrak{H}$. Then, ${\delta }_R(\wp (a,b))=0$; that is, the composition ${\mathcal{C}}_R^0(V,\mathfrak{L})\stackrel{{\delta }_R}{⟶}{\mathcal{C}}_R^1(V,\mathfrak{L})\stackrel{{\delta }_R}{⟶}{\mathcal{C}}_R^3(V,\mathfrak{L})$ is the zero map.

Proof. 

For any $v_1,v_2,v_3\in V$, first, evidently $\alpha (\wp (a,b)\left(v_1\right))=\wp (a,b)\left(\beta \left(v_1\right)\right)$. Next, we have

$$\begin{array}{cc}\hfill & {\delta }_R\wp (a,b)(v_1,v_2,v_3)(\mathrm{by}\mathrm{Equations}\left(4\right),\left(8\right),\left(9\right)\mathrm{and}\left(14\right)–\left(16\right))\hfill \\ \hfill =& {\theta }_R(v_2,v_3)\wp (a,b)\left(v_1\right)-{\theta }_R(v_1,v_3)\wp (a,b)\left(v_2\right)+D_R(v_1,v_2)\wp (a,b)\left(v_3\right)-\wp (a,b)\left({[v_1,v_2,v_3]}_R\right)\hfill \\ \hfill =& {\theta }_R(v_2,v_3)RD(a,b){\beta }^{-1}\left(v_1\right)-{\theta }_R(v_2,v_3)[a,b,R{\beta }^{-1}\left(v_1\right)]+{\theta }_R(v_2,v_3)R\mathfrak{H}(a,b,R{\beta }^{-1}\left(v_1\right))\hfill \\ \hfill & -{\theta }_R(v_1,v_3)RD(a,b){\beta }^{-1}\left(v_2\right)+{\theta }_R(v_1,v_3)[a,b,R{\beta }^{-1}\left(v_2\right)]-{\theta }_R(v_1,v_3)R\mathfrak{H}(a,b,R{\beta }^{-1}\left(v_2\right))\hfill \\ \hfill & +D_R(v_1,v_2)RD(a,b){\beta }^{-1}\left(v_3\right)-D_R(v_1,v_2)[a,b,R{\beta }^{-1}\left(v_3\right)]+D_R(v_1,v_2)R\mathfrak{H}(a,b,R{\beta }^{-1}\left(v_3\right))\hfill \\ \hfill & -RD(a,b){\beta }^{-1}\left({[v_1,v_2,v_3]}_R\right)+[a,b,R{\beta }^{-1}\left({[v_1,v_2,v_3]}_R\right)]-R\mathfrak{H}(a,b,R{\beta }^{-1}\left({[v_1,v_2,v_3]}_R\right))\hfill \\ \hfill =& {\alpha }^{-1}([RD(a,b)v_1,\alpha \left(R{v}_2\right),\left(R{v}_3\right)]+R\theta (RD(a,b)v_1,\alpha \left(R{v}_3\right))\beta \left(v_2\right)-RD(RD(a,b)v_1,\alpha \left(R{v}_2\right))\beta \left(v_3\right)\hfill \\ \hfill & -R\mathfrak{H}(RD(a,b)v_1,\alpha \left(R{v}_2\right),\alpha \left(R{v}_3\right))-[[a,b,R{v}_1],\alpha \left(R{v}_2\right),\alpha \left(R{v}_3\right)]-R\theta ([a,b,R{v}_1],\alpha \left(R{v}_3\right))\beta \left(v_2\right)\hfill \\ \hfill & +RD([a,b,R{v}_1],\alpha \left(R{v}_2\right))\beta \left(v_3\right)+R\mathfrak{H}([a,b,R{v}_1],\alpha \left(R{v}_2\right),\alpha \left(R{v}_3\right))+[R\mathfrak{H}(a,b,R{v}_1),\alpha \left(R{v}_2\right),\alpha \left(R{v}_3\right)]\hfill \\ \hfill & +R\theta (R\mathfrak{H}(a,b,R{v}_1),\alpha \left(R{v}_3\right))\beta \left(v_2\right)-RD(R\mathfrak{H}(a,b,R{v}_1),\alpha \left(R{v}_2\right))\beta \left(v_3\right)-R\mathfrak{H}(R\mathfrak{H}(a,b,R{v}_1),\alpha \left(R{v}_2\right),\alpha \left(R{v}_3\right))\hfill \\ \hfill & -[RD(a,b)v_2,\alpha \left(R{v}_1\right),\alpha \left(R{v}_3\right)]-R\theta (RD(a,b)v_2,\alpha \left(R{v}_3\right))\beta \left(v_1\right)+RD(RD(a,b)v_2,\alpha \left(R{v}_1\right))\beta \left(v_3\right)\hfill \\ \hfill & +R\mathfrak{H}(RD(a,b)v_2,\alpha \left(R{v}_1\right),\alpha \left(R{v}_3\right))+[[a,b,R{v}_2],\alpha \left(R{v}_1\right),\alpha \left(R{v}_3\right)]+R\theta ([a,b,R{v}_2],\alpha \left(R{v}_3\right))\beta \left(v_1\right)\hfill \\ \hfill & -RD([a,b,R{v}_2],\alpha \left(R{v}_1\right))\beta \left(v_3\right)-R\mathfrak{H}([a,b,R{v}_2],\alpha \left(R{v}_1\right),\alpha \left(R{v}_3\right))-[R\mathfrak{H}(a,b,R{v}_2),\alpha \left(R{v}_1\right),\alpha \left(R{v}_3\right)]\hfill \\ \hfill & -R\theta (R\mathfrak{H}(a,b,R{v}_2),\alpha \left(R{v}_3\right))\beta \left(v_1\right)+RD(R\mathfrak{H}(a,b,R{v}_2),\alpha \left(R{v}_1\right))\beta \left(v_3\right)+R\mathfrak{H}(R\mathfrak{H}(a,b,R{v}_2),\alpha \left(R{v}_1\right),\alpha \left(R{v}_3\right))\hfill \\ \hfill & +[\alpha \left(R{v}_1\right),\alpha \left(R{v}_2\right),RD(a,b)v_3]+R\theta (\alpha \left(R{v}_1\right),RD(a,b)v_3)\beta \left(v_2\right)-R\theta (\alpha \left(R{v}_2\right),RD(a,b)v_3)\beta \left(v_1\right)\hfill \\ \hfill & -R\mathfrak{H}(\alpha \left(R{v}_1\right),\alpha \left(R{v}_2\right),RD(a,b)v_3)-[\alpha \left(R{v}_1\right),\alpha \left(R{v}_2\right),[a,b,R{v}_3]]-R\theta (\alpha \left(R{v}_1\right),[a,b,R{v}_3])\beta \left(v_2\right)\hfill \\ \hfill & +R\theta (\alpha \left(R{v}_2\right),[a,b,R{v}_3])\beta \left(v_1\right)+R\mathfrak{H}(\alpha \left(R{v}_1\right),\alpha \left(R{v}_2\right),[a,b,R{v}_3])+[\alpha \left(R{v}_1\right),\alpha \left(R{v}_2\right),R\mathfrak{H}(a,b,R{v}_3)]\hfill \\ \hfill & +R\theta (\alpha \left(R{v}_1\right),R\mathfrak{H}(a,b,R{v}_3))\beta \left(v_2\right)-R\theta (\alpha \left(R{v}_2\right),R\mathfrak{H}(a,b,R{v}_3))\beta \left(v_1\right)-R\mathfrak{H}(\alpha \left(R{v}_1\right),\alpha \left(R{v}_2\right),R\mathfrak{H}(a,b,R{v}_3))\hfill \\ \hfill & -RD(a,b)D(R{v}_1,R{v}_2)v_3+RD(a,b)\theta (R{v}_1,R{v}_3)v_2-RD(a,b)\theta (R{v}_2,R{v}_3)v_1-RD(a,b)\mathfrak{H}(R{v}_1,R{v}_2,R{v}_3)\hfill \\ \hfill & +[a,b,[R{v}_1,R{v}_2,R{v}_3]]-R\mathfrak{H}(a,b,[R{v}_1,R{v}_2,R{v}_3])\left)(\mathrm{by}\mathrm{Equations}\right(9\left)\mathrm{and}\right(14\left)\right)\hfill \\ \hfill =& {\alpha }^{-1}\left(\right(-[[a,b,R{v}_1],\alpha \left(R{v}_2\right),\alpha \left(R{v}_3\right)]+[[a,b,R{v}_2],\alpha \left(R{v}_1\right),\alpha \left(R{v}_3\right)]-[\alpha \left(R{v}_1\right),\alpha \left(R{v}_2\right),[a,b,R{v}_3]]\hfill \\ \hfill & +[a,b,[R{v}_1,R{v}_2,R{v}_3]])+R(D(\alpha \left(R{v}_1\right),\alpha \left(R{v}_2\right))\mathfrak{H}(a,b,R{v}_3)-\theta (\alpha \left(R{v}_1\right),\alpha \left(R{v}_3\right))\mathfrak{H}(a,b,R{v}_2)\hfill \\ \hfill & +\mathfrak{H}([a,b,R{v}_1],\alpha \left(R{v}_2\right),\alpha \left(R{v}_3\right))+\mathfrak{H}(\alpha \left(R{v}_1\right),\alpha \left(R{v}_2\right),[a,b,R{v}_3])-\mathfrak{H}(a,b,[R{v}_1,R{v}_2,R{v}_3])\hfill \\ \hfill & -D(a,b)\mathfrak{H}(R{v}_1,R{v}_2,R{v}_3)+\theta (\alpha \left(R{v}_2\right),\alpha \left(R{v}_3\right))D(a,b)v_1-\theta (\alpha \left(R{v}_1\right),\alpha \left(R{v}_3\right))D(a,b)v_2\hfill \\ \hfill & +D(\alpha \left(R{v}_1\right),\alpha \left(R{v}_2\right))D(a,b)v_3+D([a,b,R{v}_1],\alpha \left(R{v}_2\right))\beta \left(v_3\right)-\theta ([a,b,R{v}_1],\alpha \left(R{v}_3\right))\beta \left(v_2\right)\hfill \\ \hfill & -D([a,b,R{v}_2],\alpha \left(R{v}_1\right))\beta \left(v_3\right)+\theta ([a,b,R{v}_2],\alpha \left(R{v}_3\right))\beta \left(v_1\right)+\mathfrak{H}(\alpha \left(R{v}_1\right),[a,b,R{v}_2],\alpha \left(R{v}_3\right))\hfill \\ \hfill & +\theta (\alpha \left(R{v}_2\right),[a,b,R{v}_3])\beta \left(v_1\right)-\theta (\alpha \left(R{v}_1\right),[a,b,R{v}_3])\beta \left(v_2\right)-D(a,b)\theta (R{v}_2,R{v}_3)v_1\hfill \\ \hfill & +D(a,b)\theta (R{v}_1,R{v}_3)v_2-D(a,b)D(R{v}_1,R{v}_2)v_3\left)\right)(\mathrm{by}\mathrm{Equations}\left(3\right)\mathrm{and}\left(7\right))\hfill \\ \hfill =& {\alpha }^{-1}\left(R\right(\theta (\alpha \left(R{v}_2\right),\alpha \left(R{v}_3\right))D(a,b)v_1-\theta (\alpha \left(R{v}_1\right),\alpha \left(R{v}_3\right))D(a,b)v_2+D(\alpha \left(R{v}_1\right),\alpha \left(R{v}_2\right))D(a,b)v_3\hfill \\ \hfill & +D([a,b,R{v}_1],\alpha \left(R{v}_2\right))\beta \left(v_3\right)-\theta ([a,b,R{v}_1],\alpha \left(R{v}_3\right))\beta \left(v_2\right)-D([a,b,R{v}_2],\alpha \left(R{v}_1\right))\beta \left(v_3\right)\hfill \\ \hfill & +\theta ([a,b,R{v}_2],\alpha \left(R{v}_3\right))\beta \left(v_1\right)+\theta (\alpha \left(R{v}_2\right),[a,b,R{v}_3])\beta \left(v_1\right)-\theta (\alpha \left(R{v}_1\right),[a,b,R{v}_3])\beta \left(v_2\right)\hfill \\ \hfill & -D(a,b)\theta (R{v}_2,R{v}_3)v_1+D(a,b)\theta (R{v}_1,R{v}_3)v_2-D(a,b)D(R{v}_1,R{v}_2)v_3\left)\right)(\mathrm{by}\mathrm{Equations}\left(4\right)\mathrm{and}\left(5\right))\hfill \\ \hfill =& 0.\hfill \end{array}$$

Therefore, ${\delta }_R\wp (a,b)=0.$ □

Definition 4. 

Let $R:V\to \mathfrak{L}$ be a generalized Reynolds operator on a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ associated to $(V,\beta ;\theta )$ and 3-cocycle $\mathfrak{H}$. Then, the cochain complex $({\mathcal{C}}_R^{•}(V,\mathfrak{L}),{\delta }_R)=({\oplus }_{n=1}^{+\infty }{\mathcal{C}}_R^{2n-1}(V,\mathfrak{L})\oplus {\mathcal{C}}_R^0(V,\mathfrak{L}),{\delta }_R)$ is called the cochain complex of the generalized Reynolds operator R.

The set

$${\mathcal{Z}}_R^{2n-1}(V,\mathfrak{L})=\{f\in {\mathcal{C}}_R^{2n-1}(V,\mathfrak{L})|{\delta }_{R}f=0\},n\ge 1$$

is called the space of $(2n-1)$-cocycles of R.

The set

$${\mathcal{B}}_R^1(V,\mathfrak{L})=\left\{{\delta }_R(a,b)\right|(a,b)\in {\mathcal{C}}_R^0(V,\mathfrak{L})\}$$

is called the space of 1-coboundaries of R.

The set

$${\mathcal{B}}_R^{2n-1}(V,\mathfrak{L})=\left\{{\delta }_{R}f\right|f\in {\mathcal{C}}_R^{2n-3}(V,\mathfrak{L})\},n\ge 2$$

is called the space of $(2n-1)$-coboundaries of R.

Then, the $(2n-1)$-th cohomology group of the generalized Reynolds operator R are defined as

$${\mathcal{H}}_R^{2n-1}(V,\mathfrak{L})=\frac{{\mathcal{Z}}_R^{2n-1}(V,\mathfrak{L})}{{\mathcal{B}}_R^{2n-1}(V,\mathfrak{L})},n\ge 1.$$

Remark 2. 

The cohomology theory for generalized Reynolds operators on Hom-Lie triple systems enjoys certain functorial properties. Let $R,R^{\prime }:V\to \mathfrak{L}$ be two generalized Reynolds operators on a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ associated to $(V,\beta ;\theta )$ and 3-cocycle $\mathfrak{H}$, and let $(\eta ,\zeta )$ be a homomorphism from R to $R^{\prime }$, in which ζ is invertible. Define a linear map $\mathrm{\Phi }:{\mathcal{C}}_R^{2n-1}(V,\mathfrak{L})\to {\mathcal{C}}_{R^{\prime }}^{2n-1}(V,\mathfrak{L})$ by

$$\mathrm{\Phi }\left(f\right)(v_1,\cdots ,v_{2n-1})=\eta \left(f({\zeta }^{-1}\left(v_1\right),\cdots ,{\zeta }^{-1}\left(v_{2n-1}\right))\right),$$

for any $f\in {\mathcal{C}}_R^{2n-1}(V,\mathfrak{L})$ and $v_1,\cdots ,v_{2n-1}\in V.$ Then, it is straightforward to deduce that Φ is a cochain map from the cochain complex $({\oplus }_{n=1}^{\infty }{\mathcal{C}}_R^{2n-1}(V,\mathfrak{L}),{\delta }_R)$ to the cochain complex $({\oplus }_{n=1}^{\infty }{\mathcal{C}}_{R^{\prime }}^{2n-1}(V,\mathfrak{L}),{\delta }_{R^{\prime }})$. Consequently, it induces a homomorphism ${\mathrm{\Phi }}^{\ast }$ from the cohomology group ${\mathcal{H}}_R^{2n-1}(V,\mathfrak{L})$ to ${\mathcal{H}}_{R^{\prime }}^{2n-1}(V,\mathfrak{L})$.

5. Deformations of Generalized Reynolds Operators on Hom-Lie Triple Systems

In this section, we study linear deformations and higher order deformations of generalized Reynolds operators on Hom-Lie triple systems via the cohomology theory established in the former section.

First, we use the cohomology constructed to characterize the linear deformations of generalized Reynolds operators on Hom-Lie triple systems.

Definition 5. 

Let $R:V\to \mathfrak{L}$ be a generalized Reynolds operator on a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ associated to $(V,\beta ;\theta )$ and 3-cocycle $\mathfrak{H}$. A linear deformation of R is a generalized Reynolds operator of the form $R_t=R+t{R}_1$, where $R_1:V\to \mathfrak{L}$ is a linear map and t is a parameter.

Suppose $R+t{R}_1$ is a linear deformation of R; direct deduction shows that $R_1\in {\mathcal{C}}_R^1(V,\mathfrak{L})$ is a 1-cocycle of the generalized Reynolds operator R. So the cohomology class of $R_1$ defines an element in ${\mathcal{H}}_R^1(V,\mathfrak{L})$. Furthermore, the 1-cocycle $R_1$ is called the infinitesimal of the linear deformation $R_t$ of R.

Definition 6. 

Let $R:V\to \mathfrak{L}$ be a generalized Reynolds operator on a regular Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ associated to regular representation $(V,\beta ;\theta )$ and 3-cocycle $\mathfrak{H}$. Two linear deformations $R_t=R+t{R}_1$ and $R_t^{\prime }=R+t{R}_1^{\prime }$ are called equivalent if there exist two elements $a,b\in \mathfrak{L}$, such that $\alpha \left(a\right)=a,\alpha \left(b\right)=b$ and the pair $(I{d}_{\mathfrak{L}}+t{\alpha }^{-1}(\mathcal{L}(a,b)-),I{d}_V+t{\beta }^{-1}(D(a,b)-)+t{\beta }^{-1}\left(\mathfrak{H}(a,b,R-)\right))$ is a homomorphism from $R_t$ to $R_t^{\prime }$.

Suppose $R_t$ and $R_t^{\prime }$ are equivalent; then, Equation (11) yields

$$(I{d}_{\mathfrak{L}}+t{\alpha }^{-1}(\mathcal{L}(a,b)-))R_{t}u=R_t^{\prime }(I{d}_V+t{\beta }^{-1}(D(a,b)-)+t{\beta }^{-1}\left(\mathfrak{H}(a,b,R-)\right))u,\forall u\in V,$$

which means that

$$\begin{array}{cc}\hfill R_{1}u-R_1^{\prime }u=& R{\beta }^{-1}\left(D(a,b)u\right)-{\alpha }^{-1}\left([a,b,Ru]\right)+R{\beta }^{-1}\left(\mathfrak{H}(a,b,Ru)\right)\hfill \\ \hfill =& RD(a,b){\beta }^{-1}\left(u\right)-[a,b,R{\beta }^{-1}\left(u\right)]+R\mathfrak{H}(a,b,R{\beta }^{-1}\left(u\right)).\hfill \end{array}$$

By Proposition 2, we have $R_1-R_1^{\prime }=\wp (a,b)={\delta }_R(a,b)\in {\mathcal{B}}_R^1(V,\mathfrak{L})$. So their cohomology classes are the same in ${\mathcal{H}}_R^1(V,\mathfrak{L})$.

Conversely, any 1-cocycle $R_1$ gives rise to the linear deformation $R+t{R}_1$. To sum up, we have the following result.

Proposition 3. 

Let $R:V\to \mathfrak{L}$ be a generalized Reynolds operator on a regular Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ associated to regular representation $(V,\beta ;\theta )$ and 3-cocycle $\mathfrak{H}$. Then, there is a bijection between the set of all equivalence classes of linear deformation of R and the first cohomology group ${\mathcal{H}}_R^1(V,\mathfrak{L})$.

Next, we introduce a special cohomology class associated to an order n deformation of a generalized Reynolds operator, and show that an order n deformation of a generalized Reynolds operator is extendable if and only if this cohomology class in the third cohomology group vanishes.

Definition 7. 

Let $R:V\to \mathfrak{L}$ be a generalized Reynolds operator on a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ associated to $(V,\beta ;\theta )$ and 3-cocycle $\mathfrak{H}$. If $R_t={\sum }_{i=0}^n{t}^i{R}_i$ with $R_0=R,R_i\in \mathrm{Hom}(V,\mathfrak{L}),i=1,\cdots ,n$, defines a $\mathbb{K}\left[\left[t\right]\right]/\left(t^{n+1}\right)$-module map from $V\left[\left[t\right]\right]/\left(t^{n+1}\right)$ to the Hom-Lie triple system $\mathfrak{L}\left[\left[t\right]\right]/\left(t^{n+1}\right)$ satisfying

$$\begin{array}{cc}\hfill & R_t\circ \beta =\alpha \circ R_t,\hfill \\ \hfill & [R_{t}u,R_{t}v,R_{t}w]=R_t(\theta (R_{t}v,R_{t}w)u+D(R_{t}u,R_{t}v)w-\theta (R_{t}u,R_{t}w)v+\mathfrak{H}(R_{t}u,R_{t}v,R_{t}w)),\hfill \end{array}$$

for any $u,v,w\in V$, we say that $R_t$ is an order n deformation of R.

Definition 8. 

Let $R:V\to \mathfrak{L}$ be a generalized Reynolds operator on a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ associated to $(V,\beta ;\theta )$ and 3-cocycle $\mathfrak{H}$. Let $R_t={\sum }_{i=0}^n{t}^i{R}_i$ be an order n deformation of R. If there is an $R_{n+1}\in {\mathcal{C}}_R^1(V,\mathfrak{L})$, such that $R_t^{\prime }=R_t+t^{n+1}{R}_{n+1}$ is an order $(n+1)$ deformation of R, then we say that $R_t$ is extendable.

Proposition 4. 

Let $R:V\to \mathfrak{L}$ be a generalized Reynolds operator on a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ associated to $(V,\beta ;\theta )$ and 3-cocycle $\mathfrak{H}$. Let $R_t={\sum }_{i=0}^n{t}^i{R}_i$ be an order n deformation of R. Then, $R_t$ is extendable if and only if the cohomology class $\left[Ob{s}^n\right]\in {\mathcal{H}}_R^3(V,\mathfrak{L})$ vanishes, where

$$\begin{array}{cc}\hfill & Ob{s}^n(u_1,u_2,u_3)=\sum _{\begin{array}{c}i+j+k=n+1\\ 0\le i,j,k\le n\end{array}}([R_i{u}_1,R_j{u}_2,R_k{u}_3]-R_i(D(R_j{u}_1,R_k{u}_2)u_3\hfill \\ \hfill & -\theta (R_j{u}_1,R_k{u}_3)u_2+\theta (R_j{u}_2,R_k{u}_3)u_1\left)\right)-\sum _{\begin{array}{c}i+j+k+l=n+1\\ 0\le i,j,k,l\le n\end{array}}{R}_i\mathfrak{H}(R_j{u}_1,R_k{u}_2,R_l{u}_3).\hfill \end{array}$$

Proof. 

Let $R_t^{\prime }=R_t+t^{n+1}{R}_{n+1}$ be the extension of $R_t$; then, for all $u_1,u_2,u_3\in V$

$$\begin{array}{cc}\hfill [R_t^{\prime }{u}_1,R_t^{\prime }{u}_2,R_t^{\prime }{u}_3]=& R_t^{\prime }(D(R_t^{\prime }{u}_1,R_t^{\prime }{u}_2)u_3-\theta (R_t^{\prime }{u}_1,R_t^{\prime }{u}_3)u_2+\theta (R_t^{\prime }{u}_2,R_t^{\prime }{u}_3)u_1+\hfill \\ \hfill & \mathfrak{H}(R_t^{\prime }{u}_1,R_t^{\prime }{u}_2,R_t^{\prime }{u}_3)).\hfill \end{array}$$

(17)

Expanding the equation and comparing the coefficients of $t^{n+1}$ yields:

$$\begin{array}{cc}\hfill & \sum _{\begin{array}{c}i+j+k=n+1\\ 0\le i,j,k\le n+1\end{array}}\left([R_i{u}_1,R_j{u}_2,R_k{u}_3]-R_i(D(R_j{u}_1,R_k{u}_2)u_3-\theta (R_j{u}_1,R_k{u}_3)u_2+\theta (R_j{u}_2,R_k{u}_3)u_1)\right)\hfill \\ \hfill & -\sum _{\begin{array}{c}i+j+k+l=n+1\\ 0\le i,j,k,l\le n+1\end{array}}{R}_i\mathfrak{H}(R_j{u}_1,R_k{u}_2,R_l{u}_3)=0,\hfill \end{array}$$

which is equivalent to

$$\begin{array}{cc}\hfill & \sum _{\begin{array}{c}i+j+k=n+1\\ 0\le i,j,k\le n\end{array}}\left([R_i{u}_1,R_j{u}_2,R_k{u}_3]-R_i(D(R_j{u}_1,R_k{u}_2)u_3-\theta (R_j{u}_1,R_k{u}_3)u_2+\theta (R_j{u}_2,R_k{u}_3)u_1)\right)\hfill \\ \hfill & -\sum _{\begin{array}{c}i+j+k+l=n+1\\ 0\le i,j,k,l\le n\end{array}}{R}_i\mathfrak{H}(R_j{u}_1,R_k{u}_2,R_l{u}_3)+[R_{n+1}{u}_1,R{u}_2,R{u}_3]+[R{u}_1,R_{n+1}{u}_2,R{u}_3]\hfill \\ \hfill & +[R{u}_1,R{u}_2,R_{n+1}{u}_3]-R_{n+1}(D(R{u}_1,R{u}_2)u_3-\theta (R{u}_1,R{u}_3)u_2+\theta (R{u}_2,R{u}_3)u_1+\mathfrak{H}(R{u}_1,R{u}_2,R{u}_3))\hfill \\ \hfill & -R(D(R_{n+1}{u}_1,R{u}_2)u_3+D(R{u}_1,R_{n+1}{u}_2)u_3-\theta (R_{n+1}{u}_1,R{u}_3)u_2-\theta (R{u}_1,R_{n+1}{u}_3)u_2\hfill \\ \hfill & +\theta (R_{n+1}{u}_2,R{u}_3)u_1+\theta (R{u}_2,R_{n+1}{u}_3)u_1+\mathfrak{H}(R_{n+1}{u}_1,R{u}_2,R{u}_3)\hfill \\ \hfill & +\mathfrak{H}(R{u}_1,R_{n+1}{u}_2,R{u}_3)+\mathfrak{H}(R{u}_1,R{u}_2,R_{n+1}{u}_3))=0,\hfill \end{array}$$

that is, $Ob{s}^n(u_1,u_2,u_3)+{\delta }_R{R}_{n+1}(u_1,u_2,u_3)=0$. Hence, $Ob{s}^n=-{\delta }_R{R}_{n+1}$; furthermore, ${\delta }_{R}Ob{s}^n=0,$ which implies that the cohomology class $\left[Ob{s}^n\right]\in {\mathcal{H}}_R^3(V,\mathfrak{L})$ vanishes.

Conversely, suppose that the cohomology class $\left[Ob{s}^n\right]$ vanishes; then, there exists a 1-cochain $R_{n+1}\in {\mathcal{C}}_R^1(V,\mathfrak{L})$, such that $Ob{s}^n=-{\delta }_R{R}_{n+1}$. Set $R_t^{\prime }=R_t+t^{n+1}{R}_{n+1}$. Then, $R_t^{\prime }$ satisfies

$$\begin{array}{cc}\hfill & \sum _{\begin{array}{c}i+j+k=d\end{array}}\left([R_i{u}_1,R_j{u}_2,R_k{u}_3]-R_i(D(R_j{u}_1,R_k{u}_2)u_3-\theta (R_j{u}_1,R_k{u}_3)u_2+\theta (R_j{u}_2,R_k{u}_3)u_1)\right)\hfill \\ \hfill & -\sum _{\begin{array}{c}i+j+k+l=d\end{array}}{R}_i\mathfrak{H}(R_j{u}_1,R_k{u}_2,R_l{u}_3)=0,0\le d\le n+1,\hfill \end{array}$$

which implies that Equation (17) holds; that is, $R_t^{\prime }$ is an order $(n+1)$ deformation of R. So it is an extension of $R_t.$ □

6. Hom-NS-Lie Triple Systems

In this section, we introduce the notion of Hom-NS-Lie triple systems, which is the underlying algebraic structure of generalized Reynolds operators. Moreover, we show that there exists a Hom-Lie triple-system structure on a Hom-NS-Lie triple system.

Definition 9. 

(i) A Hom-NS-Lie triple system $(\mathfrak{L},\{-,-,-\},[-,-,-],\alpha )$ consists of a vector space $\mathfrak{L}$ with trilinear products $\{-,-,-\},[-,-,-]:\mathfrak{L}\otimes \mathfrak{L}\otimes \mathfrak{L}\to \mathfrak{L}$ and an algebra morphism $\alpha :\mathfrak{L}\to \mathfrak{L}$, such that

$$\begin{array}{cc}\hfill [a_1,a_2,a_3]=& -[a_2,a_1,a_3],\hfill \end{array}$$

(18)

$$\begin{array}{cc}\hfill {\circlearrowright }_{a_1,a_2,a_3}[a_1,a_2,a_3]=& 0,\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill \{\alpha \left(b_1\right),\alpha \left(b_2\right),\left[[a_1,a_2,a_3]\right]\}=& \{\{b_1,b_2,a_1\},\alpha \left(a_2\right),\alpha \left(a_3\right)\}-\{\{b_1,b_2,a_2\},\alpha \left(a_1\right),\alpha \left(a_3\right)\}\hfill \\ \hfill & +{\{\alpha \left(a_1\right),\alpha \left(a_2\right),\{b_1,b_2,a_3\}\}}^{\ast },\hfill \end{array}$$

(20)

$$\begin{array}{cc}\hfill {\{\alpha \left(b_1\right),\alpha \left(b_2\right),\{a_1,a_2,a_3\}\}}^{\ast }=& \{{\{b_1,b_2,a_1\}}^{\ast },\alpha \left(a_2\right),\alpha \left(a_3\right)\}+\{\alpha \left(a_1\right),\left[[b_1,b_2,a_2]\right],\alpha \left(a_3\right)\}\hfill \\ \hfill & +\{\alpha \left(a_1\right),\alpha \left(a_2\right),\left[[b_1,b_2,a_3]\right]\},\hfill \end{array}$$

(21)

$$\begin{array}{cc}\hfill \alpha \left(b_1\right),\alpha \left(b_2\right),\left[[a_1,a_2,a_3]\right]]=& [\left[[b_1,b_2,a_1]\right],\alpha \left(a_2\right),\alpha \left(a_3\right)]+[\alpha \left(a_1\right),\left[[b_1,b_2,a_2]\right],\alpha \left(a_3\right)]\hfill \\ \hfill & +[\alpha \left(a_1\right),\alpha \left(a_2\right),\left[[b_1,b_2,a_3]\right]]+\{[b_1,b_2,a_1],\alpha \left(a_2\right),\alpha \left(a_3\right)\}\hfill \\ \hfill & -\{[b_1,b_2,a_2],\alpha \left(a_1\right),\alpha \left(a_3\right)\}+{\{\alpha \left(a_1\right),\alpha \left(a_2\right),[b_1,b_2,a_3]\}}^{\ast }\hfill \\ \hfill & -{\{\alpha \left(b_1\right),\alpha \left(b_2\right),[a_1,a_2,a_3]\}}^{\ast },\hfill \end{array}$$

(22)

where $a_1,a_2,a_3,b_1,b_2\in \mathfrak{L}$, ${\{-,-,-\}}^{\ast }$ and $\left[\right[-,-,-\left]\right]$ are defined to be

$$\begin{array}{cc}\hfill {\{a_1,a_2,a_3\}}^{\ast }=& \{a_3,a_2,a_1\}-\{a_3,a_1,a_2\},\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill \left[\right[a_1,a_2,a_3\left]\right]=& {\{a_1,a_2,a_3\}}^{\ast }+\{a_1,a_2,a_3\}-\{a_2,a_1,a_3\}+[a_1,a_2,a_3].\hfill \end{array}$$

(24)

(ii) A homomorphism between two Hom-NS-Lie triple systems $({\mathfrak{L}}_1,{\{-,-,-\}}_1,{[-,-,-]}_1,$${\alpha }_1)$ and $({\mathfrak{L}}_2,{\{-,-,-\}}_2,{[-,-,-]}_2,{\alpha }_2)$ is a linear map $\phi :{\mathfrak{L}}_1\to {\mathfrak{L}}_2$ satisfying $\phi \left({\alpha }_1\left(a_1\right)\right)={\alpha }_2\left(\phi \left(a_1\right)\right),\phi \left({\{a_1,a_2,a_3\}}_1\right)={\{\phi \left(a_1\right),\phi \left(a_2\right),\phi \left(a_3\right)\}}_2,\phi \left({[a_1,a_2,a_3]}_1\right)={[\phi \left(a_1\right),\phi \left(a_2\right),\phi \left(a_3\right)]}_2.$

Remark 3. 

(i) Let $(\mathfrak{L},\{-,-,-\},[-,-,-],\alpha )$ be a Hom-NS-Lie triple system. If the bracket $\{-,-,-\}=0,$ then we ascertain that $(\mathfrak{L},[-,-,-],\alpha )$ is a Hom-Lie triple system.

(ii) An NS-Lie triple system is a Hom-NS-Lie triple system with $\alpha ={\mathrm{id}}_{\mathfrak{L}}$. See [30] for more details about NS-Lie triple systems.

Proposition 5. 

Let $(\mathfrak{L},\{-,-,-\},[-,-,-],\alpha )$ be a Hom-NS-Lie triple system. Then:

(i) the triple $(\mathfrak{L},[[-,-,-]],\alpha )$ is a Hom-Lie triple system, which is called the adjacent Hom-Lie triple system.

(ii) the triple $(\mathfrak{L},\alpha ;\vartheta )$ is a representation of the adjacent Hom-Lie triple system $(\mathfrak{L},[[-,$$-,-\left]\right],\alpha )$, where

$$\vartheta :\mathfrak{L}\otimes \mathfrak{L}\to \mathrm{End}\left(\mathfrak{L}\right),(a_1,a_2)\mapsto (b\mapsto \{b,a_1,a_2\}),\forall a_1,a_2,b\in \mathfrak{L}.$$

Proof. 

(i) Evidently, for any $a_1,a_2,a_3\in \mathfrak{L}$, by Equations (18), (19), (23) and (24), we have $\left[[a_1,a_2,a_3]\right]=-\left[[a_2,a_1,a_3]\right]$ and ${\circlearrowright }_{a_1,a_2,a_3}\left[[a_1,a_2,a_3]\right]=0$. Furthermore, for any $a_1,a_2,a_3,b_1,b_2\in \mathfrak{L}$, we have

$$\begin{array}{cc}\hfill & \left[[\alpha \left(b_1\right),\alpha \left(b_2\right),\left[[a_1,a_2,a_3]\right]]\right]-\left[[\left[[b_1,b_2,a_1]\right],\alpha \left(a_2\right),\alpha \left(a_3\right)]\right]-\left[[\alpha \left(a_1\right),\left[[b_1,b_2,a_2]\right],\alpha \left(a_3\right)]\right]\hfill \\ \hfill & -\left[[\alpha \left(a_1\right),\alpha \left(a_2\right),\left[[b_1,b_2,a_3]\right]]\right](\mathrm{by}\mathrm{Equation}\left(24\right))\hfill \\ \hfill =& {\{\alpha \left(b_1\right),\alpha \left(b_2\right),\left[[a_1,a_2,a_3]\right]\}}^{\ast }+\{\alpha \left(b_1\right),\alpha \left(b_2\right),\left[[a_1,a_2,a_3]\right]\}-\{\alpha \left(b_2\right),\alpha \left(b_1\right),\left[[a_1,a_2,a_3]\right]\}\hfill \\ \hfill & +[\alpha \left(b_1\right),\alpha \left(b_2\right),\left[[a_1,a_2,a_3]\right]]-{\{\left[[b_1,b_2,a_1]\right],\alpha \left(a_2\right),\alpha \left(a_3\right)\}}^{\ast }-\{\left[[b_1,b_2,a_1]\right],\alpha \left(a_2\right),\alpha \left(a_3\right)\}\hfill \\ \hfill & +\{\alpha \left(a_2\right),\left[[b_1,b_2,a_1]\right],\alpha \left(a_3\right)\}-[\left[[b_1,b_2,a_1]\right],\alpha \left(a_2\right),\alpha \left(a_3\right)]-{\{\alpha \left(a_1\right),\left[[b_1,b_2,a_2]\right],\alpha \left(a_3\right)\}}^{\ast }\hfill \\ \hfill & -\{\alpha \left(a_1\right),\left[[b_1,b_2,a_2]\right],\alpha \left(a_3\right)\}+\{\left[[b_1,b_2,a_2]\right],\alpha \left(a_1\right),\alpha \left(a_3\right)\}-[\alpha \left(a_1\right),\left[[b_1,b_2,a_2]\right],\alpha \left(a_3\right)]\hfill \\ \hfill & -{\{\alpha \left(a_1\right),\alpha \left(a_2\right),\left[[b_1,b_2,a_3]\right]\}}^{\ast }-\{\alpha \left(a_1\right),\alpha \left(a_2\right),\left[[b_1,b_2,a_3]\right]\}+\{\alpha \left(a_2\right),\alpha \left(a_1\right),\left[[b_1,b_2,a_3]\right]\}\hfill \\ \hfill & -[\alpha \left(a_1\right),\alpha \left(a_2\right),\left[[b_1,b_2,a_3]\right]](\mathrm{by}\mathrm{Equations}\left(23\right)\mathrm{and}\left(24\right))\hfill \end{array}$$

$$\begin{array}{cc}\hfill =& (-\{\alpha \left(b_2\right),\alpha \left(b_1\right),\left[[a_1,a_2,a_3]\right]\}+\{\{b_2,b_1,a_1\},\alpha \left(a_2\right),\alpha \left(a_3\right)\}-\{\{b_2,b_1,a_2\},\alpha \left(a_1\right),\alpha \left(a_3\right)\}\hfill \\ \hfill & +{\{\alpha \left(a_1\right),\alpha \left(a_2\right),\{b_2,b_1,a_3\}\}}^{\ast })+(\{\alpha \left(b_1\right),\alpha \left(b_2\right),\left[[a_1,a_2,a_3]\right]\}-\{\{b_1,b_2,a_1\},\alpha \left(a_2\right),\alpha \left(a_3\right)\}\hfill \\ \hfill & +\{\{b_1,b_2,a_2\},\alpha \left(a_1\right),\alpha \left(a_3\right)\}-{\{\alpha \left(a_1\right),\alpha \left(a_2\right),\{b_1,b_2,a_3\}\}}^{\ast })+(-{\{\alpha \left(b_1\right),\alpha \left(b_2\right),\{a_2,a_1,a_3\}\}}^{\ast }\hfill \\ \hfill & +\{{\{b_1,b_2,a_2\}}^{\ast },\alpha \left(a_1\right),\alpha \left(a_3\right)\}+\{\alpha \left(a_2\right),\left[[b_1,b_2,a_1]\right],\alpha \left(a_3\right)\}+\{\alpha \left(a_2\right),\alpha \left(a_1\right),\left[[b_1,b_2,a_3]\right]\})\hfill \\ \hfill & +({\{\alpha \left(b_1\right),\alpha \left(b_2\right),\{a_1,a_2,a_3\}\}}^{\ast }-\{{\{b_1,b_2,a_1\}}^{\ast },\alpha \left(a_2\right),\alpha \left(a_3\right)\}-\{\alpha \left(a_1\right),\left[[b_1,b_2,a_2]\right],\alpha \left(a_3\right)\}\hfill \\ \hfill & -\{\alpha \left(a_1\right),\alpha \left(a_2\right),\left[[b_1,b_2,a_3]\right]\})+({\{\alpha \left(b_1\right),\alpha \left(b_2\right),{\{a_1,a_2,a_3\}}^{\ast }\}}^{\ast }-{\{\left[[b_1,b_2,a_1]\right],\alpha \left(a_2\right),\alpha \left(a_3\right)\}}^{\ast }\hfill \\ \hfill & -{\{\alpha \left(a_1\right),\left[[b_1,b_2,a_2]\right],\alpha \left(a_3\right)\}}^{\ast }-{\{\alpha \left(a_1\right),\alpha \left(a_2\right),{\{b_1,b_2,a_3\}}^{\ast }\}}^{\ast })\hfill \\ \hfill & +([\alpha \left(b_1\right),\alpha \left(b_2\right),\left[[a_1,a_2,a_3]\right]]-[\left[[b_1,b_2,a_1]\right],\alpha \left(a_2\right),\alpha \left(a_3\right)]-[\alpha \left(a_1\right),\left[[b_1,b_2,a_2]\right],\alpha \left(a_3\right)]\hfill \\ \hfill & -[\alpha \left(a_1\right),\alpha \left(a_2\right),\left[[b_1,b_2,a_3]\right]]-\{[b_1,b_2,a_1],\alpha \left(a_2\right),\alpha \left(a_3\right)\}+\{[b_1,b_2,a_2],\alpha \left(a_1\right),\alpha \left(a_3\right)\}\hfill \\ \hfill & -{\{\alpha \left(a_1\right),\alpha \left(a_2\right),[b_1,b_2,a_3]\}}^{\ast }+{\{\alpha \left(b_1\right),\alpha \left(b_2\right),[a_1,a_2,a_3]\}}^{\ast }\left)(\mathrm{by}\mathrm{Equations}\right(20)–(23\left)\right)\hfill \\ \hfill =& 0.\hfill \end{array}$$

Hence, $(\mathfrak{L},[[-,-,-]],\alpha )$ is a Hom-Lie triple system.

(ii) For all $a_1,a_2,a_3\in \mathfrak{L}$, we have

$$\begin{array}{cc}\hfill & \mathfrak{D}(a_1,a_2)a_3=\vartheta (a_2,a_1)a_3-\vartheta (a_1,a_2)a_3=\{a_3,a_2,a_1\}-\{a_3,a_1,a_2\}={\{a_1,a_2,a_3\}}^{\ast }.\hfill \end{array}$$

Evidently, $\vartheta (\alpha \left(a_1\right),\alpha \left(a_2\right))\alpha \left(a_3\right)=\alpha \left(\vartheta (a_1,a_2)a_3\right).$ Furthermore, for any $a_1,a_2,a_3,b_1,b_2\in \mathfrak{L}$, we obtain

$$\begin{array}{cc}\hfill & \vartheta (\alpha \left(b_1\right),\alpha \left(b_2\right))\vartheta (a_1,a_2)a_3-\vartheta (\alpha \left(a_2\right),\alpha \left(b_2\right))\vartheta (a_1,b_1)a_3-\vartheta (\alpha \left(a_1\right),\left[[a_2,b_1,b_2]\right])\alpha \left(a_3\right)\hfill \\ \hfill & +\mathfrak{D}(\alpha \left(a_2\right),\alpha \left(b_1\right))\vartheta (a_1,b_2)a_3\hfill \\ \hfill =& \{\{a_3,a_1,a_2\},\alpha \left(b_1\right),\alpha \left(b_2\right)\}-\{\{a_3,a_1,b_1\},\alpha \left(a_2\right),\alpha \left(b_2\right)\}-\{\alpha \left(a_3\right),\alpha \left(a_1\right),\left[[a_2,b_1,b_2]\right]\}\hfill \\ \hfill & +{\{\alpha \left(a_2\right),\alpha \left(b_1\right),\{a_3,a_1,b_2\}\}}^{\ast }(\mathrm{by}\mathrm{Equation}\left(20\right))\hfill \\ \hfill =& 0,\hfill \\ \hfill & \vartheta (\alpha \left(b_1\right),\alpha \left(b_2\right))\mathfrak{D}(a_1,a_2)a_3-\mathfrak{D}(\alpha \left(a_1\right),\alpha \left(a_2\right))\vartheta (b_1,b_2)a_3+\vartheta (\left[[a_1,a_2,b_1]\right],\alpha \left(b_2\right))\alpha \left(a_3\right)\hfill \\ \hfill & +\vartheta (\alpha \left(b_1\right),\left[[a_1,a_2,b_2]\right])\alpha \left(a_3\right)\hfill \\ \hfill =& \{{\{a_1,a_2,a_3\}}^{\ast },\alpha \left(b_1\right),\alpha \left(b_2\right)\}-{\{\alpha \left(a_1\right),\alpha \left(a_2\right),\{a_3,b_1,b_2\}\}}^{\ast }+\{\alpha \left(a_3\right),\left[[a_1,a_2,b_1]\right],\alpha \left(b_2\right)\}\hfill \\ \hfill & +\{\alpha \left(a_3\right),\alpha \left(b_1\right),\left[[a_1,a_2,b_2]\right]\}(\mathrm{by}\mathrm{Equation}\left(21\right))\hfill \\ \hfill =& 0.\hfill \end{array}$$

Therefore, $(\mathfrak{L},\alpha ;\vartheta )$ is a representation of the adjacent Hom-Lie triple system $(\mathfrak{L},[[-,-,-]],\alpha )$. □

Corollary 1. 

Let $\phi :({\mathfrak{L}}_1,{\{-,-,-\}}_1,{[-,-,-]}_1,{\alpha }_1)\to ({\mathfrak{L}}_2,{\{-,-,-\}}_2,{[-,-,-]}_2,{\alpha }_2)$ be a Hom-NS-Lie triple-system homomorphism. Then, φ is also a Hom-Lie triple-system homomorphism between the subadjacent Hom-Lie triple system from $({\mathfrak{L}}_1,{\left[[-,-,-]\right]}_1,{\alpha }_1)$ to $({\mathfrak{L}}_2,{\left[[-,-,-]\right]}_2,{\alpha }_2)$.

The following proposition illustrates that Hom-NS-Lie triple systems can be viewed as the underlying algebraic structures of generalized Reynolds operators on Hom-Lie triple systems.

Proposition 6. 

Let $R:V\to \mathfrak{L}$ be a generalized Reynolds operator on a Hom-Lie triple system $(\mathfrak{L},[-,-,-],\alpha )$ associated to $(V,\beta ;\theta )$ and 3-cocycle $\mathfrak{H}$. Then, the 4-tuple $(V,{\{-,-,-\}}_{\theta },{[-,-,-]}_{\mathfrak{H}},\beta )$ is a Hom-NS-Lie triple system, where

$${\{u,v,w\}}_{\theta }=\theta (Rv,Rw)u,{[u,v,w]}_{\mathfrak{H}}=\mathfrak{H}(Ru,Rv,Rw),\forall u,v,w\in V.$$

Proof. 

For any $u,v,w,s,t\in V$, first, evidently, we have ${[u,v,w]}_{\mathfrak{H}}=-{[v,u,w]}_{\mathfrak{H}}$ and ${\circlearrowright }_{u,v,w}{[u,v,w]}_{\mathfrak{H}}=0$. On the one hand,

$$\begin{array}{cc}\hfill {\{u,v,w\}}_{\theta }^{\ast }=& {\{w,v,u\}}_{\theta }-{\{w,u,v\}}_{\theta }=\theta (Rv,Ru)w-\theta (Ru,Rv)w=D(Ru,Rv)w,\hfill \\ \hfill {\left[\right[u,v,w\left]\right]}_V=& {\{u,v,w\}}_{\theta }^{\ast }+{\{u,v,w\}}_{\theta }-{\{v,u,w\}}_{\theta }+{[u,v,w]}_{\mathfrak{H}}\hfill \\ \hfill =& D(Ru,Rv)w+\theta (Rv,Rw)u-\theta (Ru,Rw)v+\mathfrak{H}(Ru,Rv,Rw).\hfill \end{array}$$

On the other hand, we obtain

$$\begin{array}{cc}\hfill & {\{\beta \left(s\right),\beta \left(t\right),{\left[[u,v,w]\right]}_V\}}_{\theta }-{\{{\{s,t,u\}}_{\theta },\beta \left(v\right),\beta \left(w\right)\}}_{\theta }+{\{{\{s,t,v\}}_{\theta },\beta \left(u\right),\beta \left(w\right)\}}_{\theta }\hfill \\ \hfill & -{\{\beta \left(u\right),\beta \left(v\right),{\{s,t,w\}}_{\theta }\}}_{\theta }^{\ast }\hfill \\ \hfill =& \theta (R\beta \left(t\right),R{\left[[u,v,w]\right]}_V)\beta \left(s\right)-\theta (R\beta \left(v\right),R\beta \left(w\right))\theta (Rt,Ru)s+\theta (R\beta \left(u\right),R\beta \left(w\right))\theta (Rt,Rv)s\hfill \\ \hfill & -D\left(R\beta \right(u),R\beta (v\left)\right)\theta (Rt,Rw)s(\mathrm{by}\mathrm{Equations}\left(8\right)\mathrm{and}\left(9\right))\hfill \\ \hfill =& \theta \left(\alpha \right(Rt),[Ru,Rv,Rw\left]\right)\beta \left(s\right)-\theta \left(\alpha \right(Rv),\alpha (Rw\left)\right)\theta (Rt,Ru)s+\theta \left(\alpha \right(Ru),\alpha (Rw\left)\right)\theta (Rt,Rv)s\hfill \\ \hfill & -D\left(\alpha \right(Ru),\alpha (Rv\left)\right)\theta (Rt,Rw)s(\mathrm{by}\mathrm{Equation}\left(5\right))\hfill \\ \hfill =& 0,\hfill \\ \hfill & {\{{\{s,t,u\}}_{\theta }^{\ast },\beta \left(v\right),\beta \left(w\right)\}}_{\theta }+{\{\beta \left(u\right),{\left[[s,t,v]\right]}_V,\beta \left(w\right)\}}_{\theta }+{\{\beta \left(u\right),\beta \left(v\right),{\left[[s,t,w]\right]}_V\}}_{\theta }\hfill \\ \hfill & -{\{\beta \left(s\right),\beta \left(t\right),{\{u,v,w\}}_{\theta }\}}_{\theta }^{\ast }\hfill \\ \hfill =& \theta (R\beta \left(v\right),R\beta \left(w\right))D(Rs,Rt)u+\theta (R{\left[[s,t,v]\right]}_V,R\beta \left(w\right))\beta \left(u\right)+\theta (R\beta \left(v\right),R{\left[[s,t,w]\right]}_V)\beta \left(u\right)\hfill \\ \hfill & -D\left(R\beta \right(s),R\beta (t\left)\right)\theta (Rv,Rw)u(\mathrm{by}\mathrm{Equations}\left(8\right)\mathrm{and}\left(9\right))\hfill \\ \hfill =& \theta \left(\alpha \right(Rv),\alpha (Rw\left)\right)D(Rs,Rt)u+\theta \left(\right[Rs,Rt,Rv],\alpha (Rw\left)\right)\beta \left(u\right)+\theta \left(\alpha \right(Rv),[Rs,Rt,Rw\left]\right)\beta \left(u\right)\hfill \\ \hfill & -D\left(\alpha \right(Rs),\alpha (Rt\left)\right)\theta (Rv,Rw)u(\mathrm{by}\mathrm{Equation}\left(6\right))\hfill \\ \hfill =& 0,\hfill \\ \hfill & {[{\left[[s,t,u]\right]}_V,\beta \left(v\right),\beta \left(w\right)]}_{\mathfrak{H}}+{[\beta \left(u\right),{\left[[s,t,v]\right]}_V,\beta \left(w\right)]}_{\mathfrak{H}}+{[\beta \left(u\right),\beta \left(v\right),{\left[[s,t,w]\right]}_V]}_{\mathfrak{H}}\hfill \\ \hfill & +{\{{[s,t,u]}_{\mathfrak{H}},\alpha \left(v\right),\alpha \left(w\right)\}}_{\theta }-{\{{[s,t,v]}_{\mathfrak{H}},\beta \left(u\right),\beta \left(w\right)\}}_{\theta }+{\{\beta \left(u\right),\beta \left(v\right),{[s,t,w]}_{\mathfrak{H}}\}}_{\theta }^{\ast }\hfill \\ \hfill & -{\{\beta \left(s\right),\beta \left(t\right),{[u,v,w]}_{\mathfrak{H}}\}}_{\theta }^{\ast }-{[\beta \left(s\right),\beta \left(t\right),{\left[[u,v,w]\right]}_V]}_{\mathfrak{H}}\hfill \\ \hfill =& \mathfrak{H}(R{\left[[s,t,u]\right]}_V,R\beta \left(v\right),R\beta \left(w\right))+\mathfrak{H}(R\beta \left(u\right),R{\left[[s,t,v]\right]}_V,R\beta \left(w\right))+\mathfrak{H}(R\beta \left(u\right),R\beta \left(v\right),R{\left[[s,t,w]\right]}_V)\hfill \\ \hfill & +\theta \left(R\beta \right(v),R\beta (w\left)\right)\mathfrak{H}(Rs,Rt,Ru)-\theta \left(R\beta \right(u),R\beta (w\left)\right)\mathfrak{H}(Rs,Rt,Rv)+D\left(R\beta \right(u),R\beta (v\left)\right)\mathfrak{H}(Rs,Rt,Rw)\hfill \\ \hfill & -D(R\beta \left(s\right),R\beta \left(t\right))\mathfrak{H}(Ru,Rv,Rw)-\mathfrak{H}(R\beta \left(s\right),R\beta \left(t\right),R{\left[[u,v,w]\right]}_V)(\mathrm{by}\mathrm{Equations}\left(8\right)\mathrm{and}\left(9\right))\hfill \\ \hfill =& \mathfrak{H}\left(\right[Rs,Rt,Ru],\alpha (Rv),\alpha (Rw\left)\right)+\mathfrak{H}\left(\alpha \right(Ru),[Rs,Rt,Rv],\alpha (Rw\left)\right)+\mathfrak{H}\left(\alpha \right(Ru),\alpha (Rv),[Rs,Rt,Rw\left]\right)\hfill \\ \hfill & +\theta \left(\alpha \right(Rv),\alpha (Rw\left)\right)\mathfrak{H}(Rs,Rt,Ru)-\theta \left(\alpha \right(Ru),\alpha (Rw\left)\right)\mathfrak{H}(Rs,Rt,Rv)+D\left(\alpha \right(Ru),\alpha (Rv\left)\right)\mathfrak{H}(Rs,Rt,Rw)\hfill \\ \hfill & -D\left(\alpha \right(Rs),\alpha (Rt\left)\right)\mathfrak{H}(Ru,Rv,Rw)-\mathfrak{H}\left(\alpha \right(Rs),\alpha (Rt),[Ru,Rv,Rw\left]\right)(\mathrm{by}\mathrm{Equation}\left(7\right))\hfill \\ \hfill =& 0.\hfill \end{array}$$

Thus, $(V,{\{-,-,-\}}_{\theta },{[-,-,-]}_{\mathfrak{H}},\beta )$ is a Hom-NS-Lie triple system. □

Example 9. 

Let $(\mathfrak{L},[-,-,-],\alpha )$ be a Hom-Lie triple system and $N:\mathfrak{L}\to \mathfrak{L}$ be a Nijenhuis operator. Then, $(\mathfrak{L},{\{-,-,-\}}_{\theta },{[-,-,-]}_{\mathfrak{H}},$$\alpha )$ is a Hom-NS-Lie triple system, where

$$\begin{array}{cc}\hfill {\{a,b,c\}}_{\theta }=& [a,Nb,Nc],\hfill \\ \hfill {[a,b,c]}_{\mathfrak{H}}=& -N([Na,b,c]+[a,Nb,c]+[a,b,Nc])+N^2[a,b,c],\forall a,b,c\in \mathfrak{L}.\hfill \end{array}$$

Proposition 7. 

Let $R_1:V_1\to {\mathfrak{L}}_1$ (resp. $R_2:V_2\to {\mathfrak{L}}_2$) be a generalized Reynolds operator on a Hom-Lie triple system $({\mathfrak{L}}_1,{[-,-,-]}_1,{\alpha }_1)$ (resp. $({\mathfrak{L}}_2,{[-,-,-]}_2,{\alpha }_2)$ ) associated to $(V_1,{\beta }_1;{\theta }_1)$ (resp. $(V_2,{\beta }_2;{\theta }_2)$) and 3-cocycle ${\mathfrak{H}}_1$ (resp. ${\mathfrak{H}}_2$), and let $(\eta ,\zeta )$ be a homomorphism from $R_1$ to $R_2$. Let $(V_1,{\{-,-,-\}}_{{\theta }_1},{[-,-,-]}_{{\mathfrak{H}}_1},{\beta }_1)$ and $(V_2,{\{-,-,-\}}_{{\theta }_2},{[-,-,-]}_{{\mathfrak{H}}_2},{\beta }_2)$ be the induced Hom-NS-Lie triple systems, respectively. Then, ζ is a homomorphism from the Hom-NS-Lie triple system $(V_1,{\{-,-,-\}}_{{\theta }_1},{[-,-,-]}_{{\mathfrak{H}}_1},{\beta }_1)$ to $(V_2,{\{-,-,-\}}_{{\theta }_2},[-,-,$${-]}_{{\mathfrak{H}}_2},{\beta }_2)$.

Proof. 

For any $u,v,w\in V$, by Equations (10)–(13), we have

$$\begin{array}{cc}\hfill \zeta \left({\{u,v,w\}}_{{\theta }_1}\right)=& \zeta \left({\theta }_1(R_{1}v,R_{1}w)u\right)={\theta }_2(\eta \left(R_{1}v\right),\eta \left(R_{1}w\right))\zeta \left(u\right)\hfill \\ \hfill =& {\theta }_2(R_2\zeta \left(v\right),R_2\zeta \left(w\right))\zeta \left(u\right)\hfill \\ \hfill =& {\{\zeta \left(u\right),\zeta \left(v\right),\zeta \left(w\right)\}}_{{\theta }_2},\hfill \\ \hfill \zeta \left({[u,v,w]}_{{\mathfrak{H}}_1}\right)=& \zeta \left({\mathfrak{H}}_1(R_{1}u,R_{1}v,R_{1}w)\right)={\mathfrak{H}}_2(\eta \left(R_{1}u\right),\eta \left(R_{1}v\right),\eta \left(R_{1}w\right))\hfill \\ \hfill =& {\mathfrak{H}}_2(R_2\zeta \left(u\right),R_2\zeta \left(v\right),R_2\zeta \left(w\right))\hfill \\ \hfill =& {[\zeta \left(u\right),\zeta \left(v\right),\zeta \left(w\right)]}_{{\mathfrak{H}}_2},\hfill \end{array}$$

which implies that $\zeta$ is a homomorphism from $(V_1,{\{-,-,-\}}_{{\theta }_1},{[-,-,-]}_{{\mathfrak{H}}_1},{\beta }_1)$ to $(V_2,$${\{-,-,-\}}_{{\theta }_2},{[-,-,-]}_{{\mathfrak{H}}_2},{\beta }_2)$. □

7. Conclusions

In the current research, we introduce the concept of generalized Reynolds operators on Hom-Lie triple systems, and give some examples. Subsequently, we construct the cohomology of generalized Reynolds operators on Hom-Lie triple systems. Furthermore, we show that any linear deformation of a generalized Reynolds operator is classified by the first cohomology group. Also, we prove that an order n deformation of a generalized Reynolds operator is extendable if and only if this cohomology class in the third cohomology group vanishes. Finally, we introduce a new algebraic structure, in connection with generalized Reynolds operators on a Hom-Lie triple system, called Hom-NS-Lie triple system.