A Fractional Magnetic System with Critical Nonlinearities

Abstract

In the present paper, we investigate a fractional magnetic system involving critical concave–convex nonlinearities with Laplace operators. Specifically, ${(-\Delta )}_A^s{u}_1={\lambda }_1{|u_1|}^{q-2}{u}_1$ + $\frac{2{\alpha }_1}{{\alpha }_1+{\beta }_1}|u_1{|}^{{\alpha }_1-2}{u}_1{|u_2|}^{{\beta }_1}$ in $\Omega$, ${(-\Delta )}_A^s{u}_2={\lambda }_2|u_2{|}^{q-2}{u}_2+\frac{2{\beta }_1}{{\alpha }_1+{\beta }_1}|u_2{|}^{{\beta }_1-2}{u}_2{|u_1|}^{{\alpha }_1}$ in $\Omega$, $u_1=u_2=0$ in ${\mathbb{R}}^n\setminus \Omega$, where $\Omega$ is a bounded set with Lipschitz boundary $\partial \Omega$ in ${\mathbb{R}}^n$, $1<q<2<\frac{n}{s}$ with $s\in (0,1)$, ${\lambda }_1, {\lambda }_2$ are two real positive parameters, ${\alpha }_1>1,{\beta }_1>1$, ${\alpha }_1+{\beta }_1=2_s^{\ast }=\frac{2n}{n-2s}$, $2_s^{\ast }$ is the fractional critical Sobolev exponent, and ${(-\Delta )}_A^s$ is a fractional magnetic Laplace operator. By using Lusternik–Schnirelmann’s theory, we prove the existence result of infinitely many solutions for the magnetic fractional system.

1. Introduction

During a short period of 20 years, a number of researchers have paid attention to nonlocal problems. The applications of nonlocal operator go far beyond the classical local operator. It has wide-ranging applications in many fields, and plays an important role in physics, probability, finance, dynamics, game theory, and so on. Naturally, as a typical nonlocal problem, the fractional elliptic equations (or fractional elliptic systems) have become a research hotspot of scholars, and many interesting results about the existence, nonexistence, and multiplicity of this class of equations are obtained. At the same time, the problems with magnetic potential function have have also been considered by many scholars. The famous nonlinear magnetic Schrödger equation

$$-{(▽-iA)}^{2}u+V\left(x\right)u=f(x,|u|)u,x\in {\mathbb{R}}^N$$

has been extensively studied, where $-{(▽-iA)}^2$ is a magnetic Schrödger operator. In many cases, the microscopic particles are not only affected by the potential field, but also by the magnetic field; thus, the magnetic potential function $A\left(x\right)\ne 0$. The Schrödinger equation plays a very important role in quantum mechanics. It describes the changing laws of particle states in the microcosmic world and has important applications in nuclear physics, solid physics, and other fields. As research continues, a number of experts have already started studying the problems with operator like this ${(-\Delta )}_A^{s}u$. Usually, the operator ${(-\Delta )}_A^{s}u$ is called fractional magnetic Laplace operator, in a suitable sense, ${(-▵)}_A^{s}u$ converges to $-{(▽-iA)}^{2}u$ in the limit $s\uparrow 1$. In the work of Fiscella, Pinamonti, and Vecchi [1], a fractional boundary value problem involving external magnetic filed was investigated, with different assumptions about the nonlinear term, and the results of multiplicity were obtained. Wang and Xiang [2], for a Choquard equation with fractional magnetic potential, proved the multiplicity of solutions applying variational methods. In [3], Xiang et al. studied fractional elliptic equations with an external magnetic field, combining the direct methods with the Nehari method, as well as the method of mountain pass theorem. Thw lowest energy solution results were obtained for this latter equation. Meanwhile, employing the symmetric mountain pass theorem, the existence of infinitely many solutions has also been obtained under some appropriate conditions. In [4], Yang et al. considered a class of magnetic fractional Schrödinger–Kirchhoff equations in ${\mathbb{R}}^n$, and proved the existence results for the equations under some suitable conditions. In [5], under some mild assumptions on f and V, Jin et al. obtained the existence of at least one ground state solution for a critical fractional magnetic Choquard equation. If you want to learn more works about the equations with magnetic fractional Laplace operators and magnetic Laplace operators, there are many useful results, such as [6,7,8,9,10,11,12,13,14,15,16] and the references contained within.

However, as far as we know, there are few works about magnetic fractional systems. Moreover, the fractional magnetic operator is a kind of non-local operator and it is defined on the complex field, resulting in complicated and difficult estimates.

Influenced by the above works, in the present paper, we investigated a system involving critical concave–convex nonliearities driven by a magnetic fractional operator,

$$\left\{\begin{array}{cc}{(-\Delta )}_A^s{u}_1={\lambda }_1|u_1{|}^{q-2}{u}_1+\frac{2{\alpha }_1}{{\alpha }_1+{\beta }_1}|u_1{|}^{{\alpha }_1-2}{u}_1{|u_2|}^{{\beta }_1}& \mathrm{in}\Omega ,\hfill \\ {(-\Delta )}_A^s{u}_2={\lambda }_2|u_2{|}^{q-2}{u}_2+\frac{2{\beta }_1}{{\alpha }_1+{\beta }_1}|u_2{|}^{{\beta }_1-2}{u}_2{|u_1|}^{{\alpha }_1}& \mathrm{in}\Omega ,\hfill \\ u_1=u_2=0& \mathrm{in}{\mathbb{R}}^n\setminus \Omega ,\hfill \end{array}\right.$$

(1)

where $\Omega$ is an open bounded set in ${\mathbb{R}}^n$, the boundary $\partial \Omega$ fulfills the Lipschitz condition, ${\lambda }_1>0, {\lambda }_2>0$ are two real parameters, $1<q<2<\frac{n}{s}$ with $s\in (0,1)$, $2_s^{\ast }=\frac{2n}{n-2s}$ is the critical exponent of fractional Sobolev space, let ${\alpha }_1+{\beta }_1=2_s^{\ast }$, ${\alpha }_1>1$ and ${\beta }_1>1$, the function $A:{\mathbb{R}}^n\to {\mathbb{R}}^n$ is a $L_{loc}^{\infty }$–vector field, and ${(-\Delta )}_A^s$ is so called magnetic fractional Laplace operator. For any complex-valued function $\varphi \in C_0^{\infty }({\mathbb{R}}^n,\mathbb{C})$, which is defined by

$$\begin{array}{c}{(-\Delta )}_A^s\varphi \left(x\right)=2\underset{ϵ\to 0^{+}}{lim}{\int }_{{\mathbb{R}}^n\setminus B_{ϵ}\left(x\right)}\frac{\varphi \left(x\right)-e^{i(x-y)·A(\frac{x+y}{2})}\varphi \left(y\right)}{{|x-y|}^{n+2s}}dy,x\in {\mathbb{R}}^n,\end{array}$$

(2)

where $B_{ϵ}\left(x\right)$ denotes a ball in ${\mathbb{R}}^n$, x is the center of the ball, and $ϵ$ is the radius. Actually, the operator ${(-\Delta )}_A^s$ can be seen as the fractional counterpart of magnetic Laplace operator $-{(\nabla -iA)}^2$; for more details, the readers can see [17,18]. If $A\equiv 0$, the operator given by (2) is consistent with the usual fractional Laplace operator

$$\begin{array}{c}{(-\Delta )}^s\varphi \left(x\right)=\underset{\epsilon \to 0^{+}}{lim}{\int }_{{\mathbb{R}}^n\setminus B_{\epsilon }\left(x\right)}\frac{\varphi \left(x\right)-\varphi \left(y\right)}{{|x-y|}^{n+2s}}dy,x\in {\mathbb{R}}^n,\end{array}$$

(3)

we refer the readers to reference [19] for details and properties about this operator. And when $A\equiv 0$, the problem (1) becomes

$$\left\{\begin{array}{cc}{(-\Delta )}^s{u}_1={\lambda }_1|u_1{|}^{q-2}{u}_1+\frac{2{\alpha }_1}{{\alpha }_1+{\beta }_1}|u_1{|}^{{\alpha }_1-2}{u}_1{|u_2|}^{{\beta }_1}& \mathrm{in}\Omega ,\hfill \\ {(-\Delta )}^s{u}_2={\lambda }_2|u_2{|}^{q-2}{u}_2+\frac{2{\beta }_1}{{\alpha }_1+{\beta }_1}|u_2{|}^{{\beta }_1-2}{u}_2{|u_1|}^{{\alpha }_1}& \mathrm{in}\Omega ,\hfill \\ u_1=u_2=0& \mathrm{in}{\mathbb{R}}^n\setminus \Omega .\hfill \end{array}\right.$$

(4)

When $A\equiv 0$, $u_1=u_2, {\alpha }_1={\beta }_1, {\alpha }_1+{\beta }_1=p, {\lambda }_1={\lambda }_2$, the problem (1) becomes

$$\left\{\begin{array}{cc}{(-\Delta )}^s{u}_1={\lambda }_1|u_1{|}^{q-2}{u}_1+{|u_1|}^{p-2}{u}_1& \mathrm{in}\Omega ,\hfill \\ u_1=0& \mathrm{in}{\mathbb{R}}^n\setminus \Omega .\hfill \end{array}\right.$$

(5)

And the more general forms of (4) and (5) are p-Laplace operators. For this class of system, in [20], Chen and Deng obtained at least two nontrivial solutions for case p with the Nehari manifold method. For the study of problems driven by fractional Laplace and p-Laplace operators, there are many valuable results, for example [21,22,23,24,25,26,27,28,29,30,31,32,33,34,35]. When $A\equiv 0$, $s=1$, the problem (1) becomes the critical elliptic equations involving concave–convex nonlinearities. For classical works, see [36,37].

2. Main Result

In order to describe our research, some notations, basic properties, and definitions about working spaces are needed, all of these will be used in the subsequent sections.

Let $L^2(\Omega ,\mathbb{R})$ be the space of the measurable real function $\phi :\Omega \to \mathbb{R}$,

$$L^2(\Omega ,\mathbb{R}):=\left\{\phi |\phi :\Omega \to \mathbb{R}\mathrm{is}\mathrm{a}\mathrm{real}\mathrm{measurable}\mathrm{function},\mathrm{with}{\int }_{\Omega }{|\phi \left(x\right)|}^{2}dx<\infty \right\},$$

equipped with norm

$${\parallel \phi \parallel }_{L^2}={\left({\int }_{\Omega }{|\phi \left(x\right)|}^{2}dx\right)}^{\frac{1}{2}}\mathrm{for}\mathrm{all}\phi \in L^2(\Omega ,\mathbb{R}).$$

Denote the usual fractional Sobolev space by $H^s(\Omega )$, that is

$$H^s(\Omega ):=\left\{\phi |\phi \in L^2(\Omega ,\mathbb{R}),{\left[\phi \right]}_s<\infty \right\},$$

with norm

$${\parallel \phi \parallel }_{H^s}={\left({\parallel \phi \parallel }_{L^2}^2+{\left[\phi \right]}_s^2\right)}^{\frac{1}{2}},$$

where ${\left[\phi \right]}_s^2={\int }_{\Omega \times \Omega }\frac{{|\phi \left(x\right)-\phi \left(y\right)|}^2}{{|x-y|}^{n+2s}}dxdy$, and ${\left[\phi \right]}_s$ is the Gagliardo semi-norm.

Let $v:\Omega \to \mathbb{C}$ be a complex-valued function, which is measurable. Denote $L^2(\Omega ,\mathbb{C})$ as a space of v, and define the real scalar product as follows:

$${\langle v,\phi \rangle }_{L^2(\Omega ,\mathbb{C})}=\mathfrak{R}{\int }_{\Omega }v\overline{\phi }dx\mathrm{for}\mathrm{all}v,\phi \in L^2(\Omega ,\mathbb{C}),$$

where $\overline{\phi }$ is the complex conjugate of $\phi$, and $\mathfrak{R}\phi$ is the real part of $\phi$. The norm of $L^2(\Omega ,\mathbb{C})$ endowed with

$${\parallel v\parallel }_{L^2(\Omega ,\mathbb{C})}={\left({\int }_{\Omega }{|v\left(x\right)|}^{2}dx\right)}^{\frac{1}{2}}\mathrm{for}\mathrm{all}v\in L^2(\Omega ,\mathbb{C}).$$

Let $L^{2_s^{\ast }}(\Omega ,\mathbb{C})$ be the space of $v:\Omega \to \mathbb{C}$, which is a Lebesgue space, and it has the following scalar product

$${\langle v,\phi \rangle }_{L^{2_s^{\ast }}(\Omega ,\mathbb{C})}=\mathfrak{R}{\int }_{\Omega }{|u\left(x\right)|}^{2_s^{\ast }-2}v\overline{\phi }dx,$$

for all v, $\phi \in L^{2_s^{\ast }}(\Omega ,\mathbb{C})$. The norm is defined by

$${\parallel v\parallel }_{L^{2_s^{\ast }}(\Omega ,\mathbb{C})}={\left({\int }_{\Omega }{|v\left(x\right)|}^{2_s^{\ast }}dx\right)}^{\frac{1}{2_s^{\ast }}}\mathrm{for}\mathrm{all}v\in L^{2_s^{\ast }}(\Omega ,\mathbb{C}).$$

Set

$${\left[\upsilon \right]}_A^2={\int }_{\Omega \times \Omega }\frac{|\upsilon \left(x\right)-e^{i(x-y)·A\left(\frac{x+y}{2}\right)}{\upsilon \left(y\right)|}^2}{{|x-y|}^{n+2s}}dxdy,$$

where ${\left[\upsilon \right]}_A$ is so called magnetic Gagliardo semi-norm.

Let

$$H_A^s(\Omega ,\mathbb{C}):=\{\upsilon |\upsilon \in L^2(\Omega ,\mathbb{C}),{\left[\upsilon \right]}_A<\infty \}.$$

A scalar product on the space $H_A^s(\Omega ,\mathbb{C})$ is defined by

$$\begin{array}{c}{\langle \varphi ,\upsilon \rangle }_{s,A}={\langle \varphi ,\upsilon \rangle }_{L^2(\Omega ,\mathbb{C})}+\mathfrak{R}{\int }_{\Omega \times \Omega }\frac{[\varphi \left(x\right)-e^{i(x-y)·A\left(\frac{x+y}{2}\right)}\varphi \left(y\right)]·\overline{[\upsilon \left(x\right)-e^{i(x-y)·A(\frac{x+y}{2})}\upsilon \left(y\right)]}}{{|x-y|}^{n+2s}}dxdy,\end{array}$$

and normed

$${\parallel \upsilon \parallel }_{H_A^s}={\left({\parallel \upsilon \parallel }_{L^2(\Omega ,\mathbb{C})}^2+{\left[\upsilon \right]}_A^2\right)}^{\frac{1}{2}}.$$

(6)

By the above definitions, we see that the space $H_A^s(\Omega ,\mathbb{C})$ is consistent with the space $H^s(\Omega )$ when the function $A\equiv 0$. Moreover, according to [18], we have $C_0^{\infty }({\mathbb{R}}^n,\mathbb{C})\subseteq H_A^s({\mathbb{R}}^n,\mathbb{C})$.

Set $Q={\mathbb{R}}^{2n}\setminus (C\Omega \times C\Omega )$, and $C\Omega ={\mathbb{R}}^n\setminus \Omega$, then, we let

$$H:=\left\{\upsilon |\upsilon \in L^2(\Omega ,\mathbb{C}),{\left[\upsilon \right]}_Q<\infty \right\},$$

where ${\left[\upsilon \right]}_Q^2={\int }_Q\frac{|\upsilon \left(x\right)-e^{i(x-y)·A\left(\frac{x+y}{2}\right)}{\upsilon \left(y\right)|}^2}{{|x-y|}^{n+2s}}dxdy$, the norm is given by

$${\parallel \upsilon \parallel }_H={\left({\parallel \upsilon \parallel }_{L^2(\Omega ,\mathbb{C})}^2+{\left[\upsilon \right]}_Q^2\right)}^{\frac{1}{2}}.$$

(7)

Next, we define the space $H_0$ as

$$H_0:=\left\{\upsilon |\upsilon \in H,\upsilon =0\mathrm{a}.\mathrm{e}.\mathrm{in}{\mathbb{R}}^n\setminus \Omega \right\}.$$

As the space introduced in [38], we generalize it to magnetic framework, and the real scalar product is defined as

$$\begin{array}{c}{\langle \varphi ,\upsilon \rangle }_{H_0}=\mathfrak{R}{\int }_Q\frac{[\varphi \left(x\right)-e^{i(x-y)·A\left(\frac{x+y}{2}\right)}\varphi \left(y\right)]·\overline{[\upsilon \left(x\right)-e^{i(x-y)·A(\frac{x+y}{2})}\upsilon \left(y\right)]}}{{|x-y|}^{n+2s}}dxdy,\end{array}$$

and normed with

$${\parallel \upsilon \parallel }_{H_0}={\left[\upsilon \right]}_Q={\left({\int }_Q\frac{|\upsilon \left(x\right)-e^{i(x-y)·A\left(\frac{x+y}{2}\right)}{\upsilon \left(y\right)|}^2}{{|x-y|}^{n+2s}}dxdy\right)}^{\frac{1}{2}}.$$

(8)

Similar to [1], we know that the norm defined by (7) is equivalent to (8), and we observe that $\left(H_0,{\langle ·,·\rangle }_{H_0}\right)$ is a real Hilbert space, and is also separable.

Now, we formulate the main result of our paper as follows.

Theorem 1. 

Let $\Omega \in {\mathbb{R}}^n$ be an open bounded set, $1<q<2<\frac{n}{s}$ with $s\in (0,1)$, ${\alpha }_1+{\beta }_1=\frac{2n}{n-2s}=2_s^{\ast }$. Then, a positive number Λ can be founded, such that for every $0<{\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}}<\Lambda$, the problem (1) possesses infinitely many solutions.

Remark 1. 

The main novelty of this result is that we provide a new method to obtain the result. As far as we know, there are not many results about the fractional system with external magnetic potential, and the conclusion of Theorem 1 is the first result for a magnetic fractional system with critical concave–convex powers.

Remark 2. 

When $2<{\alpha }_1+{\beta }_1<2_s^{\ast }$, the same result also can be obtained by using our method. The proof of this case is easier than the critical case, because Sobolev embedding has compactness in this subcritical case. The Palais–Smale ($PS$) condition holds easily for the energy functional. In this paper, we discuss the critical case, then Sobolev embedding compactness is lost, the energy functional does not satisfy the Palais–Smale ($PS$) condition at all levels. So in this case, we must find suitable range of c, such that the energy functional satisfies the Palais–Smale ($PS$) condition in this range.

Remark 3. 

As mentioned before, when $A\equiv 0$, the problem (1) becomes (4), for the system (4), our result of Theorem 1 also holds. Furthermore, using the present method, infinitely many solutions also can be obtained for this class p-Laplacian system. Moreover, our result is new, even for local cases.

Now, let us state the arrangement of the present paper. In Section 3, we provide some previous knowledge, recall some properties for the working space, and set up the functional framework. In the Section 4, we prove our main result. In the Section 5, we provide the conclusion.

3. Preliminary Knowledge and Functional Setting

To prove the main result, in this section, we introduce some lemmas related to the space and recall some preliminary knowledge.

Let $W=H_0\times H_0$, we know that W is a separable Hilbert space, and the norm of space W is defined by

$$\parallel (u_1,u_2)\parallel ={\left(\parallel u_1{\parallel }_{H_0}^2+{\parallel u_2\parallel }_{H_0}^2\right)}^{\frac{1}{2}}.$$

(9)

To simplify the writing mode, we provide the notation

$$\langle L\left(u_1\right),\upsilon \rangle =\mathfrak{R}{\int }_Q\frac{[u_1\left(x\right)-e^{i(x-y)·A\left(\frac{x+y}{2}\right)}{u}_1\left(y\right)]·\overline{[\upsilon \left(x\right)-e^{i(x-y)·A(\frac{x+y}{2})}\upsilon \left(y\right)]}}{{|x-y|}^{n+2s}}dxdy.$$

Then, we provide the following definition.

Definition 1. 

If $(u_1,u_2)\in W$, we have

$$\begin{array}{cc}\hfill \langle L\left(u_1\right),\upsilon \rangle +\langle L\left(u_2\right),\nu \rangle & =\mathfrak{R}{\int }_{\Omega }({\lambda }_1|u_1{|}^{q-2}{u}_1\overline{\upsilon }+{\lambda }_2|u_2{|}^{q-2}{u}_2\overline{\nu })dx\hfill \\ & +\mathfrak{R}\left[\frac{2{\alpha }_1}{{\alpha }_1+{\beta }_1}{\int }_{\Omega }|u_1{|}^{{\alpha }_1-2}{u}_1\overline{\upsilon }|u_2{|}^{{\beta }_1}dx+\frac{2{\beta }_1}{{\alpha }_1+{\beta }_1}{\int }_{\Omega }|u_1{|}^{{\alpha }_1}{|u_2|}^{{\beta }_1-2}{u}_2\overline{\nu }dx\right]\hfill \end{array}$$

for all $(\upsilon ,\nu )\in W$, then $(u_1,u_2)$ is a weak solution of the problem (1).

Now, let us define the functional $I(u_1,u_2):W\to \mathbb{R}$ associated with the problem (1) as

$$I(u_1,u_2)=\frac{1}{2}\parallel (u_1,u_2){\parallel }^2-\frac{1}{q}{\int }_{\Omega }({\lambda }_1|u_1{|}^q+{\lambda }_2|u_2{|}^q)dx-\frac{2}{2_s^{\ast }}{\int }_{\Omega }|u_1{|}^{{\alpha }_1}{|u_2|}^{{\beta }_1}dx.$$

(10)

We see that $I(u_1,u_2)\in C^1(W,\mathbb{R})$ and $I(u_1,u_2)$ is even, in addition

$$\begin{array}{cc}\hfill \langle I^{\prime }(u_1,u_2),(\upsilon ,\nu )\rangle & =\langle L\left(u_1\right),\upsilon \rangle +\langle L\left(u_2\right),\nu \rangle -\mathfrak{R}{\int }_{\Omega }({\lambda }_1|u_1{|}^{q-2}{u}_1\overline{\upsilon }+{\lambda }_2|u_2{|}^{q-2}{u}_2\overline{\nu })dx\hfill \\ & +\mathfrak{R}\left[\frac{2{\alpha }_1}{2_s^{\ast }}{\int }_{\Omega }|u_1{|}^{{\alpha }_1-2}{u}_1\overline{\upsilon }|u_2{|}^{{\beta }_1}dx+\frac{2{\beta }_1}{2_s^{\ast }}{\int }_{\Omega }|u_1{|}^{{\alpha }_1}{|u_2|}^{{\beta }_1-2}{u}_2\overline{\nu }dx\right].\hfill \end{array}$$

It is clear that the weak solutions of the problem (1) are all the critical points of functional $I(u_1,u_2)$.

To prove the main result of Theorem 1, the following lemmas are also needed.

Lemma 1. 

For any $\phi \in H_0$, we have $|\phi |\in H^S(\Omega )$ and ${\parallel |\phi |\parallel }_{H^s}\le {\parallel \phi \parallel }_{H_0}$.

Proof. 

Using the same process as in [3], we can obtain the result of Lemma 1 easily; so, the proving details are omitted here. □

Lemma 2 

([1]). If $\Omega \subset {\mathbb{R}}^n$ is an open bounded set, then we have that $H_0\hookrightarrow L^p(\Omega ,\mathbb{C})$ is a continuous embedding for any $p\in [1,2_s^{\ast }]$; If $\Omega \subset {\mathbb{R}}^n$ is an open bounded set and the boundary $\partial \Omega$ is Lipschitz, then we have that $H_0\hookrightarrow \hookrightarrow L^p(\Omega ,\mathbb{C})$ is a compact embedding for any $p\in [1,2_s^{\ast })$.

Next, we recall the definition of the Palais–Smale condition. For convenience, from then on, we write it as the (PS) condition for short.

Definition 2. 

((PS) sequence) Let E be a Banach space, the functional $I\in C^1(E,\mathbb{R})$, $c\in \mathbb{R}$, if $\underset{n\to \infty }{lim}I({u_1}_n,{u_2}_n)=c$ in E and $\underset{n\to \infty }{lim}{I}^{\prime }({u_1}_n,{u_2}_n)=0$ in $E^{\ast }$, then we have $\left\{({u_1}_n,{u_2}_n)\right\}\subset E$ is a (PS) sequence of I, where $E^{\ast }$ is the dual space of E.

Definition 3. 

((PS) condition) If any (PS) sequence $\left\{({u_1}_n,{u_2}_n)\right\}\subset E$ has a convergent subsequence, the functional I satisfies the (PS) condition.

Remark 4. 

By Lemma 2, we know that the embedding $H_0\hookrightarrow L^{2_s^{\ast }}(\Omega ,\mathbb{C})$ is not compact for the critical case, the functional $I(u_1,u_2)$ does not satisfy the (PS) condition at all levels, then more strategies are needed to overcome this difficulty. Hence, we need to prove that $I(u_1,u_2)$ satisfies the (PS) condition at a certain level. The following definition is needed,

$$\begin{array}{c}S=\underset{v\in H_0\setminus \left\{0\right\}}{inf}\frac{{\parallel v\parallel }_{H_0}^2}{{\parallel v\parallel }_{2_s^{\ast }}^2}.\end{array}$$

(11)

It is easy to see that $S>0$ is well defined.

In two steps, we show that the function I satisfies the (PS) condition. Firstly, we prove the boundedness of the (PS) sequence.

Lemma 3. 

Let $\left\{({u_1}_n,{u_2}_n)\right\}\subset W$ be a (PS) sequence of $I(u_1,u_2)$, we have that $\left\{({u_1}_n,{u_2}_n)\right\}$ is bounded.

Proof. 

Let us prove it by contradiction; assume that $\parallel ({u_1}_n,{u_2}_n)\parallel \to \infty$ as $n\to \infty$. Since $({u_1}_n,{u_2}_n)$ is a (PS) sequence of $I(u_1,u_2)$, there exists a constant $C>0$, such that $|I({u_1}_n,{u_2}_n)|\le C$ and $|\langle I^{\prime }({u_1}_n,{u_2}_n),({u_1}_n,{u_2}_n)\rangle |\le C\parallel ({u_1}_n,{u_2}_n)\parallel$, we get

$$\begin{array}{cc}\hfill C(1+\parallel ({u_1}_n,{u_2}_n)\parallel )& \ge I({u_1}_n,{u_2}_n)-\frac{1}{2_s^{\ast }}\langle I^{\prime }({u_1}_n,{u_2}_n),({u_1}_n,{u_2}_n)\rangle \hfill \\ & =(\frac{1}{2}-\frac{1}{2_s^{\ast }})\parallel ({u_1}_n,{u_2}_n){\parallel }^2-(\frac{1}{q}-\frac{1}{2_s^{\ast }}){\int }_{\Omega }{\lambda }_1|{u_1}_n{|}^q+{\lambda }_2{|{u_2}_n|}^{q}dx.\hfill \end{array}$$

(12)

According to (11) and the Hölder inequality, we obtain

$$\begin{array}{cc}\hfill {\int }_{\Omega }{\lambda }_1|{u_1}_n{|}^q+{\lambda }_2{|{u_2}_n|}^{q}dx& \le {|\Omega |}^{\frac{2_s^{\ast }-q}{2_s^{\ast }}}({\lambda }_1\parallel {u_1}_n{\parallel }_{L^{2_s^{\ast }}(\Omega ,\mathbb{C})}^q+{\lambda }_2\parallel {u_2}_n{\parallel }_{L^{2_s^{\ast }}(\Omega ,\mathbb{C})}^q)\hfill \\ & \le {|\Omega |}^{\frac{2_s^{\ast }-q}{2_s^{\ast }}}{S}^{-\frac{q}{2}}({\lambda }_1\parallel {u_1}_n{\parallel }_{H_0}^q+{\lambda }_2\parallel {u_2}_n{\parallel }_{H_0}^q)\hfill \\ & \le {|\Omega |}^{\frac{2_s^{\ast }-q}{2_s^{\ast }}}{S}^{-\frac{q}{2}}{({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})}^{\frac{2-q}{2}}{\parallel (u_1,u_2)\parallel }^q.\hfill \end{array}$$

(13)

Placing (13) into (12), we have

$$\begin{array}{cc}\hfill C(1+\parallel ({u_1}_n,{u_2}_n)\parallel )& \ge (\frac{1}{2}-\frac{1}{2_s^{\ast }}){\parallel ({u_1}_n,{u_2}_n)\parallel }^2\hfill \\ & -(\frac{1}{q}-\frac{1}{2_s^{\ast }}){|\Omega |}^{\frac{2_s^{\ast }-q}{2_s^{\ast }}}{S}^{-\frac{q}{2}}{({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})}^{\frac{2-q}{2}}{\parallel ({u_1}_n,{u_2}_n)\parallel }^q.\hfill \end{array}$$

(14)

Obviously,

$$\begin{array}{cc}\hfill C(\frac{1}{\parallel ({u_1}_n,{u_2}_n){\parallel }^2}+\frac{1}{\parallel ({u_1}_n,{u_2}_n)\parallel })& \ge (\frac{1}{2}-\frac{1}{2_s^{\ast }})\hfill \\ & -(\frac{1}{q}-\frac{1}{2_s^{\ast }}){|\Omega |}^{\frac{2_s^{\ast }-q}{2_s^{\ast }}}{S}^{-\frac{q}{2}}{({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})}^{\frac{2-q}{2}}\frac{1}{\parallel ({u_1}_n,{u_2}_n){\parallel }^{2-q}}.\hfill \end{array}$$

(15)

According to the assumptions, we obtain that $\parallel ({u_1}_n,{u_2}_n)\parallel \to \infty$ as $n\to \infty$. In (15), we take the limit as $n\to \infty$, there must be

$$\begin{array}{c}0<\frac{1}{2}-\frac{1}{2_s^{\ast }}\le 0\end{array}$$

(16)

as $1<q<2<2_s^{\ast }$. This is contradictory; the sequence $\left\{({u_1}_n,{u_2}_n)\right\}$ must be bounded in W. The proof is completed. □

Lemma 4. 

If $\left\{({u_1}_n,{u_2}_n)\right\}$ is a (PS) sequence of I, and $({u_1}_n,{u_2}_n)\rightharpoonup (u_1,u_2)$ in W, then $I^{\prime }(u_1,u_2)=0$, and $I(u_1,u_2)\ge -C^{\ast }({\lambda }_1^{\frac{2}{2-r}}+{\lambda }_2^{\frac{2}{2-r}})$, where constant $C^{\ast }$ is positive and

$$C^{\ast }=(\frac{1}{q}-\frac{1}{2_s^{\ast }})\frac{2-q}{2}{\left(\frac{2_s^{\ast }-q}{2_s^{\ast }-2}\right)}^{\frac{2}{2-q}}{||}^{\frac{2(2_s^{\ast }-q)}{2_s^{\ast }(2-q)}}{S}^{-\frac{q}{2-q}}.$$

Proof. 

Considering that $\left\{({u_1}_n,{u_2}_n)\right\}$ is a (PS) sequence of I in W, Let $(\upsilon ,\nu )\in W$, we have

$$\begin{array}{cc}\hfill \langle I^{\prime }({u_1}_n,{u_2}_n)-I^{\prime }(u_1,u_2),(\upsilon ,\nu )\rangle & =\langle L\left({u_1}_n\right),\upsilon \rangle -\langle L\left(u_1\right),\upsilon \rangle +\langle L\left({u_2}_n\right),\nu \rangle -\langle L\left(u_2\right),\nu \rangle \hfill \\ & -{\lambda }_1\mathfrak{R}{\int }_{\Omega }(|{u_1}_n{|}^{q-2}{u_1}_n-|u_1{|}^{q-2}{u}_1)\overline{\upsilon }dx\hfill \\ & -{\lambda }_2\mathfrak{R}{\int }_{\Omega }(|{u_2}_n{|}^{q-2}{u_2}_n-|u_2{|}^{q-2}{u}_2)\overline{\nu }dx\hfill \\ & -\frac{2{\alpha }_1}{2_s^{\ast }}\mathfrak{R}{\int }_{\Omega }(|{u_1}_n{|}^{{\alpha }_1-2}{u_1}_n|{u_2}_n{|}^{{\beta }_1}-|u_1{|}^{{\alpha }_1-2}{u}_1|u_2{|}^{{\beta }_1})\overline{\upsilon }dx\hfill \\ & -\frac{2{\beta }_1}{2_s^{\ast }}\mathfrak{R}{\int }_{\Omega }(|{u_1}_n{|}^{{\alpha }_1}|{u_2}_n{|}^{{\beta }_1-2}{u_2}_n-|u_1{|}^{{\alpha }_1}|u_2{|}^{{\beta }_1-2}{u}_2)\overline{\nu }dx,\hfill \end{array}$$

(17)

where $\langle L(·),·\rangle$ is given as before. Meanwhile, by Lemma 3, we know that $({u_1}_n,{u_2}_n)$ is bounded, and by Lemma 2, we have that there exists a subsequence (we still denote it by $\left\{({u_1}_n,{u_2}_n)\right\}$), such that

$$\begin{array}{cc}& ({u_1}_n,{u_2}_n)\rightharpoonup (u_1,u_2)\mathrm{in}W,{u_1}_n\rightharpoonup u_1,{u_2}_n\rightharpoonup u_2\mathrm{in}{L}^{2_s^{\ast }}(\Omega ,\mathbb{C}),\hfill \\ & {u_1}_n\to u_1,{u_2}_n\to u_2\mathrm{in}{L}^r(\Omega ,\mathbb{C}),1\le r<2_s^{\ast },\hfill \\ & {u_1}_n\to u_1,{u_2}_n\to u_2\mathrm{a}.\mathrm{e}.\mathrm{in}{\mathbb{R}}^n,\hfill \end{array}$$

(18)

we have

$$\begin{array}{c}\frac{{u_1}_n\left(x\right)-e^{i(x-y)A\left(\frac{x+y}{2}\right)}{u_1}_n\left(y\right)}{{|x-y|}^{\frac{n+2s}{2}}}\to \frac{u_1\left(x\right)-e^{i(x-y)A\left(\frac{x+y}{2}\right)}{u}_1\left(y\right)}{{|x-y|}^{\frac{n+2s}{2}}}\mathrm{in}{L}^2(\Omega ,\mathbb{C})\end{array}$$

and

$$\begin{array}{c}\frac{{u_2}_n\left(x\right)-e^{i(x-y)A\left(\frac{x+y}{2}\right)}{u_2}_n\left(y\right)}{{|x-y|}^{\frac{n+2s}{2}}}\to \frac{u_2\left(x\right)-e^{i(x-y)A\left(\frac{x+y}{2}\right)}{u}_2\left(y\right)}{{|x-y|}^{\frac{n+2s}{2}}}\mathrm{in}{L}^2(\Omega ,\mathbb{C}).\end{array}$$

Since

$$\begin{array}{c}\frac{\upsilon \left(x\right)-e^{i(x-y)A\left(\frac{x+y}{2}\right)}\upsilon \left(y\right)}{{|x-y|}^{\frac{n+2s}{2}}}\in L^2(\Omega ,\mathbb{C}),\\ \frac{\nu \left(x\right)-e^{i(x-y)A\left(\frac{x+y}{2}\right)}\nu \left(y\right)}{{|x-y|}^{\frac{n+2s}{2}}}\in L^2(\Omega ,\mathbb{C})\end{array}$$

for all $\upsilon ,\nu \in H_0$, we claim that

$$\begin{array}{c}\underset{n\to \infty }{lim}\langle L\left({u_1}_n\right),\upsilon \rangle =\langle L\left(u_1\right),\upsilon \rangle ,\underset{n\to \infty }{lim}\langle L\left({u_2}_n\right),\nu \rangle =\langle L\left(u_2\right),\nu \rangle .\end{array}$$

In addition, we know

$$\begin{array}{c}|{u_1}_n{|}^{q-2}{u_1}_n\to |u_1{|}^{q-2}{u}_1,|{u_2}_n{|}^{q-2}{u_2}_n\to {|u_2|}^{q-2}{u}_2\mathrm{in}{L}^{\frac{q}{q-1}}(\Omega ,\mathbb{C})\end{array}$$

and

$$\begin{array}{cc}& |{u_1}_n{|}^{{\alpha }_1-2}{u_1}_n|{u_2}_n{|}^{{\beta }_1}\to |u_1{|}^{{\alpha }_1-2}{u}_1{|u_2|}^{{\beta }_1},\hfill \\ & |{u_1}_n{|}^{{\alpha }_1}|{u_2}_n{|}^{{\beta }_1-2}{u_2}_n\to |u_1{|}^{{\alpha }_1}{|u_2|}^{{\beta }_1-2}{u}_2\mathrm{in}{L}^{\frac{2_s^{\ast }}{2_s^{\ast }-1}}(\Omega ,\mathbb{C})\hfill \end{array}$$

for all $\upsilon ,\nu \in L^q(\Omega ,\mathbb{C})\cap L^{2_s^{\ast }}(\Omega ,\mathbb{C})$, as $n\to \infty$. Therefore, we can obtain

$$\begin{array}{cc}& \mathfrak{R}{\int }_{\Omega }(|{u_1}_n{|}^{{\alpha }_1-2}{u_1}_n|{u_2}_n{|}^{{\beta }_1}-|u_1{|}^{{\alpha }_1-2}{u}_1|u_2{|}^{{\beta }_1})\overline{\upsilon }dx\to 0,\hfill \\ & \mathfrak{R}{\int }_{\Omega }(|{u_2}_n{|}^{q-2}{u_2}_n-|u_2{|}^{q-2}{u}_2)\overline{\nu }dx\to 0\hfill \end{array}$$

and

$$\begin{array}{cc}& \mathfrak{R}{\int }_{\Omega }(|{u_1}_n{|}^{q-2}{u_1}_n-|u_1{|}^{q-2}{u}_1)\overline{\upsilon }dx\to 0,\hfill \\ & \mathfrak{R}{\int }_{\Omega }(|{u_1}_n{|}^{{\alpha }_1}|{u_2}_n{|}^{{\beta }_1-2}{u_2}_n-|u_1{|}^{{\alpha }_1}|u_2{|}^{{\beta }_1-2}{u}_2)\overline{\nu }dx\to 0\hfill \end{array}$$

as $n\to \infty$. Hence,

$$\begin{array}{c}\langle I^{\prime }({u_1}_n,{u_2}_n)-I^{\prime }(u_1,u_2),(\upsilon ,\nu )\rangle \to 0,as\to \infty ,\mathrm{for}\mathrm{all}(\upsilon ,\nu )\in W.\end{array}$$

(19)

As $I^{\prime }({u_1}_n,{u_2}_n)=o\left(1\right)$, as $n\to \infty$, we have the conclusion

$$\begin{array}{c}{I}^{\prime }(u_1,u_2)=0.\end{array}$$

(20)

Specially, $\langle I^{\prime }(u_1,u_2)(u_1,u_2)\rangle =0$, we have

$$\begin{array}{c}2{\int }_{\Omega }|u_1{|}^{{\alpha }_1}|u_2{|}^{{\beta }_1}dx=\parallel (u_1,u_2){\parallel }^2-\int ({\lambda }_1|u_1{|}^q+{\lambda }_2|u_2{|}^q)dx.\end{array}$$

(21)

So

$$\begin{array}{cc}\hfill I(u_1,u_2)& =I(u_1,u_2)-\frac{1}{2_s^{\ast }}\langle I^{\prime }(u_1,u_2)(u_1,u_2)\rangle \hfill \\ & =(\frac{1}{2}-\frac{1}{2_s^{\ast }})\parallel (u_1,u_2){\parallel }^2-(\frac{1}{q}-\frac{1}{2_s^{\ast }}){\int }_{\Omega }({\lambda }_1|u_1{|}^q+{\lambda }_2|u_2{|}^q)dx.\hfill \end{array}$$

(22)

By Hölder’s inequality, Young’s inequality, and the definition of S, we have

$$\begin{array}{cc}\hfill {\int }_{\Omega }({\lambda }_1|u_1{|}^q+{\lambda }_2|u_2{|}^q)dx& \le {|\Omega |}^{\frac{2_s^{\ast }-q}{2_s^{\ast }}}{S}^{-\frac{q}{2}}({\lambda }_1\parallel u_1{\parallel }_{H_0}^q+{\lambda }_2\parallel u_2{\parallel }_{H_0}^q)\hfill \\ & =({[\frac{2}{q}\frac{s}{n}{(\frac{1}{q}-\frac{1}{2_s^{\ast }})}^{-1}]}^{\frac{q}{2}}\parallel u_1{\parallel }_{H_0}^q)({[\frac{2}{q}\frac{s}{n}{(\frac{1}{q}-\frac{1}{2_s^{\ast }})}^{-1}]}^{-\frac{q}{2}}{|\Omega |}^{\frac{2_s^{\ast }-q}{2_s^{\ast }}}{S}^{-\frac{q}{2}}{\lambda }_1\hfill \\ & +({[\frac{2}{q}\frac{s}{n}{(\frac{1}{q}-\frac{1}{2_s^{\ast }})}^{-1}]}^{\frac{q}{2}}\parallel u_2{\parallel }_{H_0}^q)({[\frac{2}{q}\frac{s}{n}{(\frac{1}{q}-\frac{1}{2_s^{\ast }})}^{-1}]}^{-\frac{q}{2}}{|\Omega |}^{\frac{2_s^{\ast }-q}{2_s^{\ast }}}{S}^{-\frac{q}{2}}{\lambda }_2\hfill \\ & \le \frac{s}{n}{(\frac{1}{q}-\frac{1}{2_s^{\ast }})}^{-1}{\parallel (u_1,u_2)\parallel }^2+C_1({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}}),\hfill \end{array}$$

(23)

where

$$\begin{array}{cc}{C}_1& =\frac{2-q}{2}{\left({[\frac{2}{q}\frac{s}{n}{(\frac{1}{q}-\frac{1}{2_s^{\ast }})}^{-1}]}^{-\frac{q}{2}}{|\Omega |}^{\frac{2_s^{\ast }-q}{2_s^{\ast }}}{S}^{-\frac{q}{2}}\right)}^{\frac{2}{2-q}}\hfill \\ & =\frac{2-q}{2}{\left(\frac{2_s^{\ast }-q}{2_s^{\ast }-2}\right)}^{\frac{q}{2-q}}{|\Omega |}^{\frac{2(2_s^{\ast }-q)}{2_s^{\ast }(2-q)}}{S}^{-\frac{q}{2-q}}.\hfill \end{array}$$

Putting (23) into (22), we have

$$\begin{array}{c}I(u_1,u_2)\ge -C^{\ast }({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}}),\end{array}$$

(24)

where

$$\begin{array}{c}{C}^{\ast }=(\frac{1}{q}-\frac{1}{2_s^{\ast }})C_1=(\frac{1}{q}-\frac{1}{2_s^{\ast }})\frac{2-q}{2}{(\frac{2_s^{\ast }-q}{2_s^{\ast }-2})}^{\frac{q}{2-q}}{|\Omega |}^{\frac{2(2_s^{\ast }-q)}{2_s^{\ast }(2-q)}}{S}^{-\frac{q}{2-q}}.\end{array}$$

As $1<q<2<2_s^{\ast }$, the constant $C^{\ast }>0$. The result of Lemma 4 is obtained. □

Next, we need to prove that the (PS) sequence of I in W admits a convergent subsequence. Let

$$\begin{array}{c}{S}_{2_s^{\ast }}:=\underset{(u_1,u_2)\in W\setminus \left\{(0,0)\right\}}{inf}\frac{\parallel (u_1,u_2){\parallel }^2}{({\int }_{\Omega }|u_1{|}^{{\alpha }_1}|u_2{{|}^{{\beta }_1}dx)}^{\frac{2}{2_s^{\ast }}}}.\end{array}$$

(25)

Lemma 5. 

The function I satisfies the (PS) condition in W for $-\infty <c<\delta$, where $\delta =\frac{2s}{n}{(\frac{S_{2_s^{\ast }}}{2})}^{\frac{n}{2s}}-C^{\ast }({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})$, $C^{\ast }$ is provided by Lemma 4 and $S_{2_s^{\ast }}$ is provided by (25).

Proof. 

We know that $\left\{({u_1}_n,{u_2}_n)\right\}\subset W$ is a (PS) sequence; there must be

$$\begin{array}{c}\frac{1}{2}\parallel ({u_1}_n,{u_2}_n){\parallel }^2-\frac{1}{q}{\int }_{\Omega }({\lambda }_1|{u_1}_n{|}^q+{\lambda }_2|{u_2}_n{|}^q)dx-\frac{2}{2_s^{\ast }}{\int }_{\Omega }|{u_1}_n{|}^{{\alpha }_1}{|{u_2}_n|}^{{\beta }_1}dx=c+o\left(1\right),\end{array}$$

(26)

and

$$\begin{array}{c}\parallel ({u_1}_n,{u_2}_n){\parallel }^2-{\int }_{\Omega }({\lambda }_1|{u_1}_n{|}^q+{\lambda }_2|{u_2}_n{|}^q)dx-2{\int }_{\Omega }|{u_1}_n{|}^{{\alpha }_1}|{u_2}_n{|}^{{\beta }_1}dx=o(\parallel ({u_1}_n,{u_2}_n)\parallel ).\end{array}$$

(27)

According to Lemma 3, the boundedness of $\left\{({u_1}_n,{u_2}_n)\right\}$ is obtained. Hence, we know that there exists a subsequence of $\left\{({u_1}_n,{u_2}_n)\right\}$ (still denoted by $\left\{({u_1}_n,{u_2}_n)\right\}$), $({u_1}_n,{u_2}_n)\rightharpoonup (u_1,u_2)$ in W. By Lemma 4, we obtain that $(u_1,u_2)\in W$ is a critical point of $I(u_1,u_2)$. We also know that ${u_1}_n\to u_1$ and ${u_2}_n\to u_2$ in $L^r(\Omega ,\mathbb{C})$ with $1\le r<2_s^{\ast }$, then

$$\begin{array}{c}{\int }_{\Omega }({\lambda }_1|{u_1}_n{|}^q+{\lambda }_2|{u_2}_n{|}^q)dx\to {\int }_{\Omega }({\lambda }_1|u_1{|}^q+{\lambda }_2|u_2{|}^q)dx,\end{array}$$

(28)

as $n\to \infty$. Considering the famous Brézis-Lieb lemma, we have

$$\begin{array}{c}\parallel ({u_1}_n,{u_2}_n)\parallel =\parallel ({u_1}_n-u_1,{u_2}_n-u_2)\parallel +\parallel (u_1,u_2)\parallel +o\left(1\right)\end{array}$$

(29)

and

$$\begin{array}{c}{\int }_{\Omega }|{u_1}_n{|}^{{\alpha }_1}|{u_2}_n{|}^{{\beta }_1}dx={\int }_{\Omega }|{u_1}_n-u_1{|}^{{\alpha }_1}|{u_2}_n-u_2{|}^{{\beta }_1}dx+{\int }_{\Omega }|u_1{|}^{{\alpha }_1}{|u_2|}^{{\beta }_1}dx+o\left(1\right).\end{array}$$

(30)

Taking (29) and (30) into (26), we have

$$\begin{array}{c}\frac{1}{2}\parallel ({u_1}_n-u_1,{u_2}_n-u_2)\parallel -\frac{2}{2_s^{\ast }}{\int }_{\Omega }|{u_1}_n-u_1{|}^{{\alpha }_1}{|{u_2}_n-u_2|}^{{\beta }_1}dx=c-I(u_1,u_2)+o\left(1\right),\end{array}$$

(31)

furthermore, we place (29) and (30) into (27), and we find

$$\begin{array}{c}\parallel ({u_1}_n-u_1,{u_1}_n-u_2){\parallel }^2=2{\int }_{\Omega }|{u_1}_n-u_1{|}^{{\alpha }_1}{|{u_2}_n-u_2|}^{{\beta }_1}dx+o\left(1\right).\end{array}$$

(32)

Let us assume

$$\begin{array}{c}\parallel ({u_1}_n-u_1),({u_2}_n-u_2){\parallel }^2\to a,2{\int }_{\Omega }|{u_1}_n-u_1{|}^{{\alpha }_1}{|{u_2}_n-u_2|}^{{\beta }_1}dx\to b\end{array}$$

(33)

as $n\to \infty$. If $a=0$, we complete the proof. Next, assuming $a>0$ and combining (32) with (33), we find $a\le 2S_{2_s^{\ast }}^{-\frac{2_s^{\ast }}{2}}·a^{\frac{2_s^{\ast }}{2}}$; hence,

$$\begin{array}{c}a\ge 2{\left(\frac{S_{2_s^{\ast }}}{2}\right)}^{-\frac{n}{2s}}.\end{array}$$

(34)

Take the limit in (31) as $n\to \infty$, by (33), (34), and Lemma 4, we have

$$\begin{array}{cc}\hfill c& =\frac{1}{2}a-\frac{m}{2_s^{\ast }}+I(u_1,u_2)=(\frac{1}{2}-\frac{1}{2_s^{\ast }})a+I(u_1,u_2)\hfill \\ & \ge \frac{s}{n}a+I(u_1,u_2)\ge \frac{2s}{n}{\left(\frac{S_{2_s^{\ast }}}{2}\right)}^{-\frac{n}{2s}}-C^{\ast }({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})=\delta .\hfill \end{array}$$

(35)

As $-\infty <c<\delta$, we have the contradiction. Hence, the proof is competed. □

4. Proof of Main Result

Now, we provide the proof of Theorem 1 in this section. Firstly, we also consider the functional $I(u_1,u_2)$, by the Hölder inequality and $S_{2_s^{\ast }}$ given by (25), we have

$$\begin{array}{cc}\hfill I(u_1,u_2)& \ge \frac{1}{2}\parallel (u_1,u_2){\parallel }^2-\frac{2}{2_s^{\ast }}{S}_{2_s^{\ast }}^{-\frac{2_s^{\ast }}{2}}\parallel (u_1,u_2){\parallel }^{2_s^{\ast }}-\frac{1}{q}{|\Omega |}^{\frac{2_s^{\ast }-q}{2_s^{\ast }}}{({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})}^{\frac{2-q}{2}}{\parallel (u_1,u_2)\parallel }^q\hfill \\ & =\frac{1}{2}\parallel (u_1,u_2){\parallel }^2-C_1\parallel (u_1,u_2){\parallel }^{2_s^{\ast }}-C_2{({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})}^{\frac{2-q}{2}}{\parallel (u_1,u_2)\parallel }^q,\hfill \end{array}$$

(36)

where $C_1=\frac{2}{2_s^{\ast }}{S}_{2_s^{\ast }}^{-\frac{2_s^{\ast }}{2}}$ and $C_2=\frac{1}{q}{|\Omega |}^{\frac{2_s^{\ast }-q}{2_s^{\ast }}}$. Assume that

$$\begin{array}{c}f\left(t\right)=\frac{1}{2}{t}^2-C_1{t}^{2_s^{\ast }}-C_2{({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})}^{\frac{2-q}{2}}{t}^q,t\ge 0,\end{array}$$

(37)

where $C_1$ and $C_2$ as above. Obviously,

$$\begin{array}{c}I(u_1,u_2)\ge h(\parallel (u_1,u_2)\parallel ).\end{array}$$

(38)

As $1<q<2<2_s^{\ast }$, we can have that $f\left(t\right)<0$ for t near 0, and $f\left(t\right)\to -\infty$ as $t\to \infty$. From the structure of $f\left(t\right)$, we know that there exist a small $R_0>0$ and

$$\begin{array}{c}\Lambda =C_2^{-\frac{2}{2-q}}{(\frac{1}{2}{R}_0^{2-q}-C_1{R}_0^{2_s^{\ast }-q})}^{\frac{2}{2-q}}\end{array}$$

(39)

such that $f_{\Lambda }\left(R_0\right)>0$ for $0<{\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}}<\Lambda$. Let

$$R_1=max\{t\in (0,R_0):f_{\Lambda }\left(t\right)\le 0\},R_2=min\{t>R_0:f_{\Lambda }\left(t\right)\le 0\}.$$

Hence, we have $0<R_1<R_2$ and $f\left(R_1\right)=f\left(R_2\right)=0$. It can be seen that there must be a $\Lambda >0$, for $0<{\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}}<\Lambda$, we obtain

$\left(i\right)$

$f\left(t\right)$ has a positive maximum;

$\left(ii\right)$

$\delta \ge 0$, where $\delta =\frac{2s}{n}{(\frac{S_{2_s^{\ast }}}{2})}^{\frac{n}{2s}}-C^{\ast }({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})$ is given in Lemma 5, $C^{\ast }$ is given by Lemma 4 and $S_{2_s^{\ast }}$ is given by (25).

Clearly, if $0\le t\le R_1$, $f\left(t\right)\le 0$; if $R_1<t<R_2$, $f\left(t\right)>0$; if $t\ge R_2$, $f\left(t\right)\le 0$. Let $e\left(t\right)\in C^{\infty }$ be a non-increasing function such that $0\le e\left(t\right)\le 1$ for all $t\in {\mathbb{R}}_0^{+}$. Define $e\left(t\right)=1$, if $0\le t\le R_1$; $e\left(t\right)=0$, if $t\ge R_2$. Set

$$g(u_1,u_2)=e(\parallel (u_1,u_2)\parallel ).$$

Considering the same idea as in [39,40], we define the truncation functional

$$J(u_1,u_2)=\frac{1}{2}\parallel (u_1,u_2){\parallel }^2-\frac{1}{q}{\int }_{\Omega }({\lambda }_1|u_1{|}^q+{\lambda }_2|u_2{|}^q)dx-\frac{2}{2_s^{\ast }}{\int }_{\Omega }|u_1{|}^{{\alpha }_1}{|u_2|}^{{\beta }_1}g(u_1,u_2)dx.$$

(40)

Let

$$\begin{array}{c}\tilde{f}\left(t\right)=\frac{1}{2}{t}^2-c_1{t}^{2_s^{\ast }}e\left(t\right)-c_2{({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})}^{\frac{2-q}{2}}{t}^q,t\ge 0.\end{array}$$

(41)

As $0\le e\left(t\right)\le 1$, we can see that $\tilde{f}\left(t\right)\ge f\left(t\right)$ for all $t\ge 0$. In the same way as (36), we have

$$\begin{array}{c}\hfill J(u_1,u_2)\ge \frac{1}{2}\parallel (u_1,u_2){\parallel }^2-c_{1}e(\parallel (u_1,u_2)\parallel )\parallel (u_1,u_2){\parallel }^{2_s^{\ast }}-c_2{({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})}^{\frac{2-q}{2}}{\parallel (u_1,u_2)\parallel }^q.\end{array}$$

(42)

It is easy to see that

$$\begin{array}{c}J(u_1,u_2)\ge \tilde{f}(\parallel (u_1,u_2)\parallel ).\end{array}$$

(43)

By the definitions of $f\left(t\right)$, $\tilde{f}\left(t\right)$, and $e\left(t\right)$, we find

$$\tilde{f}\left(t\right)=f\left(t\right),if0\le t\le R_1,$$

$$\tilde{f}\left(t\right)\ge f\left(t\right)\ge 0,if{R}_1\le t\le R_2,$$

and if $t>R_2$, we have

$$\begin{array}{cc}\hfill \tilde{f}\left(t\right)& =\frac{1}{2}{t}^2-c_2{({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})}^{\frac{2-q}{2}}{t}^q\hfill \\ & =t^q\left(\frac{1}{2}{t}^{2-q}-c_2{({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})}^{\frac{2-q}{2}}\right).\hfill \end{array}$$

When $0<{\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}}<\Lambda$, we have $\tilde{f}\left(t\right)\ge 0$. Hence,

$$\begin{array}{c}\tilde{f}\left(t\right)\ge 0\mathrm{for}t>R_1.\end{array}$$

(44)

To show our main result, we give two lemmas, which will be used in the next proof.

Lemma 6. 

There exists a constant $\Lambda >0$, for any $0<{\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}}<\Lambda$, we have

$\left(i\right)$

$J(u_1,u_2)\in C^1(W,\mathbb{R})$;

$\left(ii\right)$

If $J(u_1,u_2)<0$, then $\parallel (u_1,u_2)\parallel <R_1$ and $J(\varphi ,\phi )=I(\varphi ,\phi )$ for all $(\varphi ,\phi )$ is in a small neighborhood of $(u_1,u_2)\in W$.

$\left(iii\right)$

$J(u_1,u_2)$ satisfies the (PS) condition on W.

Proof. 

$\left(i\right)$

According to the definition of $e\left(t\right)$, we can see that $e\left(t\right)\equiv 1$ for $(u_1,u_2)$ near $(0,0)$; hence, $J(u_1,u_2)\in C^1(W,\mathbb{R})$ holds;

$\left(ii\right)$

In contradiction, we suppose that $J(u_1,u_2)<0$ and $\parallel (u_1,u_2)\parallel \ge R_1$. From (43) and (44), we know that $J(u_1,u_2)\ge \tilde{f}(\parallel (u_1,u_2)\parallel )$ and $\tilde{f}\left(t\right)\ge 0$, $fort\ge R_1$. We have $J(u_1,u_2)\ge 0$, which contradicts $J(u_1,u_2)<0$, so $\parallel (u_1,u_2)\parallel <R_1$.

$\left(iii\right)$

Let $\left\{({u_1}_n,{u_2}_n)\right\}\subset W$ be a (PS) sequence of $J(u_1,u_2)$. If $c<0$, we can assume that $J({u_1}_n,{u_2}_n)<0$, $J^{\prime }({u_1}_n,{u_2}_n)\to 0$ as $n\to \infty$. With the result of $\left(ii\right)$, that is $J(u_1,u_2)<0$ and $\parallel (u_1,u_2)\parallel <R_1$, we obtain $I({u_1}_n,{u_2}_n)=J({u_1}_n,{u_2}_n)$ and $I^{\prime }({u_1}_n,{u_2}_n)=J^{\prime }({u_1}_n,{u_2}_n)$. Thus, by Lemma 5, we deduce that $J(u_1,u_2)$ satisfies the (PS) condition for $c<0$.

□

Lemma 7. 

Given $n\in \mathbb{N}$, there exists a $\xi =\xi \left(n\right)>0$, such that

$$\gamma \left(\{(u_1,u_2)\in W:J(u_1,u_2)\le -\xi \}\right)\ge n,$$

γ denotes the genus of the set.

Proof. 

Fix $n\in \mathbb{N}$, set $W_n\subset W$ is a $n$–dimensional subspace, we know that $W_n$ is a finite space. Let $(u_1,u_2)\in W_n$ with $\parallel (u_1,u_2)\parallel =1$, for $0<t<R_1$, we have

$$\begin{array}{c}I(t{u}_1,t{u}_2)=J(t{u}_1,t{u}_2)=\frac{1}{2}{t}^2-\frac{t^q}{q}{\int }_{\Omega }({\lambda }_1|u_1{|}^q+{\lambda }_2|u_2{|}^q)dx-\frac{2t^{2_s^{\ast }}}{2_s^{\ast }}{\int }_{\Omega }|u_1{|}^{{\alpha }_1}{|u_2|}^{{\beta }_1}g(u_1,u_2)dx.\end{array}$$

Define

$${{\beta }_1}_n=inf\{{\int }_{\Omega }|u_1{|}^{q}dx|(u_1,u_2)\in W_n,\parallel (u_1,u_2)\parallel =1\}>0.$$

It is clear that all the norms are equivalent in the finite space, we find

$$J(t{u}_1,t{u}_2)\le \frac{1}{2}{t}^2-\frac{1}{q}{{\beta }_1}_{n}min({\lambda }_1,{\lambda }_2)t^q.$$

For any $\xi >0$ and $0<t<R_1$ such that $J(t{u}_1,t{u}_2)\le -\xi$ for $(u_1,u_2)\in W_n$ with $\parallel (u_1,u_2)\parallel =1$. Denote

$$S_t=\{(u_1,u_2)\in Wand\parallel (u_1,u_2)\parallel =t\},$$

then $\gamma (S_t\cap W_n)=n$ and

$$S_t\cap W_n\subset \{(u_1,u_2)\in W:J(u_1,u_2)\le -\xi \}.$$

Using the monotonicity property of genus, the result

$$\gamma \left(\{(u_1,u_2)\in W:J(u_1,u_2)\le -\xi \}\right)\ge \gamma (S_t\cap W_n)=n$$

holds. □

Now, let us give the proof of Theorem 1.

Proof of Theorem 1. 

Let

$${\omega }_n=\{B|B\in W\setminus \left\{(0,0)\right\},B\mathrm{is}\mathrm{closed}\mathrm{and}\mathrm{symmetric},\gamma \left(B\right)\ge n\},$$

where $\gamma \left(B\right)$ denotes the genus of set B. We also define

$$c_n=\underset{B\in {\omega }_n,(u_1,u_2)\in B}{inf}supJ(u_1,u_2),$$

and

$${\Gamma }_c=\{(u_1,u_2)\in W|J(u_1,u_2)=c,J^{\prime }(u_1,u_2)=0\}.$$

Assume that $0<{\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}}<\Lambda$, where $\Lambda$ is given as above, we show that if $n,k\in \mathbb{N}$ such that $c=c_n=c_{n+1}=c_{n+2}=\cdots =c_{n+k}$, then $\gamma \left({\Gamma }_c\right)\ge k+1$.

Next, we provide the details. Let

$$J^{-\xi }=\{(u_1,u_2)\in W:J(u_1,u_2)\le -\xi \}.$$

By Lemma 7, we obtain that there exists a $\xi \left(n\right)>0$, such that $\gamma \left(J^{-\xi }\right)\ge n$. As $J(u_1,u_2)$ is continuous and even, $J^{-\xi }\in {\omega }_n$ and $c_n\le \xi \left(n\right)<0$.

In addition, for $\parallel (u_1,u_2)\parallel \ge R_2$, by the Hölder inequality, we have

$$\begin{array}{cc}\hfill J(u_1,u_2)& =\frac{1}{2}\parallel (u_1,u_2){\parallel }^2-\frac{1}{q}{\int }_{\Omega }({\lambda }_1|u_1{|}^q+{\lambda }_2|u_2{|}^q)dx\hfill \\ & \ge \frac{1}{2}\parallel (u_1,u_2){\parallel }^2-\frac{1}{q}{|\Omega |}^{\frac{2_s^{\ast }-q}{2_s^{\ast }}}{({\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}})}^{\frac{2-q}{2}}{\parallel (u_1,u_2)\parallel }^q.\hfill \end{array}$$

As $q<2$, it is easy to see $J(u_1,u_2)$ is coercive and bounded from below. We know that $c_n>-\infty$ for all $n\in \mathbb{N}$.

We may assume that $c=c_n=c_{n+1}=\cdots =c_{n+k}<0$. According to Lemma 6, function $J(u_1,u_2)$ satisfies the (PS) condition, and from the definition of ${\Gamma }_c$, we see that ${\Gamma }_c$ is a compact set.

Through contradiction, if $\gamma ({\Gamma }_c\le k)$; then, there is a set U with ${\Gamma }_c\subset U$, where U is a closed and symmetric set. As the genus has a continuity property (see the details in [41]), we have $\gamma \left(U\right)\le k$. Considering that $c<0$, we can let

$$U\subset J^0=\{(u_1,u_2)\in W:J(u_1,u_2)\le 0\}.$$

According to [42], for some $0<\tau <-c$, there is an odd homeomorphism $h:W\to W$, such that $h(J^{c+n}-U)\subset J^{c-\tau }$. In fact $c=c_{n+k}=\underset{B\in {\omega }_n}{inf}\underset{(u_1,u_2)\in B}{sup}J(u_1,u_2)$, there exists a set $B\in {\omega }_n$, such that $\underset{(u_1,u_2)\in B}{sup}J(u_1,u_2)<c+\tau$, that is

$$h(B-U)\subset h(J^{c+\tau }-U)\subset J^{c-\tau },$$

we have

$$\begin{array}{c}\underset{(u_1,u_2)\in h(B-U)}{sup}J(u_1,u_2)\le c-\tau .\end{array}$$

(45)

By $\gamma \left(U\right)\le k$, we have

$$\begin{array}{c}\gamma (h(\overline{(B-U)}))\ge \gamma (\overline{(B-U)}\ge \gamma \left(B\right)-\gamma \left(U\right)\ge n.\end{array}$$

(46)

From this, we can obtain

$$h(\overline{(B-U)})\in {\omega }_n\mathrm{and}\underset{(u_1,u_2)\in h(B-U)}{sup}J(u_1,u_2)\ge c_n=c.$$

That is a contradiction of (45); we have showed that $\gamma \left({\Gamma }_c\right)\ge k+1$. Naturally, if for all $n\in N$, we have ${\omega }_{n+1}\subset {\omega }_n$ and $c_n\le c_{n+1}\le 0$. If all $c_n$ are distinct, there must be a result that $\gamma \left({\Gamma }_c\right)\ge 1$, and each element of the sequence $\left\{c_n\right\}$ is a negative critical value of $J(u_1,u_2)$.

If for some $n_0$, there exists a $k\ge 1$, such that

$$c=c_{n_0}=c_{n_0+1}=\cdots =c_{n_0+k},$$

and as above, we know $\gamma \left({\Gamma }_c\right)\ge k+1$, this tells us that infinitely many distinct elements are contained in ${\Gamma }_c$. This means that J has infinitely many distinct critical values.

From Lemma 6, we know that $I(u_1,u_2)=J(u_1,u_2)$ if $J(u_1,u_2)<0$. It is easy to see that the above critical values of J are also the critical values of I. We have proven that problem (1) has infinitely distinct weak solutions. We finish the proof. □

5. Conclusions

By using Lusternik–Schnirelmann’s theory, let $\Omega \in {\mathbb{R}}^n$ be an open bounded set, under the conditions $1<q<2<\frac{n}{s}$, $s\in (0,1)$, ${\alpha }_1+{\beta }_1=\frac{2n}{n-2s}=2_s^{\ast }$, we find a positive number $\Lambda$, when $0<{\lambda }_1^{\frac{2}{2-q}}+{\lambda }_2^{\frac{2}{2-q}}<\Lambda$, for the fractional magnetic system involving critical concave–convex nonlinearities with Laplace operators, the existence result of infinitely many solutions is obtained.