Concentrating Solutions for Fractional Schrödinger–Poisson Systems with Critical Growth

Abstract

In this paper, we consider a class of fractional Schrödinger–Poisson systems ${(-\mathsf{\Delta })}^{s}u+\lambda V(x)u+\varphi u=f(u)+{|u|}^{2_s^{*}-2}u$ and ${(-\mathsf{\Delta })}^t\varphi =u^2$ in ${\mathbb{R}}^3$, where $s,t\in (0,1)$ with $2s+2t>3$, $\lambda >0$ denotes a parameter, $V:{\mathbb{R}}^3\to \mathbb{R}$ admits a potential well $\mathsf{\Omega }\triangleq \mathrm{int}{V}^{-1}(0)$ and $2_s^{*}\triangleq {\displaystyle \frac{6}{3-2s}}$ is the fractional Sobolev critical exponent. Given some reasonable assumptions as to the potential V and the nonlinearity f, with the help of a constrained manifold argument, we conclude the existence of positive ground state solutions for some sufficiently large $\lambda$. Upon relaxing the restrictions on V and f, we utilize the minimax technique to show that the system has a positive mountain-pass type by introducing some analytic tricks. Moreover, we investigate the asymptotical behavior of the obtained solutions when $\lambda \to +\infty$.

1. Introduction

1.1. Overview

In this article, we investigate the existence and concentration of nontrivial solutions for the following fractional Schrödinger–Poisson system with critical growth

$$\left\{\begin{array}{cc}{(-\mathsf{\Delta })}^{s}u+\lambda V(x)u+\varphi u=f(u)+{|u|}^{2_s^{*}-2}u,\hfill & x\in {\mathbb{R}}^3,\hfill \\ {(-\mathsf{\Delta })}^t\varphi =u^2,\hfill & x\in {\mathbb{R}}^3,\hfill \end{array}\right.$$

(1)

where $s,t\in (0,1)$ with $2s+2t>3$, $\lambda >0$ denotes a parameter, $V:{\mathbb{R}}^3\to \mathbb{R}$ admits a potential well $\mathsf{\Omega }\triangleq \mathrm{int}{V}^{-1}(0)$, and $2_s^{*}\triangleq {\displaystyle \frac{6}{3-2s}}$ is the fractional Sobolev critical exponent. The fractional Laplacian ${(-\mathsf{\Delta })}^s$ is a nonlocal pseudo-differential operator which is defined by

$${(-\mathsf{\Delta })}^{s}u(x)=C_s\mathrm{P}.\mathrm{V}.{\mathsf{\int }}_{{\mathbb{R}}^3}{\displaystyle \frac{u(x)-u(y)}{{|x-y|}^{3+2s}}}dy=C_s\underset{\epsilon \to 0^{+}}{lim}{\mathsf{\int }}_{{\mathbb{R}}^3\setminus B_{\epsilon }(0)}{\displaystyle \frac{u(x)-u(y)}{{|x-y|}^{3+2s}}}dy$$

where $C_s$ is a normalization constant and P.V. is the Cauchy principal value. For the potential V, we will first make the following assumptions

(V₁) 

$V\in \mathcal{C}({\mathbb{R}}^3,\mathbb{R})$ with $V\ge 0$ on ${\mathbb{R}}^3$;

(V₂) 

There is a constant $c>0$ such that the set $\mathsf{\Sigma }\triangleq \left\{x\in {\mathbb{R}}^3:V(x)<c\right\}$ has a positive finite Lebesgue measure;

(V₃) 

$\mathsf{\Omega }=\mathrm{int}{V}^{-1}(0)$ is nonempty with a smooth boundary with $\overline{\mathsf{\Omega }}=V^{-1}(0)$, $V^{-1}(0)\triangleq \{x:V(x)=0\}$.

In celebrated papers, Bartsch and his collaborators initially proposed the above hypotheses to study the nonlinear Schrödinger equations; see [1,2]. As is generally known, the harmonic trapping potential

$$V(x)=\left\{\begin{array}{cc}{\omega }_1|x_1{|}^2+{\omega }_2|x_2{|}^2+{\omega }_3{|x_3|}^2-\omega ,\hfill & \mathrm{if}|(\sqrt{{\omega }_1}{x}_1,\sqrt{{\omega }_2}{x}_2,\sqrt{{\omega }_3}{x}_3){|}^2\ge \omega ,\hfill \\ 0,\hfill & \mathrm{if}|(\sqrt{{\omega }_1}{x}_1,\sqrt{{\omega }_2}{x}_2,\sqrt{{\omega }_3}{x}_3){|}^2\le \omega ,\hfill \end{array}\right.$$

with $\omega >0$ satisfying $(V_1)$–$(V_3)$, where ${\omega }_i>0$ is called the anisotropy factor of the trap in quantum physics and the trapping frequency of the ith-direction in mathematics; see, e.g., [3,4,5]. Indeed, the potential $\lambda V$, instead of V, given assumptions $(V_1)$–$(V_3)$ can be read as a steep potential.

Over the past several decades, considerable attention has been paid to the standing, or solitary, wave solutions of Schrödinger–Poisson systems of the type

$$\left\{\begin{array}{cc}{\displaystyle i\frac{\partial \psi }{\partial t}=\mathsf{\Delta }\psi -W(x)\psi +\varphi \psi +\tilde{g}(|\psi |)\psi ,}\hfill & \mathrm{in}{\mathbb{R}}^{+}\times {\mathbb{R}}^3,\hfill \\ {\displaystyle -\mathsf{\Delta }\varphi ={|\psi |}^2,}\hfill & \mathrm{in}{\mathbb{R}}^3,\hfill \end{array}\right.$$

(2)

where $\psi :{\mathbb{R}}^3\times \mathbb{R}\to \mathbb{C}$ is the time-dependent wave function, $W:{\mathbb{R}}^3\to \mathbb{R}$ stands for the real external potential, $\varphi$ represents an internal potential for a nonlocal self-interaction of wave function, and nonlinear term $g(\psi )\triangleq \tilde{g}(|\psi |)\psi$ describes the interaction effect among particles. By inserting the standing wave ansatz $\psi (x,t)=exp(-i\omega t)u(x)$ with $\omega \in \mathbb{R}$ and $x\in {\mathbb{R}}^3$ into (2), then $u:{\mathbb{R}}^3\to \mathbb{R}$ satisfies the Schrödinger–Poisson system

$$\left\{\begin{array}{cc}{\displaystyle -\mathsf{\Delta }u+\overline{W}(x)u+\varphi u=g(u),}\hfill & \mathrm{in}{\mathbb{R}}^3,\hfill \\ {\displaystyle -\mathsf{\Delta }\varphi =u^2,}\hfill & \mathrm{in}{\mathbb{R}}^3,\hfill \end{array}\right.$$

(3)

where and in the sequel $\overline{W}(x)=W(x)+\omega$ for all $x\in {\mathbb{R}}^3$. We refer the interested readers to [6,7] and the references therein for more about the physical background of (2). There are many interesting works about the existence of positive solutions, positive ground states, multiple solutions, sign-changing solutions and semiclassical states to system (3), see, e.g., [8,9,10,11,12,13,14,15] and references therein.

In [16], Jiang and Zhou first applied the steep potential well to the Schrödinger–Poisson system and proved the existence of nontrivial solutions and ground state solutions. Subsequently, by using the linking theorem [17,18], the authors in [19] considered the existence and concentration of nontrivial solutions for the following Schrödinger–Poisson system

$$\left\{\begin{array}{cc}-\mathsf{\Delta }u+\lambda V(x)u+K(x)\varphi u={|u|}^{p-2}u,\hfill & x\in {\mathbb{R}}^3,\hfill \\ -\mathsf{\Delta }\varphi =K(x)u^2,\hfill & x\in {\mathbb{R}}^3,\hfill \end{array}\right.$$

(4)

under the following conditions:

$(\tilde{V})$

$V\in C({\mathbb{R}}^3,\mathbb{R})$ and V is bounded from below;

and $(V_2)$–$(V_3)$ with some suitable assumptions on $K:{\mathbb{R}}^3\to \mathbb{R}$ for $4\le p<6$. It is worth mentioning that in particular, they investigated the existence and concentration of nontrivial solutions to (4) by the monotonicity trick due to Jeanjean [20] under the conditions $(V_1)$–$(V_3)$, $K\in L_{\mathrm{loc}}^{\infty }({\mathbb{R}}^3)\cap L^2({\mathbb{R}}^3)$ and

$(\stackrel{-}{V})$

V is weakly differentiable such that $(x,\nabla V)\in L^{p_1}({\mathbb{R}}^3)$ for some $p_1\in [{\displaystyle \frac{3}{2}},\infty ]$, and

$$2V(x)+(x,\nabla V)\ge 0\mathrm{for}\mathrm{a}.\mathrm{e}.x\in {\mathbb{R}}^3,$$

where $(·,·)$ is the usual inner product in ${\mathbb{R}}^3$.

$(\stackrel{-}{K})$

K is weakly differentiable such that $(x,\nabla K)\in L^{p_2}({\mathbb{R}}^3)$ for some $p_2\in [2,\infty ]$, and

$${\displaystyle \frac{2(p-3)}{p}}K(x)+(x,\nabla K)\ge 0\mathrm{for}\mathrm{a}.\mathrm{e}.x\in {\mathbb{R}}^3.$$

Whereas, the related research on fractional Schrödinger–Poisson systems like (1) are not as rich as the classic Schrödinger–Poisson system (3). Actually, we shall reach the system (1) by supposing $s=t=1$ and $K(x)\equiv 1$ for each $x\in {\mathbb{R}}^3$ in the system (4). As a consequence, one of the aims in this paper is to generalize the corresponding results obtained in [19] to the fractional case, which makes the studies interesting.

When it comes to the fractional-order operators, the following fractional Schrödinger equation

$${(-\mathsf{\Delta })}^{\alpha }u+V(x)u=f(x,u),x\in {\mathbb{R}}^N,$$

(5)

is usually used to study the standing wave solutions $\psi (x,t)=u(x)e^{-i\omega t}$ for the equation

$$i\hslash {\displaystyle \frac{\partial \psi }{\partial t}}={\hslash }^2{(-\mathsf{\Delta })}^{\alpha }\psi +W(x)\psi -f(x,\psi ),x\in {\mathbb{R}}^N,$$

where ℏ is the Planck’s constant, $W:{\mathbb{R}}^N\to \mathbb{R}$ is an external potential and f is a suitable nonlinearity. Because the fractional Schrödinger equation appears in problems involving nonlinear optics, plasma physics and condensed matter physics, it is one of the main objects of the fractional quantum mechanic. Equation (5) has been firstly proposed by Laskin [21,22] as a result of expanding the Feynman path integral from the Brownian-like to the Lévy-like quantum mechanical paths. In [23], Caffarelli and Silvestre transformed the nonlocal operator ${(-\mathsf{\Delta })}^{\alpha }$ to a Dirichlet–Neumann boundary value problem for a certain elliptic problem with local differential operators defined on the upper half space. This technique is a powerful tool to deal with the equations involving fractional operators in the respects of regularity and variational methods; please see [10,24] and their references for example. When the conditions $(V_1)$–$(V_3)$ are satisfied, Yang and Liu [25] established the multiplicity and concentration of solutions for the following fractional Schrödinger equation

$${(-\mathsf{\Delta })}^{\alpha }u+\lambda V(x)u=f(x,u)+g(x){|u|}^{v-2}u,x\in {\mathbb{R}}^N,$$

involving a k-order asymptotically linear term $f(x,u)$, where $s\in (0,1)$, $2s<N$, $1\le k<2_s^{*}-1={\displaystyle \frac{N+2s}{N-2s}}$ and $g\in L^{{\displaystyle \frac{v}{2-v}}}({\mathbb{R}}^N)$ with $1<v<2$. There exist some other meaningful results in [26,27] and their references on fractional Schrödinger equations.

Recently, Teng [28] contemplated the existence of ground state solutions to the following fractional Schrödinger–Poisson system

$$\left\{\begin{array}{cc}{(-\mathsf{\Delta })}^{s}u+V(x)u+\varphi u={|u|}^{p-2}u+\mu {|u|}^{2_s^{*}-2}u,\hfill & x\in {\mathbb{R}}^3,\hfill \\ {(-\mathsf{\Delta })}^t\varphi =u^2,\hfill & x\in {\mathbb{R}}^3,\hfill \end{array}\right.$$

where the potential $V:{\mathbb{R}}^3\to {\mathbb{R}}^{+}$ satisfies some technical assumptions, $\mu =1$ and $2<p<2_s^{*}$. Later on, Shen and Yao [29] improved the corresponding results for the case $\mu =0$. In the meanwhile, the authors in [30] disposed of the semiclassic ground state for the following fractional Schrödinger–Poisson system

$$\left\{\begin{array}{cc}{\epsilon }^{2s}{(-\mathsf{\Delta })}^{s}u+V(x)u+\varphi u=f(u)+{|u|}^{2_s^{*}-2}u,\hfill & x\in {\mathbb{R}}^3,\hfill \\ {\epsilon }^{2t}{(-\mathsf{\Delta })}^t\varphi =u^2,\hfill & x\in {\mathbb{R}}^3.\hfill \end{array}\right.$$

Other meaningful results of the fractional Schrödinger–Poisson system could be found in [28,30,31,32,33,34] and their references therein.

1.2. Main Results

Motivated by all the works above, particularly by [32], we shall focus on the existence and concentration results for (1) with steep potential well. Because we are interested in positive solutions, without loss of generality, we assume that $f\in {\mathcal{C}}^0(\mathbb{R},\mathbb{R})$ vanishes in $(-\infty ,0)$ and satisfies the following conditions

(f₁) 

$f\in {\mathcal{C}}^0(\mathbb{R},{\mathbb{R}}^{+})$ and $f(z)=o(z)$ as $z\to 0$, where ${\mathbb{R}}^{+}=[0,+\infty )$;

(f₂) 

$|f(z)|\le C_0{(1+|z|}^{q-1})$ for some constants $C_0>0$ and $2<q<2_s^{*}$;

(f₃) 

There are some $p\in \left({\displaystyle \frac{4s+2t}{s+t}},2_s^{*}\right)$, $\widehat{\mu }>0$ and ${\mu }_0>0$ such that $F(z)\ge \widehat{\mu }{z}^p-{\mu }_0{z}^2$ for all $z\in {\mathbb{R}}^{+}$;

(f₄) 

There is a $\gamma >{\displaystyle \frac{4s+2t}{s+t}}$ such that $zf(z)-\gamma F(z)\ge 0$ for all $z\in {\mathbb{R}}^{+}$, where $F(z)={\mathsf{\int }}_0^{z}f(s)ds$;

(f₅) 

The map $z\mapsto {\displaystyle \frac{(s+t)f(z)z-3F(z)}{z^{\frac{4s+2t}{s+t}}}}$ is nondecreasing on $z\in (0,+\infty )$.

Our first main result can be stated as follows.

Theorem 1.

Let $s,t\in (0,1)$ satisfy $2s+2t>3$. Suppose that $(V_1)$–$(V_3)$ and $(f_1)$–$(f_5)$ as well as the following conditions hold

(V₄) 

V is weakly differentiable and satisfies the inequality below

$$(s+t)(\gamma -2)V(x)+(x,\nabla V)\ge 0;$$

(V₅) 

The map $\theta \mapsto {\theta }^{2s}[(2s+2t-3)V(\theta x)-(\nabla V(\theta x),\theta x)]$ is nondecreasing on $\theta \in (0,+\infty )$ and $(2s+2t-3)V(x)\ge 2(\nabla V,x)\ge 0$ for all $x\in {\mathbb{R}}^3$.

If one of the following assumptions on p and $\widehat{\mu }$ appearing in $(f_3)$ holds true

$$\left\{\begin{array}{c}{\displaystyle (\mathrm{I}):s>\frac{3}{4},\frac{4s}{3-2s}<p<2_s^{*}\mathrm{and}\mathrm{for}\mathrm{all}\widehat{\mu }>0;}\hfill \\ {\displaystyle (\mathrm{II}):s>\frac{3}{4},\frac{4s+2t}{s+t}<p\le \frac{4s}{3-2s}\mathrm{and}\mathrm{for}\mathrm{all}\mathrm{sufficiently}\mathrm{large}\widehat{\mu }>0;}\hfill \\ {\displaystyle (\mathrm{III}):\frac{1}{2}<s\le \frac{3}{4},\frac{4s+2t}{s+t}<p<2_s^{*}\mathrm{and}\mathrm{for}\mathrm{all}\widehat{\mu }>0,}\hfill \end{array}\right.$$

(6)

then there exists a $\mathsf{\Lambda }>0$ such that the system (1) admits at least one positive ground state solution (namely, it has the minimum energy among the set ${\mathcal{M}}_{\lambda }$ defined in (19) below) for all $\lambda >\mathsf{\Lambda }$.

Remark 1.

There exist many functions f that satisfy the assumptions $(f_1)$–$(f_5)$ above, for example, $f(z)={|z|}^{\gamma -2}z$ for all $z\in {\mathbb{R}}^{+}$ and $f(z)=0$ for all $z<0$. Obviously, it would occur that $\gamma <4$ which results in some unpleasant difficulties. As to the potential V, without loss of generality, we are indeed assuming that it is of class ${\mathcal{C}}^1$ at almost everywhere at the point in ${\mathbb{R}}^3$ and provide an example as follows

$$V(x)=\left\{\begin{array}{cc}0,\hfill & \mathrm{if}|x|\le 1,\hfill \\ {|x|}^{{\displaystyle \frac{2s+2t-3}{2}}},\hfill & \mathrm{if}|x|>1.\hfill \end{array}\right.$$

The reader is invited to infer that the restriction (6) is just used to restore the compactness. Moreover, we prefer to believe that the example on V above is not sharp, but it reveals that the existence result in Theorem 1 seems reasonable.

Inspired by the results in [1,19], we obtain the following concentration result:

Theorem 2.

Let $(u_{\lambda },{\varphi }_{u_{\lambda }})\in H^s({\mathbb{R}}^3)\times D^{t,2}({\mathbb{R}}^3)$ be the ground state solution obtained by Theorem 1, then $u_{\lambda }\to u_0$ in $H^s({\mathbb{R}}^3)$ and ${\varphi }_{u_{\lambda }}\to {\varphi }_{u_0}$ in $D^{t,2}({\mathbb{R}}^3)$ along a subsequence as $\lambda \to +\infty$, where $u_0\in H_0^s(\mathsf{\Omega })$ is a ground state solution of

$$\left\{\begin{array}{cc}{\displaystyle {(-\mathsf{\Delta })}^{s}u+c_t\left({\mathsf{\int }}_{\mathsf{\Omega }}\frac{u^2(y)}{{|x-y|}^{3-2t}}dy\right)u=f(u)+{|u|}^{2_s^{*}-2}u,}\hfill & \mathrm{in}\mathsf{\Omega },\hfill \\ {\displaystyle u=0,}\hfill & \mathrm{on}\partial \mathsf{\Omega }.\hfill \end{array}\right.$$

(7)

Here, $c_t>0$ is a constant given by (15) below.

As pointed out in Remark 1, the assumptions on f and V required in Theorem 1 are somehow restrictive. It is natural to ask that whether the existence result remains true when $(f_5)$ and $(V_5)$ are absent. Thus, our next main result shows an affirmative answer.

Theorem 3.

Let $s,t\in (0,1)$ satisfy $2s+2t>3$. Suppose that $(V_1)$–$(V_4)$ and $(f_1)$–$(f_4)$ hold. If one of the assumptions in (6) holds true, then there exists a $\widehat{\mathsf{\Lambda }}>0$ such that the system (1) has at least one positive solution for all $\lambda >\widehat{\mathsf{\Lambda }}$.

Remark 2.

It is worth pointing out here that even if we only consider the case $s=t=1$ in Theorem 3, in contrast to ([19], Theorem 1.3), there are three main contributions:

(1) 

Firstly, the more general nonlinearity is dealt with and it needs some more careful calculations;

(2) 

Secondly, the critical term in the nonlinearity is involved and so we have to take some deep and delicate analysis to restore the compactness;

(3) 

Last but not the least, we do not assume a weight function K in the front of the Poisson term in (1). Actually, if we follow the arguments adopted in this quoted paper, the weight function K with $K\in L^{{\displaystyle \frac{6}{4s+2t-3}}}({\mathbb{R}}^3)$ seems indispensable. So, we can relax the constraint assumption in this direction.

Proceeding as the same way in Theorem 2, we can also derive the asymptotical behavior of solutions obtained in Theorem 3. More precisely, we shall demonstrate the following result whose detailed proof is omitted.

Theorem 4.

Let $(u_{\lambda },{\varphi }_{u_{\lambda }})\in H^s({\mathbb{R}}^3)\times D^{t,2}({\mathbb{R}}^3)$ denote the positive solution in Theorem 3, then $u_{\lambda }\to u_0$ in $H^s({\mathbb{R}}^3)$ and ${\varphi }_{u_{\lambda }}\to {\varphi }_{u_0}$ in $D^{t,2}({\mathbb{R}}^3)$ along a subsequence as $\lambda \to +\infty$, where $u_0\in H_0^s(\mathsf{\Omega })$ is a positive solution of (7).

As far as we are concerned, the main results in this article seem new by now. Alternatively, it should be mentioned that this paper could be regarded as a continuation of our latest work in [35], where the existence and concentrating results of a planar Schrödinger–Poisson equation with steep potential well were established. Here, there are two essential differences. On the one hand, due to the different geometry structures of the two variational functionals, we must take advantage of sone new techniques to restore the compactness. On the other hand, since we consider the existence of ground state solutions in Theorem 1, a suitable constraint minimization argument will be used instead of depending on the mountain-pass theorem in [35]. Finally, when the critical term ${|u|}^{2_s^{*}-2}u$ in the system (1) disappears, one may be curious about the case that the potential is strongly indefinite according to [36]. Of course, we are also working hard in this direction, and it would be contemplated in our further studies.

The paper is organized as follows. In Section 2, we mainly introduce some preliminary results. In Section 3 and Section 4, we show some crucial lemmas and exhibit the detailed proofs of Theorems 1, 2 and 3, respectively.

Notations: From now on in this paper, unless otherwise mentioned, we ultilize the following notations:

• $C,C_1,C_2,\cdots$ denote any positive constant, whose value is not relevant and ${\mathbb{R}}^{+}\triangleq (0,+\infty )$.
• Let $(Z,\parallel ·{\parallel }_Z)$ be a Banach space with dual space $(Z^{-1},\parallel ·{\parallel }_{Z^{-1}})$ and $\mathsf{\Phi }$ be functional on Z.
• The (PS) sequence at a level $c\in \mathbb{R}$ (${(PS)}_c$ sequence in short) corresponding to $\mathsf{\Phi }$ means that $\mathsf{\Phi }(x_n)\to c$ and ${\mathsf{\Phi }}^{\prime }(x_n)\to 0$ in $Z^{-1}$ as $n\to \infty$, where $\left\{x_n\right\}\subset Z$.
• ${||}_p$ stands for the usual norm of the Lebesgue space $L^p({\mathbb{R}}^N)$ for all $p\in [1,+\infty ]$, and ${\parallel \parallel }_{H^{\alpha }({\mathbb{R}}^N)}$ denotes the usual norm of the Sobolev space $H^{\alpha }({\mathbb{R}}^N)$ for $\alpha \in (0,1)$.
• For any $\varrho >0$ and every $x\in {\mathbb{R}}^3$, $B_{\varrho }(x)\triangleq \{y\in {\mathbb{R}}^3:|y-x|<\varrho \}$.
• $o_n(1)$ denotes the real sequences with $o_n(1)\to 0$ as $n\to +\infty$.
• $“\to ”$ and $“\rightharpoonup ”$ stand for the strong and weak convergence in the related function spaces, respectively;

2. Preliminary Stuff

2.1. Variational Setting

In this section, according to the explorations about the fractional Sobolev spaces in [37], we first bring in some necessary variational settings which permit us to treat the problems variationally. We denote the fractional Sobolev space $W^{\alpha ,p}({\mathbb{R}}^N)$ for any $p\in [1,+\infty )$ and $\alpha \in (0,1)$ by

$$W^{\alpha ,p}({\mathbb{R}}^N)=\left\{u\in L^p({\mathbb{R}}^N):{\mathsf{\int }}_{{\mathbb{R}}^N}{\mathsf{\int }}_{{\mathbb{R}}^N}{\displaystyle \frac{{|u(x)-u(y)|}^p}{{|x-y|}^{N+\alpha p}}}dxdy<+\infty \right\}$$

equipped with the natural norm

$${\parallel u\parallel }_{W^{\alpha ,p}({\mathbb{R}}^N)}={\left({\mathsf{\int }}_{{\mathbb{R}}^N}{\mathsf{\int }}_{{\mathbb{R}}^N}{\displaystyle \frac{{|u(x)-u(y)|}^p}{{|x-y|}^{N+\alpha p}}}dxdy+{\mathsf{\int }}_{{\mathbb{R}}^N}{|u|}^{p}dx\right)}^{{\displaystyle \frac{1}{p}}}.$$

In particular, the fractional Sobolev space $W^{\alpha ,2}({\mathbb{R}}^N)$ would be simply relabeled by $H^{\alpha }({\mathbb{R}}^N)$ if $p=2$. As a matter of fact, the Hilbert space $H^{\alpha }({\mathbb{R}}^N)$ can also be described by the Fourier transform, that is,

$$H^{\alpha }({\mathbb{R}}^N)=\left\{u\in L^2({\mathbb{R}}^N):{\mathsf{\int }}_{{\mathbb{R}}^N}{|\xi |}^{2\alpha }|\widehat{u}(\xi ){|}^2+{|\widehat{u}(\xi )|}^{2}d\xi <+\infty \right\},$$

where $\widehat{u}$ denotes the usual Fourier transform of u. When we take the definition of the fractional Sobolev space $H^{\alpha }({\mathbb{R}}^N)$ by the Fourier transform, the inner product and the norm for $H^{\alpha }({\mathbb{R}}^N)$ are defined as

$${(u,v)}_{H^{\alpha }({\mathbb{R}}^N)}={\mathsf{\int }}_{{\mathbb{R}}^N}{|\xi |}^{2\alpha }\widehat{u}(\xi )\widehat{v}(\xi )+\widehat{u}(\xi )\widehat{v}(\xi )d\xi ,\forall u,v\in H^{\alpha }({\mathbb{R}}^N).$$

and

$${\parallel u\parallel }_{H^{\alpha }({\mathbb{R}}^N)}={\left({\mathsf{\int }}_{{\mathbb{R}}^N}{|\xi |}^{2\alpha }|\widehat{u}{(\xi )|}^2+{|\widehat{u}(\xi )|}^{2}d\xi \right)}^{{\displaystyle \frac{1}{2}}},\forall u,v\in H^{\alpha }({\mathbb{R}}^N).$$

Thanks to the Plancherel’s theorem, we have ${|u|}_2={|\widehat{u}|}_2$ and ${|(-\mathsf{\Delta })}^{{\displaystyle \frac{\alpha }{2}}}{u|}_2={||\xi |}^{\alpha }\widehat{u}{|}_2$. Hence,

$${\parallel u\parallel }_{H^{\alpha }({\mathbb{R}}^N)}={\left({\mathsf{\int }}_{{\mathbb{R}}^N}{|(-\mathsf{\Delta })}^{{\displaystyle \frac{\alpha }{2}}}{u|}^2+{|u|}^{2}dx\right)}^{{\displaystyle \frac{1}{2}}},\forall u\in H^{\alpha }({\mathbb{R}}^N).$$

(8)

We can infer from ([37], Proposition 3.4 and Proposition 3.6) that

$${|(-\mathsf{\Delta })}^{{\displaystyle \frac{\alpha }{2}}}{u|}_2={\left({\mathsf{\int }}_{{\mathbb{R}}^N}{|\xi |}^{2\alpha }{|\widehat{u}(\xi )|}^{2}d\xi \right)}^{{\displaystyle \frac{1}{2}}}={\left({\displaystyle \frac{1}{C_N(\alpha )}}{\mathsf{\int }}_{{\mathbb{R}}^N}{\mathsf{\int }}_{{\mathbb{R}}^N}{\displaystyle \frac{{|u(x)-u(y)|}^2}{{|x-y|}^{N+2\alpha }}}dxdy\right)}^{{\displaystyle \frac{1}{2}}}.$$

showing that the norm in (8) makes sense for the fractional Sobolev space. Moreover, we introduce the homogeneous fractional Sobolev space $D^{\alpha ,2}({\mathbb{R}}^N)$ by

$$D^{\alpha ,2}({\mathbb{R}}^N)=\left\{u\in L^{2_{\alpha }^{*}}({\mathbb{R}}^N):{|\xi |}^{\alpha }\widehat{u}(\xi )\in L^{2_{\alpha }^{*}}({\mathbb{R}}^N)\right\}\mathrm{with}{2}_{\alpha }^{*}={\displaystyle \frac{2N}{N-2\alpha }}\mathrm{and}N\ge 3,$$

which is the completion of $C_0^{\infty }({\mathbb{R}}^N)$ under the norm

$${\parallel u\parallel }_{D^{\alpha ,2}({\mathbb{R}}^N)}={\left({\mathsf{\int }}_{{\mathbb{R}}^N}{\left|{(-\mathsf{\Delta })}^{{\displaystyle \frac{\alpha }{2}}}u\right|}^{2}dx\right)}^{{\displaystyle \frac{1}{2}}}={\left({\mathsf{\int }}_{{\mathbb{R}}^N}{|\xi |}^{2\alpha }{|\widehat{u}(\xi )|}^{2}d\xi \right)}^{{\displaystyle \frac{1}{2}}},\forall u\in D^{\alpha ,2}({\mathbb{R}}^N).$$

Taking into account the imbedding theorem $H^{\alpha }({\mathbb{R}}^N)\hookrightarrow L^r({\mathbb{R}}^N)$ for every $r\in [2,2_{\alpha }^{*})$, there exists a constant $C_r>0$ such that

$${\parallel u\parallel }_{H^{\alpha }({\mathbb{R}}^N)}\le C_r{|u|}_r,\forall u\in H^{\alpha }({\mathbb{R}}^N)\mathrm{and}2\le r<2_{\alpha }^{*}.$$

(9)

Also, there exists a best constant $S_{\alpha }>0$ (see, e.g., [38]) such that

$$S_{\alpha }=\underset{u\in D^{\alpha ,2}({\mathbb{R}}^N)\setminus \left\{0\right\}}{inf}{\displaystyle \frac{{\mathsf{\int }}_{{\mathbb{R}}^N}{|{(-\mathsf{\Delta })}^{\frac{\alpha }{2}}u|}^{2}dx}{{\left({\mathsf{\int }}_{{\mathbb{R}}^N}{|u|}^{2_{\alpha }^{*}}dx\right)}^{\frac{2}{2_{\alpha }^{*}}}}}.$$

(10)

Throughout this paper, for $s\in (0,1)$ and the dimension $N=3$, we define the space

$$E\triangleq \left\{u\in H^s({\mathbb{R}}^3):{\mathsf{\int }}_{{\mathbb{R}}^3}V(x)u^{2}dx<+\infty \right\}.$$

By using $(V_1)$, it is easy to verify that it is a Hilbert space equipped with the inner product and norm

$${(u,v)}_E={\mathsf{\int }}_{{\mathbb{R}}^3}{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}u{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}v+V(x)uvdx\mathrm{and}{\parallel u\parallel }_E={\left({\mathsf{\int }}_{{\mathbb{R}}^3}{|{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}u|}^2+V(x)u^{2}dx\right)}^{{\displaystyle \frac{1}{2}}}$$

for any $u,v\in E$. Particularly, one can deduce that the imbedding $E\hookrightarrow H^s({\mathbb{R}}^3)$ is continuous. Indeed, combining $(V_2)$ and (10), one has

$$\begin{array}{cc}\hfill {\mathsf{\int }}_{{\mathbb{R}}^3}{u}^{2}dx& ={\mathsf{\int }}_{{\mathbb{R}}^3\setminus \mathsf{\Sigma }}{u}^{2}dx+{\mathsf{\int }}_{\mathsf{\Sigma }}{u}^{2}dx\le {\displaystyle \frac{1}{c}}{\mathsf{\int }}_{{\mathbb{R}}^3\setminus \mathsf{\Sigma }}V(x)u^{2}dx+{|\mathsf{\Sigma }|}^{{\displaystyle \frac{2_s^{*}-2}{2_s^{*}}}}{\left({\mathsf{\int }}_{\mathsf{\Sigma }}{|u|}^{2_s^{*}}dx\right)}^{{\displaystyle \frac{2}{2_s^{*}}}}\hfill \\ \hfill & \le max\left\{{\displaystyle \frac{1}{c}},{|\mathsf{\Sigma }|}^{{\displaystyle \frac{2_s^{*}-2}{2_s^{*}}}}\right\}{\parallel u\parallel }_E^2,\hfill \end{array}$$

where $|\mathsf{\Sigma }|$ stands for the Lebesgue measure of a Lebesgue measurable set $\mathsf{\Sigma }\subset {\mathbb{R}}^3$. As a consequence of (9) and (10), there exists a constant $d_r>0$ such that

$${|u|}_r\le d_r{\parallel u\parallel }_E,\forall u\in E\mathrm{and}2\le r\le 2_s^{*}.$$

(11)

For any $\lambda >0$, define the Hilbert space $E_{\lambda }{\triangleq (E,\parallel ·\parallel }_{E_{\lambda }})$ with inner product and norm given by

$${(u,v)}_{E_{\lambda }}={\mathsf{\int }}_{{\mathbb{R}}^3}{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}u{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}v+\lambda V(x)uvdx\mathrm{and}{\parallel u\parallel }_{E_{\lambda }}={\left({\mathsf{\int }}_{{\mathbb{R}}^3}{|(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u|}^2+\lambda V(x){|u|}^{2}dx\right)}^{{\displaystyle \frac{1}{2}}}$$

for all $u,v\in E$. Obviously, if $\lambda \ge 1$, one sees ${\parallel u\parallel }_E\le {\parallel u\parallel }_{E_{\lambda }}$ for all $u\in E$. Using $(V_2)$ again,

$$\left\{\begin{array}{c}{\displaystyle {\mathsf{\int }}_{\mathsf{\Sigma }}{|u|}^{2}dx\le {|\mathsf{\Sigma }|}^{\frac{2_s^{*}-2}{2_s^{*}}}{|u|}_{2_s^{*}}^2\le {|\mathsf{\Sigma }|}^{\frac{2_s^{*}-2}{2_s^{*}}}{S}_s^{-1}{\parallel u\parallel }_{E_{\lambda }}^2,}\hfill \\ {\displaystyle {\mathsf{\int }}_{{\mathbb{R}}^3\setminus \mathsf{\Sigma }}{|u|}^{2}dx\le \frac{1}{\lambda c}{\mathsf{\int }}_{{\mathbb{R}}^3\setminus \mathsf{\Sigma }}{\lambda V(x)|u|}^{2}dx\le \frac{1}{\lambda c}{\mathsf{\int }}_{{\mathbb{R}}^3}{\lambda V(x)|u|}^{2}dx\le \frac{1}{\lambda c}{\parallel u\parallel }_{E_{\lambda }}^2.}\hfill \end{array}\right.$$

From which, for any $r\in [2,2_s^{*}]$, there holds

$$\begin{array}{cc}\hfill {\mathsf{\int }}_{{\mathbb{R}}^3}{|u|}^{r}dx& \le {\left({\mathsf{\int }}_{{\mathbb{R}}^3}{|u|}^{2}dx\right)}^{{\displaystyle \frac{2_s^{*}-r}{2_s^{*}-2}}}{\left({\mathsf{\int }}_{{\mathbb{R}}^3}{|u|}^{2_s^{*}}dx\right)}^{{\displaystyle \frac{r-2}{2_s^{*}-2}}}\hfill \\ \hfill & \le {\left(max\{S_s^{-1}{|\mathsf{\Sigma }|}^{{\displaystyle \frac{2_s^{*}-2}{2_s^{*}}}},{\displaystyle \frac{1}{\lambda c}}\}{\parallel u\parallel }_{E_{\lambda }}^2\right)}^{{\displaystyle \frac{2_s^{*}-r}{2_s^{*}-2}}}{\left(S_s^{-{\displaystyle \frac{2_s^{*}}{2}}}{\parallel u\parallel }_{E_{\lambda }}^{2_s^{*}}\right)}^{{\displaystyle \frac{r-2}{2_s^{*}-2}}}.\hfill \end{array}$$

Hence, for all $r\in [2,2_s^{*}]$, we reach

$${\mathsf{\int }}_{{\mathbb{R}}^3}{|u|}^{r}dx\le {|\mathsf{\Sigma }|}^{{\displaystyle \frac{2_s^{*}-r}{2_s^{*}}}}{S}_s^{-{\displaystyle \frac{r}{2}}}{\parallel u\parallel }_{E_{\lambda }}^r\mathrm{whenever}\lambda \ge c^{-1}{|\mathsf{\Sigma }|}^{-{\displaystyle \frac{2_s^{*}-2}{2_s^{*}}}}{S}_s.$$

(12)

When the work space $E_{\lambda }$ is built, we turn to find the variational structure of system (1). Following the classic Schrödinger–Poisson system, it can reduce to be a single equation. Actually, according to the Hölder’s inequality, for every $u\in H^s({\mathbb{R}}^3)$ and $v\in D^{t,2}({\mathbb{R}}^3)$, one has

$$\begin{array}{cc}\hfill {\mathsf{\int }}_{{\mathbb{R}}^3}{u}^{2}vdx& \le {\left({\mathsf{\int }}_{{\mathbb{R}}^3}{|u|}^{{\displaystyle \frac{12}{3+2t}}}dx\right)}^{{\displaystyle \frac{3+2t}{6}}}{\left({\mathsf{\int }}_{{\mathbb{R}}^3}{|v|}^{{\displaystyle \frac{6}{3-2t}}}dx\right)}^{{\displaystyle \frac{3-2t}{6}}}\hfill \\ \hfill & \le S_t^{-{\displaystyle \frac{1}{2}}}{\parallel u\parallel }_{H^s({\mathbb{R}}^3)}^2{\parallel v\parallel }_{D^{t,2}({\mathbb{R}}^3)}\le {C\parallel u\parallel }_{H^s({\mathbb{R}}^3)}^2{\parallel v\parallel }_{D^{t,2}({\mathbb{R}}^3)},\hfill \end{array}$$

(13)

where we have used the continuous imbedding $H^s({\mathbb{R}}^3)\hookrightarrow L^{{\displaystyle \frac{12}{3+2t}}}({\mathbb{R}}^3)$ since $4s+2t>3$ and $t\in (0,1)$.

Given $u\in H^s({\mathbb{R}}^3)$, one can use the Lax–Milgram theorem, and then there exists a unique ${\varphi }_u^t\in D^{t,2}({\mathbb{R}}^3)$ such that

$${\mathsf{\int }}_{{\mathbb{R}}^3}{(-\mathsf{\Delta })}^t{\varphi }_u^{t}vdx={\mathsf{\int }}_{{\mathbb{R}}^3}{(-\mathsf{\Delta })}^{{\displaystyle \frac{t}{2}}}{\varphi }_u^t{(-\mathsf{\Delta })}^{{\displaystyle \frac{t}{2}}}vdx={\mathsf{\int }}_{{\mathbb{R}}^3}{u}^{2}vdx,\forall v\in D^{t,2}({\mathbb{R}}^3),$$

(14)

showing that ${\varphi }_u^t$ satisfies the Poisson equation

$${(-\mathsf{\Delta })}^t{\varphi }_u^t=u^2,x\in {\mathbb{R}}^3.$$

In view of [37], its integral expression can be characterized by

$${\varphi }_u^t(x)=c_t{\mathsf{\int }}_{{\mathbb{R}}^3}{\displaystyle \frac{u^2(y)}{{|x-y|}^{3-2t}}}dx,x\in {\mathbb{R}}^3,$$

(15)

which is called the t-Riesz potential, where

$$c_t={\pi }^{-{\displaystyle \frac{3}{2}}}{2}^{-2t}{\displaystyle \frac{\mathsf{\Gamma }(\frac{3}{2}-2t)}{\mathsf{\Gamma }(t)}}.$$

It follows from (15) that ${\varphi }_u^t(x)\ge 0$ for all $x\in {\mathbb{R}}^3$. Taking $v={\varphi }_u^t$ in (13) and (14), we derive

$$\parallel {\varphi }_u^t{\parallel }_{D^{t,2}({\mathbb{R}}^3)}\le C{\parallel u\parallel }_{H^s({\mathbb{R}}^3)}^2.$$

(16)

Substituting (15) into (1), one can rewrite (1) in the following equivalent form

$${(-\mathsf{\Delta })}^{s}u+\lambda V(x)u+{\varphi }_u^{t}u=f(u)+{|u|}^{2_s^{*}-2}u,x\in {\mathbb{R}}^3.$$

(17)

The variational functional $I_{\lambda }:E_{\lambda }\to \mathbb{R}$ associated with the problem (17) is given by

$$I_{\lambda }(u)={\displaystyle \frac{1}{2}}{\parallel u\parallel }_{E_{\lambda }}^2+{\displaystyle \frac{1}{4}}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_u^t{u}^{2}dx-{\mathsf{\int }}_{{\mathbb{R}}^3}F(u)dx-{\displaystyle \frac{1}{2_s^{*}}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|u|}^{2_s^{*}}dx.$$

(18)

It would be simply verified that $I_{\lambda }$ is well defined in $E_{\lambda }$ and belongs to ${\mathcal{C}}^1(E_{\lambda },\mathbb{R})$ whose derivative is given by

$$I_{\lambda }^{\prime }(u)v={\mathsf{\int }}_{{\mathbb{R}}^3}[{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}u{(-\mathsf{\Delta })}^{\frac{s}{2}}v+\lambda V(x)uv]dx+{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_u^{t}uvdx-{\mathsf{\int }}_{{\mathbb{R}}^3}{(f(u)+|u|}^{2_s^{*}-2}u)vdx$$

for any $u,v\in E_{\lambda }$. It is clear to see that if u is a critical point of $I_{\lambda }$, then the pair $(u,{\varphi }_u^t)$ is a solution of system (1).

2.2. Basic Lemmas

It is similar to the proof of ([28], Proposition 2.1) that we can derive the following

Lemma 1

(Pohožaev identity). Let $u\in E_{\lambda }$ be a critical point of the functional $I_{\lambda }$, then the identity $P_{\lambda }(u)\equiv 0$ holds true, where the functional $P_{\lambda }:E_{\lambda }\to \mathbb{R}$ is defined by

$$\begin{array}{cc}\hfill P_{\lambda }(u)& \triangleq {\displaystyle \frac{3-2s}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u|}^{2}dx+{\displaystyle \frac{1}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}[3V(x)+(\nabla V,x)]{|u|}^{2}dx+{\displaystyle \frac{2t+3}{4}}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_u^t{u}^{2}dx\hfill \\ \hfill & -3{\mathsf{\int }}_{{\mathbb{R}}^3}F(u)dx-{\displaystyle \frac{3}{2_s^{*}}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|u|}^{2_s^{*}}dx.\hfill \end{array}$$

Now, let us define the functional $N:E_{\lambda }\to \mathbb{R}$ by

$$N(u)={\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_u^t{u}^{2}dx,\forall u\in E_{\lambda }.$$

We gather the results in ([29], Lemmas 9 and 10) to introduce the properties associated with N below.

Lemma 2.

Let $s,t\in (0,1)$ satisfy $4s+2t>3$, then the following properties are true:

$(1)$

For all $u\in E_{\lambda }$ and we set $u_{\theta }(·)\triangleq {\theta }^{s+t}u(\theta ·)$ for $\theta \in {\mathbb{R}}^{+}$, then $N(u_{\theta })={\theta }^{4s+2t-3}N(u)$.

$(2)$

${\varphi }_{u(·+y)}^t={\varphi }_u^t(·+y)$ for all $y\in {\mathbb{R}}^3$.

$(3)$

If $u_n\rightharpoonup u$ in $E_{\lambda }$, then $N(u_n)-N(u_n-u)-N(u)=o_n(1)$ in $E_{\lambda }$, $N^{\prime }(u_n)-N^{\prime }(u_n-u)-N^{\prime }(u)=o_n(1)$ in ${(E_{\lambda })}^{-1}$.

We conclude this section by the following vanishing lemma associated with the fractional Sobolev space.

Lemma 3

(see, e.g., ([39], Lemma)). Assume $(u_n)$ is a bounded sequence in $H^{\alpha }({\mathbb{R}}^3)$ with $\alpha \in (0,1)$. If

$$\underset{n\to \infty }{lim}\underset{y\in {\mathbb{R}}^3}{sup}{\mathsf{\int }}_{B_{\varrho }(y)}{|u_n|}^{2}dx=0$$

for some $\varrho >0$, then $u_n\to 0$ in $L^q({\mathbb{R}}^3)$ for all $2<p<2_{\alpha }^{*}$.

3. Existence and Concentration

In this section, we focus on the existence and concentration of ground state solutions for (1). First of all, to look for a ground state solution, we shall consider the following minimization problem

$$m_{\lambda }\triangleq \underset{u\in {\mathcal{M}}_{\lambda }}{inf}{I}_{\lambda }(u),$$

(19)

where ${\mathcal{M}}_{\lambda }=\{u\in E_{\lambda }\setminus \left\{0\right\}:G_{\lambda }(u)=0\}$ with the functional $G_{\lambda }:E_{\lambda }\to \mathbb{R}$ defined by

$$\begin{array}{cc}\hfill G_{\lambda }(u)& ={\displaystyle \frac{4s+2t-3}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}u|}^{2}dx+{\displaystyle \frac{1}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}\lambda [(2s+2t-3)V(x)-(\nabla V,x)]u^{2}dx\hfill \\ \hfill & +{\displaystyle \frac{4s+2t-3}{4}}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_u^t{u}^{2}dx-{\mathsf{\int }}_{{\mathbb{R}}^3}[(s+t)f(u)u-3F(u)]dx-{\displaystyle \frac{2_s^{*}(s+t)-3}{2_s^{*}}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|u|}^{2_s^{*}}dx.\hfill \end{array}$$

Recalling the functional $P_{\lambda }$ in Lemma 1, one sees that $G_{\lambda }(u)=(s+t)I_{\lambda }^{\prime }(u)u-P_{\lambda }(u)$ for all $u\in E_{\lambda }$. In other words, if $u\in E_{\lambda }$ is a critical point of $I_{\lambda }$, then we are derived from Lemma 1 that $G_{\lambda }(u)=0$. As a consequence, the set ${\mathcal{M}}_{\lambda }$ is a natural constraint, and we then begin showing some properties for it and the minimization constant $m_{\lambda }$.

Before exhibiting them, we need the following elementary facts:

$$\xi (\theta ,x)\triangleq V(x)-{\theta }^{2s+2t-3}V({\theta }^{-1}x)-{\displaystyle \frac{1-{\theta }^{4s+2t-3}}{4s+2t-3}}[(2s+2t-3)V(x)-(\nabla V,x)]\ge 0$$

(20)

for all $(\theta ,x)\in (0,+\infty )\times {\mathbb{R}}^3$ and

$$\zeta (\theta ,z)\triangleq {\displaystyle \frac{1-{\theta }^{4s+2t-3}}{4s+2t-3}}[(s+t)f(z)z-3F(z)]+{\theta }^{-3}F({\theta }^{s+t}z)-F(z)\ge 0$$

(21)

for all $(\theta ,z)\in (0,+\infty )\times {\mathbb{R}}^{+}$.

Actually, since V is weakly differentiable by $(V_4)$, one uses $(V_5)$ to see that

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial }{\partial \theta }}\xi (\theta ,x)& ={\theta }^{4s+2t-4}\left\{[(2s+2t-3)V(x)-(\nabla V,x)]-{\displaystyle \frac{(2s+2t-3)V({\theta }^{-1}x)-(\nabla V({\theta }^{-1}x),{\theta }^{-1}x)}{{\theta }^{2s}}}\right\}\hfill \\ \hfill & \left\{\begin{array}{cc}\le 0,\hfill & \mathrm{if}\theta \in (0,1]\hfill \\ \ge 0,\hfill & \mathrm{if}\theta \in [1,+\infty ).\hfill \end{array}\right.\hfill \end{array}$$

Hence, the function $\theta \mapsto \xi (\theta ,x)$ is decreasing on $(0,1)$ and increasing on $(1,+\infty )$ for all $x\in {\mathbb{R}}^3$, which indicates that $\xi (\theta ,x)\ge \underset{\theta >0}{min}\xi (\theta ,x)=\xi (1,x)=0$ for all $(\theta ,x)\times (0,+\infty )\in {\mathbb{R}}^3$. Similarly, we are able to apply $(f_5)$ to derive

$$\begin{array}{cc}\hfill {\displaystyle \frac{\partial }{\partial \theta }}\zeta (\theta ,z)& ={\theta }^{-4}[(s+t)f({\theta }^{s+t}z){\theta }^{s+t}z-3F({\theta }^{s+t}z)]-{\theta }^{4s+2t-4}[(s+t)f(z)z-3F(z)]\hfill \\ \hfill & ={\theta }^{4s+2t-4}{z}^{{\displaystyle \frac{4s+2t}{s+t}}}\left[{\displaystyle \frac{(s+t)f({\theta }^{s+t}z){\theta }^{s+t}z-3F({\theta }^{s+t}z)}{{({\theta }^{s+t}z)}^{\frac{4s+2t}{s+t}}}}-{\displaystyle \frac{(s+t)f(z)z-3F(z)}{z^{\frac{4s+2t}{s+t}}}}\right]\hfill \\ \hfill & \left\{\begin{array}{cc}\le 0,\hfill & \mathrm{if}\theta \in (0,1]\hfill \\ \ge 0,\hfill & \mathrm{if}\theta \in [1,+\infty ).\hfill \end{array}\right.\hfill \end{array}$$

It therefore infers that $\zeta (\theta ,z)\ge \underset{\theta >0}{min}\zeta (\theta ,z)=\zeta (1,z)=0$ for all $(\theta ,z)\times (0,+\infty )\in {\mathbb{R}}^{+}$.

Lemma 4.

Let $s,t\in (0,1)$ satisfy $2s+2t>3$. Assume $(V_1)$–$(V_3)$ with $(V_4)$–$(V_5)$ and $(f_1)$–$(f_3)$ with $(f_5)$ hold, then for any nonzero $u\in E_{\lambda }$, there is a unique $\overline{\theta }=\overline{\theta }(u)>0$ such that $u_{\overline{\theta }}={\overline{\theta }}^{s+t}u(\overline{\theta }·)\in {\mathcal{M}}_{\lambda }$ for suitably large $\lambda >0$, where $I_{\lambda }(u_{\overline{\theta }})=\underset{\theta >0}{max}{I}_{\lambda }(u_{\theta })$. In particular, there holds

$$m_{\lambda }=d_{\lambda }\triangleq \underset{u\in E_{\lambda }\setminus (0)}{inf}\underset{\theta >0}{max}{I}_{\lambda }(u_{\theta }).$$

Proof. 

For any $u\in E_{\lambda }\setminus (0)$ and $\theta >0$, we define $\tau (\theta )=I_{\lambda }(u_{\theta })$, where

$$\begin{array}{cc}\hfill \tau (\theta )& ={\displaystyle \frac{{\theta }^{4s+2t-3}}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}u|}^{2}dx+{\displaystyle \frac{{\theta }^{2s+2t-3}}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}\lambda V({\theta }^{-1}x)u^{2}dx+{\displaystyle \frac{{\theta }^{4s+2t-3}}{4}}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_u^t{u}^{2}dx\hfill \\ \hfill & -{\theta }^{-3}{\mathsf{\int }}_{{\mathbb{R}}^3}F({\theta }^{s+t}u)dx-{\displaystyle \frac{{\theta }^{2_s^{*}(s+t)-3}}{2_s^{*}}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|u|}^{2_s^{*}}dx.\hfill \end{array}$$

It is simple to observe that

$${\tau }^{\prime }(\theta )=0⟺{\theta }^{-1}{G}_{\lambda }(u_{\theta })=0⟺G_{\lambda }(u_{\theta })=0⟺u_{\theta }\in {\mathcal{M}}_{\lambda }.$$

Since $4s+2t<2_s^{*}(s+t)$ and $\underset{\theta \to 0^{+}}{lim}{\theta }^{-3}F({\theta }^{s+t}z)=0$ for all $z\in \mathbb{R}$ by $(f_3)$, we can derive $\underset{\theta \to 0^{+}}{lim}\tau (\theta )>0$. Without loss of generality, we are assuming that $0\in \mathsf{\Omega }$ in $(V_3)$ and thus $\underset{\theta \to +\infty }{lim}{\mathsf{\int }}_{{\mathbb{R}}^3}\lambda V({\theta }^{-1}x)u^{2}dx=0$. Adopting $4s+2t<2_s^{*}(s+t)$ and $(f_3)$ again, it holds that $\underset{\theta \to +\infty }{lim}\tau (\theta )=-\infty$. As a consequence, with the above two facts in hands, we take advantage of $4s+2t<2_s^{*}(s+t)$ and $(f_3)$ to demonstrate that $\tau (\theta )$ possesses a critical point which corresponds to its maximum; that is, there exists a constant $\overline{\theta }>0$ such that ${\tau }^{\prime }(\overline{\theta })=0$. We next verify that $\overline{\theta }$ is unique. Arguing it indirectly, we would assume that there exist two constants ${\theta }_1,{\theta }_2>0$ with ${\theta }_1\ne {\theta }_2$ such that $u_{{\theta }_i}\in {\mathcal{M}}_{\lambda }$ for $i\in \{1,2\}$. It concludes from some elementary computations that

$$\begin{array}{c}{I}_{\lambda }(u_{{\theta }_1})-I_{\lambda }(u_{{\theta }_2})-{\displaystyle \frac{{\theta }_1^{4s+2t-3}-{\theta }_2^{4s+2t-3}}{(4s+2t-3){\theta }_1^{4s+2t-3}}}{G}_{\lambda }(u_{{\theta }_1})\hfill \\ ={\displaystyle \frac{{\theta }_1^{2s+2t-3}}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}\xi \left({\displaystyle \frac{{\theta }_2}{{\theta }_1}},{\theta }_1^{-1}x\right)u^{2}dx+{\theta }_1^{-3}{\mathsf{\int }}_{{\mathbb{R}}^3}\zeta \left({\displaystyle \frac{{\theta }_2}{{\theta }_1}},{\theta }_1^{s+t}u\right)dx\hfill \\ +{\displaystyle \frac{{\theta }_1^{2_s^{*}(s+t)-3}}{2_s^{*}}}\left[{\displaystyle \frac{1-{\left(\frac{{\theta }_2}{{\theta }_1}\right)}^{4s+2t-3}}{4s+2t-3}}[2_s^{*}(s+t)-3]+{\left({\displaystyle \frac{{\theta }_2}{{\theta }_1}}\right)}^{2_s^{*}(s+t)-3}-1\right]{\mathsf{\int }}_{{\mathbb{R}}^3}{|u|}^{2_s^{*}}dx\hfill \end{array}$$

and

$$\begin{array}{c}{I}_{\lambda }(u_{{\theta }_2})-I_{\lambda }(u_{{\theta }_1})-{\displaystyle \frac{{\theta }_2^{4s+2t-3}-{\theta }_1^{4s+2t-3}}{(4s+2t-3){\theta }_2^{4s+2t-3}}}{G}_{\lambda }(u_{{\theta }_2})\hfill \\ ={\displaystyle \frac{{\theta }_2^{2s+2t-3}}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}\xi \left({\displaystyle \frac{{\theta }_1}{{\theta }_2}},{\theta }_{2}x\right)u^{2}dx+{\theta }_2^{-3}{\mathsf{\int }}_{{\mathbb{R}}^3}\zeta \left({\displaystyle \frac{{\theta }_1}{{\theta }_2}},{\theta }_2^{s+t}u\right)dx\hfill \\ +{\displaystyle \frac{{\theta }_2^{2_s^{*}(s+t)-3}}{2_s^{*}}}\left[{\displaystyle \frac{1-{\left(\frac{{\theta }_1}{{\theta }_2}\right)}^{4s+2t-3}}{4s+2t-3}}[2_s^{*}(s+t)-3]+{\left({\displaystyle \frac{{\theta }_1}{{\theta }_2}}\right)}^{2_s^{*}(s+t)-3}-1\right]{\mathsf{\int }}_{{\mathbb{R}}^3}{|u|}^{2_s^{*}}dx.\hfill \end{array}$$

In view of (20) and (21), combining the above two formulas with $G_{\lambda }(u_{{\theta }_i})=0$ for $i\in \{1,2\}$, we arrive at a contradiction if ${\theta }_1\ne {\theta }_2$. Finally, the result $d_{\lambda }\le m_{\lambda }$ is a direct consequence of the inequality

$$I_{\lambda }(u)-I_{\lambda }(u_{\theta })-{\displaystyle \frac{1-{\theta }^{4s+2t-3}}{4s+2t-3}}{G}_{\lambda }(u)\ge 0,\forall u\in E_{\lambda }\mathrm{and}\theta >0,$$

(22)

we immediately finish the proof of this lemma. □

The following results can be found in [28].

Lemma 5.

Let $u_{\epsilon }$ be defined by (28) in the proof of Lemma 6 below, then

$${\mathsf{\int }}_{{\mathbb{R}}^3}{|{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u}_{\epsilon }|}^{2}dx\le S_s^{{\displaystyle \frac{3}{2s}}}+O({\epsilon }^{3-2s}),$$

(23)

and

$${\mathsf{\int }}_{{\mathbb{R}}^3}{|u_{\epsilon }|}^{2_s^{*}}dx=S_s^{{\displaystyle \frac{3}{2s}}}+O({\epsilon }^3).$$

(24)

For all $q\in [2,2_s^{*})$, there holds

$${\mathsf{\int }}_{{\mathbb{R}}^3}{|u_{\epsilon }|}^{q}dx=\left\{\begin{array}{cc}O\left({\epsilon }^{3-{\displaystyle \frac{3-2s}{2}}q}\right),\hfill & \mathrm{for}q>{\displaystyle \frac{3}{3-2s}},\hfill \\ O\left({\epsilon }^{{\displaystyle \frac{3}{2}}}|log\epsilon |\right),\hfill & \mathrm{for}q={\displaystyle \frac{3}{3-2s}},\hfill \\ O\left({\epsilon }^{{\displaystyle \frac{3-2s}{2}}q}\right),\hfill & \mathrm{for}q<{\displaystyle \frac{3}{3-2s}}.\hfill \end{array}\right.$$

(25)

According to Lemma 4, we know that ${\mathcal{M}}_{\lambda }$ is a nonempty set for some suitably large $\lambda >0$. The following lemma ensures that the minimization constant $m_{\lambda }$ would be well defined. More precisely, we further show that $m_{\lambda }$ is uniformly bounded from below and above by some positive constants which are independent of some suitably large $\lambda >0$.

Lemma 6.

Let $s,t\in (0,1)$ satisfy $2s+2t>3$. Assume that $(V_1)$–$(V_5)$ and $(f_1)$–$(f_5)$ hold, there is a $\rho >0$ independent of $\lambda >{\mathsf{\Lambda }}_0$ such that

$$\underset{\lambda >{\mathsf{\Lambda }}_0}{inf}{m}_{\lambda }\ge \rho ,$$

(26)

where ${\mathsf{\Lambda }}_0\triangleq max\{1,c^{-1}{|\mathsf{\Sigma }|}^{-{\displaystyle \frac{2_s^{*}-2}{2_s^{*}}}}{S}_s\}$. If in addition one of the assumptions in (6) holds true, then

$$\underset{\lambda >{\mathsf{\Lambda }}_0}{sup}{m}_{\lambda }<{\displaystyle \frac{s}{3}}{S}_s^{{\displaystyle \frac{3}{2s}}}.$$

(27)

Proof. 

For all $u\in {\mathcal{M}}_{\lambda }$, we are derived from $(f_4)$ and $(\nabla V,x)\ge 0$ for all $x\in {\mathbb{R}}^3$ in $(V_5)$ that

$$\begin{array}{cc}\hfill & I_{\lambda }(u)=I_{\lambda }(u)-{\displaystyle \frac{1}{(s+t)\gamma -3}}{G}_{\lambda }(u)\hfill \\ \hfill & ={\displaystyle \frac{(s+t)\gamma -(4s+2t)}{2\left[(s+t)\gamma -3\right]}}{|{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}u|}_2^2+{\displaystyle \frac{1}{2\left[(s+t)\gamma -3\right]}}{\mathsf{\int }}_{{\mathbb{R}}^3}\lambda \left[(s+t)(\gamma -2)V(x)+(\nabla V,x)\right]u^{2}dx\hfill \\ \hfill & +{\displaystyle \frac{(s+t)\gamma -(4s+2t)}{4\left[(s+t)\gamma -3\right]}}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_u^t{u}^{2}dx+{\displaystyle \frac{s+t}{(s+t)\gamma -3}}{\mathsf{\int }}_{{\mathbb{R}}^3}\left[uf(u)-\gamma F(u)\right]dx+{\displaystyle \frac{2_s^{*}-\gamma }{2_s^{*}[(s+t)-3]}}{|u|}_{2_s^{*}}^{2_s^{*}}\hfill \\ \hfill & \ge {\displaystyle \frac{(s+t)\gamma -(4s+2t)}{2\left[(s+t)\gamma -3\right]}}{\parallel u\parallel }_{E_{\lambda }}^2.\hfill \end{array}$$

It follows from $(f_1)$–$(f_2)$ and (12) that

$${\mathsf{\int }}_{{\mathbb{R}}^3}[(s+t)f(u)u-3F(u)]dx\le {\displaystyle \frac{2s+2t-3}{4}}{\parallel u\parallel }_{E_{\lambda }}^2+C_1{\parallel u\parallel }_{E_{\lambda }}^q.$$

From which, combining $(2s+2t-3)V(x)\ge 2(\nabla V,x)\ge 0$ for all $x\in {\mathbb{R}}^3$ in $(V_5)$ and (10), we see that

$${\displaystyle \frac{2s+2t-3}{4}}{\parallel u\parallel }_{E_{\lambda }}^2\le C_1{\parallel u\parallel }_{E_{\lambda }}^q+S_s^{-{\displaystyle \frac{2}{2_s^{*}}}}{\parallel u\parallel }_{E_{\lambda }}^{2_s^{*}},\forall u\in {\mathcal{M}}_{\lambda },$$

yielding that ${\parallel u\parallel }_{E_{\lambda }}\ge C_2$ for some $C_2>0$ independent of $\lambda$. So, we arrive at (26).

On the other hand, we begin verifying (27). Without loss of generality, we are assuming that $0\in \mathsf{\Omega }$. Because $\mathsf{\Omega }$ is an open subset of ${\mathbb{R}}^3$, it holds that $B_{r_0}(0)\subset \mathsf{\Omega }$ for some $r_0>0$. Given a constant ${\widehat{r}}_0>0$ which will be determined later, we choose a cutoff function $\psi \in C_0^{\infty }({\mathbb{R}}^3)$ in such a way that $\psi (x)\equiv 1$ if $|x|\le {\widehat{r}}_0$ and $\psi (x)\equiv 0$ if $|x|\ge 2{\widehat{r}}_0$. For all $\epsilon >0$, we define

$$u_{\epsilon }(x)=\psi (x)U_{\epsilon }(x),\forall x\in {\mathbb{R}}^3,$$

(28)

where $U_{\epsilon }(x)={\epsilon }^{-{\displaystyle \frac{3-2s}{2}}}{u}^{*}\left({\displaystyle \frac{x}{\epsilon }}\right)$, $u^{*}(x)={\displaystyle \frac{U\left(x/S_s^{\frac{1}{2s}}\right)}{{\parallel U\parallel }_{2s}^{*}}}$ and $U(x)={\displaystyle \frac{\kappa }{({\tau }^2+{|x|}^2{)}^{\frac{3-2s}{2}}}}$ with $\kappa \ne 0$ and $\tau >0$. Due to Lemma 4 and (26), there exists a ${\theta }_{\epsilon }>0$ such that

$$0<m_{\lambda }\le \underset{\theta >0}{max}{I}_{\lambda }(u_{\theta })=I_{\lambda }({(u_{\epsilon })}_{{\theta }_{\epsilon }}).$$

Next, we shall prove that there exist two constants ${\theta }_{*}$, ${\theta }^{*}>0$ such that ${\theta }_{*}\le {\theta }_{\epsilon }\le {\theta }^{*}$. First, we claim that ${\theta }_{\epsilon }$ is bounded from below by a positive constant. Otherwise, there is a sequence ${\epsilon }_n\to 0$ such that ${\theta }_{{\epsilon }_n}\to 0$. Then, we conclude that ${(u_{{\epsilon }_n})}_{{\epsilon }_n}\to 0$ in $E_{\lambda }$. So, we have

$$0<m_{\lambda }\le I_{\lambda }({(u_{\epsilon })}_{{\theta }_{\epsilon }})\to I_{\lambda }(0)=0,$$

a contradiction. Taking some similar calculations in the proof of Lemma 4, one has $\underset{{\theta }_{\epsilon }\to +\infty }{lim}{I}_{\lambda }({(u_{\epsilon })}_{{\theta }_{\epsilon }})=-\infty$ which is absurd, too. Thus, we conclude the claim. Letting ${\widehat{r}}_0={\displaystyle \frac{1}{2}}{\theta }_{*}{r}_0$, then

$${\mathsf{\int }}_{{\mathbb{R}}^3}V({\theta }_{\epsilon }^{-1}x)u_{\epsilon }^{2}dx={\mathsf{\int }}_{B_{{\theta }_{\epsilon }{r}_0}(0)}V({\theta }_{\epsilon }^{-1}x)u_{\epsilon }^{2}dx+{\mathsf{\int }}_{{\mathbb{R}}^3\setminus B_{{\theta }_{\epsilon }{r}_0}(0)}V({\theta }_{\epsilon }^{-1}x)u_{\epsilon }^{2}dx=0$$

from where it follows that

$$\begin{array}{cc}\hfill I_{\lambda }({(u_{\epsilon })}_{{\theta }_{\epsilon }})& ={\displaystyle \frac{{\theta }_{\epsilon }^{4s+2t-3}}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u}_{\epsilon }|}^{2}dx+{\displaystyle \frac{{\theta }_{\epsilon }^{4s+2t-3}}{4}}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_{u_{\epsilon }}^t{u}_{\epsilon }^{2}dx\hfill \\ \hfill & -{\theta }^{-3}{\mathsf{\int }}_{{\mathbb{R}}^3}F({\theta }^{s+t}{u}_{\epsilon })dx-{\displaystyle \frac{{\theta }^{2_s^{*}(s+t)-3}}{2_s^{*}}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|u_{\epsilon }|}^{2_s^{*}}dx.\hfill \end{array}$$

Clearly, the proof of (27) would be complete if $I_{\lambda }({(u_{\epsilon })}_{{\theta }_{\epsilon }})<{\displaystyle \frac{s}{3}}{S}_s^{{\displaystyle \frac{3}{2s}}}$ for some suitably small $\epsilon >0$. Let us adopt the useful estimates in Lemma 5 and apply $(f_3)$ to reach

$$\begin{array}{cc}\hfill I_{\lambda }({(u_{\epsilon })}_{{\theta }_{\epsilon }})& \le \left({\displaystyle \frac{{\theta }_{\epsilon }^{4s+2t-3}}{2}}-{\displaystyle \frac{{\theta }_{\epsilon }^{(s+t)2_s^{*}-3}}{2_s^{*}}}\right)S_s^{{\displaystyle \frac{3}{2s}}}+O({\epsilon }^{3-2s})+C|u_{\epsilon }{|}_2^2+C|u_{\epsilon }{|}_{{\displaystyle \frac{12}{3+2t}}}^4-C\widehat{\mu }{|u_{\epsilon }|}_p^p\hfill \\ \hfill & \le {\displaystyle \frac{s}{3}}{S}_s^{{\displaystyle \frac{3}{3s}}}+O({\epsilon }^{3-2s})+C|u_{\epsilon }{|}_2^2+C|u_{\epsilon }{|}_{{\displaystyle \frac{12}{3+2t}}}^4-C\widehat{\mu }{|u_{\epsilon }|}_p^p,\hfill \end{array}$$

where we have used the following inequality

$${\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_{u_{\epsilon }}^t{u}_{\epsilon }^{2}dx\le C{\left({\mathsf{\int }}_{{\mathbb{R}}^3}{|u_{\epsilon }|}^{{\displaystyle \frac{12}{3+2t}}}dx\right)}^{{\displaystyle \frac{3+2t}{3}}}.$$

To continue the proof, we divide the following three different cases.

Case 1. $2<{\displaystyle \frac{3}{3-2s}}$ which is equivalent to $s>{\displaystyle \frac{3}{4}}$. Then,

$$I_{\lambda }({(u_{\epsilon })}_{{\theta }_{\epsilon }})\le {\displaystyle \frac{s}{3}}{S}_s^{{\displaystyle \frac{3}{3s}}}+O({\epsilon }^{3-2s})+C|u_{\epsilon }{|}_{{\displaystyle \frac{12}{3+2t}}}^4-C\widehat{\mu }{|u_{\epsilon }|}_p^p.$$

Case 2. $2={\displaystyle \frac{3}{3-2s}}$ which is equivalent to $s={\displaystyle \frac{3}{4}}$. Then,

$$I_{\lambda }({(u_{\epsilon })}_{{\theta }_{\epsilon }})\le {\displaystyle \frac{s}{3}}{S}_s^{{\displaystyle \frac{3}{3s}}}+O({\epsilon }^{2s}|log\epsilon |)+C|u_{\epsilon }{|}_{{\displaystyle \frac{12}{3+2t}}}^4-C\widehat{\mu }{|u_{\epsilon }|}_p^p.$$

Case 3. $2>{\displaystyle \frac{3}{3-2s}}$ which is equivalent to $s<{\displaystyle \frac{3}{4}}$. Then,

$$I_{\lambda }({(u_{\epsilon })}_{{\theta }_{\epsilon }})\le {\displaystyle \frac{s}{3}}{S}_s^{{\displaystyle \frac{3}{3s}}}+O({\epsilon }^{2s})+C|u_{\epsilon }{|}_{{\displaystyle \frac{12}{3+2t}}}^4-C\widehat{\mu }{|u_{\epsilon }|}_p^p.$$

We note that ${\displaystyle \frac{3s+t}{s+t}}<{\displaystyle \frac{2s}{3-2s}}={\displaystyle \frac{3}{3-2s}}-1$ for any $s>{\displaystyle \frac{3}{4}}$ and ${\displaystyle \frac{3s+t}{s+t}}\ge {\displaystyle \frac{2s}{3-2s}}={\displaystyle \frac{3}{3-2s}}-1$ for any $s\le {\displaystyle \frac{3}{4}}$. Thereby,

(a) If $s>{\displaystyle \frac{3}{4}}$ in Case 1. It follows from (25) that

$${\displaystyle \underset{\epsilon \to 0^{+}}{lim}\frac{|u_{\epsilon }{|}_{\frac{12}{3+2t}}^4}{{\epsilon }^{3-2s}}\le \left\{\begin{array}{cc}{\displaystyle \underset{\epsilon \to 0^{+}}{lim}\frac{O({\epsilon }^{4s+2t-3})}{{\epsilon }^{3-2s}}=0,}\hfill & \frac{12}{3+2t}>\frac{3}{3-2s},\hfill \\ {\displaystyle \underset{\epsilon \to 0^{+}}{lim}\frac{O({\epsilon }^{4s+2t-3}){|log\epsilon |}^{\frac{3+2t}{3}}}{{\epsilon }^{3-2s}}=0,}\hfill & \frac{12}{3+2t}=\frac{3}{3-2s},\hfill \\ {\displaystyle \underset{\epsilon \to 0^{+}}{lim}\frac{O({\epsilon }^{2(3-2s)})}{{\epsilon }^{3-2s}}=0,}\hfill & \frac{12}{3+2t}<\frac{3}{3-2s}.\hfill \end{array}\right.}$$

Moreover, since ${\displaystyle \frac{4s}{3-2s}}<p<{\displaystyle \frac{6}{3-2s}}$ gives that $2s-{\displaystyle \frac{3-2s}{2}}p<0$, one infers from (25) again that

$$\underset{\epsilon \to 0^{+}}{lim}{\displaystyle \frac{\widehat{\mu }{|u_{\epsilon }|}_p^p}{{\epsilon }^{3-2s}}}=\left\{\begin{array}{cc}{\displaystyle \underset{\epsilon \to 0}{lim}\widehat{\mu }\frac{O({\epsilon }^{3-\frac{3-2s}{2}p})}{{\epsilon }^{3-2s}}=+\infty ,}\hfill & {\displaystyle \frac{4s}{3-2s}}<p<{\displaystyle \frac{6}{3-2s}},\hfill \\ {\displaystyle \underset{\epsilon \to 0^{+}}{lim}\widehat{\mu }\frac{O({\epsilon }^{3-\frac{3-2s}{2}p})}{{\epsilon }^{3-2s}},}\hfill & {\displaystyle \frac{3}{3-2s}}<p\le {\displaystyle \frac{4s}{3-2s}},\hfill \\ {\displaystyle \underset{\epsilon \to 0^{+}}{lim}\widehat{\mu }\frac{O({\epsilon }^{3-\frac{3-2s}{2}p})|log\epsilon |}{{\epsilon }^{3-2s}},}\hfill & p={\displaystyle \frac{3}{3-2s}},\hfill \\ {\displaystyle \underset{\epsilon \to 0^{+}}{lim}\widehat{\mu }\frac{O({\epsilon }^{\frac{3-2s}{2}p})}{{\epsilon }^{3-2s}},}\hfill & {\displaystyle \frac{4s+2t}{s+t}}<p<{\displaystyle \frac{3}{3-2s}}.\hfill \end{array}\right.$$

Choosing $\widehat{\mu }={\epsilon }^{-2s}$, then the above three unknown limits would also be $+\infty$.

(b) If $s={\displaystyle \frac{3}{4}}$ in Case 2. Since ${\displaystyle \frac{12}{3+2t}}>2-{\displaystyle \frac{3}{3-2s}}$, there holds

$$\underset{\epsilon \to 0^{+}}{lim}{\displaystyle \frac{|u_{\epsilon }{|}_{\frac{12}{3+2t}}^4}{{\epsilon }^{2s}|log\epsilon |}}\le \underset{\epsilon \to 0^{+}}{lim}{\displaystyle \frac{O({\epsilon }^{4s+2t-3})}{{\epsilon }^{2s}|log\epsilon |}}=0.$$

By ${\displaystyle \frac{3}{3-2s}}=2<{\displaystyle \frac{4s+2t}{s+t}}<p$, for any $\widehat{\mu }>0$, we have that

$$\underset{\epsilon \to 0^{+}}{lim}{\displaystyle \frac{\widehat{\mu }{|u_{\epsilon }|}_p^p}{{\epsilon }^{2s}|log\epsilon |}}=\underset{\epsilon \to 0}{lim}\widehat{\mu }{\displaystyle \frac{O({\epsilon }^{3-\frac{3-2s}{2}p})}{{\epsilon }^{2s}|log\epsilon |}}=+\infty ,{\displaystyle \frac{4s+2t}{s+t}}<p<{\displaystyle \frac{6}{3-2s}}.$$

(c) If $s<{\displaystyle \frac{3}{4}}$ in Case 3. Since ${\displaystyle \frac{3}{3-2s}}\in \left({\displaystyle \frac{3}{2}},2\right)$, then ${\displaystyle \frac{12}{3+2t}}>{\displaystyle \frac{3}{3-2s}}$ and ${\displaystyle \frac{3}{3-2s}}<{\displaystyle \frac{4s+2t}{s+t}}<p<{\displaystyle \frac{6}{3-2s}}$. Hence,

$$\underset{\epsilon \to 0^{+}}{lim}{\displaystyle \frac{|u_{\epsilon }{|}_{\frac{12}{3+2t}}^4}{{\epsilon }^{2s}}}\le \underset{\epsilon \to 0^{+}}{lim}{\displaystyle \frac{O({\epsilon }^{4s+2t-3})}{{\epsilon }^{2s}}}=0$$

and for any $\widehat{\mu }>0$, there holds

$$\underset{\epsilon \to 0^{+}}{lim}{\displaystyle \frac{\widehat{\mu }{|u_{\epsilon }|}_p^p}{{\epsilon }^{2s}}}=\underset{\epsilon \to 0}{lim}\widehat{\mu }{\displaystyle \frac{O({\epsilon }^{3-\frac{3-2s}{2}p})}{{\epsilon }^{2s}}}=+\infty ,{\displaystyle \frac{4s+2t}{s+t}}<p<{\displaystyle \frac{6}{3-2s}}.$$

At this stage, we will apply Point (a) to Case 1, from (I) and (II) in (6); Point (b) to Case 2 and Point (c) to Case 3, from (III) in (6); there exists a sufficiently small $\epsilon >0$ to arrive at the desired result. The proof is completed. □

As a by-product of Lemma 6, we conclude that $m_{\lambda }$ is well defined. Before looking for a minimizer for it, we shall derive the following result which permits us to show that the weak limit of the minimizing sequence of $m_{\lambda }$ is nontrivial.

Lemma 7.

Let $s,t\in (0,1)$ satisfy $2s+2t>3$. Assume that $(V_1)$–$(V_5)$ and $(f_1)$–$(f_5)$ hold. Let $\lambda >{\mathsf{\Lambda }}_0$ and $(u_n)\subset E_{\lambda }$ be a minimizing sequence sequence of $m_{\lambda }$, then there exist $r\in \left(2,{\displaystyle \frac{3(3-s)}{3-2s}}\right)$ and ${\sigma }_0>0$, independent of λ, such that $|u_n{|}_r\ge {\sigma }_0$, for all $n\ge 1$.

Proof. 

First of all, we can show that $(u_n)$ is uniformly bounded in $n\in \mathbb{N}$ for all $\lambda >{\mathsf{\Lambda }}_0$, see, e.g., Lemma 8 below in detail. Let us divide the proof into intermediate steps.

Step I: Let $\lambda >{\Lambda }_0$ and $(u_n)\subset E_{\lambda }$ be a minimizing sequence of $m_{\lambda }$, then there exist $r\in \left(2,{\displaystyle \frac{3(3-s)}{3-2s}}\right)$ and $\sigma =\sigma (\lambda )>0$ such that $|u_n{|}_r\ge \sigma$ for all $n\ge 1$.

Suppose, by contradiction, that $u_n\to 0$ in $L^r({\mathbb{R}}^3)$ for each $r\in \left(2,{\displaystyle \frac{3(3-s)}{3-2s}}\right)$. Due to the boundedness of $(u_n)$ in $E_{\lambda }$, we see that $(u_n)$ is uniformly bounded in $L^q({\mathbb{R}}^3)$ for all $q\in (2,2_s^{*})$, too. As a consequence, one simply arrives at

$$\underset{n\to \infty }{lim}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_{u_n}^t{u}_n^{2}dx=0,\underset{n\to \infty }{lim}{\mathsf{\int }}_{{\mathbb{R}}^3}f(u_n)u_{n}dx=0\mathrm{and}\underset{n\to \infty }{lim}{\mathsf{\int }}_{{\mathbb{R}}^3}F(u_n)dx=0.$$

(29)

Without loss of generality, we could assume that $\parallel u_n{\parallel }_{E_{\lambda }}^2\to l$ as $n\to \infty$. Obviously, we derive $l>0$. Otherwise, $\parallel u_n{\parallel }_{E_{\lambda }}^2\to 0$ and hence $|u_n{|}_{2_s^{*}}^{2_s^{*}}\to 0$ as $n\to \infty$ by (10). Combining these facts and (29), it holds that $m_{\lambda }=\underset{n\to \infty }{lim}{I}_{\lambda }(u_n)=0$, which is absurd because of (26). Now, we claim that $\underset{n\to \infty }{lim}{|u_n|}_{2_s^{*}}^{2_s^{*}}=l$. Indeed, according to $G_{\lambda }(u_n)=0$, (29) and ${\displaystyle \frac{4s+2t-3}{2}}={\displaystyle \frac{2_s^{*}(s+t)-3}{2_s^{*}}}$ with $(V_5)$, we obtain the desired result. Using (10) again, then $l\le S_s^{-{\displaystyle \frac{2_s^{*}}{2}}}{l}^{{\displaystyle \frac{2_s^{*}}{2}}}$ which gives that $l\ge S_s^{{\displaystyle \frac{3}{2s}}}$. So, it follows from (29) that

$$m_{\lambda }=\underset{n\to \infty }{lim}{I}_{\lambda }(u_n)=\left({\displaystyle \frac{1}{2}}-{\displaystyle \frac{1}{2_s^{*}}}\right)l\ge {\displaystyle \frac{s}{3}}{S}_s^{{\displaystyle \frac{3}{2s}}}$$

reaching a contradiction with (27).

Step II: Conclusion.

Let $r\in \left(2,{\displaystyle \frac{3(3-s)}{3-2s}}\right)$ be as in Step I. Suppose by contradiction that the uniform control from below of $L^r({\mathbb{R}}^3)$-norm is false. Then, for every $k\in \mathbb{N}$, $k\ne 0$, there exist ${\lambda }_k>{\mathsf{\Lambda }}_0$ and a minimizing sequence $(u_{k,n})$ of $m_{{\lambda }_k}$ such that

$$|u_{k,n}{|}_r<{\displaystyle \frac{1}{k}},\mathrm{definitely}.$$

Then, by a diagonalization argument, for any $k\ge 1$, we can find an increasing sequence $(n_k)$ in $\mathbb{N}$ and $u_{n_k}\in E_{{\lambda }_{n_k}}$ such that

$$u_{n_k}\in {\mathcal{M}}_{{\lambda }_k},J_{n_k}(u_{n_k})=m_{{\lambda }_{n_k}}+o_k(1),{|u_{n_k}|}_r=o_k(1),$$

where $o_k(1)$ is a positive quantity which goes to zero as $k\to +\infty$. Then, we are able to arrive at the same contradiction in Step I with (27), again. The proof is completed. □

Lemma 8.

Let $s,t\in (0,1)$ satisfy $2s+2t>3$. Assume that $(V_1)$–$(V_5)$ and $(f_1)$–$(f_5)$ with one of the assumptions in (6) holding, then there is a $\mathsf{\Lambda }>0$ such that $m_{\lambda }$ can be attained for all $\lambda >\mathsf{\Lambda }$.

Proof. 

Let $(u_n)\subset {\mathcal{M}}_{\lambda }$ be a sequence satisfying $I_{\lambda }(u_n)\to m_{\lambda }$ as $n\to \infty$. First of all, we claim that $(u_n)$ is uniformly bounded in $E_{\lambda }$ with respect to $n\in \mathbb{N}$ for all $\lambda >{\mathsf{\Lambda }}_0$. Indeed, since $(u_n)\subset {\mathcal{M}}_{\lambda }$ gives that $G_{\lambda }(u_n)=0$ and so

$$\begin{array}{cc}\hfill m_{\lambda }& =I_{\lambda }(u_n)+o_n(1)=I_{\lambda }(u_n)-{\displaystyle \frac{1}{(s+t)\gamma -3}}{G}_{\lambda }(u_n)+o_n(1)\hfill \\ \hfill & ={\displaystyle \frac{(s+t)\gamma -(4s+2t)}{2\left[(s+t)\gamma -3\right]}}{|{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u}_n|}_2^2+{\displaystyle \frac{1}{2\left[(s+t)\gamma -3\right]}}{\mathsf{\int }}_{{\mathbb{R}}^3}\lambda \left[(s+t)(\gamma -2)V(x)+(\nabla V,x)\right]u_n^{2}dx\hfill \\ \hfill & +{\displaystyle \frac{(s+t)\gamma -(4s+2t)}{4\left[(s+t)\gamma -3\right]}}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_{u_n}^t{u}_n^{2}dx+{\displaystyle \frac{s+t}{(s+t)\gamma -3}}{\mathsf{\int }}_{{\mathbb{R}}^3}\left[u_{n}f(u_n)-\gamma F(u_n)\right]dx+o_n(1)\hfill \\ \hfill & +{\displaystyle \frac{2_s^{*}-\gamma }{2_s^{*}[(s+t)-3]}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|u_n|}^{2_s^{*}}dx+o_n(1)\hfill \\ \hfill & \ge {\displaystyle \frac{(s+t)\gamma -(4s+2t)}{2\left[(s+t)\gamma -3\right]}}{|{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u}_n|}_2^2+{\displaystyle \frac{(s+t)\gamma -(4s+2t)}{4\left[(s+t)\gamma -3\right]}}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_{u_n}^t{u}_n^{2}dx+o_n(1)\hfill \end{array}$$

(30)

which together with (27) implies that ${|(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u}_n{|}_2$ is uniformly bounded in $n\in \mathbb{N}$ for all $\lambda >{\mathsf{\Lambda }}_0$. By means of the interpolation inequality, for $q\in (2,2_s^{*})$, we combine (10) and (12) to derive

$$\begin{array}{cc}\hfill |u_n{|}_q^q& \le |u_n{|}_2^{2\nu }|u_n{|}_{2_s^{*}}^{2_s^{*}(1-\nu )}\le {C\parallel u\parallel }_{E_{\lambda }}^{2\nu }{|u_n|}_{2_s^{*}}^{2(1-\nu )}\hfill \\ \hfill & \le C\parallel u_n{\parallel }_{E_{\lambda }}^{2\nu }{|(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u}_n{|}_2^{1-\nu }\le C{\parallel u_n\parallel }_{E_{\lambda }}^{2\nu },\hfill \end{array}$$

(31)

where $\nu ={\displaystyle \frac{2_s^{*}-q}{2_s^{*}-2}}\in (0,1)$. Therefore, using $(f_1)$–$(f_2)$, it follows from (31), (10) and (27) that

$$\begin{array}{cc}\hfill m_{\lambda }& =I_{\lambda }(u_n)+o_n(1)\ge {\displaystyle \frac{1}{4}}\parallel u_n{\parallel }_{E_{\lambda }}^2-C|u_n{|}_q^q-C{|{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u}_n|}_2^{2_s^{*}}\hfill \\ \hfill & \ge {\displaystyle \frac{1}{4}}\parallel u_n{\parallel }_{E_{\lambda }}^2-C{\parallel u_n\parallel }_{E_{\lambda }}^{2\nu }-C,\hfill \end{array}$$

yielding that $(u_n)$ is uniformly bounded in $E_{\lambda }$ with respect to $n\in \mathbb{N}$ for all $\lambda >{\mathsf{\Lambda }}_0$ since $\xi \in (0,1)$. So, up to a subsequence if necessary, there is a $u\in E_{\lambda }$ such that $u_n\rightharpoonup u$ in $E_{\lambda }$, $u_n\to u$ in $L_{\mathrm{loc}}^p({\mathbb{R}}^3)$ for all $2<p<2_s^{*}$ and $u_n\to u$ a.e. in ${\mathbb{R}}^3$.

Secondly, we shall find a suitably large $\mathsf{\Lambda }>0$ such that $u\ne 0$ for all $\lambda >\mathsf{\Lambda }$. Owing to the above discussions, we know that $\parallel u_n{\parallel }_{E_{\lambda }}^2\le C^{*}$ for a suitable $C^{*}>0$, for any $n\ge 1$ and $\lambda >{\mathsf{\Lambda }}_0$. Let $r>2$ and ${\sigma }_0>0$ be given as in Lemma 7, recalling $(V_3)$, there is a sufficiently large constant $\overline{R}>1$ such that

$${\mathsf{\int }}_{B_{\overline{R}}^c(0)\cap \mathsf{\Sigma }}{|u_n|}^{r}dx\le {\displaystyle \frac{{\sigma }_0}{4}},\mathrm{for}\mathrm{all}\lambda >{\mathsf{\Lambda }}_0\mathrm{and}\mathrm{for}\mathrm{all}n\ge 1.$$

(32)

Since $V(x)\ge c$ on ${\mathsf{\Sigma }}^c$ by $(V_3)$, we have

$${\mathsf{\int }}_{B_{\overline{R}}^c(0)\cap {\mathsf{\Sigma }}^c}|u_n{|}^{2}dx\le {\displaystyle \frac{1}{\lambda c}}{\mathsf{\int }}_{B_{\overline{R}}^c(0)\cap {\mathsf{\Sigma }}^c}\lambda V(x){|u_n|}^{2}dx\le {\displaystyle \frac{C^{*}}{\lambda c}}$$

It easily infers that

$${\mathsf{\int }}_{B_{\overline{R}}^c(0)\cap {\mathsf{\Sigma }}^c}{|u_n|}^{r}dx\le {\left({\mathsf{\int }}_{B_{\overline{R}}^c(0)\cap {\mathsf{\Sigma }}^c}{|u_n|}^{2}dx\right)}^{{\displaystyle \frac{1}{2}}}{\left({\mathsf{\int }}_{B_{\overline{R}}^c(0)\cap {\mathsf{\Sigma }}^c}{|u_n|}^{2(r-1)}dx\right)}^{{\displaystyle \frac{1}{2}}},$$

and so one can find a $\mathsf{\Lambda }>{\mathsf{\Lambda }}_0$ such that

$${\mathsf{\int }}_{B_{\overline{R}}^c(0)\cap {\mathsf{\Sigma }}^c}{|u_n|}^{r}dx\le {\displaystyle \frac{{\sigma }_0}{4}},\mathrm{for}\mathrm{all}\lambda >\mathsf{\Lambda }\mathrm{and}\mathrm{for}\mathrm{all}n\ge 1.$$

(33)

Finally, we fix $\lambda >{\mathsf{\Lambda }}_0$, if $u_n\rightharpoonup u\equiv 0$, we can deduce that

$${\mathsf{\int }}_{B_{\overline{R}}(0)}{|u_n|}^{r}dx\le {\displaystyle \frac{{\sigma }_0}{4}},\mathrm{for}\mathrm{all}n\mathrm{sufficiently}\mathrm{large}.$$

(34)

Clearly, (32), (33) and (34) are in contradiction with Lemma 7.

Finally, we conclude that $u_n\to u$ along a subsequence as $n\to \infty$ for all $\lambda >\mathsf{\Lambda }$. Define $w_n\triangleq u_n-u$, then thanks to Lemmas 2-(3) and the Brézis–Lieb lemma,

$$\underset{n\to \infty }{lim}{I}_{\lambda }(w_n)=\underset{n\to \infty }{lim}\left[I_{\lambda }(u_n)-I_{\lambda }(u)\right]=m_{\lambda }-I_{\lambda }(u)$$

(35)

and

$$\underset{n\to \infty }{lim}{G}_{\lambda }(w_n)=\underset{n\to \infty }{lim}\left[G_{\lambda }(u_n)-G_{\lambda }(u)\right]=-G_{\lambda }(u).$$

(36)

We claim that $G_{\lambda }(u)\le 0$. Otherwise, it has that $\underset{n\to \infty }{lim}{G}_{\lambda }(w_n)<0$ by (36). Without loss of generality, we are assuming that $G_{\lambda }(w_n)<0$ for all $n\in \mathbb{N}$. From which, one knows that $w_n\ne 0$ and so Lemma 4 permits us to determine a ${\theta }_n>0$ such that $G_{\lambda }({(w_n)}_{{\theta }_n})=0$. Combining (22) and (35) and (36),

$$\begin{array}{c}{m}_{\lambda }-I_{\lambda }(u)+{\displaystyle \frac{1}{4s+2t-3}}{G}_{\lambda }(u)={\displaystyle \underset{n\to \infty }{lim}}\left[I_{\lambda }(w_n)-{\displaystyle \frac{1}{4s+2t-3}}{G}_{\lambda }(w_n)\right]\hfill \\ \ge {\displaystyle \underset{n\to \infty }{lim}}\left[I_{\lambda }\left({(w_n)}_{{\theta }_n}\right)-{\displaystyle \frac{{\theta }_n^{4s+2t-3}}{4s+2t-3}}{G}_{\lambda }(w_n)\right]>{\displaystyle \underset{n\to \infty }{lim}}{I}_{\lambda }({(w_n)}_{{\theta }_n})\ge m_{\lambda },\hfill \end{array}$$

which gives that

$$I_{\lambda }(u)-{\displaystyle \frac{1}{4s+2t-3}}{G}_{\lambda }(u)<0.$$

It is similar to (30) that we would obtain a contradiction. Hence, we have arrived at $G_{\lambda }(u)\le 0$. Adopting Lemma 4 again, there exists a $\theta >0$ such that $u_{\theta }\in {\mathcal{M}}_{\lambda }$. Owing to (22) and Fatou’s lemma,

$$\begin{array}{cc}\hfill m_{\lambda }& =\underset{n\to \infty }{lim}{I}_{\lambda }(u_n)=\underset{n\to \infty }{lim}\left[I_{\lambda }(u_n)-{\displaystyle \frac{1}{4s+2t-3}}{G}_{\lambda }(u_n)\right]\ge I_{\lambda }(u)-{\displaystyle \frac{1}{4s+2t-3}}{G}_{\lambda }(u)\hfill \\ \hfill & \ge I_{\lambda }(u_{\theta })-{\displaystyle \frac{{\theta }^{4s+2t-3}}{4s+2t-3}}{G}_{\lambda }(u)\ge I_{\lambda }(u_{\theta })\ge m_{\lambda },\hfill \end{array}$$

which yields that $u_n\to u$ in $E_{\lambda }$. Consequently, $I_{\lambda }(u)=m_{\lambda }$ and $G_{\lambda }(u)=0$. The proof is completed. □

4. Proof of Main Theorems

4.1. Proof of Theorem 1

Now, we are in a position to show the proof of Theorem 1.

The proof would be complete if u obtained in Lemma 8 satisfies $I_{\lambda }^{\prime }(u)=0$ in $E_{\lambda }^{-1}$. Motivated by [40], we argue it indirectly. If $I_{\lambda }^{\prime }(u)\ne 0$, there exists a $\phi \in C_0^{\infty }({\mathbb{R}}^3)$ such that $I_{\lambda }^{\prime }(u)\phi <-1$. Let $\epsilon >0$ be small enough and satisfy

$$I_{\lambda }^{\prime }(u_{\theta }+\tau \phi )\phi \le -{\displaystyle \frac{1}{2}},\mathrm{for}|\theta -1|+|\tau |\le \epsilon .$$

(37)

Let $\chi \in C_0^{\infty }(\mathbb{R},[0,1])$ be a cut-off function satisfying $\chi (\theta )\equiv 1$ for every $|\theta -1|\le {\displaystyle \frac{\epsilon }{2}}$ and $\chi (\theta )\equiv 0$ for all $|\theta -1|\ge \epsilon$. For any $\theta >0$, we define

$$\eta (\theta )\triangleq \left\{\begin{array}{cc}{u}_{\theta },\hfill & \mathrm{if}|\theta -1|\ge \epsilon ,\hfill \\ u_{\theta }+\epsilon \chi (\theta )\phi ,\hfill & \mathrm{if}|\theta -1|<\epsilon .\hfill \end{array}\right.$$

Obviously, $\eta \in \mathcal{C}(E_{\lambda })$ and one can fix $\epsilon >0$ sufficiently small such that ${\parallel \eta (\theta )\parallel }_{E_{\lambda }}>0$ for $|\theta -1|<\epsilon$. By (37), it is easy to show that

$$\underset{\theta >0}{max}{I}_{\lambda }(\eta (\theta ))<m_{\lambda }.$$

Proceeding as the proof of Lemma 4, we have $G_{\lambda }(\eta (1-\epsilon ))>0$ and $G_{\lambda }(\eta (1+\epsilon ))<0$. Since $G_{\lambda }(\eta (\theta ))$ is continuous, there exists ${\theta }_0\in (1-\epsilon ,1+\epsilon )$ such that $G_{\lambda }(\eta ({\theta }_0))=0$, which is $\eta ({\theta }_0)\in {\mathcal{M}}_{\lambda }$. Therefore, $m_{\lambda }\le I_{\lambda }(\eta ({\theta }_0))\le \underset{\theta >0}{max}{I}_{\lambda }(\eta (\theta ))<m_{\lambda }$, which is a contradiction. As to the positivity of u, it is standard and we omit it here. The proof is completed.

Next, we will deal with the concentrating behavior of ground state solutions obtained in Theorem 1. For any $u\in H_0^s(\mathsf{\Omega })$, we denote by $\tilde{u}\in H^s({\mathbb{R}}^3)$ its trivial extension, namely

$$\tilde{u}\triangleq \left\{\begin{array}{cc}u\hfill & \mathrm{in}\mathsf{\Omega },\hfill \\ 0\hfill & \mathrm{in}{\mathsf{\Omega }}^c=\{x:x\in {\mathbb{R}}^3\setminus \mathsf{\Omega }\}.\hfill \end{array}\right.$$

We now define $I_0{|}_{\mathsf{\Omega }}:H_0^s(\mathsf{\Omega })\to \mathbb{R}$ as

$$\begin{array}{cc}\hfill I_0{|}_{\mathsf{\Omega }}(u)& ={\displaystyle \frac{1}{2}}{\mathsf{\int }}_{\mathsf{\Omega }}{|(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u|}^{2}dx+{\displaystyle \frac{c_t}{4}}{\mathsf{\int }}_{\mathsf{\Omega }}{\mathsf{\int }}_{\mathsf{\Omega }}{\displaystyle \frac{u^2(x)u^2(y)}{{|x-y|}^{3-2t}}}dxdy-{\mathsf{\int }}_{\mathsf{\Omega }}f(u)udx-{\displaystyle \frac{1}{2_s^{*}}}{\mathsf{\int }}_{\mathsf{\Omega }}{|u|}^{2_s^{*}}dx\hfill \end{array}$$

and consider the minimization problem

$$m_0{|}_{\mathsf{\Omega }}\triangleq \underset{u\in {\mathcal{M}}_0{|}_{\mathsf{\Omega }}}{inf}{I_0|}_{\mathsf{\Omega }}(u)$$

where

$${\mathcal{M}}_0{|}_{\mathsf{\Omega }}=\{u\in H_0^s(\mathsf{\Omega })\setminus \left\{0\right\}:G_0{|}_{\mathsf{\Omega }}(u)=0\}$$

denotes the corresponding manifold and $G_0{|}_{\mathsf{\Omega }}:H_0^s(\mathsf{\Omega })\to \mathbb{R}$ is given by

$$\begin{array}{cc}\hfill G_0{|}_{\mathsf{\Omega }}(u)& ={\displaystyle \frac{4s+2t-3}{2}}{\mathsf{\int }}_{\mathsf{\Omega }}{|{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}u|}^{2}dx+{\displaystyle \frac{4s+2t-3}{4}}{c}_t{\mathsf{\int }}_{\mathsf{\Omega }}{\mathsf{\int }}_{\mathsf{\Omega }}{\displaystyle \frac{u^2(x)u^2(y)}{{|x-y|}^{3-2t}}}dxdy\hfill \\ \hfill & -{\mathsf{\int }}_{\mathsf{\Omega }}[(s+t)f(u)u-3F(u)]dx-{\displaystyle \frac{2_s^{*}(s+t)-3}{2_s^{*}}}{\mathsf{\int }}_{\mathsf{\Omega }}{|u|}^{2_s^{*}}dx.\hfill \end{array}$$

We note that up to the above trivial extension, there holds that ${\mathcal{M}}_0{|}_{\mathsf{\Omega }}\subset {\mathcal{M}}_{\lambda }$ for all $\lambda >0$.

For each $\lambda >{\mathsf{\Lambda }}_0$, we denote by $u_{\lambda }\in E_{\lambda }$ a ground state solution of system (1); that is, $I_{\lambda }^{\prime }(u_{\lambda })=0$ and $I_{\lambda }(u_{\lambda })=m_{\lambda }$. Then, we prove Theorem 2 as follows.

4.2. Proof of Theorem 2

Let ${\lambda }_n\to +\infty$ as $n\to +\infty$ and $(u_{{\lambda }_n})\subset E_{{\lambda }_n}$ be a sequence of ground state solutions of system (1); that is, $I_{{\lambda }_n}^{\prime }(u_{{\lambda }_n})=0$ and $I_{{\lambda }_n}(u_{{\lambda }_n})=m_{{\lambda }_n}$. Up to a subsequence if necessary, by (26) and ${\mathcal{M}}_0{|}_{\mathsf{\Omega }}\subset {\mathcal{M}}_{\lambda }$, for all $\lambda >0$,

$$0<\rho \le \underset{n\to \infty }{lim}{I}_{{\lambda }_n}(u_{{\lambda }_n})\triangleq {\tilde{m}}_{\mathsf{\Omega }}\le m_0{|}_{\mathsf{\Omega }}<+\infty .$$

(38)

Clearly, $(u_{{\lambda }_n})$ is bounded in $H^s({\mathbb{R}}^3)$. Thereby, up to a subsequence if necessary, there is a $u_0\in H^s({\mathbb{R}}^3)$ such that $u_{{\lambda }_n}\rightharpoonup u_0$ in $H^s({\mathbb{R}}^3)$ and $u_{{\lambda }_n}\to u_0$ a.e. in ${\mathbb{R}}^3$. By means of Lemmas 2-(3), we conclude that $I_0{|}_{\mathsf{\Omega }}^{\prime }(u_0)=0$. We claim that $u\equiv 0$ in ${\mathsf{\Omega }}^c$. Otherwise, there is a compact subset ${\Theta }_{u_0}\subset {\mathsf{\Omega }}^c$ with $\mathrm{dist}({\Theta }_{u_0},\partial {\mathsf{\Omega }}^c)>0$ such that $u_0\ne 0$ on ${\Theta }_{u_0}$ and by Fatou’s lemma

$$\underset{n\to \infty }{lim\; inf}{\mathsf{\int }}_{{\mathbb{R}}^3}{u}_n^{2}dx\ge {\mathsf{\int }}_{{\Theta }_{u_0}}{u_0}^{2}dx>0.$$

(39)

Moreover, there exists ${\epsilon }_0>0$ such that $V(x)\ge {\epsilon }_0$ for any $x\in {\Theta }_{u_0}$ by the assumptions $(V_1)$ and $(V_2)$. Combining $(f_4)$ with $\gamma >2$ and (38) and (39), we reach

$$\begin{array}{cc}\hfill c_{\mathsf{\Omega }}& \ge \underset{n\to \infty }{lim\; inf}\left\{{\displaystyle \frac{\gamma -2}{2\gamma }}{\mathsf{\int }}_{{\mathbb{R}}^2}{\lambda }_{n}V(x)u_n^{2}dx-{\displaystyle \frac{|\gamma -4|}{4\gamma }}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_{u_{{\lambda }_n}}^t{u}_{{\lambda }_n}^2-{\displaystyle \frac{2_s^{*}-\gamma }{2_s^{*}\gamma }}{\mathsf{\int }}_{{\mathbb{R}}^3}{|u_{{\lambda }_n}|}^{2_s^{*}}dx\right\}\hfill \\ \hfill & \ge {\displaystyle \frac{(q-2){\epsilon }_0}{2q}}{\mathsf{\int }}_{{\Theta }_u}{u_0}^{2}dx\underset{n\to \infty }{lim\; inf}{\lambda }_n-\widehat{C}=+\infty ,\hfill \end{array}$$

a contradiction, where $\widehat{C}>0$ is independent of $n\in \mathbb{N}$. Therefore, $u_0\in H_0^s(\mathsf{\Omega })$ by the fact that $\partial \mathsf{\Omega }$ is smooth and $I_0{|}_{\mathsf{\Omega }}^{\prime }(u_0)=0$. Similar to the proof of Lemma 8, one knows $u_0\ne 0$. Proceeding as the proof of Lemma 1, it holds that $G_0{|}_{\mathsf{\Omega }}(u_0)=0$. In view of (38), by $u_0\in H_0^s(\mathsf{\Omega })$, we use Fatou’s lemma to obtain

$$\begin{array}{cc}\hfill m_0{|}_{\mathsf{\Omega }}& \ge {\tilde{m}}_{\mathsf{\Omega }}=\underset{n\to \infty }{lim\; inf}\left[I_{{\lambda }_n}(u_{{\lambda }_n})-{\displaystyle \frac{1}{4s+2t-3}}{G}_{{\lambda }_n}(u_{{\lambda }_n})\right]\hfill \\ \hfill & \ge I_0{|}_{\mathsf{\Omega }}(u_0)-{\displaystyle \frac{1}{4s+2t-3}}{G}_0{|}_{\mathsf{\Omega }}(u_0)=I_0{{|}_{\mathsf{\Omega }}(u_0)\ge m_0|}_{\mathsf{\Omega }}\hfill \end{array}$$

yielding that $u_{{\lambda }_n}\to u_0$ in $H^s({\mathbb{R}}^3)$ and $I_0{{|}_{\mathsf{\Omega }}(u_0)=m_0|}_{\mathsf{\Omega }}$. The proof is finished.

4.3. Proof of Theorem 3

In this section, we are going to contemplate the existence of positive solutions for system (1) with a wider class of V and f. Without $(V_5)$ and $(f_5)$, one could not take advantage of the minimization constraint manifold method explored in Section 3. Whereas, because of $(f_4)$, it seems impossible to prove that the $(PS)$ sequence is uniformly bounded. As a consequence, we shall depend on an indirect approach developed by Jeanjean [20].

Proposition 1

(see ([20], Theorem 1.1 and Lemma 2.3)). Let $(X,\parallel ·\parallel )$ be a Banach space and $T\subset R^{+}$ be an interval, consider a family of $C^1$ functionals on X of the form

$${\Phi }_{\mu }(u)=A(u)-\mu B(u),\forall \mu \in T,$$

with $B(u)\ge 0$ and either $A(u)\to +\infty$ or $B(u)\to +\infty$ as $\parallel u\parallel \to +\infty$. Assume that there exists two points $v_1,v_2\in X$ such that

$$c_{\mu }=\underset{\gamma \in \mathsf{\Gamma }}{inf}\underset{\theta \in [0,1]}{sup}{\Phi }_{\mu }(\gamma (\theta ))>max\{{\Phi }_{\mu }(v_1),{\Phi }_{\mu }(v_1)\},\forall \mu \in T,$$

where

$$\mathsf{\Gamma }=\{\gamma \in C([0,1],X):\gamma (0)=v_1,\gamma (1)=v_2\}.$$

Then, for almost every $\mu \in T$, there is a sequence $(u_n(\mu ))\subset X$ such that

$(a)$

$(u_n(\mu ))$ is bounded in X;

$(b)$

${\Phi }_{\mu }(u_n(\mu ))\to c_{\mu }$ and ${\Phi }_{\mu }^{\prime }(u_n(\mu ))\to 0$;

$(c)$

the map $\mu \to c_{\mu }$ is non-increasing and left continuous.

Letting $T=[\delta ,1]$, where $\delta \in (0,1)$ is a positive constant. To apply Proposition 1, we will introduce a family of ${\mathcal{C}}^1$ functionals on $X=E_{\lambda }$ with the form

$$I_{\lambda ,\mu }(u)={\displaystyle \frac{1}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}{[|(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u|}^2+{\lambda V(x)|u|}^2]dx+{\displaystyle \frac{1}{4}}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_u^t{u}^{2}dx-\mu {\mathsf{\int }}_{{\mathbb{R}}^3}G(u)dx,$$

(40)

where and in the sequel $G(z)=F(z)+{\displaystyle \frac{1}{2_s^{*}}}{|z|}^{2_s^{*}}$ for all $z\in \mathbb{R}$. Define $I_{\lambda ,\mu }(u)=A(u)-\mu B(u)$, where

$$A(u)={\displaystyle \frac{1}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}{[|(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u|}^2+{\lambda V(x)|u|}^2]dx+{\displaystyle \frac{1}{4}}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_u^t{u}^{2}dx\to +\infty \mathrm{as}{\parallel u\parallel }_{E_{\lambda }}\to +\infty ,$$

and

$$B(u)={\mathsf{\int }}_{{\mathbb{R}}^3}G(u)dx\ge 0.$$

Clearly, $I_{\lambda ,\mu }$ is of class ${\mathcal{C}}^1$ functionals with

$$I_{\lambda ,\mu }^{\prime }(u)v={\mathsf{\int }}_{{\mathbb{R}}^3}[{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}u{(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}v+\lambda V(x)uv]dx+{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_u^{t}uvdx-\mu {\mathsf{\int }}_{{\mathbb{R}}^3}g(u)vdx$$

for all $u,v\in E_{\lambda }$, where $g(z)=f(z)+{|z|}^{2_s^{*}-2}z$ for all $z\in \mathbb{R}$.

For simplicity, from now on until the end of this section, we shall always suppose the assumptions in Theorem 3 when there is no misunderstanding.

Lemma 9.

The functional $I_{\lambda ,\mu }$ possesses a mountain-pass geometry, that is,

$(a)$

There exists $v\in E_{\lambda }\setminus \left\{0\right\}$ independent of μ such that $I_{\lambda ,\mu }(v)\le 0$ for all $\mu \in [\delta ,1]$;

$(b)$

$c_{\lambda ,\mu }\triangleq \underset{\eta \in \Gamma }{inf}\underset{\theta \in [0,1]}{sup}{I}_{\lambda ,\mu }(\gamma (\eta ))>max\{I_{\lambda ,\mu }(0),I_{\lambda ,\mu }(v)\}$ for all $\mu \in [\delta ,1]$, where

$$\mathsf{\Gamma }=\{\eta \in \mathcal{C}([0,1],E_{\lambda }):\eta (0)=0,\eta (1)=v\}.$$

Proof. 

The proof is very similar to the calculations on finding the existence of critical points in the proof of Lemma 4, so we omit the details. □

Repeating the arguments explored in Lemma 6, there is a constant $\widehat{\rho }>0$ such that

$$\widehat{\rho }\le \underset{\lambda >{\mathsf{\Lambda }}_0}{inf}{c}_{\lambda ,\mu }\le \underset{\lambda >{\mathsf{\Lambda }}_0}{sup}{c}_{\lambda ,\mu }<{\displaystyle \frac{s}{3{\mu }^{\frac{3-2s}{2s}}}}{S}_s^{{\displaystyle \frac{3}{2s}}},\forall \mu \in [\delta ,1].$$

(41)

Lemma 10.

Let $(u_n)$ be a bounded $(PS)$ sequence of the functional $I_{\lambda ,\mu }$ at the level $c>0$, then for each ${\displaystyle \widehat{M}\in \left(c,\frac{s}{3{\mu }^{\frac{3-2s}{2s}}}{S}_s^{\frac{3}{2s}}\right)}$, there exists a $\widehat{\mathsf{\Lambda }}=\mathsf{\Lambda }(\widehat{M})>0$ such that $(u_n)$ contains a strongly convergent subsequence in $E_{\lambda }$ for all $\lambda >\widehat{\mathsf{\Lambda }}$.

Proof. 

Since $(u_n)$ is bounded in $E_{\lambda }$, then there exists a $u\in E_{\lambda }$ such that $u_n\rightharpoonup u$ in $E_{\lambda }$, $u_n\to u$ in $L_{\mathrm{loc}}^p({\mathbb{R}}^3)$ with $p\in [1,2_s^{*})$ and $u_n\to u$ $a.e.$ in ${\mathbb{R}}^3$. To show the proof clearly, we shall split it into several steps:

Step 1: $I_{\lambda ,\mu }^{\prime }(u)=0$ and $I_{\lambda ,\mu }(u)\ge 0$.

To show $I_{\lambda }^{\prime }(u)=0$, since $C_0^{\infty }({\mathbb{R}}^3)$ is dense in $E_{\lambda }$, then it suffices to exhibit that $I_{\lambda ,\mu }^{\prime }(u)\phi =0$ for every $\phi \in C_0^{\infty }({\mathbb{R}}^3)$. Thanks to Lemma 2-(3), it is a direct conclusion. Because u is a critical point of $I_{\lambda ,\mu }$, according to Lemma 1, there holds $P_{\lambda ,\mu }(u)\equiv 0$, where

$$\begin{array}{cc}\hfill P_{\lambda ,\mu }(u)& \triangleq {\displaystyle \frac{3-2s}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u|}^{2}dx+{\displaystyle \frac{1}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}[3V(x)+(\nabla V,x)]{|u|}^{2}dx+{\displaystyle \frac{2t+3}{4}}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_u^t{u}^{2}dx\hfill \\ \hfill & -3\mu {\mathsf{\int }}_{{\mathbb{R}}^3}F(u)dx-{\displaystyle \frac{\mu }{2_s^{*}}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|u|}^{2_s^{*}}dx.\hfill \end{array}$$

Moreover, one easily sees that $I_{\lambda ,\mu }^{\prime }(u)u=0$ and so

$$I_{\lambda ,\mu }(u)=I_{\lambda ,\mu }(u)-{\displaystyle \frac{1}{(s+t)\gamma -3}}\left[(s+t)I_{\lambda ,\mu }^{\prime }(u)u-P_{\lambda .\mu }(u)\right]\ge 0$$

proving the Step 1.

Step 2: Define $v_n\triangleq u_n-u$, then there exists a $\widehat{\mathsf{\Lambda }}=\mathsf{\Lambda }(\widehat{M})>0$ such that $v_n\to 0$ in $L^q({\mathbb{R}}^3)$ for all $q\in (2,2_s^{*})$ along a subsequence as $n\to \infty$ when $\lambda >\widehat{\mathsf{\Lambda }}$.

Actually, since $(v_n)$ is uniformly bounded in $n\in \mathbb{N}$ for all $\lambda >{\mathsf{\Lambda }}_0$, then we have one of the following two possibilities for some $r>0$:

$$\left\{\begin{array}{c}{\displaystyle (\mathrm{i})\underset{n\to \infty }{lim}\underset{y\in {\mathbb{R}}^3}{sup}{\mathsf{\int }}_{B_r(y)}{|v_n|}^{2}dx>0,}\hfill \\ {\displaystyle (\mathrm{ii})\underset{n\to \infty }{lim}\underset{y\in {\mathbb{R}}^3}{sup}{\mathsf{\int }}_{B_r(y)}{|v_n|}^{2}dx=0.}\hfill \end{array}\right.$$

As a consequence, the conclusion would be clear if we could demonstrate that the case (i) cannot occur for sufficiently large $\lambda >0$. Now, we suppose, by contradiction, that (i) was true. Proceeding as the very similar way in Lemma 8, there is a constant $\widehat{\delta }>0$ independent of $\lambda >{\mathsf{\Lambda }}_0$ such that

$$\underset{n\to \infty }{lim}\underset{y\in {\mathbb{R}}^3}{sup}{\mathsf{\int }}_{B_r(y)}{|v_n|}^{2}dx\ge \widehat{\delta }$$

for some $r>0$. Since $(u_n)$ is uniformly bounded in $E_{\lambda }$, without loss of generality, we can assume that $\underset{n\to \infty }{lim}{\parallel u_n\parallel }_{E_{\lambda }}^2\le \Theta$ for some $\Theta \in (0,+\infty )$. Clearly, there holds $\underset{n\to \infty }{lim}{\parallel v_n\parallel }_{E_{\lambda }}^2\le 4\Theta$. Recalling $v_n\to 0$ in $L_{\mathrm{loc}}^q({\mathbb{R}}^3)$ with $q\in (2,2_s^{*})$ and $|{\mathcal{A}}_R|\to 0$ as $R\to +\infty$ by $(V_2)$, where ${\mathcal{A}}_R\triangleq \{x\in {\mathbb{R}}^3\setminus B_R(0):V(x)<c\}$, we can determine a sufficiently large but fixed $R>0$ to satisfy

$$\underset{n\to \infty }{lim\; sup}{\mathsf{\int }}_{B_R(0)}{|v_n|}^{2}dx<{\displaystyle \frac{\widehat{\delta }}{4}}$$

(42)

and

$$|{\mathcal{A}}_R|<{\left({\displaystyle \frac{\widehat{\delta }{S}_s}{16\Theta }}\right)}^{{\displaystyle \frac{q}{q-2}}}{|\mathsf{\Sigma }|}^{-{\displaystyle \frac{2(2_s^{*}-q)}{2_s^{*}(q-2)}}}.$$

(43)

Combining (12) and (43), one sees that

$$\underset{n\to \infty }{lim\; sup}{\mathsf{\int }}_{{\mathcal{A}}_R}|v_n{|}^{2}dx\le \underset{n\to \infty }{lim\; sup}{\left({\mathsf{\int }}_{{\mathcal{A}}_R}{|v_n|}^{q}dx\right)}^{{\displaystyle \frac{2}{q}}}|{\mathcal{A}}_R{|}^{{\displaystyle \frac{q-2}{q}}}\le {4\Theta |\mathsf{\Sigma }|}^{{\displaystyle \frac{2(2_s^{*}-q)}{2_s^{*}q}}}{S}_s^{-1}{|{\mathcal{A}}_R|}^{{\displaystyle \frac{q-2}{q}}}<{\displaystyle \frac{\widehat{\delta }}{4}}.$$

(44)

Let us choose $\widehat{\mathsf{\Lambda }}=max\left\{1,{\mathsf{\Lambda }}_0,{\displaystyle \frac{16\Theta }{\widehat{\delta }c}}\right\}$, then for all $\lambda >\widehat{\mathsf{\Lambda }}$, we reach

$$\underset{n\to \infty }{lim\; sup}{\mathsf{\int }}_{{\mathcal{B}}_R}|v_n{|}^{2}dx\le \underset{n\to \infty }{lim\; sup}{\displaystyle \frac{1}{\lambda c}}{\mathsf{\int }}_{{\mathcal{B}}_R}\lambda V(x){|v_n|}^{2}dx\le {\displaystyle \frac{4\Theta }{\lambda c}}<{\displaystyle \frac{\widehat{\delta }}{4}},$$

(45)

where ${\mathcal{B}}_R\triangleq \{x\in {\mathbb{R}}^3\setminus B_R(0):V(x)\ge c\}$. We gather (42), (43) and (45) to derive

$$\begin{array}{cc}\hfill \widehat{\delta }& \le \underset{n\to \infty }{lim}\underset{y\in {\mathbb{R}}^3}{sup}{\mathsf{\int }}_{B_r(y)}|v_n{|}^{2}dx\le \underset{n\to \infty }{lim\; sup}{\mathsf{\int }}_{{\mathbb{R}}^3}{|v_n|}^{2}dx\hfill \\ \hfill & =\underset{n\to \infty }{lim\; sup}\left({\mathsf{\int }}_{{\mathbb{R}}^3\setminus B_R(0)}|v_n{|}^{2}dx+{\mathsf{\int }}_{B_R(0)}{|v_n|}^{2}dx\right)\le {\displaystyle \frac{3\widehat{\delta }}{4}}\hfill \end{array}$$

which is impossible. The proof of this step is completed.

Step 3: Passing to a subsequence if necessary, $u_n\to u$ in $E_{\lambda }$ as $n\to \infty$. Since $v_n\triangleq u_n-u$, by Lemma 2-(3) and the Brézis–Lieb lemma, one has

$$I_{\lambda ,\mu }(v_n)=I_{\lambda ,\mu }(u_n)-I_{\lambda ,\mu }(u)+o_n(1)\mathrm{and}{I}_{\lambda ,\mu }^{\prime }(v_n)=I_{\lambda ,\mu }^{\prime }(u_n)+o_n(1).$$

(46)

According to Step 2, we take advantage of (14) and $(f_1)$–$(f_2)$ to deduce that

$$\underset{n\to \infty }{lim}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_{v_n}^t{v}_n^{2}dx=0\mathrm{and}\underset{n\to \infty }{lim}{\mathsf{\int }}_{{\mathbb{R}}^3}f(v_n)v_{n}dx=0$$

jointly with Lemma 2-(3) and the Brézis–Lieb lemma indicate that

$$o_n(1)=I_{\lambda }^{\prime }(u_n)(u_n-u)-I_{\lambda }^{\prime }(u)(u_n-u)=\parallel v_n{\parallel }_{E_{\lambda }}^2-\mu {|v_n|}_{2_s^{*}}^{2_s^{*}}.$$

Let us suppose that $\parallel v_n{\parallel }_{E_{\lambda }}^2\to l$ and $\mu |v_n{|}_{2_s^{*}}^{2_s^{*}}\to l$ along some subsequences and so

$$c\ge c-I_{\lambda }(u)=\underset{n\to \infty }{lim}{I}_{\lambda }(v_n)=\left({\displaystyle \frac{1}{2}}-{\displaystyle \frac{1}{2_s^{*}}}\right)l,$$

(47)

where we have used Step 1 and (46). In view of (10), it holds that

$${({\mu }^{-1}l)}^{{\displaystyle \frac{2}{2_s^{*}}}}\le S_s^{-1}l.$$

(48)

If $l\ne 0$, that is, $l>0$, then $l\ge {\mu }^{-{\displaystyle \frac{3-2s}{2s}}}{S}_s^{{\displaystyle \frac{3}{2s}}}$ by (48). As a consequence, with the help of (47), we arrive at $c\ge {\displaystyle \frac{s}{3{\mu }^{\frac{3-2s}{2s}}}}{S}_s^{{\displaystyle \frac{3}{2s}}}$, a contradiction. Therefore, $l=0$ which is the desired result. The proof is completed. □

Let us recall Proposition 1 and Lemmas 9 and 10; there are two sequences $({\mu }_n)\subset [\delta ,1]$ and $(u_n)\subset E_{\lambda }\setminus \left\{0\right\}$ such that

$$I_{\lambda ,{\mu }_n}^{\prime }(u_n)=0,I_{\lambda ,{\mu }_n}(u_n)=c_{\lambda ,{\mu }_n}\mathrm{and}{\mu }_n\to 1^{-}.$$

(49)

With (49) in hand, we are able to derive the proof of Theorem 3.

Proof of Theorem 3.

First of all, since $I_{\lambda ,{\mu }_n}^{\prime }(u_n)=0$, we are derived from a similar argument in Lemma 1 that $P_{\lambda ,{\mu }_n}(u_n)\equiv 0$, where

$$\begin{array}{cc}\hfill P_{\lambda ,{\mu }_n}(u)& \triangleq {\displaystyle \frac{3-2s}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|(-\mathsf{\Delta })}^{{\displaystyle \frac{s}{2}}}{u}_n{|}^{2}dx+{\displaystyle \frac{1}{2}}{\mathsf{\int }}_{{\mathbb{R}}^3}[3V(x)+(\nabla V,x)]{|u_n|}^{2}dx+{\displaystyle \frac{2t+3}{4}}{\mathsf{\int }}_{{\mathbb{R}}^3}{\varphi }_{u_n}^t{u}_n^{2}dx\hfill \\ \hfill & -3{\mu }_n{\mathsf{\int }}_{{\mathbb{R}}^3}F(u_n)dx-{\displaystyle \frac{{\mu }_n}{2_s^{*}}}{\mathsf{\int }}_{{\mathbb{R}}^3}{|u|}^{2_s^{*}}dx.\hfill \end{array}$$

Proceeding as the proof of Lemma 8, one sees that $(u_n)$ is uniformly bounded in $E_{\lambda }$ for all $\lambda >{\mathsf{\Lambda }}_0$.

Then, we claim that $(u_n)$ is a ${(PS)}_{c_{\lambda ,1}}$ sequence of the functional $I_{\lambda }=I_{\lambda ,1}$. Actually, taking into account ${\mu }_n\to 1^{-}$ and Proposition 1-(c),

$$\underset{n\to \infty }{lim}{I}_{\lambda ,1}(u_n)=\left(\underset{n\to \infty }{lim}{I}_{\lambda ,{\mu }_n}(u_n)+({\mu }_n-1){\mathsf{\int }}_{{\mathbb{R}}^3}G(u_n)dx\right)=\underset{n\to \infty }{lim}{c}_{\lambda ,{\mu }_n}=c_{\lambda ,1},$$

where we have used the fact that $(G(u_n))$ is uniformly bounded in $L^1({\mathbb{R}}^3)$. Similarly,

$$\begin{array}{cc}\hfill \underset{n\to \infty }{lim}{\displaystyle \frac{|I_{\lambda ,1}^{\prime }(u_n)\psi |}{{\parallel \psi \parallel }_{E_{\lambda }}}}& =\underset{n\to \infty }{lim}{\displaystyle \frac{|I_{\lambda ,{\mu }_n}^{\prime }(u_n)\psi +({\mu }_n-1){\mathsf{\int }}_{{\mathbb{R}}^3}g(u_n)\psi dx|}{{\parallel \psi \parallel }_{E_{\lambda }}}}\hfill \\ \hfill & \le \underset{n\to \infty }{lim}{\displaystyle \frac{|{\mu }_n-1||{\mathsf{\int }}_{{\mathbb{R}}^3}g(u_n)\psi dx|}{{\parallel \psi \parallel }_{E_{\lambda }}}}=0,\forall \psi \in E_{\lambda }.\hfill \end{array}$$

As a consequence, one has that $(u_n)$ is a ${(PS)}_{c_{\lambda ,1}}$ sequence of the functional $I_{\lambda }=I_{\lambda ,1}$.

Finally, combining the above two steps and (41), we can apply Lemma 10 to finish the proof. □

5. Conclusions

In this paper, we have considered the existence and concentrating behavior of positive solutions for the following fractional Schrödinger–Poisson system with critical growth

$$\left\{\begin{array}{cc}{(-\mathsf{\Delta })}^{s}u+\lambda V(x)u+\varphi u=f(u)+{|u|}^{2_s^{*}-2}u,\hfill & x\in {\mathbb{R}}^3,\hfill \\ {(-\mathsf{\Delta })}^t\varphi =u^2,\hfill & x\in {\mathbb{R}}^3,\hfill \end{array}\right.$$

where $s,t\in (0,1)$ with $2s+2t>3$, $\lambda >0$ denotes a parameter, $V:{\mathbb{R}}^3\to \mathbb{R}$ admits a potential well $\mathsf{\Omega }\triangleq \mathrm{int}{V}^{-1}(0)$ and $2_s^{*}\triangleq {\displaystyle \frac{6}{3-2s}}$ is the fractional Sobolev critical exponent. Combining the constrained manifold argument and minimax techniques, we introduce some new analytic tricks to prove that the system possesses a positive ground state solution and a mountain-pass type solution, respectively. Actually, what we want to mention here is that the restrictions on V and f play some crucial roles in the existence of solutions. Furthermore, we believe that the studies in this paper would prompt related research on fractional Schrödinger–Poisson systems.