Ground State Solutions of Fractional Choquard Problems with Critical Growth

Abstract

In this article, we investigate a class of fractional Choquard equation with critical Sobolev exponent. By exploiting a monotonicity technique and global compactness lemma, the existence of ground state solutions for this equation is obtained. In addition, we demonstrate the existence of ground state solutions for the corresponding limit problem.

1. Introduction

In this paper, we study the following fractional Choquard equation

$$\begin{array}{c}\hfill {(-\mathrm{\Delta })}^{\alpha }u+K\left(x\right)u=(I_{\beta }\ast G\left(u\right))g\left(u\right)+{\left|u\right|}^{2_{\alpha }^{*}-2}u,x\in {\mathbb{R}}^3,\end{array}$$

(1)

where $\alpha \in (\frac{3}{4},1),\beta \in (2\alpha ,3),2_{\alpha }^{*}=\frac{6}{3-2\alpha }$. ${(-\mathrm{\Delta })}^{\alpha }$ is defined as

$$\begin{array}{c}\hfill {(-\mathrm{\Delta })}^{\alpha }u\left(x\right):=C_{\alpha }P.V.{\int }_{{\mathbb{R}}^3}\frac{u\left(x\right)-u\left(y\right)}{{|x-y|}^{3+2\alpha }}dy,x\in {\mathbb{R}}^3,\end{array}$$

where $C_{\alpha }={\left({\int }_{{\mathbb{R}}^3}\frac{1-cos{\zeta }_1}{{\left|\zeta \right|}^{3+2\alpha }}d\zeta \right)}^{-1}$ and $P.V.$ is an abbreviation for Cauchy principal value. The Riesz potential $I_{\beta }:{\mathbb{R}}^3\setminus \left\{0\right\}\to \mathbb{R}$ is defined by

$$\begin{array}{c}\hfill I_{\beta }\left(x\right):=\frac{\mathrm{\Gamma }\left(\frac{3-\beta }{2}\right)}{\mathrm{\Gamma }\left(\frac{\beta }{2}\right){\pi }^{\frac{3}{2}}{2}^{\beta }{\left|x\right|}^{3-\beta }},x\in {\mathbb{R}}^3\setminus \left\{0\right\}.\end{array}$$

Let us state some hypothesises on K and g:

(K₁)

$K\in {\mathcal{C}}^1\left({\mathbb{R}}^3\right)\cap L^{\infty }\left({\mathbb{R}}^3\right)$ and $\alpha K\left(x\right)+(\nabla K(x),x)⩾0$ for any $x\in {\mathbb{R}}^3$;

(K₂)

$K\left(x\right)⩽\underset{\left|y\right|\to +\infty }{lim\; inf}K\left(y\right)=K_{\infty }\in {\mathbb{R}}^{+}$ for all $x\in {\mathbb{R}}^3$ and the inequality is strict in a subset of positive Lebesgue measure;

(K₃)

there is a positive constant $a_0$ such that

$$\begin{array}{c}\hfill a_0=\underset{u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}}{inf}\frac{{\int }_{{\mathbb{R}}^3}\left({|(-\mathrm{\Delta })}^{\frac{\alpha }{2}}{u|}^2+K\left(x\right){\left|u\right|}^2\right)dx}{{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2}dx}>0;\end{array}$$

(H₁)

$g\in \mathcal{C}(\mathbb{R},\mathbb{R}),G\left(\tau \right)={\int }_0^{\tau }g\left(u\right)du\ge 0$ for all $\tau \in \mathbb{R}$ and there exits $C_0>0$ and $1+\frac{\beta }{3}<q<2_{\beta ,\alpha }^{*}$ such that for every $\tau \in \mathbb{R}$,

$$\begin{array}{c}\hfill \left|g\left(\tau \right)\right|⩽C_0{\left(\right|\tau |}^{\frac{\beta }{3}}+{\left|\tau \right|}^{q-1}),\end{array}$$

where $2_{\beta ,\alpha }^{*}=\frac{\beta +3}{3-2\alpha }$;

(H₂)

$\underset{\left|\tau \right|\to 0}{lim}\frac{g\left(\tau \right)}{{\tau }^{\frac{\beta }{3}}}=\underset{\left|\tau \right|\to +\infty }{lim}\frac{g\left(\tau \right)}{{\tau }^{2_{\beta ,\alpha }^{*}-1}}=0$;

(H₃)

$[4\alpha g\left(\tau \right)\tau -(3+\beta )G\left(\tau \right)]/{\tau \left|\tau \right|}^{(\beta +6-4\alpha )/4\alpha }$ is non-decreasing on both $(0,+\infty )$ and $(-\infty ,0)$;

(H₄)

there exist $\nu >0$ and $p\in (2,2_{\beta ,\alpha }^{*})$ such that $g\left(\tau \right)⩾\nu {\tau }^{p-1}$ for any $\tau ⩾0$.

When $\alpha =1,G\left(u\right)={\left|u\right|}^p,K\left(x\right)\equiv 1$, Equation (1) is known as following Choquard equation

$$\begin{array}{c}\hfill -\mathrm{\Delta }u+u=(I_{\beta }{\ast \left|u\right|}^p{\left)\right|u|}^{p-2}u+f\left(u\right),u\in H^1\left({\mathbb{R}}^3\right).\end{array}$$

(2)

For the case $\beta =2,p=2$ and $f\left(u\right)=0$, Equation (2) is called the Choquard-Pekar equation. It was first proposed by Pekar [1]. Later, Choquard rediscovered it as an approximation of Hartree-Fock’s theory of single-component plasma [2]. For more physical interpretation, one can refer to Penrose [3], Diosi [4], Lewin-Nam-Rougerie [5,6] and Jones [7,8]. Moroz-Van Schaftingen [9] obtained a solution to (2) when $1+\frac{\beta }{3}<p<3+\beta$, where $3+\beta$ and $1+\frac{\beta }{3}$ are the upper and lower critical exponents. Gao-Yang [10] investigated the existence of solution for the upper critical exponent case. Yao-Chen-Radulescu-Sun [11] studied the existence of results for the double critical case. Du-Gao-Yang [12] and Yang-Radulescu-Zhou [13] considered the existence and properties of solutions with Stein-Weiss type nonlinearities. Moroz-Van Schaftingen [14] investigated the semi-classical states for $p⩾2$, and set the rest case $p<2$ for open problem. Cingolani-Tanaka [15] given a pisitive answer to the open problem. Furthermore, Su-Liu [16,17] extended the semi-classical results to the upper critical case. Afterwards, these results were extended to the more general function G or the more general potential K, see [18,19,20,21,22,23,24,25,26,27,28].

Recently, researchers are increasingly concerned the existence results of fractional Choquard equation

$$\begin{array}{c}\hfill {(-\mathrm{\Delta })}^{\alpha }u+K\left(x\right)u=(I_{\beta }{\ast \left|u\right|}^p{\left)\right|u|}^{p-2}u,\mathrm{in}{\mathbb{R}}^3,\end{array}$$

where $\frac{3+\beta }{3}<p<\frac{3+\beta }{3-2\alpha }$. For instance, Shen-Gao-Yang [29] established the existence of non-negative ground state solution by supposing that the nonlinearities satisfied the general Berestycki-Lions type conditions. Su et al. [30] studied the existence of results for the double critical case. Li, Zhang, Wang and Teng [31] considered

$$\begin{array}{c}\hfill {(-\mathrm{\Delta })}^{\alpha }u+K\left(x\right)u={\left[\right|x|}^{-\beta }{\ast G\left(u\right)]g\left(u\right)+\lambda [\left|x\right|}^{-\beta }{\ast \left|u\right|}^p{\left]p\right|u|}^{p-2}u,\mathrm{in}{\mathbb{R}}^N,\end{array}$$

(3)

where $0<\alpha <1,N>2\alpha ,0<\beta <2\alpha ,$ and $p\ge 2_{\beta ,\alpha }^{*}$. They obtained that there existed ground state solutions to Equation (3) with critical or supercritical growth by the variational methods.

We notice that the work in the above literature focuses on the existence of ground state solutions to fractional Choquard equations with a Hardy-Littlewood-Sobolev critical exponent. We determine to investigate a class of fractional order Choquard equations with a critical local term. Inspired by the above work, we first deal with the case of a constant potential. Specifically, by taking a minimizing sequence from a Nehari-Pohožaev manifold, we obtain a minimum value and prove that the minimum value is a critical point. Therefore, we prove the existence of the Nehari-Pohožaev type ground state solution for the fractional Choquard equation with a constant potential. On this basis, we use a monotonicity technique and a global compactness lemma to obtain the existence of a Nehari-Pohožaev type ground state solution to Equation (1). As we’ve seen there is nearly not any result for the existence of nonnegative least energy solutions for the fractional Choquard Equation (1) with critical growth.

Therefore, Let’s first study the limit problem

$$\begin{array}{c}\hfill {(-\mathrm{\Delta })}^{\alpha }u+K_{\infty }u=(I_{\beta }\ast G\left(u\right))g\left(u\right)+{\left|u\right|}^{2_{\alpha }^{*}-2}u,x\in {\mathbb{R}}^3.\end{array}$$

(4)

We obtain the following result.

Theorem 1. 

Assume that $\alpha \in (\frac{3}{4},1)$, $\beta \in (2\alpha ,3)$, $\left(H_1\right)$–$\left(H_4\right)$ hold.

(i) 

If $p\in (2,2_{\alpha }^{*}-1)$, there exists ${\nu }_1>0$ such that for $\nu >{\nu }_1$, Equation (4) has a ground state solution.

(ii) 

If $p\in (2_{\alpha }^{*}-1,2_{\beta ,\alpha }^{*})$, for any $\nu >0$, Equation (4) has a ground state solution.

Then, we can obtain the next main result.

Theorem 2. 

Assume that $\alpha \in (\frac{3}{4},1)$, $\beta \in (2\alpha ,3)$, $\left(K_1\right)$–$\left(K_3\right)$ and $\left(H_1\right)$–$\left(H_4\right)$ hold.

(i) 

If $p\in (2,2_{\alpha }^{*}-1)$, there exists ${\nu }_1>0$ such that for $\nu >{\nu }_1$, Equation (1) has a ground state solution.

(ii) 

If $p\in (2_{\alpha }^{*}-1,2_{\beta ,\alpha }^{*})$, for any $\nu >0$, Equation (1) has a ground state solution.

Remark 1. 

We will use a global compactness lemma, Jeanjean’s monotonicity trick, and a general minimax principle to prove the main results. There are some troubles in proving the main theorems. The first trouble is that f doesn’t satisfy Ambrosetti-Rabinowtiz condition, we can’t use the Nehari manifold to obtain the ground state solution of Equation (1). Moreover, It is difficult to acquire the boundedness of (PS) sequence. To conquer it, we shall use Jeanjean’s monotonicity technique [32] and establish the Pohožaev identity. The second problem is the lack of the compactness induced by the critical term. We will use some new estimates to obtain a global compactness lemma to overcome this difficulty. Because of the fractional Laplace operator and convolutional nonlinearity, these estimates are complex. Moreover, the potential $K\left(x\right)$ is not a constant, we consider the limit problem of Equation (1).

The rest of this paper is organized as follows. In Section 2, we introduce some preliminaries results. In Section 3, we investigate the limit problem. Section 4 is devoted to the existence of ground state solutions to Equation (1). Section 5 is a brief conclusion of this paper.

2. Preliminaries

The fractional Sobolev space ${\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)$ is defined by

$$\begin{array}{c}\hfill {\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right):=\left\{u\in L^{2_{\alpha }^{*}}\left({\mathbb{R}}^3\right):\frac{\left|u\right(x)-u(y\left)\right|}{{|x-y|}^{\frac{3+2\alpha }{2}}}\in L^2\left({\mathbb{R}}^3\times {\mathbb{R}}^3\right)\right\}\end{array}$$

with the norm

$$\begin{array}{c}\hfill {\parallel u\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}:={\left({\int }_{{\mathbb{R}}^3}{\left|{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u\right|}^{2}dx\right)}^{\frac{1}{2}}.\end{array}$$

It follows from Propositions 3.4 and 3.6 in [33] that

$$\begin{array}{c}\hfill {\int }_{{\mathbb{R}}^3}{\left|{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u\right|}^{2}dx=\frac{1}{C\left(\alpha \right)}{\int }_{{\mathbb{R}}^3}{\int }_{{\mathbb{R}}^3}\frac{{|u\left(x\right)-u\left(y\right)|}^2}{{|x-y|}^{3+2\alpha }}dxdy.\end{array}$$

It is well known that the embedding ${\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)\hookrightarrow L^{2_{\alpha }^{*}}\left({\mathbb{R}}^3\right)$ is continuous.The constant $S_{\alpha }$ is defined as

$$\begin{array}{c}\hfill S_{\alpha }=\underset{u\in {\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}}{inf}\frac{{\int }_{{\mathbb{R}}^3}{\left|{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u\right|}^{2}dx}{{(\int }_{{\mathbb{R}}^3}{\left|u\left(x\right)\right|}^{2_{\alpha }^{*}}{dx)}^{\frac{2}{2_{\alpha }^{*}}}}.\end{array}$$

(5)

The fractional Sobolev space $H^{\alpha }\left({\mathbb{R}}^3\right)$ is defined by

$$\begin{array}{c}\hfill H^{\alpha }\left({\mathbb{R}}^3\right):=\left\{u\in L^2\left({\mathbb{R}}^3\right):\frac{\left|u\right(x)-u(y\left)\right|}{{|x-y|}^{\frac{3+2\alpha }{2}}}\in L^2({\mathbb{R}}^3\times {\mathbb{R}}^3)\right\}\end{array}$$

endowed with the norm

$$\begin{array}{c}\hfill {\parallel u\parallel }_{H^{\alpha }\left({\mathbb{R}}^3\right)}={\left[{\int }_{{\mathbb{R}}^3}\left({|(-\mathrm{\Delta })}^{\frac{\alpha }{2}}{u|}^2+u^2\right)dx\right]}^{\frac{1}{2}}.\end{array}$$

Define the work space of Equation (1) by

$$\begin{array}{c}\hfill H_K^{\alpha }\left({\mathbb{R}}^3\right):=\left\{u\in H^{\alpha }\left({\mathbb{R}}^3\right):{\int }_{{\mathbb{R}}^3}K\left(x\right){\left|u\right|}^{2}dx<+\infty \right\},\end{array}$$

with the inner product

$$\begin{array}{c}\hfill \langle u,v\rangle :={\int }_{{\mathbb{R}}^3}{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}vdx+{\int }_{{\mathbb{R}}^3}K\left(x\right)uvdx,\end{array}$$

and the norm

$$\begin{array}{c}\hfill {\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}:={\left({\int }_{{\mathbb{R}}^3}{\left|{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u\right|}^{2}dx+{\int }_{{\mathbb{R}}^3}K\left(x\right)u^{2}dx\right)}^{\frac{1}{2}}.\end{array}$$

From $\left(K_2\right)$–$\left(K_3\right)$, the norms ${\parallel ·\parallel }_{H^{\alpha }\left({\mathbb{R}}^3\right)}$ and ${\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}$ are equivalent.

From $\left(H_1\right)$, Hardy-Littlewood-Sobolev inequality [34] and Sobolev embedding theorem, we can conclude that

$$\begin{array}{cc}\hfill \left|{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))G\left(u\right)dx\right|& \le {C\parallel G\left(u\right)\parallel }_{L^{\frac{6}{3+\beta }}\left({\mathbb{R}}^3\right)}^2\hfill \\ \hfill & \le C{\left[{\int }_{}{\left({\left|u\right|}^{\frac{3+\beta }{3}}+{\left|u\right|}^q\right)}^{\frac{6}{3+\beta }}\right]}^{\frac{3+\beta }{3}}\hfill \\ \hfill & \le C\left({\parallel u\parallel }_{L^2\left({\mathbb{R}}^3\right)}^{\frac{2(3+\beta )}{3}}+{\parallel u\parallel }_{L^{\frac{6q}{3+\beta }}\left({\mathbb{R}}^3\right)}^{2q}\right),\hfill \end{array}$$

(6)

and

$$\begin{array}{cc}\hfill & \left|{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))g\left(u\right)vdx\right|\hfill \\ \hfill & \le C\left(\beta \right){\left({\int }_{{\mathbb{R}}^3}{\left|G\left(u\right)\right|}^{\frac{6}{3+\beta }}dx\right)}^{\frac{3+\beta }{6}}{\left({\int }_{{\mathbb{R}}^3}{\left|g\left(u\right)v\right|}^{\frac{6}{3+\beta }}dx\right)}^{\frac{3+\beta }{6}}\hfill \\ \hfill & \le C\left({\parallel u\parallel }_{L^2\left({\mathbb{R}}^3\right)}^{\frac{3+\beta }{3}}+{\parallel u\parallel }_{L^{\frac{6q}{3+\beta }}\left({\mathbb{R}}^3\right)}^q\right)\left({\parallel u\parallel }_{L^2\left({\mathbb{R}}^3\right)}^{\frac{\beta }{3}}{\parallel v\parallel }_{L^2\left({\mathbb{R}}^3\right)}+{\parallel u\parallel }_{L^{\frac{6q}{3+\beta }}\left({\mathbb{R}}^3\right)}^{q-1}{\parallel v\parallel }_{L^{\frac{6q}{3+\beta }}\left({\mathbb{R}}^3\right)}\right).\hfill \end{array}$$

(7)

Hence, the energy functional $E:H_K^{\alpha }\left({\mathbb{R}}^3\right)\to \mathbb{R}$ associated with Equation (1)

$$\begin{array}{c}\hfill E\left(u\right)=\frac{1}{2}{\int }_{{\mathbb{R}}^3}{\left(\right|(-\mathrm{\Delta })}^{\frac{\alpha }{2}}{u|}^2+K\left(x\right)u^2)dx-\frac{1}{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))G\left(u\right)dx-\frac{1}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\end{array}$$

is well defined on $H_K^{\alpha }\left({\mathbb{R}}^3\right)$ and $E\in {\mathcal{C}}^1(H_K^{\alpha }\left({\mathbb{R}}^3\right),\mathbb{R})$. Moreover, for any $v\in H_K^{\alpha }\left({\mathbb{R}}^3\right),$

$$\begin{array}{c}\hfill \langle E^{\prime }\left(u\right),v\rangle ={\int }_{{\mathbb{R}}^3}\left[{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}v+K\left(x\right)uv-(I_{\beta }\ast G\left(u\right))g\left(u\right)v-{\left|u\right|}^{2_{\alpha }^{*}-1}uv\right]dx.\end{array}$$

Similar to Proposition 2.10 in [35], we get the Pohožaev type identity.

Lemma 1. 

Assume that $\left(K_1\right)$–$\left(K_2\right)$, $\left(H_1\right)$ hold, and $u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)$ is a solution to Equation (1). Then the Pohožaev identity holds true

$$\begin{array}{cc}\hfill & \frac{3-2\alpha }{2}{\parallel u\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+\frac{3}{2}{\int }_{{\mathbb{R}}^3}K\left(x\right)u^{2}dx+\frac{1}{2}{\int }_{{\mathbb{R}}^3}(\nabla K\left(x\right),x)u^{2}dx\hfill \\ \hfill & =\frac{3+\beta }{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))G\left(u\right)dx+\frac{3}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx.\hfill \end{array}$$

Define the Nehari-Pohožaev manifold by

$$\begin{array}{c}\hfill \Pi :=\{u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}:J\left(u\right):=2\alpha \langle E^{\prime }\left(u\right),u\rangle -\mathcal{L}\left(u\right)=0\},\end{array}$$

where

$$\begin{array}{cc}\hfill \mathcal{L}\left(u\right)& :=\frac{3-2\alpha }{2}{\parallel u\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+\frac{3}{2}{\int }_{{\mathbb{R}}^3}K\left(x\right)u^{2}dx+\frac{1}{2}{\int }_{{\mathbb{R}}^3}(\nabla K\left(x\right),x)u^{2}dx\hfill \\ \hfill & -\frac{3+\beta }{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))G\left(u\right)dx-\frac{3}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\hfill \end{array}$$

and

$$\begin{array}{cc}\hfill J\left(u\right)& :=\frac{6\alpha -3}{2}{\parallel u\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+\frac{1}{2}{\int }_{{\mathbb{R}}^3}[(4\alpha -3)K\left(x\right)-(\nabla K\left(x\right),x)]u^{2}dx\hfill \\ \hfill & +\frac{1}{2}\left\{{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))[(3+\beta )G\left(u\right)-4\alpha g\left(u\right)u]dx-(6\alpha -3){\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\right\}.\hfill \end{array}$$

By combining Lemma 2.2 in [10] and [30], we are able to get the next Brezis-Lieb type lemma.

Lemma 2. 

If $u_n\rightharpoonup u$ in $H_K^{\alpha }\left({\mathbb{R}}^3\right)$ with $\alpha \in (\frac{3}{4},1)$ and $u_n\to u$ a.e. in ${\mathbb{R}}^3$, then

$$\begin{array}{c}\hfill E\left(u_n\right)=E\left(u\right)+E(u_n-u)+o\left(1\right),J\left(u_n\right)=J\left(u\right)+J(u_n-u)+o\left(1\right),\end{array}$$

and

$$\begin{array}{c}\hfill E^{\prime }\left(u_n\right)=E^{\prime }\left(u\right)+E^{\prime }(u_n-u)+o\left(1\right),\end{array}$$

and

$$\begin{array}{c}\hfill \langle E^{\prime }\left(u_n\right),u_n\rangle =\langle E^{\prime }\left(u\right),u\rangle +\langle E^{\prime }(u_n-u),u_n-u\rangle +o\left(1\right).\end{array}$$

At the end of this section, we set up some crucial inequalities.

Lemma 3. 

Assume that $\left(H_1\right)$ and $\left(H_3\right)$ hold. Then for all $\vartheta >0$ and $\tau \in \mathbb{R}$,

$$\begin{array}{c}\hfill f(\vartheta ,\tau ):=\frac{3}{{\vartheta }^{\frac{3+\beta }{2}}}G\left({\vartheta }^{2\alpha }\tau \right)+\left(1-{\vartheta }^{\frac{3}{2}}\right)[4\alpha g\left(\tau \right)\tau -(3+\beta )G\left(\tau \right)]-3G\left(\tau \right)\ge 0.\end{array}$$

Proof. 

Clearly, $f(\vartheta ,0)\ge 0$ for $\vartheta >0$. From $\left(H_3\right)$, for $\tau \ne 0$, we obtain

$$\begin{array}{cc}\hfill & \frac{d}{d\vartheta }f(\vartheta ,\tau )\hfill \\ \hfill =& \frac{3{\vartheta }^{\frac{1}{2}}{\left|\tau \right|}^{\frac{\beta +6}{4\alpha }}}{2}\left[\frac{4\alpha g\left({\vartheta }^{2\alpha }\tau \right){\vartheta }^{2\alpha }\tau -(3+\beta )G\left({\vartheta }^{2\alpha }\tau \right)}{{|\vartheta }^{2\alpha }{\tau |}^{\frac{\beta +6}{4\alpha }}}-\frac{4\alpha g\left(\tau \right)\tau -(3+\beta )G\left(\tau \right)}{{\left|\tau \right|}^{\frac{\beta +6}{4\alpha }}}\right]\hfill \\ \hfill & \left\{\begin{array}{cc}\ge 0,\hfill & \vartheta \ge 1\hfill \\ <0,\hfill & 0<\vartheta <1,\hfill \end{array}\right.\hfill \end{array}$$

which implies that $f(\vartheta ,\tau )\ge f(1,\tau )=0$ for all $\vartheta \in (0,+\infty )$ and $\tau \in (-\infty ,0)\cup (0,+\infty )$. □

Lemma 4. 

Assume that $\alpha \in (\frac{3}{4},1)$, $\left(H_1\right)$–$\left(H_3\right)$ hold. Then

$$\frac{G\left(\vartheta \right)}{{\vartheta \left|\vartheta \right|}^{(8\alpha +\beta -6)/4\alpha }}\mathrm{is}\mathrm{nondecreasing}\mathrm{on}\mathrm{both}(-\infty ,0)\mathrm{and}(0,+\infty ).$$

Proof. 

Together with $\left(H_2\right)$, Lemma 3 and $\alpha >\frac{3}{4}$, we obtain

$$\underset{\vartheta \to 0}{lim}f(\vartheta ,\tau )=4\alpha g\left(\tau \right)\tau -(6+\beta )G\left(\tau \right)\ge 0.$$

(8)

It follows from (8) that

$$\frac{d}{d\vartheta }\left(\frac{G\left(\vartheta \right)}{{\vartheta \left|\vartheta \right|}^{(8\alpha +\beta -6)/4\alpha }}\right)=\frac{1}{{4\alpha \left|\vartheta \right|}^{(16\alpha +\beta -6)/4\alpha }}[4\alpha g\left(\vartheta \right)\vartheta -(12\alpha +\beta -6)G\left(\vartheta \right)]\ge 0.$$

□

Lemma 5. 

Assume that $\left(H_1\right)$–$\left(H_3\right)$ hold. Then

$$\begin{array}{cc}\hfill k(\vartheta ,v)& :={\int }_{{\mathbb{R}}^3}\{\frac{6\alpha -3}{{\vartheta }^{3+\beta }}(I_{\beta }\ast G\left({\vartheta }^{2\alpha }v\right))G\left({\vartheta }^{2\alpha }v\right)\hfill \\ & +(1-{\vartheta }^{6\alpha -3})(I_{\beta }\ast G\left(v\right))[4\alpha g\left(v\right)v-(3+\beta )G\left(v\right)]-(6\alpha -3)(I_{\beta }\ast G\left(v\right))G\left(v\right)\}dx\hfill \\ & \ge 0,\forall \vartheta >0,v\in H_K^{\alpha }\left({\mathbb{R}}^3\right).\hfill \end{array}$$

Proof. 

It follows from $\left(H_1\right)$ and Lemma 4 that

$$I_{\beta }\ast \left(\frac{G\left({\vartheta }^{2\alpha }v\right)}{{\left|\vartheta \right|}^{(12\alpha +\beta -6)/2}}\right)-I_{\beta }\ast G\left(v\right)\left\{\begin{array}{cc}\ge 0,\hfill & \vartheta \ge 1,\hfill \\ \le 0,\hfill & 0<\vartheta <1.\hfill \end{array}\right.$$

(9)

From $\left(H_1\right)$, $\left(H_3\right)$ and (9), we obtain

$$\begin{array}{cc}\hfill & \frac{d}{d\vartheta }k(\vartheta ,v)\hfill \\ \hfill & ={\int }_{{\mathbb{R}}^3}\{\frac{(6\alpha -3)4\alpha }{{\vartheta }^{3+\beta }}(I_{\beta }\ast G\left({\vartheta }^{2\alpha }v\right))g\left({\vartheta }^{2\alpha }v\right){\vartheta }^{2\alpha -1}v\hfill \\ \hfill & -\frac{(6\alpha -3)(3+\beta )}{{\vartheta }^{\beta +4}}(I_{\beta }\ast G\left({\vartheta }^{2\alpha }u\right))G\left({\vartheta }^{2\alpha }v\right)\hfill \\ \hfill & -(6\alpha -3){\vartheta }^{6\alpha -4}(I_{\beta }\ast G\left(v\right))[4\alpha g\left(v\right)v-(3+\beta )G\left(v\right)]\}dx\hfill \\ \hfill & =(6\alpha -3){\vartheta }^{6\alpha -4}{\int }_{{\mathbb{R}}^3}{\left|v\right|}^{(\beta +6)/4\alpha }\{I_{\beta }\ast \left(\frac{G\left({\vartheta }^{2\alpha }v\right)}{{\left|\vartheta \right|}^{(12\alpha +\beta -6)/2}}\right)\frac{4\alpha g\left({\vartheta }^{2\alpha }v\right){\vartheta }^{2\alpha }v-(3+\beta )G\left({\vartheta }^{2\alpha }v\right)}{|{\vartheta }^{2\alpha }{v|}^{(\beta +6)/4\alpha }}\hfill \\ \hfill & -(I_{\beta }\ast G\left(v\right))\frac{4\alpha g\left(v\right)v-(3+\beta )G\left(v\right)}{{\left|v\right|}^{(\beta +6)/4\alpha }}\}dx\hfill \\ \hfill & \left\{\begin{array}{cc}\ge 0,\hfill & \vartheta \ge 1,\hfill \\ \le 0,\hfill & 0<\vartheta <1,\hfill \end{array}\right.\hfill \end{array}$$

which yields $k(\vartheta ,v)\ge k(1,v)=0$ for all $\vartheta >0$ and $v\in H_K^{\alpha }\left({\mathbb{R}}^3\right)$. □

3. The Limit Problem

Noting that $\underset{\left|x\right|\to \infty }{lim}K\left(x\right)=K_{\infty }$, we consider the limit problem (4), whose energy functional is defined by

$$\begin{array}{cc}\hfill E_{\infty }\left(u\right)=& \frac{1}{2}{\parallel u\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+\frac{1}{2}{K}_{\infty }{\parallel u\parallel }_{L^2\left({\mathbb{R}}^3\right)}^2\hfill \\ \hfill & -\frac{1}{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))G\left(u\right)dx-\frac{1}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx.\hfill \end{array}$$

(10)

By Lemma 1, if u is the critical point of $E_{\infty }$ in $H_K^{\alpha }\left({\mathbb{R}}^3\right)$, then it satisfies the Pohožaev identity

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\infty }\left(u\right)& :=\frac{3-2\alpha }{2}{\parallel u\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+\frac{3}{2}{K}_{\infty }{\parallel u\parallel }_{L^2\left({\mathbb{R}}^3\right)}^2-\frac{3+\beta }{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))G\left(u\right)dx-\frac{3}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\hfill \\ & =0.\hfill \end{array}$$

Set $u_{\vartheta }={\vartheta }^{2\alpha }u\left(\vartheta x\right)$. By direct calculation, we deduce that

$$\begin{array}{cc}\hfill h\left(\vartheta \right)=E_{\infty }\left(u_{\vartheta }\right)& =\frac{{\vartheta }^{6\alpha -3}}{2}{\parallel u\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+\frac{{\vartheta }^{4\alpha -3}}{2}{K}_{\infty }{\parallel u\parallel }_{L^2\left({\mathbb{R}}^3\right)}^2\hfill \\ \hfill & -\frac{1}{2{\vartheta }^{3+\beta }}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left({\vartheta }^{2\alpha }u\right))G\left({\vartheta }^{2\alpha }u\right)dx-\frac{{\vartheta }^{\frac{3(6\alpha -3)}{3-2\alpha }}}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx,\hfill \end{array}$$

(11)

which implies that $E_{\infty }\left(u_{\vartheta }\right)\to -\infty$ as $\vartheta \to +\infty$. As a consequence, we obtain the next lemma.

Lemma 6. 

$E_{\infty }$ is not bounded from below.

Define ${\Pi }_{\infty }=\{u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}:J_{\infty }\left(u\right)=0\}$, where

$$\begin{array}{cc}\hfill J_{\infty }\left(u\right)& =2\alpha \langle E_{\infty }^{\prime }\left(u\right),u\rangle -{\mathcal{L}}_{\infty }\left(u\right)\hfill \\ \hfill & =\frac{6\alpha -3}{2}{\parallel u\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+\frac{4\alpha -3}{2}{K}_{\infty }{\parallel u\parallel }_{L^2\left({\mathbb{R}}^3\right)}^2-\frac{6\alpha -3}{2}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\hfill \\ \hfill & +\frac{3+\beta }{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))G\left(u\right)dx-2\alpha {\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))g\left(u\right)udx\hfill \\ \hfill & =\frac{d{E}_{\infty }\left(u_{\vartheta }\right)}{d\vartheta }{|}_{\vartheta =1}.\hfill \end{array}$$

(12)

Set

$$\xi \left(\vartheta \right):=\frac{1-{\vartheta }^{4\alpha -3}}{2}-\frac{(4\alpha -3)(1-{\vartheta }^{6\alpha -3})}{2(6\alpha -3)},\zeta \left(\vartheta \right):=\frac{1-{\vartheta }^{6\alpha -3}}{2}-\frac{1-{\vartheta }^{\frac{3(6\alpha -3)}{3-2\alpha }}}{2_{\alpha }^{*}},\forall \vartheta >0.$$

Clearly,

$$\xi \left(\vartheta \right)>0,\zeta \left(\vartheta \right)>0,\vartheta \in (0,1)\cup (1,+\infty ).$$

(13)

Lemma 7. 

For any $u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)$ and $\vartheta >0$,

$$E_{\infty }\left(u\right)\ge E_{\infty }\left(u_{\vartheta }\right)+\frac{1-{\vartheta }^{6\alpha -3}}{6\alpha -3}{J}_{\infty }\left(u\right)+\xi \left(\vartheta \right){\int }_{{\mathbb{R}}^3}{K}_{\infty }{u}^{2}dx+\zeta \left(\vartheta \right){\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx.$$

Proof. 

From (10)–(13) and Lemma 5, we get that

$$\begin{array}{cc}& E_{\infty }\left(u\right)-E_{\infty }\left(u_{\vartheta }\right)-\frac{1-{\vartheta }^{6\alpha -3}}{6\alpha -3}{J}_{\infty }\left(u\right)\hfill \\ & =\xi \left(\vartheta \right){\int }_{{\mathbb{R}}^3}{K}_{\infty }{u}^{2}dx+\frac{1}{2(6\alpha -3)}{\int }_{{\mathbb{R}}^3}\{\frac{6\alpha -3}{{\vartheta }^{3+\beta }}(I_{\beta }\ast G\left({\vartheta }^{2\alpha }u\right))G\left({\vartheta }^{2\alpha }u\right)\hfill \\ & +(1-{\vartheta }^{6\alpha -3})(I_{\beta }\ast G\left(u\right))[4\alpha g\left(u\right)u-(3+\beta )G\left(u\right)]-(6\alpha -3)(I_{\beta }\ast G\left(u\right))G\left(u\right)\}dx\hfill \\ & +\zeta \left(\vartheta \right){\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\hfill \\ & \ge 0,\hfill \end{array}$$

which comes to the conclusions. □

Lemma 8. 

For all $u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}$, there exists a unique ${\vartheta }_0>0$ such that $u_{{\vartheta }_0}\in {\Pi }_{\infty }$. Moreover,

$$E_{\infty }\left(u_{{\vartheta }_0}\right)=\underset{\vartheta \ge 0}{max}{E}_{\infty }\left(u_{\vartheta }\right).$$

Proof. 

Set $u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}$ be fixed, one has $h^{\prime }\left(\vartheta \right)=0$ is equivalent to $u_{\vartheta }\in {\Pi }_{\infty }, \mathrm{for}\vartheta >0.$ By $\left(K_2\right)$, $\left(H_2\right)$, (6) and $q>1+\frac{\beta }{3}>1+\frac{\beta }{4\alpha }$, we obtain ${lim}_{\vartheta \to 0^{+}}h\left(\vartheta \right)=0,$ for $\vartheta >0$ small, $h\left(\vartheta \right)>0$ and for $\vartheta$ large, $h\left(\vartheta \right)<0$. Hence, ${max}_{\vartheta >0}h\left(\vartheta \right)$ is attained at $\vartheta ={\vartheta }_0\left(u\right)>0$ such that $h^{\prime }\left({\vartheta }_0\right)=0$ and $u_{{\vartheta }_0}\in {\Pi }_{\infty }$.

Next, we show ${\vartheta }_0$ is unique for any $u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}$. Suppose on the contrary that there exist ${\vartheta }_1,{\vartheta }_2>0$ such that $h^{\prime }\left({\vartheta }_1\right)=h^{\prime }\left({\vartheta }_1\right)=0$. It follows from $J_{\infty }\left(u_{{\vartheta }_1}\right)=J_{\infty }\left(u_{{\vartheta }_2}\right)=0$ and Lemma 7 that

$$\begin{array}{cc}\hfill & E_{\infty }\left({\vartheta }_1^{2\alpha }u\left({\vartheta }_{1}x\right)\right)\hfill \\ \hfill & \ge E_{\infty }\left({\vartheta }_2^{2\alpha }u\left({\vartheta }_{2}x\right)\right)+\frac{{\vartheta }_1^{6s-3}-{\vartheta }_2^{6\alpha -3}}{(6\alpha -3){\vartheta }_1^{6\alpha -3}}{J}_{\infty }\left({\vartheta }_1^{2\alpha }u\left({\vartheta }_{1}x\right)\right)\hfill \\ \hfill & +\xi ({\vartheta }_2/{\vartheta }_1){\vartheta }_1^{4\alpha -3}{\int }_{{\mathbb{R}}^3}{K}_{\infty }{u}^{2}dx+\zeta ({\vartheta }_2/{\vartheta }_1){\vartheta }_1^{\frac{(6\alpha -3)3}{3-2\alpha }}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\hfill \\ \hfill & \ge E_{\infty }\left({\vartheta }_2^{2\alpha }u\left({\vartheta }_{2}x\right)\right)+\xi ({\vartheta }_2/{\vartheta }_1){\vartheta }_1^{4s-3}{\int }_{{\mathbb{R}}^3}{K}_{\infty }{u}^{2}dx+\zeta ({\vartheta }_2/{\vartheta }_1){\vartheta }_1^{\frac{(6\alpha -3)3}{3-2\alpha }}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\hfill \end{array}$$

(14)

and

$$\begin{array}{cc}\hfill & E_{\infty }\left({\vartheta }_2^{2\alpha }u\left({\vartheta }_{2}x\right)\right)\hfill \\ \hfill & \ge E_{\infty }\left({\vartheta }_1^{2\alpha }u\left({\vartheta }_{1}x\right)\right)+\frac{{\vartheta }_2^{6\alpha -3}-{\vartheta }_1^{6\alpha -3}}{(6\alpha -3){\vartheta }_2^{6\alpha -3}}{J}_{\infty }\left({\vartheta }_2^{2\alpha }u\left({\vartheta }_{2}x\right)\right)\hfill \\ \hfill & +\xi ({\vartheta }_1/{\vartheta }_2){\vartheta }_2^{4\alpha -3}{\int }_{{\mathbb{R}}^3}{K}_{\infty }{u}^{2}dx+\zeta ({\vartheta }_1/{\vartheta }_2){\vartheta }_2^{\frac{(6\alpha -3)3}{3-2\alpha }}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\hfill \\ \hfill & \ge E_{\infty }\left({\vartheta }_1^{2\alpha }u\left({\vartheta }_{1}x\right)\right)+\xi ({\vartheta }_1/{\vartheta }_2){\vartheta }_2^{4\alpha -3}{\int }_{{\mathbb{R}}^3}{K}_{\infty }{u}^{2}dx+\zeta ({\vartheta }_1/{\vartheta }_2){\vartheta }_2^{\frac{(6\alpha -3)3}{3-2\alpha }}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx.\hfill \end{array}$$

(15)

Therefore, from (14) and (15), we obtain ${\vartheta }_1={\vartheta }_2$. That is, ${\vartheta }_0$ is unique for any $u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}.$ □

Lemma 9. 

The manifold ${\Pi }_{\infty }$ satisfies the following properties:

(1) 

there exists $\sigma >0$ such that ${\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}\ge \sigma ,\forall u\in {\Pi }_{\infty }$.

(2) 

$c_{\infty }={inf}_{u\in {\Pi }_{\infty }}{E}_{\infty }\left(u\right)>0$.

Proof. 

(1)

By (6)–(7), we have

$$\begin{array}{cc}\hfill \frac{4\alpha -3}{2}{\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^2& \le \frac{6\alpha -3}{2}{\parallel u\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+\frac{4\alpha -3}{2}{\int }_{{\mathbb{R}}^3}{K}_{\infty }{u}^{2}dx\hfill \\ & =\frac{1}{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))[4\alpha g\left(u\right)u-(3+\beta )G\left(u\right)]dx+\frac{6\alpha -3}{2}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\hfill \\ & \le C(\parallel u{\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^{2+\frac{2\beta }{3}}+\parallel u{\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^{2q}+\parallel u{\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^{2_{\alpha }^{*}}),\hfill \end{array}$$

which implies

$${\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}\ge \sigma ,\forall u\in {\Pi }_{\infty }.$$

(16)

(2)

Let $\left\{u_n\right\}\subset {\Pi }_{\infty }$ be such that $E_{\infty }\left(u_n\right)\to c_{\infty }$. There exist two possible scenarios:

(i)

${inf}_{n\in \mathbb{N}}{\parallel u_n\parallel }_{L^2\left({\mathbb{R}}^3\right)}>0$, or

(ii)

${inf}_{n\in \mathbb{N}}{\parallel u_n\parallel }_{L^2\left({\mathbb{R}}^3\right)}=0$.

Case (i) ${inf}_{n\in \mathbb{N}}{\parallel u_n\parallel }_{L^2\left({\mathbb{R}}^3\right)}={\sigma }_1>0$. It follows from (8), (10) and (12) that

$$\begin{array}{cc}\hfill c_{\infty }& =E_{\infty }\left(u_n\right)=E_{\infty }\left(u_n\right)-\frac{1}{6s-3}{J}_{\infty }\left(u_n\right)\hfill \\ \hfill & =\frac{\alpha }{6\alpha -3}{\int }_{{\mathbb{R}}^3}{K}_{\infty }{u}_n^{2}dx+\left(\frac{1}{2}-\frac{1}{2_{\alpha }^{*}}\right){\int }_{{\mathbb{R}}^3}{\left|u_n\right|}^{2_{\alpha }^{*}}dx\hfill \\ \hfill & +\frac{1}{2(6\alpha -3)}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u_n\right))[4\alpha g\left(u_n\right)u_n-(6\alpha +\mu )G\left(u_n\right)]dx\hfill \\ \hfill & \ge \frac{\alpha K_{\infty }}{6\alpha -3}{\sigma }_1^2.\hfill \end{array}$$

(17)

Case (ii) ${inf}_{n\in \mathbb{N}}{\parallel u_n\parallel }_{L^2\left({\mathbb{R}}^3\right)}=0$. According to (16), passing to a sub-sequence, we obtain

$${\parallel u}_n{\parallel }_{L^2\left({\mathbb{R}}^3\right)}\to 0,{\parallel u_n\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}\ge \frac{\sigma }{2}.$$

(18)

It follows from $\left(H_1\right)$–$\left(H_2\right)$ that for any $ϵ>0$, there exists $C_{ϵ}>0$ such that

$$\left|G\left(\tau \right)\right|\le C_{ϵ}{\left|\tau \right|}^{1+\frac{\beta }{3}}+ϵ{\left|\tau \right|}^{2_{\beta ,\alpha }^{*}},\forall \tau \in \mathbb{R}.$$

(19)

From (5), (6) and (19), we deduce that

$${\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))G\left(u\right)\le C{C}_{ϵ}{\parallel u\parallel }_{L^2\left({\mathbb{R}}^3\right)}^{2+\frac{2\beta }{3}}+CϵS_{\alpha }^{-\frac{2_{\alpha }^{*}}{2}}{\parallel u\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^{\frac{2_{\alpha }^{*}(3+\beta )}{3}}.$$

(20)

Let ${\vartheta }_n={\left[{\left(2_{\alpha }^{*}\right)}^{\frac{3-2\alpha }{2\alpha }}{S}_{\alpha }^{\frac{3}{2\alpha }}{\parallel u_n\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^{-2}\right]}^{\frac{1}{6\alpha -3}}$. Then, due to (18), $\left\{{\vartheta }_n\right\}$ is bounded. Applying Lemma 7, (11), (18) and (20), we deduce that

$$\begin{array}{cc}\hfill c_{\infty }+o_n\left(1\right)& =E_{\infty }\left(u_n\right)\ge E_{\infty }\left({\left(u_n\right)}_{{\vartheta }_n}\right)\hfill \\ & =\frac{{\vartheta }_n^{6\alpha -3}}{2}{\parallel u_n\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+\frac{{\vartheta }_n^{4\alpha -3}}{2}{\int }_{{\mathbb{R}}^3}{K}_{\infty }{u}_n^{2}dx\hfill \\ & -\frac{1}{2{\vartheta }_n^{3+\beta }}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left({\vartheta }_n^{2\alpha }u\right))G\left({\vartheta }_n^{2\alpha }u\right)dx-\frac{{\vartheta }_n^{\frac{3(6\alpha -3)}{3-2\alpha }}}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u_n\right|}^{2_{\alpha }^{*}}dx\hfill \\ & \ge \frac{{\vartheta }_n^{6\alpha -3}}{2}{\parallel u_n\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2-C{C}_{ϵ}{\left({\vartheta }_n^{4\alpha -3}{\parallel u_n\parallel }_{L^2\left({\mathbb{R}}^3\right)}^2\right)}^{\frac{3+\beta }{3}}\hfill \\ & -CϵS_{\alpha }^{-\frac{2_{\alpha }^{*}}{2}}{\left({\vartheta }_n^{6\alpha -3}{\parallel u_n\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2\right)}^{\frac{3+\beta }{3-2\alpha }}-\frac{S_{\alpha }^{-\frac{2_{\alpha }^{*}}{2}}}{2_{\alpha }^{*}}{\left({\vartheta }_n^{6\alpha -3}{\parallel u_n\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2\right)}^{\frac{3}{3-2\alpha }}\hfill \\ & =\frac{{\vartheta }_n^{6s-3}{\parallel u_n\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2}{4}\left[2-\frac{S_{\alpha }^{-\frac{2_{\alpha }^{*}}{2}}}{2_{\alpha }^{*}}{\left({\vartheta }_n^{6\alpha -3}{\parallel u_n\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2\right)}^{\frac{2\alpha }{3-2\alpha }}\right]+o_n\left(1\right)\hfill \\ & =\frac{1}{4}{\left(2_{\alpha }^{*}\right)}^{\frac{3-2\alpha }{2\alpha }}{S}_{\alpha }^{\frac{3}{2\alpha }}+o_n\left(1\right).\hfill \end{array}$$

It follows from Case (i) and Case (ii) that $c_{\infty }={inf}_{u\in {\Pi }_{\infty }}{E}_{\infty }\left(u\right)>0$. □

By using Lemmas 8 and 9, we can find the next result.

Lemma 10. 

The following equality holds

$$c_{\infty }={\overline{c}}_{\infty }:=\underset{u\ne 0}{inf}\underset{\vartheta >0}{max}{E}_{\infty }\left(u_{\vartheta }\right).$$

We shall use Proposition 2.8 (the general mini-max principle) in [36] to obtain a Cerami sequence for the functional $E_{\infty }$ with $J_{\infty }\left(u_n\right)\to 0$, where $E_{\infty },J_{\infty }\left(u_n\right)$ are given in (10) and (12), respectively.

Lemma 11. 

There exists a sequence $\left\{u_n\right\}\subset H_K^{\alpha }\left({\mathbb{R}}^3\right)$ such that

$$E_{\infty }\left(u_n\right)\to {\check{c}}_{\infty }>0,\parallel E_{\infty }^{\prime }{\parallel }_{H_K^{-\alpha }\left({\mathbb{R}}^3\right)}(1+\parallel u_n{\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)})\to 0\mathrm{and}{J}_{\infty }\left(u_n\right)\to 0,$$

(21)

where

$${\check{c}}_{\infty }:=\underset{\mu \in \mathrm{\Gamma }}{inf}\underset{\tau \in [0,1]}{max}{E}_{\infty }\left(\mu \left(\tau \right)\right),\mathrm{\Gamma }:=\left\{\mu \in \mathcal{C}([0,1],H_K^{\alpha }\left({\mathbb{R}}^3\right)):\mu \left(0\right)=0,E_{\infty }\left(\mu \left(1\right)\right)<0\right\}.$$

Proof. 

For $u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\},$ By $\left(H_2\right)$, one has $E_{\infty }\left(\tau u\right)\to -\infty$ as $\tau \to +\infty$. By the standard arguments, we have $\mathrm{\Gamma }\ne \varnothing$ and ${\check{c}}_{\infty }<\infty$. Furthermore, it is easy to check that there exist ${\sigma }_0,{\delta }_0>0$ such that $E_{\infty }\left(u\right)\ge 0$ for all u with ${\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}\le {\sigma }_0$ and $E_{\infty }\left(u\right)\ge {\delta }_0$ for all u with ${\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}={\sigma }_0$. Together with the definition of $\mathrm{\Gamma }$, we obtain ${\parallel \mu \left(1\right)\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}>{\sigma }_0$. From the continuity of $\mu \left(\tau \right)$ and the intermediate value theorem, there exists ${\tau }_{\mu }\in (0,1)$ such that $\parallel \mu \left({\tau }_{\mu }\right){\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}={\sigma }_0$. Hence, we obtain

$$\underset{\tau \in [0,1]}{max}{E}_{\infty }\left(\mu \left(\tau \right)\right)\ge E_{\infty }\left(\mu \left({\tau }_{\mu }\right)\right)\ge {\delta }_0>0,$$

which implies

$$\infty >{\check{c}}_{\infty }=\underset{\mu \in \mathrm{\Gamma }}{inf}\underset{\tau \in [0,1]}{max}{E}_{\infty }\left(\mu \left(\tau \right)\right)\ge {\delta }_0>0.$$

Define the continuous map

$$\eta :\mathbb{R}\times H_K^{\alpha }\left({\mathbb{R}}^3\right)\to H_K^{\alpha }\left({\mathbb{R}}^3\right),\eta (\tau ,u)\left(x\right)=e^{2\alpha \tau }u\left(e^{\tau }x\right),\mathrm{for}\tau \in \mathbb{R},u\in H_K^{\alpha }\left({\mathbb{R}}^3\right),x\in {\mathbb{R}}^3,$$

where $\mathbb{R}\times H_K^{\alpha }\left({\mathbb{R}}^3\right)$ is the Banach space with the product norm ${\parallel (\tau ,u)\parallel }_{\mathbb{R}\times H_K^{\alpha }\left({\mathbb{R}}^3\right)}:={\left(\right|\tau |}^2+{\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^2{)}^{\frac{1}{2}}$. We define the following auxiliary functional:

$$\begin{array}{cc}\hfill {\tilde{E}}_{\infty }(\tau ,u)& =E_{\infty }\left(\eta (\tau ,u)\right)\hfill \\ & =\frac{1}{2}{\int }_{{\mathbb{R}}^3}{\left(\right|(-\mathrm{\Delta })}^{\frac{\alpha }{2}}{\eta (\tau ,u)|}^2+K_{\infty }{\left|\eta (\tau ,u)\right|}^2)dx\hfill \\ & -\frac{1}{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(\eta (\tau ,u)\right))G\left(\eta (\tau ,u)\right)dx-\frac{1}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|\eta (\tau ,u)\right|}^{2_{\alpha }^{*}}dx\hfill \\ & =\frac{e^{(6\alpha -3)\tau }}{2}{\parallel u\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+\frac{e^{(4\alpha -3)\tau }}{2}{K}_{\infty }{\parallel u\parallel }_{L^2\left({\mathbb{R}}^3\right)}^2\hfill \\ & -\frac{e^{-(3+\beta )\tau }}{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(e^{2\alpha \tau }u\right))G\left(e^{2\alpha \tau }u\right)dx-\frac{e^{\frac{2_{\alpha }^{*}(6\alpha -3)}{2}\tau }}{2_{\alpha }^{*}}{\parallel u\parallel }_{L^{2_{\alpha }^{*}}\left({\mathbb{R}}^3\right)}^{2_{\alpha }^{*}}.\hfill \end{array}$$

Moreover, by direct calculations, we obtain ${\tilde{E}}_{\infty }\in {\mathcal{C}}^1(\mathbb{R}\times H_K^{\alpha }\left({\mathbb{R}}^3\right),\mathbb{R})$, and

$${\partial }_{\tau }{\tilde{E}}_{\infty }(\tau ,u)=J_{\infty }\left(\eta (\tau ,u)\right),{\partial }_u{\tilde{E}}_{\infty }(\tau ,u)w=E_{\infty }^{\prime }\left(\eta (\tau ,u)\right)\eta (\tau ,w)$$

for all $\tau \in \mathbb{R}$ and $u,w\in H_K^{\alpha }\left({\mathbb{R}}^3\right)$. We define the mini-max value ${\tilde{c}}_{\infty }$ for $\tilde{E}$ by

$${\tilde{c}}_{\infty }=\underset{\tilde{\mu }\in \tilde{\mathrm{\Gamma }}}{inf}\underset{\tau \in [0,1]}{max}{\tilde{E}}_{\infty }\left(\tilde{\mu }\left(\tau \right)\right),$$

where $\tilde{\mathrm{\Gamma }}=\{\tilde{\mu }\in \mathcal{C}([0,1],\mathbb{R}\times H_K^{\alpha }\left({\mathbb{R}}^3\right)):\tilde{\mu }\left(0\right)=(0,0),{\tilde{E}}_{\infty }\left(\tilde{\mu }\left(1\right)\right)<0\}$. Since $\mathrm{\Gamma }=\{\eta \circ \tilde{\mu }:\tilde{\mu }\in \tilde{\mathrm{\Gamma }}\}$, we deduce that ${\check{c}}_{\infty }={\tilde{c}}_{\infty }$. By the definition of ${\check{c}}_{\infty }$, there exists ${\mu }_n\in \mathrm{\Gamma }$ such that for any $n\in \mathbb{N}$,

$$\underset{\tau \in [0,1]}{max}{\tilde{E}}_{\infty }(0,{\mu }_n\left(\tau \right))=\underset{\tau \in [0,1]}{max}{E}_{\infty }\left({\mu }_n\left(\tau \right)\right)\le {\check{c}}_{\infty }+\frac{1}{n^2}.$$

Applying Proposition 2.8 (the general mini-max principle) in [36] to ${\tilde{E}}_{\infty }$, setting $D=[0,1],D_0=\{0,1\},\mathbb{B}=\mathbb{R}\times H_K^{\alpha }\left({\mathbb{R}}^3\right),\sigma =\frac{1}{n^2},\delta =\frac{1}{n}$ and ${\tilde{\mu }}_n\left(\tau \right)=(0,{\mu }_n\left(\tau \right))$, we conclude that there exist $({\tau }_n,u_n)\in \mathbb{R}\times H_K^{\alpha }\left({\mathbb{R}}^3\right)$ such that

$${\tilde{E}}_{\infty }({\tau }_n,u_n)\to {\check{c}}_{\infty },$$

and

$${\parallel \tilde{E}}_{\infty }^{\prime }({\tau }_n,u_n){\parallel }_{{\mathbb{B}}^{\prime }}(1+\parallel ({\tau }_n,u_n){\parallel }_{\mathbb{B}})\to 0,$$

(22)

and

$$\mathrm{dist}(({\tau }_n,u_n),\left\{0\right\}\times {\mu }_n\left([0,1]\right))\to 0,$$

(23)

as $n\to \infty$. Thus, (23) implies ${\tau }_n\to 0.$ It is easy to see that for all $(t,w)\in \mathbb{R}\times H_K^{\alpha }\left({\mathbb{R}}^3\right)$,

$$\langle {\tilde{E}}_{\infty }^{\prime }({\tau }_n,u_n),(t,w)\rangle =\langle E_{\infty }^{\prime }\left(\eta ({\tau }_n,u_n)\right),\eta ({\tau }_n,w)\rangle +J_{\infty }\left(\eta ({\tau }_n,u_n)\right)t.$$

(24)

Set $v_n=\eta ({\tau }_n,u_n)$. If we take $t=1$ and $w=0$ in (24), we obtain $J_{\infty }\left(v_n\right)\to 0$ as $n\to \infty$. For each $u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)$, let $t=0$ and $w_n=e^{-2\alpha {\tau }_n}u\left(e^{-{\tau }_n}x\right)$ in (24). We deduce from (22)–(23) that

$$\begin{array}{cc}\hfill & \left|\langle E_{\infty }^{\prime }\left(v_n\right),u\rangle \right|(1+\parallel v_n{\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)})\hfill \\ \hfill =& \left|\langle E_{\infty }\left(\eta ({\tau }_n,u_n)\right),\eta ({\tau }_n,w_n)\rangle \right|(1+\parallel v_n{\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)})\hfill \\ \hfill =& o\left(1\right)\parallel w_n{\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)},\hfill \end{array}$$

as $n\to \infty$. Therefore, (21) holds. □

Lemma 12. 

${\check{c}}_{\infty }<\frac{\alpha }{3}{S}_{\alpha }^{\frac{3}{2\alpha }}$, where $S_{\alpha }$ is given in (5).

Proof. 

Let $\varphi \left(x\right)\in {\mathcal{C}}_0^{\infty }\left({\mathbb{R}}^3\right)$ be a cut-off function such that $0\le \varphi \left(x\right)\le 1$ in ${\mathbb{R}}^3$, $\varphi \equiv 1$ in $B_1\left(0\right)$ and $\varphi \equiv 0$ in ${\mathbb{R}}^3\setminus B_2\left(0\right)$. As is known to all, $S_{\alpha }$ is achieved by

$$\tilde{u}:=\kappa ({\sigma }^2+|x-x_0{{|}^2)}^{-\frac{3-2\alpha }{2}}$$

for any $\kappa \in \mathbb{R},\sigma >0$ and $x_0\in {\mathbb{R}}^3$. Hence, setting $x_0=0$, we define

$$u_{ϵ}\left(x\right):=\varphi \left(x\right)U_{ϵ}\left(x\right),x\in {\mathbb{R}}^3,$$

where

$$U_{ϵ}\left(x\right)={ϵ}^{-\frac{(3-2\alpha )}{2}}{u}^{*}(x/ϵ),u^{*}\left(x\right)=\frac{\tilde{u}\left(x/S_{\alpha }^{1/\left(2\alpha \right)}\right)}{\parallel \tilde{u}{\parallel }_{L^{2_{\alpha }^{*}}\left({\mathbb{R}}^3\right)}}.$$

As in [23], we obtain

$${\mathcal{A}}_{ϵ}:={\int }_{{\mathbb{R}}^3}{\left|{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}{u}_{ϵ}\right|}^{2}dx\le S_{\alpha }^{\frac{3}{2\alpha }}+O\left({ϵ}^{3-2\alpha }\right),$$

(25)

and

$${\mathcal{B}}_{ϵ}:={\int }_{{\mathbb{R}}^3}{\left|u_{ϵ}\right|}^{2_{\alpha }^{*}}dx=S_{\alpha }^{\frac{3}{2\alpha }}+O\left({ϵ}^3\right).$$

(26)

By a simple calculation, we observe

$${\mathcal{C}}_{ϵ}:={\int }_{{\mathbb{R}}^3}{\left|u_{ϵ}\right|}^{2}dx=O\left({ϵ}^{3-2\alpha }\right),$$

(27)

and

$${\mathcal{D}}_{ϵ}:={\int }_{{\mathbb{R}}^3}{\left|u_{ϵ}\right|}^{s}dx=\left\{\begin{array}{cc}O\left({ϵ}^{\frac{3(2-s)+2\alpha s}{2}}\right),\hfill & s>\frac{3}{3-2\alpha },\hfill \\ O\left({ϵ}^{\frac{3(2-s)+2\alpha s}{2}}\right|logϵ\left|\right),\hfill & s=\frac{3}{3-2\alpha },\hfill \\ O\left({ϵ}^{\frac{(3-2\alpha )s}{2}}\right),\hfill & s<\frac{3}{3-2\alpha }.\hfill \end{array}\right.$$

(28)

By virtue of (6) and (28), for any $p\in (2,2_{\alpha }^{*})$ we obtain

$$\begin{array}{cc}\hfill {\mathcal{H}}_{ϵ}& ={\int }_{{\mathbb{R}}^3}{\int }_{{\mathbb{R}}^3}\frac{|u_{ϵ}{\left(x\right)|}^p{\left|u_{ϵ}\left(y\right)\right|}^p}{{|x-y|}^{3-\beta }}dxdy\hfill \\ \hfill & \ge C{\int }_{B_1\left(\frac{x_0}{ϵ}\right)}{\int }_{B_1\left(\frac{x_0}{ϵ}\right)}|U_{ϵ}{\left(x\right)|}^p{\left|U_{ϵ}\left(y\right)\right|}^{p}dxdy\hfill \\ \hfill & =C{\left({ϵ}^{3-\frac{(3-2\alpha )p}{2}}{\int }_0^{\frac{1}{ϵS_{\alpha }^{1/\left(2\alpha \right)}}}\frac{r^2}{{({\sigma }^2+r^2)}^{\frac{(3-2\alpha )p}{2}}}dr\right)}^2\hfill \\ \hfill & =O\left({ϵ}^{6-(3-2\alpha )p}\right).\hfill \end{array}$$

(29)

Since ${sup}_{\tau \ge 0}{E}_{\infty }\left(\tau u_{ϵ}\right)=E_{\infty }\left({\tau }_{ϵ}{u}_{ϵ}\right)\ge {\delta }_0>0$, there exists $T_1>0$ such that ${\tau }_{ϵ}>T_1$. Moreover, we infer from $E_{\infty }\left(\tau u_{ϵ}\right)\to -\infty$ as $\tau \to \infty$ that there exists $T_2>0$ such that ${\tau }_{ϵ}<T_2$. Then $T_1<{\tau }_{ϵ}<T_2$. Note that

$$\begin{array}{cc}\hfill E_{\infty }\left(\tau u_{ϵ}\right)& \le \frac{{\tau }^2}{2}{\int }_{{\mathbb{R}}^3}{{\left(\right|(-\mathrm{\Delta })}^{\frac{\alpha }{2}}{u}_{ϵ}|}^2+K_{\infty }|u_{ϵ}{|}^2)dx\hfill \\ \hfill & -\frac{{\tau }^{2p}{\nu }^2}{2p^2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast |u_{ϵ}{|}^p\left)\right|u_{ϵ}{|}^{p}dx-\frac{{\tau }^{2_{\alpha }^{*}}}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u_{ϵ}\right|}^{2_{\alpha }^{*}}dx\hfill \\ \hfill & =\frac{{\tau }^2}{2}{\mathcal{A}}_{ϵ}+\frac{{\tau }^2}{2}{K}_{\infty }{\mathcal{C}}_{ϵ}-\frac{{\tau }^{2p}}{2p^2}{\nu }^2{\mathcal{H}}_{ϵ}-\frac{{\tau }^{2_{\alpha }^{*}}}{2_{\alpha }^{*}}{\mathcal{B}}_{ϵ}.\hfill \end{array}$$

(30)

Define

$$Q_{ϵ}\left(t\right):=\frac{{\tau }^2}{2}{\mathcal{A}}_{ϵ}-\frac{{\tau }^{2_{\alpha }^{*}}}{2_{\alpha }^{*}}{\mathcal{B}}_{ϵ}.$$

According to (25)–(26), it is easy to verify that

$$\underset{\tau \ge 0}{sup}{Q}_{ϵ}\left(\tau \right)\le \frac{\alpha }{3}{S}_{\alpha }^{\frac{3}{2\alpha }}+O\left({ϵ}^{3-2\alpha }\right).$$

(31)

It follows from (27), (29)–(31) that

$${\check{c}}_{\infty }\le E_{\infty }\left(\tau u_{ϵ}\right)\le \frac{\alpha }{3}{S}_{\alpha }^{\frac{3}{2\alpha }}+O\left({ϵ}^{3-2\alpha }\right)-O\left({\nu }^2{ϵ}^{6-(3-2\alpha )p}\right).$$

If $2_{\beta ,\alpha }^{*}>p>2_{\alpha }^{*}-1$, then $0<6-(3-2\alpha )p<3-2\alpha$, which implies that for any fixed ${\nu }^2>0$, ${\check{c}}_{\infty }<\frac{\alpha }{3}{S}_{\alpha }^{\frac{3}{2\alpha }}$ for $ϵ>0$ small. If $2<p\le 2_{\alpha }^{*}-1$ and $\nu \ge {ϵ}^{\frac{(3-2\alpha )p-4-2\alpha }{2}}$, we also obtain ${\check{c}}_{\infty }<\frac{\alpha }{3}{S}_{\alpha }^{\frac{3}{2\alpha }}$. □

Lemma 13. 

The following equality holds

$$\underset{\mu \in \mathrm{\Gamma }}{inf}\underset{\tau \in [0,1]}{sup}{E}_{\infty }\left(\mu \left(\tau \right)\right)={\check{c}}_{\infty }={\overline{c}}_{\infty }=\underset{u\ne 0}{inf}\underset{\vartheta >0}{max}{E}_{\infty }\left(u_{\vartheta }\right).$$

Proof. 

From Lemma 6, we can see that $E_{\infty }\left(u_{\vartheta }\right)<0$ for $u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}$ and $\vartheta$ large enough. This implies ${\check{c}}_{\infty }\le {\overline{c}}_{\infty }$. Then, we show ${\check{c}}_{\infty }\ge {\overline{c}}_{\infty }$. We claim that for any $\mu \in \mathrm{\Gamma },\mu \left([0,1]\right)\cap {\Pi }_{\infty }\ne \varnothing$. Indeed, from (17), we obtain for any $\mu \in \mathrm{\Gamma }$,

$$J_{\infty }\left(\mu \left(1\right)\right)\le (6\alpha -3)E_{\infty }\left(\mu \left(1\right)\right)-\alpha K_{\infty }{\sigma }_1^2<0.$$

For any $u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}$, from (6), (7) and (12), we deduce that

$$\begin{array}{cc}\hfill J_{\infty }\left(u\right)& =\frac{6\alpha -3}{2}{\int }_{{\mathbb{R}}^3}{\left|{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u\right|}^{2}dx+\frac{4\alpha -3}{2}{\int }_{{\mathbb{R}}^3}{K}_{\infty }{u}^{2}dx\hfill \\ & -\frac{1}{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))[4\alpha g\left(u\right)u-(3+\beta )G\left(u\right)]dx-\frac{6\alpha -3}{2}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\hfill \\ & \ge \frac{4\alpha -3}{2}{\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^2-{C\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^{2+\frac{2\beta }{3}}-{C\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^{2q}-C{\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^{2_{\alpha }^{*}},\hfill \end{array}$$

which yields there exists ${\sigma }_2\in {(0,\parallel \mu \left(1\right)\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)})$ and ${\rho }_2>0$ such that $J_{\infty }\left(u\right)\ge {\rho }_2$ for ${\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}={\sigma }_2$. This implies there exists ${\tau }_0\in (0,1)$ such that $J_{\infty }\left(\mu \left({\tau }_0\right)\right)\ge {\rho }_2$. Therefore, the curve $\mu \in \mathrm{\Gamma }$ must cross ${\Pi }_{\infty }$, which indicates ${\check{c}}_{\infty }\ge c_{\infty }$. Together with Lemma 10, we obtain ${\check{c}}_{\infty }\ge {\overline{c}}_{\infty }$. □

Proof of Theorem 1. 

By Lemma 11, there exists a sequence $\left\{u_n\right\}\subset H_K^{\alpha }\left({\mathbb{R}}^3\right)$ satisfying (21). It follows from (8) and (17) that

$$\begin{array}{cc}\hfill {\check{c}}_{\infty }& =E_{\infty }\left(u_n\right)=E_{\infty }\left(u_n\right)-\frac{1}{6\alpha -3}{J}_{\infty }\left(u_n\right)\hfill \\ \hfill \ge & \frac{\alpha }{6\alpha -3}{\int }_{{\mathbb{R}}^3}{K}_{\infty }{u}_n^{2}dx+\left(\frac{1}{2}-\frac{1}{2_{\alpha }^{*}}\right){\int }_{{\mathbb{R}}^3}{\left|u_n\right|}^{2_{\alpha }^{*}}dx.\hfill \end{array}$$

Combining with the Hölder inequality, we deduce that $\left\{u_n\right\}$ is bounded in $L^r\left({\mathbb{R}}^3\right)$ for $r\in [2,2_{\alpha }^{*}]$. Then, by $J_{\infty }\left(u_n\right)\to 0$, (6) and (7), we can see that

$$\begin{array}{cc}& \frac{6\alpha -3}{2}{\parallel u_n\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}\hfill \\ & \le \frac{1}{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u_n\right))[4\alpha g\left(u_n\right)u_n-(3+\beta )G\left(u_n\right)]dx+\frac{6\alpha -3}{2}{\int }_{{\mathbb{R}}^3}{\left|u_n\right|}^{2_{\alpha }^{*}}dx\hfill \\ & \le C(\parallel u_n{\parallel }_{L^2\left({\mathbb{R}}^3\right)}^{2+\frac{2\beta }{3}}+\parallel u_n{\parallel }_{L^{\frac{6q}{3+\beta }}\left({\mathbb{R}}^3\right)}^{2q}+\parallel u_n{\parallel }_{L^{2_{\alpha }^{*}}\left({\mathbb{R}}^3\right)}^{2_{\alpha }^{*}})\le C.\hfill \end{array}$$

This implies $\left\{u_n\right\}$ is bounded in $H_K^{\alpha }\left({\mathbb{R}}^3\right)$.

Now, we claim that

$$\underset{n\to \infty }{lim}\underset{y\in {\mathbb{R}}^3}{sup}{\int }_{B_1\left(y\right)}{\left|u_n\right|}^{2}dx>0.$$

If it does not occur, then it follows fromLemma 2.4 (a fractional version of Lions vanishing lemma) in [37] that $u_n\to 0$ in $L^r\left({\mathbb{R}}^3\right)$ for $r\in (2,2_{\alpha }^{*})$. Hence, we obtain

$$o_n\left(1\right)=\langle E_{\infty }^{\prime }\left(u_n\right),u_n\rangle =\parallel u_n{\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+K_{\infty }\parallel u_n{\parallel }_{L^2\left({\mathbb{R}}^3\right)}^2-{\parallel u_n\parallel }_{L^{2_{\alpha }^{*}}\left({\mathbb{R}}^3\right)}^{2_{\alpha }^{*}}+o_n\left(1\right),$$

as $n\to \infty$. Since $\left\{u_n\right\}$ is bounded in $H_K^{\alpha }\left({\mathbb{R}}^3\right)$ and ${\check{c}}_{\infty }>0$, we may assume that up to a sub-sequence, as $n\to \infty$, for some $l>0$,

$$\parallel u_n{\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+K_{\infty }\parallel u_n{\parallel }_{L^2\left({\mathbb{R}}^3\right)}^2\to l,{\parallel u_n\parallel }_{L^{2_{\alpha }^{*}}\left({\mathbb{R}}^3\right)}^{2_{\alpha }^{*}}\to l.$$

(32)

In view of (5) and (32), we obtain

$$\begin{array}{cc}\hfill l=& \underset{n\to \infty }{lim}\left(\parallel u_n{\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+K_{\infty }{\parallel u_n\parallel }_{L^2\left({\mathbb{R}}^3\right)}^2\right)\hfill \\ \hfill \ge & \underset{n\to \infty }{lim}{\parallel u_n\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2\hfill \\ \hfill \ge & S_{\alpha }{\parallel u_n\parallel }_{L^{2_{\alpha }^{*}}\left({\mathbb{R}}^3\right)}^2=S_{\alpha }{l}^{\frac{2}{2_{\alpha }^{*}}},\hfill \end{array}$$

which implies

$$l\ge S_{\alpha }^{\frac{3}{2\alpha }}.$$

(33)

Combining the fact

$$\begin{array}{cc}\hfill {\check{c}}_{\infty }+o_n\left(1\right)=& E_{\infty }\left(u_n\right)\hfill \\ \hfill =& \frac{1}{2}\left(\parallel u_n{\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+K_{\infty }{\parallel u_n\parallel }_{L^2\left({\mathbb{R}}^3\right)}^2\right)-\frac{1}{2_{\alpha }^{*}}{\parallel u_n\parallel }_{L^{2_{\alpha }^{*}}\left({\mathbb{R}}^3\right)}^{2_{\alpha }^{*}}+o_n\left(1\right)\hfill \\ \hfill =& \frac{\alpha }{3}l+o_n\left(1\right)\hfill \end{array}$$

and (33), we observe that ${\check{c}}_{\infty }\ge \frac{\alpha }{3}{S}_{\alpha }^{\frac{3}{2\alpha }}$, which contradicts Lemma 12. Hence, there exists $\sigma >0$ and a sequence $\left\{z_n\right\}\subset {\mathbb{R}}^3$ such that ${\int }_{B_1\left(z_n\right)}{\left|v_n\right|}^{2}dx>\sigma$. Let ${\check{v}}_n\left(x\right)=v_n(x+z_n)$, then as $n\to \infty$,

$$E_{\infty }\left({\check{v}}_n\right)\to {\check{c}}_{\infty },E_{\infty }^{\prime }\left({\check{v}}_n\right)\to 0,J_{\infty }\left({\check{v}}_n\right)\to 0$$

and ${\int }_{B_1\left(0\right)}{\left|{\check{v}}_n\right|}^{2}dx>\delta$. Hence, passing to a sub-sequence, there exists $\check{v}\in H_K^{\alpha }\left({\mathbb{R}}^3\right)$ such that

$$\left\{\begin{array}{cc}{\check{v}}_n\rightharpoonup \check{v}\hfill & \mathrm{in}{H}_K^{\alpha }\left({\mathbb{R}}^3\right),\hfill \\ {\check{v}}_n\to \check{v}\hfill & \mathrm{in}{L}_{loc}^r\left({\mathbb{R}}^3\right)\mathrm{for}r\in [1,2_{\alpha }^{*}),\hfill \\ {\check{v}}_n\to \check{v}\hfill & \mathrm{a}.\mathrm{e}.\mathrm{in}{\mathbb{R}}^3.\hfill \end{array}\right.$$

By using standard arguments, we obtain that $E_{\infty }^{\prime }\left(\check{v}\right)=0$ and $E_{\infty }\left(\check{v}\right)\ge {\overline{c}}_{\infty }$. Therefore, $\check{v}$ is a nontrivial solution to (4). In view of Lemma 13, (8) and Fatou’s Lemma, we obtain

$$\begin{array}{cc}\hfill {\overline{c}}_{\infty }={\check{c}}_{\infty }& =\underset{n\to \infty }{lim}{E}_{\infty }\left({\check{v}}_n\right)=\underset{n\to \infty }{lim}\left[E_{\infty }\left({\check{v}}_n\right)-\frac{1}{6\alpha -3}{J}_{\infty }\left({\check{v}}_n\right)\right]\hfill \\ & =\underset{n\to \infty }{lim}\left\{\frac{\alpha }{6\alpha -3}{\int }_{{\mathbb{R}}^3}{K}_{\infty }{\check{v}}_n^{2}dx+\frac{1}{2(6\alpha -3)}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))[4\alpha g\left({\check{v}}_n\right){\check{v}}_n-(6\alpha +\beta )G\left({\check{v}}_n\right)]dx\right.\hfill \\ & +\left.\left(\frac{1}{2}-\frac{1}{2_{\alpha }^{*}}\right){\int }_{{\mathbb{R}}^3}{\left|{\check{v}}_n\right|}^{2_{\alpha }^{*}}dx\right\}\hfill \\ & \ge \frac{\alpha }{6\alpha -3}{\int }_{{\mathbb{R}}^3}{K}_{\infty }{\check{v}}^{2}dx+\frac{1}{2(6\alpha -3)}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))[4\alpha g\left(\check{v}\right)\check{v}-(6\alpha +\beta )G\left(\check{v}\right)]dx\hfill \\ & +\left(\frac{1}{2}-\frac{1}{2_{\alpha }^{*}}\right){\int }_{{\mathbb{R}}^3}{\left|\check{v}\right|}^{2_{\alpha }^{*}}dx\hfill \\ & =E_{\infty }\left(\check{v}\right)-\frac{1}{6\alpha -3}{J}_{\infty }\left(\check{v}\right)\ge {\overline{c}}_{\infty },\hfill \end{array}$$

which implies that $E_{\infty }\left(\check{v}\right)={\overline{c}}_{\infty }=c_{\infty }$, recalling Lemma 10. □

4. Existence of Ground State Solution to (1)

In this section, our aim is to find ground state solution to (1), whose potential is not a constant. In order to use a delicate method exploited by Jeanjean [32] (Theorem 1.1), for $\lambda \in [\frac{1}{2},1]$, we study a family of functional $E_{\lambda }:H_K^{\alpha }\left({\mathbb{R}}^3\right)\to \mathbb{R}$ defined by

$$E_{\lambda }\left(u\right)=\frac{1}{2}{\int }_{{\mathbb{R}}^3}{\left(\right|(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u{|}^2+K\left(x\right)u^2)dx-\frac{\lambda }{2}\left[{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))G\left(u\right)dx+\frac{2}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\right].$$

We get the next lemma, which is analogue to Lemma 1.

Lemma 14. 

Assume that $\left(K_1\right)$–$\left(K_2\right)$ and $\left(H_1\right)$ hold. Let u be a critical point of $E_{\lambda }$ in $H_K^{\alpha }\left({\mathbb{R}}^3\right)$, then the next Pohožaev type identity holds

$$\begin{array}{cc}\hfill {\mathcal{L}}_{\lambda }\left(u\right)& :=\frac{3-2\alpha }{2}{\int }_{{\mathbb{R}}^3}{\left|{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u\right|}^{2}dx+\frac{3}{2}{\int }_{{\mathbb{R}}^3}K\left(x\right)u^{2}dx+\frac{1}{2}{\int }_{{\mathbb{R}}^3}(\nabla K\left(x\right),x)u^{2}dx\hfill \\ & -\frac{(3+\beta )\lambda }{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))G\left(u\right)dx-\frac{3\lambda }{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx=0.\hfill \end{array}$$

Due to Lemma 14, set $J_{\lambda }\left(u\right)=2\alpha \langle E_{\lambda }^{\prime }\left(u\right),u\rangle -{\mathcal{L}}_{\lambda }\left(u\right)$ for all $\lambda \in [\frac{1}{2},1]$. Then

$$\begin{array}{cc}\hfill J_{\lambda }\left(u\right)& =\frac{6\alpha -3}{2}{\int }_{{\mathbb{R}}^3}{\left|{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u\right|}^{2}dx+\frac{1}{2}{\int }_{{\mathbb{R}}^3}[(4\alpha -3)K\left(x\right)-(\nabla K\left(x\right),x)]u^{2}dx\hfill \\ \hfill & +\frac{\lambda }{2}\left\{{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))[(3+\beta )G\left(u\right)-4\alpha g\left(u\right)u]dx-(6\alpha -3){\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\right\}.\hfill \end{array}$$

(34)

Let us set $E_{\lambda }\left(u\right)=A\left(u\right)-\lambda B\left(u\right)$, where

$$A\left(u\right)=\frac{1}{2}{\int }_{{\mathbb{R}}^3}{\left(\right|(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u{|}^2+K\left(x\right)u^2)dx\to +\infty ,$$

as ${\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}\to +\infty$ and

$$B\left(u\right)=\frac{1}{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))G\left(u\right)dx+\frac{1}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\ge 0.$$

Lemma 15. 

(i) 

There is $e>0$ such that $E_{\lambda }\left(e\right)<0$ for any $\lambda \in [\frac{1}{2},1]$.

(ii) 

$c_{\lambda }={inf}_{\mu \in \mathrm{\Gamma }}{max}_{\tau \in [0,1]}{E}_{\lambda }\left(\mu \left(\tau \right)\right)>max\{E_{\lambda }\left(0\right),E_{\lambda }\left(e\right)\}$ for all $\lambda \in [\frac{1}{2},1]$, where

$$\mathrm{\Gamma }=\{\mu \in \mathcal{C}([0,1],H_K^{\alpha }\left({\mathbb{R}}^3\right)):\mu \left(0\right)=0,\mu \left(1\right)=e\}.$$

Proof. 

(i)

For $u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}$ fixed and $\lambda \in [\frac{1}{2},1]$, define

$$\begin{array}{cc}\hfill E_{\infty }^{\lambda }\left(u\right)& =\frac{1}{2}{\int }_{{\mathbb{R}}^3}{\left(\right|(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u{|}^2+K_{\infty }{u}^2)dx\hfill \\ \hfill & -\lambda \left[\frac{1}{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))G\left(u\right)dx+\frac{1}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\right].\hfill \end{array}$$

(35)

Thus, from $\left(K_2\right)$, we obtain $E_{\lambda }\left(u\right)\le E_{\lambda }^{\infty }\left(u\right).$ Set $u_{\vartheta }={\vartheta }^{2\alpha }u\left(\vartheta x\right),\forall \vartheta >0$. It follows from Lemma 6 that

$$E_{\lambda }^{\infty }\left(u_{\vartheta }\right)\to -\infty ,\mathrm{as}\vartheta \to +\infty .$$

Take $e=u_{\vartheta }$ for $\vartheta$ large enough, (i) follows immediately.

(ii)

From (6), we observe

$$E_{\lambda }\left(u\right)\ge \frac{1}{2}{\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^2-{C(\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^{2+\frac{2\beta }{3}}+{\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^{2q}+{\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^{2_{\alpha }^{*}}),$$

which implies that there exist $\delta >0$ and $\sigma >0$ such that

$$E_{\lambda }\left(u\right)\ge \delta >0,\forall {\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}=\sigma ,\mathrm{for}\mathrm{any}\lambda \in [\frac{1}{2},1].$$

Thus, for any $\lambda \in [\frac{1}{2},1]$, there exists ${\tau }_0\in (0,1)$ such that $\parallel \mu \left({\tau }_0\right){\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}=\sigma$ and

$$\underset{\tau \in [0,1]}{max}{E}_{\lambda }\left(\mu \left(\tau \right)\right)\ge E_{\lambda }\left(\mu \left({\tau }_0\right)\right)\ge \delta >max\{E_{\lambda }\left(0\right),E_{\lambda }\left(e\right)\},$$

which yields $c_{\lambda }>0.$

□

Taking into account of Theorem 1.1 in [32] and Lemma 15, for any $\lambda \in [\frac{1}{2},1]$, we get a sequence $\left\{v_n\right\}\subset H_K^{\alpha }\left({\mathbb{R}}^3\right)$ which is bounded and satisfies

$$E_{\lambda }\left(v_n\right)\to c_{\lambda },E_{\lambda }^{\prime }\left(v_n\right)\to 0.$$

To prove the above sequence $\left\{v_n\right\}$ satisfies the $\left(PS\right)$ condition, we consider the following limit problem

$${(-\mathrm{\Delta })}^{\alpha }v+K_{\infty }v=\lambda (I_{\beta }\ast G\left(v\right))g\left(v\right)+\lambda {\left|v\right|}^{2_{\alpha }^{*}-2}v,\mathrm{in}{\mathbb{R}}^3.$$

(36)

By Theorem 1, Equation (36) admits a ground state solution $v_{\infty }^{\lambda }\in H_K^{\alpha }\left({\mathbb{R}}^3\right)$, i.e., for any $\lambda \in [\frac{1}{2},1]$, there exists $v_{\infty }^{\lambda }\in {\Pi }_{\infty }^{\lambda }$ such that

$${\left(E_{\infty }^{\lambda }\right)}^{\prime }\left(v_{\infty }^{\lambda }\right)=0,E_{\infty }^{\lambda }\left(v_{\infty }^{\lambda }\right)=c_{\infty }^{\lambda }=\underset{v\in \in {\Pi }_{\infty }^{\lambda }}{inf}{E}_{\infty }^{\lambda }\left(u\right),$$

where $E_{\infty }^{\lambda }\left(u\right)$ defined in (35),

$${\Pi }_{\infty }^{\lambda }=\{u\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}:J_{\infty }^{\lambda }\left(u\right)=0\},$$

$$\begin{array}{cc}\hfill J_{\infty }^{\lambda }\left(u\right)& =\frac{6\alpha -3}{2}{\int }_{{\mathbb{R}}^3}{\left|{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u\right|}^{2}dx+\frac{4\alpha -3}{2}{\int }_{{\mathbb{R}}^3}{K}_{\infty }{u}^{2}dx\hfill \\ & +\frac{\lambda }{2}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))[(3+\beta )G\left(u\right)dx-4\alpha g\left(u\right)u]dx-\frac{(6\alpha -3)\lambda }{2}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx.\hfill \end{array}$$

Lemma 16. 

For any $\lambda \in [\frac{1}{2},1]$ fixed,

$$c_{\lambda }<m_{\infty }^{\lambda },$$

where $c_{\lambda }$ is defined in Lemma 15 and $m_{\infty }^{\lambda }={inf}_{u\in {\Pi }_{\infty }^{\lambda }}{E}_{\infty }^{\lambda }\left(u\right)$.

Proof. 

Take $u_{\infty }^{\lambda }$ as the minimizer of $m_{\infty }^{\lambda }$. By Lemmas 10 and 15 and $\left(K_2\right)$, we observe that for all $\lambda \in [\frac{1}{2},1]$ and $\vartheta$ large enough,

$$c_{\lambda }\le \underset{\vartheta >0}{max}{E}_{\lambda }\left({\vartheta }^{2\alpha }{u}_{\infty }^{\lambda }\left(\vartheta x\right)\right)<\underset{\vartheta >0}{max}{E}_{\infty }^{\lambda }\left({\vartheta }^{2\alpha }{u}_{\infty }^{\lambda }\left(\vartheta x\right)\right)=E_{\infty }^{\lambda }\left(u_{\infty }^{\lambda }\right)=m_{\infty }^{\lambda }.$$

□

Lemma 17. 

Let $\left\{v_n\right\}$ be a bounded ${\left(PS\right)}_{c_{\lambda }}$ sequence of $E_{\lambda }$. Then there exist a sub-sequence of $\left\{v_n\right\}$, and integer $l\in \mathbb{N}\cup \left\{0\right\}$, a sequence $\left\{z_n^j\right\}\subset {\mathbb{R}}^3,{\omega }^j\in H_K^{\alpha }\left({\mathbb{R}}^3\right)$ for $1\le j\le l$ such that

(i) 

$v_n\rightharpoonup v_{\lambda }$ with $E_{\lambda }^{{}^{\prime }}\left(v_{\lambda }\right)=0$;

(ii) 

$z_n^j\to +\infty$ and $|z_n^i-z_n^j|\to +\infty$ for $i\ne j$;

(iii) 

${\omega }^i\ne 0$ and ${\left(E_{\infty }^{\lambda }\right)}^{\prime }\left({\omega }^i\right)=0$ for $1\le i\le l$;

(iv) 

$\parallel v_n-v_{\lambda }-{\sum }_{j=1}^l{\omega }^j(·-z_n^j){\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}\to 0$;

(v) 

$E_{\lambda }\left(v_n\right)\to E_{\lambda }\left(v_{\lambda }\right)+{\sum }_{j=1}^l{E}_{\infty }^{\lambda }\left({\omega }^j\right)$.

In addition, we agree that in the case $l=0$ the above hold without $w^j$ and $z_n^j$.

Proof. 

Since $\left\{v_n\right\}\subset H_K^{\alpha }\left({\mathbb{R}}^3\right)$ is a bounded sequence satisfying

$$E_{\lambda }\left(v_n\right)\to c_{\lambda }>0,E_{\lambda }^{\prime }\left(v_n\right)\to 0.$$

Then, there exists $v_{\lambda }\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}$ satisfying

$$\left\{\begin{array}{cc}{v}_n\rightharpoonup v_{\lambda }\hfill & \mathrm{in}{H}_K^{\alpha }\left({\mathbb{R}}^3\right),\hfill \\ v_n\to v_{\lambda }\hfill & \mathrm{in}{L}_{loc}^r\left({\mathbb{R}}^3\right)\mathrm{for}r\in [1,2_s^{*}),\hfill \\ v_n\to v_{\lambda }\hfill & \mathrm{a}.\mathrm{e}.\mathrm{in}{\mathbb{R}}^3.\hfill \end{array}\right.$$

Moreover, we can show that $E_{\lambda }^{\prime }\left(v_{\lambda }\right)=0$, and so $J_{\lambda }\left(v_{\lambda }\right)=0$. We deduce from (8), (34) and ($K_1$) that

$$\begin{array}{cc}\hfill E_{\lambda }\left(v_{\lambda }\right)& =E_{\lambda }\left(v_{\lambda }\right)-\frac{1}{6\alpha -3}{J}_{\lambda }\left(v_{\lambda }\right)\hfill \\ \hfill & =\frac{1}{6\alpha -3}{\int }_{{\mathbb{R}}^3}[\alpha K\left(x\right)+(\nabla K\left(x\right),x)]v_{\lambda }^{2}dx+\frac{\alpha \lambda }{3}{\int }_{{\mathbb{R}}^3}{\left|v_{\lambda }\right|}^{2_{\alpha }^{*}}dx\hfill \\ \hfill & +\frac{\lambda }{2(6\alpha -3)}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(v_{\lambda }\right))[4\alpha g\left(u_{\lambda }\right)v_{\lambda }-(6\alpha +\beta )G\left(v_{\lambda }\right)]dx\hfill \\ \hfill & \ge 0.\hfill \end{array}$$

(37)

Set $u_n^1=v_n-v_{\lambda }$, then we have $u_n^1\rightharpoonup 0$. In the sequel, one of two conclusions of $u_n^1$ holds:

Case 1: 

$u_n^1\to 0$ in $H_K^{\alpha }\left({\mathbb{R}}^3\right)$, or

Case 2: 

there exists a sequence $\left\{y_n^1\right\}\in {\mathbb{R}}^3$, $R_0>0,\delta >0$ such that

$$\underset{n\to \infty }{lim\; inf}{\int }_{B_{R_0}\left(y_n^1\right)}{\left|u_n^1\right|}^{2}dx\ge \delta >0.$$

(38)

In fact, suppose that Case 2 does not occur. Hence, for any $R>0$, we get

$$\underset{n\to \infty }{lim}\underset{y\in {\mathbb{R}}^3}{sup}{\int }_{B_R\left(y_n^1\right)}{\left|u_n^1\right|}^{2}dx=0.$$

Thus, Lemma 2.4 in [37] implies that $u_n^1\to 0$ in $L^s\left({\mathbb{R}}^3\right),s\in (2,2_{\alpha }^{*})$. In view of (6), we see that

$$\underset{n\to +\infty }{lim}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u_n^1\right))G\left(u_n^1\right)dx=0.$$

(39)

Moreover, we infer from Lemma 2 and (37) that

$$\underset{n\to \infty }{lim}{E}_{\lambda }\left(u_n^1\right)=\underset{n\to \infty }{lim}{E}_{\lambda }\left(v_n\right)-E_{\lambda }\left(v_{\lambda }\right)\le c_{\lambda }$$

(40)

and

$$\underset{n\to \infty }{lim}{\left(E_{\lambda }\right)}^{\prime }\left(u_n^1\right)=\underset{n\to \infty }{lim}{\left(E_{\lambda }\right)}^{\prime }\left(v_n\right)-{\left(E_{\lambda }\right)}^{\prime }\left(v_{\lambda }\right)=0.$$

(41)

By virtue of (39) and (41), we see

$$0=\underset{n\to \infty }{lim}\langle {\left(E_{\lambda }\right)}^{\prime }\left(u_n^1\right),u_n^1\rangle =\underset{n\to \infty }{lim}\left({\int }_{{\mathbb{R}}^3}{|(-\mathrm{\Delta })}^{\frac{\alpha }{2}}{u}_n^1{|}^{2}dx+{\int }_{{\mathbb{R}}^3}K\left(x\right){\left(u_n^1\right)}^{2}dx-\lambda {\int }_{{\mathbb{R}}^3}{\left|u_n^1\right|}^{2_{\alpha }^{*}}dx\right).$$

Since $\left\{u_n^1\right\}$ is bounded in $H_K^{\alpha }\left({\mathbb{R}}^3\right)$, then we can suppose that up to a subsequence, as $n\to \infty$,

$${\int }_{{\mathbb{R}}^3}{|(-\mathrm{\Delta })}^{\frac{\alpha }{2}}{u}_n^1{|}^{2}dx+{\int }_{{\mathbb{R}}^3}K\left(x\right){\left(u_n^1\right)}^{2}dx\to \chi ,\lambda {\int }_{{\mathbb{R}}^3}{\left|u_n^1\right|}^{2_{\alpha }^{*}}dx\to \chi$$

(42)

for some $\chi \ge 0$. If $\chi >0$, in view of (5), we obtain

$$\begin{array}{c}\hfill {\parallel u}_n^1{\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2+{\int }_{{\mathbb{R}}^3}K\left(x\right){\left(u_n^1\right)}^{2}dx\ge \parallel u_n^1{\parallel }_{{\mathcal{D}}^{\alpha ,2}\left({\mathbb{R}}^3\right)}^2\ge S_{\alpha }{\parallel u_n^1\parallel }_{L^{2_{\alpha }^{*}}\left({\mathbb{R}}^3\right)}^2.\end{array}$$

This together with (42) gives that

$$\chi \ge S_{\alpha }^{\frac{3}{2\alpha }}{\lambda }^{-\frac{3-2\alpha }{2\alpha }},\mathrm{for}\mathrm{any}\lambda \in [\frac{1}{2},1].$$

However, (40) implies that

$$\begin{array}{cc}\hfill c_{\lambda }& \ge \underset{n\to \infty }{lim}{E}_{\lambda }\left(u_n^1\right)\hfill \\ \hfill & =\underset{n\to \infty }{lim}\left[\frac{1}{2}\left({\int }_{{\mathbb{R}}^3}{\left|{(-\mathrm{\Delta })}^{\frac{\alpha }{2}}{u}_n^1\right|}^{2}dx+{\int }_{{\mathbb{R}}^3}K\left(x\right){\left(u_n^1\right)}^{2}dx\right)-\frac{\lambda }{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u_n^1\right|}^{2_{\alpha }^{*}}dx\right]\hfill \\ \hfill & \ge \frac{\alpha }{3}{S}_{\alpha }^{\frac{3}{2\alpha }}{\lambda }^{-\frac{3-2\alpha }{2\alpha }}.\hfill \end{array}$$

(43)

By using similar argument as in Lemmas 10, 12 and 13, we show that

$$m_{\infty }^{\lambda }<\frac{\alpha }{3}{S}_{\alpha }^{\frac{3}{2\alpha }}{\lambda }^{-\frac{3-2\alpha }{2\alpha }}.$$

Combining with (43) and Lemma 16, we obtain

$$\frac{\alpha }{3}{S}_{\alpha }^{\frac{3}{2\alpha }}{\lambda }^{-\frac{3-2\alpha }{2\alpha }}\le c_{\lambda }<m_{\infty }^{\lambda }<\frac{\alpha }{3}{S}_{\alpha }^{\frac{3}{2\alpha }}{\lambda }^{-\frac{3-2\alpha }{2\alpha }},\mathrm{for}\mathrm{all}\lambda \in [\frac{1}{2},1].$$

which is a contradiction. Thus, $\chi =0$. From (42), we conclude that $\parallel u_n^1{\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}\to 0$, that is, $v_n\to v$ in $H_K^{\alpha }\left({\mathbb{R}}^3\right)$ and Lemma 17 holds with $\chi =0$ if Case 2 does not occur.

In the following, we suppose that Case 2 is true, that is (38) holds. Then, up to a sub-sequence, we obtain

$$|z_n^1|\to +\infty ,u_n^1(·+z_n^1)\rightharpoonup {\omega }^1\ne 0,{\left(E_{\lambda }^{\infty }\right)}^{\prime }{\omega }^1=0.$$

Indeed, consider $\widehat{u_n^1}:=u_n^1(·+z_n^1)$. Note that $\left\{u_n^1\right\}$ is bounded. Then together with (38), we deduce that $\widehat{u_n^1}\rightharpoonup {\omega }^1\ne 0$. Therefore, it follows from $u_n^1\rightharpoonup 0$ in $H_K^{\alpha }\left({\mathbb{R}}^3\right)$ that $\left\{z_n^1\right\}$ is unbounded, up to a subsequence, $|z_n^1|\to +\infty$. Now we shall prove ${\left(E_{\lambda }^{\infty }\right)}^{\prime }\left({\omega }^1\right)=0$. It suffices to prove that $\langle {\left(E_{\lambda }^{\infty }\right)}^{\prime }\left(\widehat{u_n^1}\right),\psi \rangle \to 0$ for any $\psi \in {\mathcal{C}}_0^{\infty }\left({\mathbb{R}}^3\right)$.

According to (41), we obtain

$$|\langle E_{\lambda }^{{}^{\prime }}\left(u_n\right),\psi \rangle -\langle E_{\lambda }^{{}^{\prime }}\left(u_{\lambda }\right),\psi \rangle -\langle E_{\lambda }^{{}^{\prime }}\left(u_n^1\right),\psi \rangle |\le o_n\left(1\right){\parallel \psi \parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)},$$

which implies $|\langle E_{\lambda }^{{}^{\prime }}\left(u_n^1\right),\psi \rangle |=o_n\left(1\right){\parallel \psi \parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}$. Note that

$$\begin{array}{cc}\hfill & \langle E_{\lambda }^{{}^{\prime }}\left(u_n^1\right),\psi (·-z_n^1)\rangle \hfill \\ \hfill & ={\int }_{{\mathbb{R}}^3}{\int }_{{\mathbb{R}}^3}\frac{(u_n^1\left(x\right)-u_n^1\left(y\right))(\psi (x-z_n^1)-\psi (y-z_n^1))}{{|x-y|}^{3+2\alpha }}dxdy\hfill \\ \hfill & +{\int }_{{\mathbb{R}}^3}K\left(x\right)u_n^1\left(x\right)\psi (x-z_n^1)dx-\lambda {\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u_n^1\right))g\left(u_n^1\right)\psi (x-z_n^1)dx\hfill \\ \hfill & -\lambda {\int }_{{\mathbb{R}}^3}{\left|u_n^1\right|}^{2_{\alpha }^{*}-2}{u}_n^1\left(x\right)\psi (x-z_n^1)dx\hfill \\ \hfill & ={\int }_{{\mathbb{R}}^3}{\int }_{{\mathbb{R}}^3}\frac{(\widehat{u_n^1}\left(x\right)-\widehat{u_n^1}\left(y\right))(\psi \left(x\right)-\psi \left(y\right))}{{|x-y|}^{3+2\alpha }}dxdy+{\int }_{{\mathbb{R}}^3}K(x+z_n^1)\widehat{u_n^1}\left(x\right)\psi \left(x\right)dx\hfill \\ \hfill & -\lambda {\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(\widehat{u_n^1}\right))g\left(\widehat{u_n^1}\right)\psi \left(x\right)dx-\lambda {\int }_{{\mathbb{R}}^3}{\left|\widehat{u_n^1}\right|}^{2_{\alpha }^{*}-2}\widehat{u_n^1}\left(x\right)\psi \left(x\right)dx\hfill \\ \hfill & =o_n\left(1\right){\parallel \psi \parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}.\hfill \end{array}$$

(44)

Since $|z_n^1|\to +\infty$ and $\psi \in {\mathcal{C}}_0^{\infty }\left({\mathbb{R}}^3\right)$, we have

$${\int }_{{\mathbb{R}}^3}[K(x+z_n^1)-K_{\infty }]\widehat{u_n^1}\left(x\right)\psi \left(x\right)dx\to 0.$$

(45)

Combining (44) and (45), we obtain that for any $\psi \in {\mathcal{C}}_0^{\infty }\left({\mathbb{R}}^3\right)$, $\langle {\left(E_{\infty }^{\lambda }\right)}^{\prime }\left(\widehat{u_n^1}\right),\psi \rangle \to 0$.

By $\left(K_2\right)$ and $v_n\to v_{\lambda }\mathrm{in}{L}_{loc}^2\left({\mathbb{R}}^3\right)$, we can see

$${\int }_{{\mathbb{R}}^3}(K\left(x\right)-K_{\infty }){\left(u_n^1\right)}^{2}dx\to 0.$$

(46)

It follows immediately from (40) and (46) that

$$E_{\lambda }\left(u_n^1\right)\to c_{\lambda }-E_{\lambda }\left(v_{\lambda }\right),E_{\lambda }\left(v_n\right)-E_{\lambda }\left(v_{\lambda }\right)-E_{\infty }^{\lambda }\left(u_n^1\right)\to 0.$$

(47)

Set $u_n^2(·):=u_n^1(·)-{\omega }^1(·-z_n^1)$, then $u_n^2\rightharpoonup 0$ in $H_K^{\alpha }\left({\mathbb{R}}^3\right)$. Noting that $\widehat{u_n^1}\rightharpoonup {\omega }^1\ne 0$, we obtain

$$\begin{array}{cc}\hfill {\int }_{{\mathbb{R}}^3}K\left(x\right){\left|u_n^2\right|}^{2}dx& ={\int }_{{\mathbb{R}}^3}K\left(x\right)|u_n^1{|}^{2}dx+{\int }_{{\mathbb{R}}^3}K(x+z_n^1){\left|{\omega }^1\left(x\right)\right|}^{2}dx\hfill \\ \hfill & -2{\int }_{{\mathbb{R}}^3}K(x+z_n^1)u_n^1(x+z_n^1){\omega }^1\left(x\right)dx\hfill \\ \hfill & ={\int }_{{\mathbb{R}}^3}K\left(x\right)|v_n{|}^{2}dx-{\int }_{{\mathbb{R}}^3}K\left(x\right)|v_{\lambda }{|}^{2}dx-{\int }_{{\mathbb{R}}^3}{K}_{\infty }{\left|{\omega }^1\right|}^{2}dx+o_n\left(1\right).\hfill \end{array}$$

(48)

From (48), Brezis-Lieb Lemma, Lemma 2.6 in [35] and Lemma 2.9 in [38], we deduce that

$$\left\{\begin{array}{c}{E}_{\lambda }\left(u_n^2\right)=E_{\lambda }\left(u_n\right)-{\Phi }_{\lambda }\left(u_{\lambda }\right)-E_{\infty }^{\lambda }\left({\omega }^1\right)+o_n\left(1\right),\hfill \\ E_{\infty }^{\lambda }\left(u_n^2\right)={\Phi }_{\lambda }\left(u_n^1\right)-E_{\infty }^{\lambda }\left({\omega }^1\right)+o_n\left(1\right),\hfill \\ \langle E_{\lambda }^{\prime }\left(u_n^2\right),\psi \rangle =\langle E_{\lambda }^{\prime }\left(v_n\right),\psi \rangle -\langle E_{\lambda }^{\prime }\left(v_{\lambda }\right),\psi \rangle -\langle {\left(E_{\infty }^{\lambda }\right)}^{\prime }\left({\omega }^1\right),\psi \rangle +o_n\left(1\right)=o_n\left(1\right).\hfill \end{array}\right.$$

Therefore, together with (47), we obtain

$$E_{\lambda }\left(v_n\right)=E_{\lambda }\left(v_{\lambda }\right)+E_{\infty }^{\lambda }\left(u_n^1\right)+o_n\left(1\right)=E_{\lambda }\left(v_{\lambda }\right)+E_{\infty }^{\lambda }\left({\omega }^1\right)+E_{\infty }^{\lambda }\left(u_n^2\right)+o_n\left(1\right).$$

It follows from (37) and Lemma 16 that

$$E_{\infty }^{\lambda }\left(u_n^2\right)=c_{\lambda }-E_{\lambda }\left(u_{\lambda }\right)-E_{\infty }^{\lambda }\left({\omega }^1\right)\le c_{\lambda }.$$

Please notice that one of Case 1 and Case 2 is true for $v_n^2$. If Case 1 holds, then Lemma 17 holds with $l=1$. If Case 2 occurs, we repeat the above arguments. By iterating this process we have sequences of $\left\{z_n^j\right\}\subset {\mathbb{R}}^3$ such that $|z_n^j|\to +\infty ,|z_n^j-z_n^i|\to +\infty$ for $i\ne j$ and $u_n^j=u_n^{j-1}-{\omega }^{j-1}(·-z_n^{j-1})$ with $j\ge 2$ satisfying

$$u_n^j\rightharpoonup 0\mathrm{in}{H}_K^{\alpha }\left({\mathbb{R}}^3\right),{\left(E_{\infty }^{\lambda }\right)}^{\prime }\left({\omega }^j\right)=0$$

and

$$\left\{\begin{array}{c}\parallel v_n{\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^2-\parallel v_{\lambda }{\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^2-\sum _{j=1}^l{\parallel {\omega }^j(·-z_n^j)\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^2\hfill \\ =\parallel v_n-v_{\lambda }-\sum _{j=1}^l{\omega }^j(·-z_n^j){\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}+o_n\left(1\right),\hfill \\ E_{\lambda }\left(v_n\right)-E_{\lambda }\left(v_{\lambda }\right)-\sum _{j=1}^{l-1}{E}_{\infty }^{\lambda }\left({\omega }^j\right)-E_{\infty }^{\lambda }\left(u_n^l\right)=o_n\left(1\right).\hfill \end{array}\right.$$

(49)

In view of $\left\{v_n\right\}$ is bounded in $H_K^{\alpha }\left({\mathbb{R}}^3\right)$, (49) yields that the iteration stops at some l. That is, $u_n^{l+1}\to 0$ in $H_K^{\alpha }\left({\mathbb{R}}^3\right)$. From (49), it is easy to check that (iv) and (v) are true. The proof is complete. □

Lemma 18. 

For almost every $\lambda \in [\frac{1}{2},1]$, let $\left\{v_n\right\}$ be a bounded ${\left(PS\right)}_{c_{\lambda }}$ sequence of $E_{\lambda }$. Then there exists a subsequence $\left\{v_n\right\}$ converges to a nontrivial $v_{\lambda }\in H_K^{\alpha }\left({\mathbb{R}}^3\right)\setminus \left\{0\right\}$ satisfying

$$E_{\lambda }\left(v_{\lambda }\right)=c_{\lambda },{\left(E_{\lambda }\right)}^{\prime }\left(v_{\lambda }\right)=0.$$

Proof. 

From Lemma 17, up to a sub-sequence, there exists $v_{\lambda }\in H_K^{\alpha }\left({\mathbb{R}}^3\right)$, nontrivial critical points ${\omega }^j,j=1,\dots ,l$ of $E_{\infty }^{\lambda }$, $l\in \mathbb{N}\cup \left\{0\right\}$ and $\left\{z_n^j\right\}\subset {\mathbb{R}}^3$ with $|z_n^j|\to +\infty ,1\le j\le l$ such that

$$E_{\lambda }^{{}^{\prime }}\left(v_{\lambda }\right)=0,v_n\rightharpoonup v_{\lambda },E_{\lambda }\left(v_n\right)\to E_{\lambda }\left(v_{\lambda }\right)+\sum _{j=1}^l{E}_{\infty }^{\lambda }\left({\omega }^j\right).$$

Together with (37), we infer that if $l\ne 0$,

$$c_{\lambda }=\underset{n\to \infty }{lim}{E}_{\lambda }\left(v_n\right)=E_{\lambda }\left(v_{\lambda }\right)+\sum _{j=1}^l{E}_{\infty }^{\lambda }\left({\omega }^j\right)\ge m_{\infty }^{\lambda },$$

which contradicts with Lemma 16. Therefore, this lemma follows. □

Proof of Theorem 2. 

Taking a sequence $\left\{{\lambda }_n\right\}\subset [\frac{1}{2},1]$ satisfying ${\lambda }_n\to 1$, from Lemma 15, there is a sequence of nontrivial critical points $v_{{\lambda }_n}$ (we may still denote by $\left\{v_n\right\}$) for $E_{{\lambda }_n}$ and $E_{{\lambda }_n}\left(v_n\right)=c_{{\lambda }_n}$. Now, we prove that $\left\{v_n\right\}$ is bounded. It follows from (8) and $\beta >2\alpha$ that for every $\tau \in \mathbb{R}$,

$$g\left(\tau \right)\tau -2G\left(\tau \right)>g\left(\tau \right)\tau -\frac{6\alpha +\beta }{4\alpha }G\left(\tau \right)\ge 0.$$

Combining $\langle E_{{\lambda }_n}^{\prime }\left(v_n\right),v_n\rangle =0$ and $3<4\alpha$ we infer that

$$\begin{array}{cc}\hfill c_{\frac{1}{2}}\ge & c_{{\lambda }_n}\hfill \\ \hfill =& E_{{\lambda }_n}\left(v_n\right)-\frac{1}{4}\langle E_{{\lambda }_n}^{\prime }\left(v_n\right),v_n\rangle \hfill \\ \hfill =& \frac{1}{4}\parallel v_n{\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^2+\frac{{\lambda }_n}{4}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(v_n\right))[g\left(v_n\right)v_n-2G\left(v_n\right)]dx+\left(\frac{\alpha }{3}-\frac{1}{4}\right){\lambda }_n{\int }_{{\mathbb{R}}^3}{\left|v_n\right|}^{2_{\alpha }^{*}}dx\hfill \\ \hfill \ge & \frac{1}{4}{\parallel v_n\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}^2,\hfill \end{array}$$

which means that $\left\{v_n\right\}$ is bounded in $H_K^{\alpha }\left({\mathbb{R}}^3\right)$. Hence, by Theorem 1.1 in [32], we obtain that

$$\begin{array}{cc}& \underset{n\to \infty }{lim}E\left(v_n\right)\hfill \\ \hfill =& \underset{n\to \infty }{lim}\left\{E_{{\lambda }_n}\left(v_n\right)+\frac{{\lambda }_n-1}{2}\left[{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(v_n\right))G\left(v_n\right))dx+\frac{2}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|v_n\right|}^{2_{\alpha }^{*}}dx\right]\right\}\hfill \\ \hfill =& \underset{n\to \infty }{lim}{c}_{{\lambda }_n}=c_1,\hfill \end{array}$$

and

$$\begin{array}{cc}& \underset{n\to \infty }{lim}\langle E^{\prime }\left(v_n\right),\psi \rangle \hfill \\ \hfill =& \underset{n\to \infty }{lim}\left\{\langle E_{{\lambda }_n}^{\prime }\left(v_n\right),\psi \rangle +({\lambda }_n-1)\left[{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(v_n\right))g\left(v_n\right)\psi dx+{\int }_{{\mathbb{R}}^3}{\left|v_n\right|}^{2_{\alpha }^{*}-2}{v}_n\psi dx\right]\right\}\hfill \\ \hfill =& 0,\hfill \end{array}$$

which implies that $\left\{v_n\right\}$ is a bounded ${\left(PS\right)}_{c_1}$ sequence of E. Hence, in view of Lemma 18, there is a nontrivial critical point $v_0\in$ for E with $E\left(v_0\right)=c_1$.

At last, we prove there is a ground state solution to Equation (1). Let

$$m=inf\{E\left(u\right):u\ne 0,E^{\prime }\left(u\right)=0\}.$$

It is easy to see that $0\le m\le E\left(v_0\right)=c_1<+\infty$. For any v satisfying $E^{\prime }\left(v\right)=0$ and $\varrho >0$, we see that ${\parallel u\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}\ge \varrho$. While, it follows from $\left(K_1\right)$, $J\left(v\right)=0$ and (8) that

$$\begin{array}{cc}\hfill E\left(v\right)& =E\left(v\right)-\frac{1}{6\alpha -3}J\left(v\right)\hfill \\ & =\frac{1}{6\alpha -3}{\int }_{{\mathbb{R}}^3}[\alpha K\left(x\right)+(\nabla K\left(x\right),x)]v^{2}dx\hfill \\ & +\frac{1}{2(6\alpha -3)}{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(v\right))[4\alpha g\left(v\right)v-(6\alpha +\beta )G\left(v\right)]dx+\frac{\alpha }{3}{\int }_{{\mathbb{R}}^3}{\left|v\right|}^{2_{\alpha }^{*}}dx,\hfill \end{array}$$

which implies $m\ge 0$. Suppose $m=0$, then one has a critical point sequence $\left\{v_n\right\}$ of E with $E\left(v_n\right)\to 0$. Consequently,

$$\underset{n\to \infty }{lim}{\parallel v_n\parallel }_{L^{2_{\alpha }^{*}}\left({\mathbb{R}}^3\right)}^{2_{\alpha }^{*}}=0.$$

(50)

Similar as (20), we infer that

$${\int }_{{\mathbb{R}}^3}{(I}_{\beta }\ast G\left(v_n\right)g\left(v_n\right))v_n\le ϵ\parallel v_n{\parallel }_{L^2\left({\mathbb{R}}^3\right)}^{2+\frac{2\beta }{3}}+C_{ϵ}{\parallel v_n\parallel }_{L^{2_{\alpha }^{*}}\left({\mathbb{R}}^3\right)}^{2q},$$

which implies that

$$\underset{n\to \infty }{lim}{\int }_{{\mathbb{R}}^3}{(I}_{\beta }\ast G\left(v_n\right))g\left(v_n\right)v_n=0,$$

as $ϵ\to 0$. Combining with (50) and $\langle E^{\prime }\left(v_n\right),v_n\rangle =0$, we obtain ${lim}_{n\to \infty }{\parallel v_n\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}=0$, which contradicts with ${\parallel v}_n{\parallel }_{H_K^{\alpha }\left({\mathbb{R}}^3\right)}\ge \varrho$. Therefore, $0<m<+\infty$. Then let $\left\{v_n\right\}$ be a sequence such that $E^{\prime }\left(v_n\right)=0,E\left(v_n\right)\to m$. Similarly, we observe that $\left\{v_n\right\}$ is bounded. Using a similar proof of Lemma 18, we infer that there is $v\in H_K^{\alpha }\left({\mathbb{R}}^3\right)$ satisfying $E^{\prime }\left(v\right)=0$, $E\left(v\right)=m$. □

5. Conclusions

The main purpose of this paper is to study the existence of ground state solution for the fractional Choquard equation with critical Sobolev exponent. To prove Theorem 1, we first establish a key inequality

$$E_{\infty }\left(u\right)\ge E_{\infty }\left(u_{\vartheta }\right)+\frac{1-{\vartheta }^{6\alpha -3}}{6\alpha -3}{J}_{\infty }\left(u\right)+\xi \left(\vartheta \right){\int }_{{\mathbb{R}}^3}{K}_{\infty }{u}^{2}dx+\zeta \left(\vartheta \right){\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx.$$

(51)

Using (51), we can prove Lemmas 8–9, which investigate some properties of ${\Pi }_{\infty }$. Then we show $m_{\infty }={\overline{c}}_{\infty }:={inf}_{u\ne 0}{max}_{\vartheta >0}{E}_{\infty }\left(u_{\vartheta }\right).$ We use the general mini-max principle Proposition 2.8 in [36] to obtain a Cerami sequence for the functional $E_{\infty }$ with $J_{\infty }\left(v_n\right)\to 0$, where $E_{\infty },J_{\infty }\left(v_n\right)$ are given in (10) and (12), respectively. Finally, we conclude that $c_{\infty }$ is achieved by using an important estimate $c_{\infty }<\frac{\alpha }{3}{S}_{\alpha }^{\frac{3}{2\alpha }}$.

Next, we aim to find ground state solution to Equation (1). Due to the potential is not a constant, we use Jeanjean’s monotonicity trick. Define a family of functional

$$E_{\lambda }\left(u\right)=\frac{1}{2}{\int }_{{\mathbb{R}}^3}{\left(\right|(-\mathrm{\Delta })}^{\frac{\alpha }{2}}u{|}^2+K\left(x\right)u^2)dx-\frac{\lambda }{2}\left[{\int }_{{\mathbb{R}}^3}(I_{\beta }\ast G\left(u\right))G\left(u\right)dx+\frac{2}{2_{\alpha }^{*}}{\int }_{{\mathbb{R}}^3}{\left|u\right|}^{2_{\alpha }^{*}}dx\right].$$

We show that

$$c_{\lambda }=\underset{\mu \in \mathrm{\Gamma }}{inf}\underset{\tau \in [0,1]}{max}{E}_{\lambda }\left(\mu \left(\tau \right)\right)>max\{E_{\lambda }\left(0\right),E_{\lambda }\left(e\right)\}$$

for all $\lambda \in [\frac{1}{2},1]$, where

$$\mathrm{\Gamma }=\{\mu \in \mathcal{C}([0,1],H_K^{\alpha }\left({\mathbb{R}}^3\right)):\mu \left(0\right)=0,\mu \left(1\right)=e\}.$$

Together with Jeanjean’s monotonicity trick, we obtain a bounded sequence $\left\{u_n\right\}\subset H_K^{\alpha }\left({\mathbb{R}}^3\right)$ such that

$$E_{\lambda }\left(u_n\right)\to c_{\lambda },E_{\lambda }^{\prime }\left(u_n\right)\to 0.$$

To prove that the above sequence $\left\{v_n\right\}$ satisfies the $\left(PS\right)$ condition, we consider the following limit problem

$${(-\mathrm{\Delta })}^{\alpha }u+K_{\infty }u=\lambda (I_{\beta }\ast G\left(u\right))g\left(u\right)+\lambda {\left|u\right|}^{2_{\alpha }^{*}-2}u,\mathrm{in}{\mathbb{R}}^3$$

and conclude that $c_{\lambda }<m_{\infty }^{\lambda }$. Then we can obtain a global compactness result, i.e., Lemma 17, which implies that there exists a nontrivial critical point v for E.

In the proof, the restriction on $\alpha$ is very crucial, we do not know whether the solution can still exist for $\alpha \in (0,\frac{3}{4})$. This is a question that we need to further consider.