Quantum Magnetism in Wannier-Obstructed Mott Insulators

Abstract

We develop a strong coupling approach towards quantum magnetism in Mott insulators for Wannier-obstructed bands. Despite the lack of Wannier orbitals, electrons can still singly occupy a set of exponentially localized but nonorthogonal orbitals to minimize the repulsive interaction energy. We develop a systematic method to establish an effective spin model from the electron Hamiltonian using a diagrammatic approach. The nonorthogonality of the Mott basis gives rise to multiple new channels of spin-exchange (or permutation) interactions beyond Hartree–Fock and superexchange terms. We apply this approach to a Kagome lattice model of interacting electrons in Wannier-obstructed bands (including both Chern bands and fragile topological bands). Due to the orbital nonorthogonality, as parameterized by the nearest-neighbor orbital overlap g, this model exhibits stable ferromagnetism up to a finite bandwidth $W\sim Ug$, where U is the interaction strength. This provides an explanation for the experimentally observed robust ferromagnetism in Wannier-obstructed bands. The effective spin model constructed through our approach also opens up the possibility for frustrated quantum magnetism around the ferromagnet-antiferromagnet crossover in Wannier-obstructed bands.

1. Introduction

Mott insulators are correlated insulators where electrons singly occupy localized orbitals to avoid the repulsive interaction. In many cases, Mott insulators further develop antiferromagnetic ordering below the charge gap due to the superexchange interaction among low-energy spin degrees of freedom. However, in recent twisted bilayer graphene (tBLG) experiments [1,2], the observation of ferromagnetic hysteresis [3] suggests that the three-quarter-filling Mott-insulating state exhibits ferromagnetism. In another experiment on twisted double bilayer graphene (tDBLG) [4,5,6], an increasing gap under an in-plane magnetic field also suggests a ferromagnetic phase. More recently, ferromagnetism has also been observed in rhombohedral multilayer graphene [7,8] and twisted transition metal dichalcogenides (TMD) [9]. Such a ferromagnetic Mott state is understood in the flat band limit [10,11,12,13], where the spin exchange term in Coulomb interaction reduces the energy of the spin-polarized state. For similar reasons, the ferromagnetic phase in Moiré superlattice systems has a large overlap with the flat band ferromagnetism [14,15,16]. Faithful treatments of correlated Moiré superlattice systems have been proposed in various ways, including projecting Coulomb interaction onto the symmetry-broken (or obstruction-free) Wannier basis [17,18,19,20,21,22,23] and analyzing the interacting effects within momentum space in the weak coupling limit [15,16,24,25,26,27,28,29,30,31]. While most analyses consistently point to a robust ferromagnetism near the flat-band limit, it remains challenging to analyze the instability of such a ferromagnetic state in competition with the antiferromagnetic superexchange away from the flat-band limit.

One major obstacle to modeling the competition between exchange and superexchange effects systematically in tBLG has to do with the Wannier obstruction [32,33], i.e., if Wannier orbitals are constructed with only the relevant bands, they cannot respect all the symmetries [34,35]. To construct the Wannier orbitals for tBLG with all symmetries taken into account, the minimal model needs to contain at least ten bands as pointed out by H. C. Po et al. [24,36,37]. If we only focus on a few bands within the energy scale of interaction, the Wannier obstruction would prevent us from constructing Wannier orbitals. With the lack of Wannier orbitals, it becomes unclear how the electrons should localize to form Mott insulators, which further obscure the derivation of low-energy effective spin (and/or valley) models. The issue of Wannier obstruction has long been identified and studied in quantum Hall systems and other Chern insulators. In the phases with nonzero Chern number, exponentially localized Wannier orbitals cannot be constructed regardless of any symmetry considerations [38]. Now, with more fragile topological systems being discovered [39,40,41], the Wannier obstruction becomes a more general issue in the study of strongly correlated electronic systems.

We begin our general discussion by considering an extended Hubbard model on an arbitrary lattice,

$$\begin{array}{cc}H& =H_t+H_U,\hfill \\ H_t& ={\displaystyle \sum _{ij}}{t}_{ij}{c}_i^{†}{c}_j,\hfill \\ H_U& =\frac{1}{2}{\displaystyle \sum _{ij}}{U}_{ij}{n}_i{n}_j.\hfill \end{array}$$

(1)

where $c_i={(c_{i,\uparrow },c_{i,\downarrow })}^{⊺}$ is the electron operator containing spin degrees of freedom and $n_i=c_i^{†}{c}_i$ is the total electron number on site i. The model may be generalized to include orbital or valley [36] degrees of freedom, but in this work, we will only focus on spins for illustration purposes. We assume a certain degree of locality in $t_{ij}$ and $U_{ij}$. Suppose that $H_t$ produces several bands, described by the dispersion relations ${ϵ}_n\left(\mathit{k}\right)$,

$$H_t={\displaystyle \sum _{n\mathit{k}}}{c}_{n\mathit{k}}^{†}{ϵ}_n\left(\mathit{k}\right)c_{n\mathit{k}}.$$

(2)

In many cases, we are only interested in a subset of these bands near the Fermi energy with total bandwidth W, and well separated from other bands by energy gap Δ. Although the full band structure has a tight-binding model description, a subset of these bands may not. H. C. Po et al. [32,33] discussed several scenarios for a subset of bands, which can be trivial, obstructed trivial, fragile topological, or stably topological. As long as these bands are not trivial, they admit the Wannier obstruction, which is an obstruction towards constructing symmetric, exponentially localized and orthogonal Wannier orbitals by linearly combining Bloch states within the subset of bands. The well-known ones are those with a Chern number where no additional symmetry is required [38,42,43]. We also have fragile topology where they are obstructed, but the obstruction can be removed by adding trivial bands (essentially site localized bands) below the Fermi energy, which can then be mixed with the bands of interest to obtain Wannier orbitals [39,40,41]. Finally, for the obstructed trivial bands, the obstruction is only due to the fact that the Wannier center does not reside on a lattice site, which can be resolved by adding empty sites. The absence of such Wannier orbitals prevents us from writing down an effective tight-binding model targeting only those subsets of bands of interest. This is precisely the case for tBLG, where the relevant conduction and valance bands are fragile topological and hence Wannier-obstructed by the $C_2\mathcal{T}$ symmetry [24,36,37].

Weak coupling approaches have been developed to treat the interaction perturbatively (as $U\ll W$) [24,25,27], such that one only needs to work with a momentum space description of the effective band structure near the Fermi surface, hence circumventing the Wannier obstruction. However, it remains challenging to understand the strong coupling physics in Wannier-obstructed bands when the energy scales are arranged in the following hierarchy

$$W\ll U\ll \Delta .$$

(3)

The band gap Δ, as the leading energy scale, protects a set of Wannier-obstructed low-energy bands from mixing with high-energy bands away from the Fermi surface. To further respect the interaction energy U, electrons should repel each other into real space localized orbitals by combining states in the low-energy bands. In the standard notion of Mott insulators, electrons are localized in Wannier orbitals with charge fluctuation gapped and spin fluctuation remained active at low energy [44,45,46,47,48,49]. Now, in the absence of Wannier orbitals, does the many-body Hamiltonian in Equation (1) still admit Mott-like ground states? If so, can we write down the trial wave function to describe the low-lying states? Can we derive an effective model to describe the spin dynamics at low energy?

Motivated by these questions, we take a closer look at the requirements of Wannier orbitals, namely symmetry, locality and orthogonality. If we sacrifice one or more of them, the Wannier obstruction can be lifted and it would be possible to construct orbitals to host electrons in a Mott-like state. If we sacrifice the symmetry requirement, we will have to fine tune the hopping parameters to fit the band structure. If we sacrifice the locality requirement, we will end up with a non-local hopping model which is hard to deal with. So, we decided to explore the possibility of sacrificing orthogonality and work with a set of nonorthogonal Wannier bases. This approach is analogous to the Maki–Zotos wavefunction [50] and the von Neumann lattice formulation [51,52,53] in quantum Hall systems with nonzero but negligible orbital overlap.

We present the general theory based on nonorthogonal Wannier basis with finite orbital overlap in Section 2. The nonorthogonality of the orbitals leads to new spin exchange channels and chiral spin exchange channels. The competition among these new channels could lead to a rich magnetic phase diagram. We discuss leading order contributions to the effective spin Hamiltonian in Section 3 and discuss several new channels emerging from the theory of nonorthogonal basis. We also propose an energetic objective function in Section 4 to construct these nonorthogonal orbitals numerically to facilitate the study of any concrete model. Within this framework, we study a toy model proposed in Ref. [33] in Section 5 to demonstrate our framework in Wannier-obstructed bands. In this model, we show that the flat-band ferromagnetism remains stable up to finite band width and a variety of magnetic phases appears around the the ferromagnet-antiferromagnet crossover.

2. Theory of Nonorthogonal Basis

In this section, we discuss how to project the Hamiltonian in Equation (1) to a nonorthogonal spin basis of low-energy states and how to treat perturbative corrections. Let us assume a set of localized and normalized but nonorthogonal orbitals ${\varphi }_I\left(i\right)$ in the real space labeled by the orbital index I, which jointly labels the unit cell and the orbital within the unit cell. We will leave the energetic criterion to optimize these orbitals and their completeness as a set of basis for later discussions in Section 4. For now, we assume that electrons will self-organize under repulsive interaction to develop such nonorthogonal localized orbitals. The nonorthogonality implies a non-trivial metric $g_{IJ}$ among these orbitals,

$$g_{IJ}\equiv {\displaystyle \sum _i}{\varphi }_I^{*}\left(i\right){\varphi }_J\left(i\right)\ne {\delta }_{IJ}.$$

(4)

Nevertheless, we assume the normalization condition $g_{II}=1$ for all I. Given these orbitals, we can define a set of fermion operators $a_{I\sigma }^{†}$ that create electrons residing on these orbitals,

$$a_{I\sigma }^{†}={\displaystyle \sum _i}{\varphi }_I\left(i\right)c_{i\sigma }^{†},(\sigma =\uparrow ,\downarrow )$$

(5)

such that they satisfy the following anticommutation relations $\{a_{I\sigma },a_{J\tau }\}=\{a_{I\sigma }^{†},a_{J\tau }^{†}\}=0$ and $\{a_{I\sigma },a_{J\tau }^{†}\}=g_{IJ}{\delta }_{\sigma \tau }$. These operators $a_{I\sigma }^{†}$ allow us to construct a set of many-body trial states from the vacuum state $|\mathrm{vac}\rangle$,

$$|{\mathsf{\Psi }}_{\mathit{\sigma }}\rangle ={\displaystyle \prod _I}{a}_{I{\sigma }_I}^{†}|\mathrm{vac}\rangle .$$

(6)

In these many-body states, every orbital is singly occupied and the spin configuration is labelled by $\mathit{\sigma }=\left\{{\sigma }_I\right\}$. The nonorthogonality of single particle orbitals also implies the nonorthogonality of these many-body states $\langle {\mathsf{\Psi }}_{\mathit{\sigma }}|{\mathsf{\Psi }}_{\mathit{\tau }}\rangle \ne {\delta }_{\mathit{\sigma }\mathit{\tau }}$. In the Mott limit $U\gg W$, the trial states $|{\mathsf{\Psi }}_{\mathit{\sigma }}\rangle$ span the low energy manifold $\mathcal{H}$. All the charge degrees of freedom are frozen, and the spin degrees of freedom are still allowed to fluctuate, resembling the Mott states. We can then derive the effective theory for these spin degrees of freedom and investigate the resulting phases.

2.1. Exchange Interactions and Beyond

To formulate an effective spin model, we first introduce the spin Hilbert space $\tilde{\mathcal{H}}$ spanned by a fictitious set of orthogonal Ising basis $|\mathit{\sigma }\rangle$, which allows us to define the spin operator ${\mathit{S}}_I=(S_I^x,S_I^y,S_I^z)$ in the conventional way $\langle \mathit{\sigma }|S_I^a|\mathit{\tau }\rangle =\frac{1}{2}{\sigma }_{{\tau }_I{\sigma }_I}^a{\prod }_{J\ne I}{\delta }_{{\tau }_J{\sigma }_J}$, where ${\sigma }_{\tau \sigma }^a$ denotes the Pauli matrix element. We would like to comment that the Ising states $|\mathit{\sigma }\rangle$ do not directly correspond to physical electronic states, but merely play a bookkeeping role to provide a convenient basis for the purpose of representing spin operators. The relation between different Hilbert spaces is illustrated in Figure 1.

We want to project the many-body Hamiltonian Equation (1) to the low energy subspace $\mathcal{H}$ of electrons and then translate it to the spin space $\tilde{\mathcal{H}}$. The complication arises from the nonorthogonality of the many-body trial states $|{\mathsf{\Psi }}_{\mathit{\sigma }}\rangle$ that span the low energy subspace $\mathcal{H}$. Here we present a systematic approach to deal with the nonorthogonality. First we introduce a non-unitary linear transformation $A:\tilde{\mathcal{H}}\to \mathcal{H}$ to map the Ising basis $|\mathit{\sigma }\rangle$ to the many-body trial states $|{\mathsf{\Psi }}_{\mathit{\sigma }}\rangle$ with corresponding spin configuration,

$$A={\displaystyle \sum _{\mathit{\sigma }}}|{\mathsf{\Psi }}_{\mathit{\sigma }}\rangle \langle \mathit{\sigma }|.$$

(7)

Next we define the adjoint operator $A^{†}={\sum }_{\mathit{\sigma }}|\mathit{\sigma }\rangle \langle {\mathsf{\Psi }}_{\mathit{\sigma }}|$, such that $\langle {\mathsf{\Psi }}_{\mathit{\sigma }}|A\mathit{\tau }\rangle =\langle A^{†}{\mathsf{\Psi }}_{\mathit{\sigma }}|\mathit{\tau }\rangle$. Note that A is not a unitary transformation, so $A^{†}\ne A^{-1}$. In fact, $A^{-1}={\sum }_{\mathit{\sigma }}|\mathit{\sigma }\rangle \langle {\overline{\mathsf{\Psi }}}_{\mathit{\sigma }}|$, where $\left\{\right|{\overline{\mathsf{\Psi }}}_{\mathit{\sigma }}\rangle \}$ is the dual basis to $\left\{\right|{\mathsf{\Psi }}_{\mathit{\sigma }}\rangle \}$ such that $\langle {\overline{\mathsf{\Psi }}}_{\mathit{\sigma }}|{\mathsf{\Psi }}_{\mathit{\tau }}\rangle ={\delta }_{\mathit{\sigma }\mathit{\tau }}$. Now, we can use the operators A and $A^{†}$ to project the identity operator $\mathbb{1}$ and Hamiltonian H to the Ising basis,

$$\begin{array}{cc}\hfill \mathbb{1}& \to A^{†}A\equiv G,\hfill \\ \hfill H& \to A^{†}HA\equiv \tilde{H}.\hfill \end{array}$$

(8)

Under the projection, the original eigen problem $H|\mathsf{\Psi }\rangle =E|\mathsf{\Psi }\rangle$ becomes a generalized eigen problem by inserting the identity operator $A{A}^{-1}$ and multipling $A^{†}$ from the left,

$$\tilde{H}|\mathsf{\Phi }\rangle =EG|\mathsf{\Phi }\rangle ,$$

(9)

where $|\mathsf{\Phi }\rangle \equiv A^{-1}|\mathsf{\Psi }\rangle$ is the representation of the many-body eigenstate $|\mathsf{\Psi }\rangle$ in the Ising basis. Now, the generalized eigen problem in Equation (9) is formulated in the spin Hilbert space $\tilde{\mathcal{H}}$ with a nice orthogonal basis.

After the projection, we can expand the many-body Gram matrix G and the projected Hamiltonian $\tilde{H}$ as linear combinations of permutation operators ${\mathcal{X}}_{\mathcal{P}}$ in the spin space,

$$\begin{array}{cc}G& ={\displaystyle \sum _{\mathcal{P}\in S_N}}{(-)}^{\mathcal{P}}{G}_{\mathcal{P}}{\mathcal{X}}_{\mathcal{P}},\hfill \\ \tilde{H}& ={\displaystyle \sum _{\mathcal{P}\in S_N}}{(-)}^{\mathcal{P}}{H}_{\mathcal{P}}{\mathcal{X}}_{\mathcal{P}},\hfill \end{array}$$

(10)

where $S_N$ denotes the permutation group over all orbitals I, ${(-)}^{\mathcal{P}}$ is the sign of the permutation $\mathcal{P}$, and ${\mathcal{X}}_{\mathcal{P}}$ is the permutation operator that permutes the spins among different orbitals ${\mathcal{X}}_{\mathcal{P}}|\left\{{\sigma }_I\right\}\rangle =|\left\{{\sigma }_{{\mathcal{P}}^{-1}\left(I\right)}\right\}\rangle$. For two-spin and three-spin permutations, ${\mathcal{X}}_{\mathcal{P}}$ can be expressed with the familiar Heisenberg term and chiral spin term,

$$\begin{array}{cc}\hfill {\mathcal{X}}_{\left(IJ\right)}& =\frac{1}{2}+2{\mathit{S}}_I·{\mathit{S}}_J,\hfill \\ \hfill {\mathcal{X}}_{\left(IJK\right)}& =\frac{1}{4}+{\mathit{S}}_I·{\mathit{S}}_J+{\mathit{S}}_J·{\mathit{S}}_K+{\mathit{S}}_K·{\mathit{S}}_I\hfill \\ \hfill & \phantom{=}-2\mathrm{i}{\mathit{S}}_I·({\mathit{S}}_J\times {\mathit{S}}_K).\hfill \end{array}$$

(11)

To specify the coefficients $G_{\mathcal{P}}$ and $H_{\mathcal{P}}$, we introduce the hopping tensor $t_{IJ}$ and the interaction tensor $U_{IJKL}$,

$$\begin{array}{cc}{t}_{IJ}& =\sum _{ij}{\varphi }_I^{*}\left(i\right)t_{ij}{\varphi }_J\left(j\right),\hfill \\ U_{IJKL}& =\sum _{ij}{\varphi }_I^{*}\left(i\right){\varphi }_J\left(i\right)U_{ij}{\varphi }_K^{*}\left(j\right){\varphi }_L\left(j\right),\hfill \end{array}$$

(12)

where $t_{ij}$ and $U_{ij}$ are the bare hopping and interaction coefficients in H, as introduced in Equation (1). Given the tensors $g_{IJ}$ in Equation (4) and $t_{IJ},U_{IJKL}$ in Equation (12), the coefficients $G_{\mathcal{P}}$, $H_{\mathcal{P}}\equiv T_{\mathcal{P}}+U_{\mathcal{P}}$ in Equation (10) are given by

$$\begin{array}{cc}{G}_{\mathcal{P}}& ={\displaystyle \prod _I}{g}_{\mathcal{P}\left(I\right)I},\hfill \\ T_{\mathcal{P}}& ={\displaystyle \sum _I}{t}_{\mathcal{P}\left(I\right)I}{\displaystyle \prod _{J\ne I}}{g}_{\mathcal{P}\left(J\right)J},\hfill \\ U_{\mathcal{P}}& =\frac{1}{2}{\displaystyle \sum _{I\ne J}}{U}_{\mathcal{P}\left(I\right)I\mathcal{P}\left(J\right)J}{\displaystyle \prod _{K\ne I,J}}{g}_{\mathcal{P}\left(K\right)K},\hfill \end{array}$$

(13)

where $T_{\mathcal{P}}$ and $U_{\mathcal{P}}$ denotes the contribution from the hopping and the interaction terms, respectively.

To simplify the notation, we introduce the diagrammatic representation: a closed loop of arrows represents a permutation cycle among the orbitals. If an arrow from I to J is labeled by t or g, it contributes a factor of $t_{IJ}$ or $g_{IJ}$; if two arrows, e.g., one from I to J and the other from K to L, are connected and labeled by U, it contributes a factor of $U_{IJKL}$. For example,

(14)

Using the diagrammatic representations, one can expand G and $\tilde{H}$ in terms of spin permutations ${\mathcal{X}}_{\mathcal{P}}$ order by order. To the order of two-spin exchange, we have

(15)

where the permutation operator ${\mathcal{X}}_{\left(IJ\right)}$ can eventually be written in terms of spin operators as in Equation (11). Higher order permutations can be included systematically based on Equation (10). For all summations appeared in Equation (15), it is assumed that the orbitals labeled by different indices do not coincide. In the orthogonal limit $g_{IJ}={\delta }_{IJ}$, all diagrams in Equation (15) that contains g-labeled (red) arrows will vanish, such that G reduces back to the identity operator $\mathbb{1}$ and $\tilde{H}$ reduces to the following three terms

(16)

The first diagram is the band energy. The second and third diagrams are, respectively, the Hartree and the Fock energies between electrons from orbitals I and J (where the Fock interaction is accompanied with the spin exchange ${\mathcal{X}}_{\left(IJ\right)}$). For nonorthogonal orbitals, the metric $g_{IJ}$ becomes non-trivial, then non-vanishing terms (containing g-labeled arrows) in Equation (15) suggest contributions beyond Hartree–Fock approximation, which lead to new channels of spin exchange interactions. Furthermore, there exist three- and even more spin interactions that are conventionally only present through super-exchange effects that are suppressed by electron interaction. The various spin permutation interactions competing with each other could result in a frustrated quantum magnet with a rather rich phase diagram. To determine the ground state of the low-energy spin degrees of freedom in Wannier-obstructed Mott insulators, one will need to solve the generalized eigen problem in Equation (9).

2.2. Superexchange Interactions from Perturbation

In the above discussion, we project the many-body Hamiltonian H to the low-energy subspace $\mathcal{H}$ spanned by the Mott states $|{\mathsf{\Psi }}_{\mathit{\sigma }}\rangle$ to derive the effective spin model in the flat-band limit $W\ll U$. Away from the flat-band limit, the electrons can virtually hop to the neighboring orbitals and back, which gives rise to the superexchange interactions among the spins. To capture such effect, we should go beyond the low-energy subspace $\mathcal{H}$ and consider the perturbation effects in orders of $(t/U)$.

In the following, we analyze the perturbative corrections to the effective spin model within our framework. In Appendix A, we review the generalized perturbation theory for nonorthogonal basis. We apply the perturbation theory to the spin dynamics in the low energy manifold $\mathcal{H}$ by treating the hopping Hamiltonian $H_t$ in Equation (2) as a small perturbation. (Strictly speaking, we should treat the bandwidth W as perturbation, such that $H_t$ corresponds to the difference $H_W-H_{W=0}$, where $H_{W=0}$ corresponds to the band-flattened version of the band structure, but this will not affect our formulation of the general approach). Consider the high energy subspace spanned by states $|n,\alpha \rangle$ with double occupancy but still the same number of electrons, where n labels the number of doubly occupied orbitals and $\alpha$ labels the configuration. Assuming that the zeroth order energy is completely determined by n and states with different n’s are orthogonal, i.e., $\langle n,\alpha |m,\beta \rangle ={\delta }_{mn}{G}_{n\alpha \beta }$, we obtain the following energy correction from the second-order perturbation theory

$$\langle {\mathsf{\Psi }}_{\mathit{\sigma }}|H^{\left(2\right)}|{\mathsf{\Psi }}_{\mathit{\tau }}\rangle =-\sum _{n>0}\frac{\langle {\mathsf{\Psi }}_{\mathit{\sigma }}|H_t|n,\alpha \rangle G_n^{\alpha \beta }\langle n,\beta |H_t|{\mathsf{\Psi }}_{\mathit{\tau }}\rangle }{E_n-E_0}$$

(17)

where $G_n^{\alpha \beta }$ is the inverse of $G_{n\alpha \beta }$. Since the dominant contribution of $\langle {\mathsf{\Psi }}_{\mathit{\sigma }}\left|H_t\right|n,\alpha \rangle$ comes from states with only one doubly occupied site, we can approximate Equation (17) by

$$\langle {\mathsf{\Psi }}_{\mathit{\sigma }}|H^{\left(2\right)}|{\mathsf{\Psi }}_{\mathit{\tau }}\rangle \approx -\frac{1}{U}\left(\langle {\mathsf{\Psi }}_{\mathit{\sigma }}|H_t^2-H_t{\mathbb{1}}_{\mathsf{\Psi }}{H}_t|{\mathsf{\Psi }}_{\mathit{\tau }}\rangle \right)$$

(18)

where U is the energy required to create one double occupancy, and ${\mathbb{1}}_{\mathsf{\Psi }}={\sum }_{\mathit{\sigma }}|{\mathsf{\Psi }}_{\mathit{\sigma }}\rangle \langle {\overline{\mathsf{\Psi }}}_{\mathit{\sigma }}|$ is the projection operator to the low energy subspace. Now, we can use the projection introduced previously to write down the Hamiltonian ${\tilde{H}}^{\left(2\right)}\equiv A^{†}{H}^{\left(2\right)}A$ in the Ising basis,

$${\tilde{H}}^{\left(2\right)}=-\frac{1}{U}\left(\widetilde{H_t^2}-{\tilde{H}}_t{G}^{-1}{\tilde{H}}_t\right),$$

(19)

where $\widetilde{H_t^2}\equiv A^{†}{H}_t^{2}A$ follows our convention, and in the second term we use $A{A}^{-1}={\left(A^{†}\right)}^{-1}{A}^{†}={\mathbb{1}}_{\mathsf{\Psi }}$. We can further write ${\tilde{H}}^{\left(2\right)}$ in terms of hopping tensors,

$$\begin{array}{cc}\hfill {\tilde{H}}^{\left(2\right)}& =-\frac{1}{U}(\sum _{\mathcal{P}\in S_N}{(-)}^{\mathcal{P}}{\left(T^2\right)}_{\mathcal{P}}{\mathcal{X}}_{\mathcal{P}}\hfill \\ \hfill & -\sum _{\mathcal{P}\circ \mathcal{Q}\in S_N}{(-)}^{\mathcal{P}\circ \mathcal{Q}}{T}_{\mathcal{P}}{G}^{-1}{T}_{\mathcal{Q}}{\mathcal{X}}_{\mathcal{P}\circ \mathcal{Q}}),\hfill \end{array}$$

(20)

where ∘ denotes the composition of permutations (such that ${\mathcal{X}}_{\mathcal{P}\circ \mathcal{Q}}={\mathcal{X}}_{\mathcal{P}}{\mathcal{X}}_{\mathcal{Q}}$), and the coefficients $T_{\mathcal{P}}$ and ${\left(T^2\right)}_{\mathcal{P}}$ are given by

$$\begin{array}{cc}\hfill T_{\mathcal{P}}& =\sum _I{t}_{\mathcal{P}\left(I\right)I}\prod _{J\ne I}{g}_{\mathcal{P}\left(J\right)J},\hfill \\ \hfill {\left(T^2\right)}_{\mathcal{P}}& =\sum _I{\left(t^2\right)}_{\mathcal{P}\left(I\right)I}\prod _{J\ne I}{g}_{\mathcal{P}\left(J\right)J}\hfill \\ \hfill & \phantom{=}+\sum _{I\ne J}{t}_{\mathcal{P}\left(I\right)I}{t}_{\mathcal{P}\left(J\right)J}\prod _{K\ne I,J}{g}_{\mathcal{P}\left(K\right)K},\hfill \end{array}$$

(21)

and ${\left(t^2\right)}_{IJ}={\sum }_{ijk}{\varphi }_I^{*}\left(i\right)t_{ik}{t}_{kj}{\varphi }_J\left(j\right)$. To the order of two-spin exchange, we have

(22)

For all summations appeared in Equation (22), it is assumed that orbitals labeled by different indices do not coincide. In the orthogonal limit $g_{IJ}={\delta }_{IJ}$, all diagrams containing g-labeled arrows will vanish, such that the second-order perturbation

(23)

contains only the usual $t^2/U$ antiferromagnetic superexchange interaction. When we allow non-trivial $g_{IJ}$, both $T_{\mathcal{P}}$ and ${\left(T^2\right)}_{\mathcal{P}}$ can give rise to new channels in spin interactions. Mediated by $g_{IJ}$, three- or higher order spin interactions can also arise even at the level of second-order perturbation in $(t/U)$. Similar treatment can be generalized to higher order perturbation theory.

3. Effective Spin Hamiltonian

In this section, we discuss how to solve the effective spin model constructed in Section 2. There are two major challenges. First, the model is presented as a generalized eigenvalue problem in Equation (9), which requires to diagonalize a complicated operator $G^{-1}\tilde{H}$ that is not guaranteed to be short-ranged on the lattice. Second, the summation over all permutation in Equation (10) is hard to track even numerically. Both challenges can be resolved by separating connected permutations and disconnected permutations. As demonstrated in Equation (15), diagrams in $\tilde{H}$ can be organized by the connected diagram (containing t or U), each followed by a series of disconnected diagrams (containing g only). The series of disconnected diagrams is similar to G, which motivates us to factor G out of $\tilde{H}$. However, residue terms are generated due to over counting diagrams with colliding indices, illustrated as follows

(24)

where ${\mathcal{P}}_0$ sums over single-cycle (i.e., connected) permutations and ${\mathcal{P}}_{1,2,\cdots }$ sum over those permutations that have non-vanishing index overlap with every other permutation (including ${\mathcal{P}}_0$) in the diagram. The explicit expression and a detailed convergence analysis of the entire series can be found in Appendix B. The rough idea is that the expansion is controlled by the small parameter $g\equiv |g_{\langle IJ\rangle } |\ll 1$ for well-localized orbitals ${\varphi }_I$. Since n-spin interaction can only be generated by ${\mathcal{X}}_{\mathcal{P}}$ with length $\left(\mathcal{P}\right)\ge n$, they are suppressed by at least $g^{n-2}$. Thus, the spin dynamics is still dominated by few-spin interactions as expected. Furthermore, for these few-spin interactions of small n, the contribution from sub-leading terms in the series Equation (24) is further suppressed by $n{g}^2$ compared to the leading $g^{n-2}$ term. Thus, for those dominating few-spin interactions, we only need to consider the leading order connected diagrams:

$$G^{-1}\tilde{H}\simeq H_c\equiv \sum _{{\mathcal{P}}_0\in S_N^{*}}{(-)}^{{\mathcal{P}}_0}{H}_{{\mathcal{P}}_0}{\mathcal{X}}_{{\mathcal{P}}_0},$$

(25)

where $S_N^{*}$ is the set of single-cycle permutations in $S_N$. With this approximation, the general eigenvalue problem in Equation (9) reduces to the ordinary eigenvalue problem

$$H_c|\mathsf{\Phi }\rangle =E|\mathsf{\Phi }\rangle$$

(26)

where $H_c$ collects the connected pieces in $\tilde{H}$. The problem reduces to solving the effective spin Hamiltonian $H_c$ on a set of orthogonal basis $|\mathit{\sigma }\rangle$, for which many well-developed analytical and numerical tools in quantum magnetism can be applied. Similar treatment applies to the perturbation theory described in Section 2.2, where the $G^{-1}$ in Equation (19) has cancelled the disconnected pieces in one of the ${\tilde{H}}_t$’s. Then, the effective spin Hamiltonian $H_c$ in Equation (26) is corrected by $H_c^{\left(2\right)}$.

In conclusion, despite of the nonorthogonality of the Mott basis and the complication of the generalized eigen problem, we can still work with an effective spin Hamiltonian $H_c$ in a ordinary eigen problem to describe the low-energy spin degrees of freedoms approximately. The full spin-rotation symmetry dictates the spin Hamiltonian $H_c$ to take the general form of

$$H_c=\sum _{\langle IJ\rangle }{J}_{IJ}{\mathit{S}}_I·{\mathit{S}}_J+\sum _{\langle IJK\rangle }{K}_{IJK}{\mathit{S}}_I·({\mathit{S}}_J\times {\mathit{S}}_K)+\cdots$$

(27)

up to three-spin interactions. Here we keep only the near neighbor interactions and analyze the coupling strengths of the Heisenberg interaction ${\mathit{S}}_I·{\mathit{S}}_J$ and the chiral spin interaction ${\mathit{S}}_I·({\mathit{S}}_J\times {\mathit{S}}_K)$. In Equation (15), among the terms attached with ${\mathcal{X}}_{\langle IJ\rangle }$, to the leading order of g, those contribute to the Heisenberg interaction are

(28)

The second term is the familiar Fock exchange term, which is always positive and thus provides inter-site Hund’s coupling. We remark that the Fock term is non-vanishing even in the orthogonal limit due to the density-density overlap between orbitals [16,20]. The first term is a new channel arising from nonorthogonality, which can be either ferromagnetic or antiferromagnetic depending on its sign. This channel could potentially provide a stronger antiferromagnetism in the strong coupling (large U) limit than the usual $t^2/U$ superexchange antiferromagnetism [54], which may enhance the magnetic frustration in the spin model. When the competition between ferromagnetism and antiferromagnetism reaches a balance in certain parameter regime, higher-order spin interactions will start to dominate the spin model. For example, there are two terms attached with ${\mathcal{X}}_{\langle IJK\rangle }$ that contribute to the three-spin ring exchange interaction,

(29)

Both terms give rise to new channels contributing to the chiral spin interaction ${\mathit{S}}_I·({\mathit{S}}_J\times {\mathit{S}}_K)$. Compared to the usual $t^3/U^2$ chiral spin interaction from the 3rd order superexchange channel [55,56], these nonorthogonality enabled exchange channels could provide stronger chiral spin interaction in the strong coupling (large U) limit, in favor of the chiral spin liquid ground state [57].

We can repeat the analysis for the second-order perturbative correction ${\tilde{H}}^{\left(2\right)}$. It can also be approximated by the connected part $H_c^{\left(2\right)}$ as argued previously. If we look at the Heisenberg interaction and the chiral spin interaction,

$$H_c^{\left(2\right)}=\sum _{\langle IJ\rangle }{J}_{IJ}^{\left(2\right)}{\mathit{S}}_I·{\mathit{S}}_J+\sum _{\langle IJK\rangle }{K}_{IJK}^{\left(2\right)}{\mathit{S}}_I·({\mathit{S}}_J\times {\mathit{S}}_K)+\cdots$$

(30)

To the leading order in g, we obtain

(31)

The Heisenberg term $J_{IJ}^{\left(2\right)}$ contains contributions from the standard superexchange channel. The chiral spin term $K_{IJK}^{\left(2\right)}$ contains contributions from a new channel as nonorthogonal ring superexchange.

Collecting all contributions from Equations (28), (29) and (31), there are three channels that contribute to the Heisenberg interaction—the $gt$ term from the nonorthogonal exchange, the U term from the conventional exchange, and the $t^2/U$ term from the superexchange; and there are four channels that contribute to the chiral spin interaction—the $g^{2}t$ and $gU$ terms from the nonorthogonal ring exchange in Equation (29), the $g{t}^2/U$ term from second-order perturbation theory, and the conventional $t^3/U^2$ term from third-order perturbation theory (which will appear in $H_c^{\left(3\right)}$). In the strong coupling (large U) limit, the novel channels originated from the orbital nonorthogonality typically dominate over the conventional superexchange and ring exchange channels. They are crucial to the analysis of the magnetism in Wannier-obstructed Mott insulators.

4. Constructing Localized Orbitals

In previous discussions, we have established the low-energy effective spin model starting from the assumption of electrons localized on a set of nonorthogonal orbitals ${\varphi }_I\left(i\right)$. Now, we come back to discuss why such arrangement is favorable and how these orbitals should be determined. In correlated materials, when the interaction energy dominates over the band width $U\gg W$, it becomes energetically favorable to recombine single-particle states in the energy band to form localized orbitals, and to arrange one electron in each localized orbital to reduce the repulsive interaction. Although the Wannier obstruction prevents us from constructing orthogonal Wannier orbitals, it does not prevent us from constructing nonorthogonal and localized orbitals, on which electrons can reside. The criterion is to minimize the total energy of the system. Thus, we start from the trial many-body state $|{\mathsf{\Psi }}_{\mathit{\sigma }}\rangle$ proposed in Equation (6), and minimize its energy $\langle {\mathsf{\Psi }}_{\mathit{\sigma }}\left|H\right|{\mathsf{\Psi }}_{\mathit{\sigma }}\rangle$ so as to optimize the localized orbitals ${\varphi }_I\left(i\right)$ that were used to construct the trial state $|{\mathsf{\Psi }}_{\mathit{\sigma }}\rangle$.

However, the energy $\langle {\mathsf{\Psi }}_{\mathit{\sigma }}\left|H\right|{\mathsf{\Psi }}_{\mathit{\sigma }}\rangle$ still depends on the spin configurations $\mathit{\sigma }$, which makes the objective function undetermined. To proceed, we focus on the “spin-independent part” of the energy, which is naturally the constant piece in the effective spin model $H_c$. It corresponds to the Hartree energy $H_{\left(\right)}$ in front of the identity operator ${\mathcal{X}}_{\left(\right)}$, as given in Equation (13),

(32)

Here ${ϵ}_n(\mathrm{i}\nabla )$ denotes a real space representation of the band structure ${ϵ}_n\left(\mathit{k}\right)$. We will assume that the nth band ${ϵ}_n$ is well separated from other bands by a large band gap Δ, which is much larger than the interaction strength U. The separating energy scales ($\Delta \gg U$) allows us to focus on the nth band only and find the optimal orbitals that can minimize the energy $H_{\left(\right)}$. For the purpose of designing lattice models, the required separation of energy scales ($\Delta \gg W$) can be realized by applying the band flattening approach [58,59,60]. The energy optimization $\delta H_{\left(\right)}/\delta {\varphi }_I=0$ boils down to solving the following Gross–Pitaevskii (GP) equation,

$$\left(\sum _n{ϵ}_n(\mathrm{i}\nabla )+\sum _{I\ne J}\sum _j{U}_{ij}{\left|{\varphi }_J\left(j\right)\right|}^2\right){\varphi }_I\left(i\right)=E{\varphi }_I\left(i\right).$$

(33)

The ground state can be found by imaginary time evolution of ${\varphi }_I\left(i\right)$ under the GP Hamiltonian (i.e., the operator on the left-hand-side of Equation (33)). In each step of the evolution, the updated ${\varphi }_I\left(i\right)$ orbital will be broadcasted to other unit cells by translation. During the evolution, we do not impose the orthogonality among Wannier orbitals, so the orbitals we eventually obtain are in general nonorthogonal. The interaction will automatically determine whether or not the optimal orbitals will spontaneously break the point group symmetry.

We make a remark on the completeness of the localized orbitals. As is well known, for a single Chern band (suppose it is isolated for simplicity), it is impossible to choose a set of Bloch states $|{\psi }_{\mathit{k}}\rangle$ such that it is normalized and smooth in the Brillouin zone torus [38]. This implies a Wannier obstruction to construct a set of exponentially localized orbitals which are orthonormal, complete and related to each other by translations. What if we relax the orthonormality constraint? Unfortunately, it is still impossible to find exponential localized orbitals which are complete and translation invariant, even if they are allowed to be nonorthogonal. If such orbitals exist, we can Fourier transform to obtain a set of unnormalized but smooth and nowhere vanishing Bloch states. Further normalizing these states leads to normalized and smooth Bloch states, causing a contradiction. It turns out that, under certain assumption which holds for the example we will be considering, it is possible to find a set of exponentially localized orbitals which are complete but nonorthogonal and also breaks the translation symmetry for one orbital. In other words, $N-1$ number of orbitals are related to each other by translations but the last orbital takes a different form, where N is the total number of orbitals. We prove this claim in Appendix C. The orbital obtained from the energy optimization procedure described above, if exponentially localized, can be used to generate these $N-1$ translation related orbitals, and a different last orbital is needed to complete a basis. We expect a single orbital to have little effect on the overall physics of the system, thus we ignore this subtlety hereafter. In Appendix C, we also show that the dual orbital formalism, which has been useful in the study of Hubbard model ferromagnetism in topologically trivial bands [11,12], is not applicable to a Chern band with the orbitals we constructed. This is one of the reasons that we take a different approach in this work.

5. Application

5.1. Kagome Lattice Model and Wannier Obstructions

In this section, we apply our theory to an interacting fermion model on a Kagome lattice, whose Hamiltonian $H=H_t+H_U$ takes the form of Equation (1). The hopping Hamiltonian $H_t$ consists of purely imaginary nearest-neighbor hopping only, with $t_{ij}=\mathrm{i}/2$ for $j\to i$ along the bond direction as specified in Figure 2a, such that each triangle carries a quantum flux $-\pi /2$ and each hexagon carries a quantum flux $\pi$. The model was introduced by Ref. [33] to demonstrate fragile topological insulators. This lattice model preserves translation symmetry and six-fold rotation symmetry $C_6$ (about the hexagon center). The single particle spectrum consists of three bands fully gapped from each other as shown in Figure 2b.

The middle band is strictly flat with zero Chern number $C=0$, which is an obstructed trivial band. Its Wannier obstruction is only due to the lack of lattice sites at the hexagon center Wyckoff position, such that the obstruction can be lifted by adding empty sites. The top and bottom bands are Chern bands of Chern number $C=\pm 1$, respectively. They combined together to form a set of fragile topological bands, which are Wannier-obstructed and do not admit an effective tight-binding model description (despite their total Chern number being zero) [33,61], because the symmetry representations at high-symmetry momentum points do not match those of any atomic insulators of the same lattice symmetry. This situation is analogous to the middle bands in the tBLG around the charge neutrality, which has a fragile topological band structure for each valley [32,37]. Placing the tBLG on the aligned hexagonal boron nitride (hBN) substrate further opens up the band gap at charge neutrality. The top and bottom bands are valley Chern bands of opposite Chern numbers [15], which resemble the Chern bands in the Kagome lattice model as in Figure 2b. Similar Chern bands also appeared in Kagome lattice materials Cs₂LiMn₃F₁₂ [62] and Co₃Sn₂S₂ [63,64]. The Chern number is correlated with the spin state due to spin–orbit coupling effects.

In the following, we will first set aside the specific material realization and focus on the Kagome lattice model itself. By applying our proposed approach to analyze this toy model, we wish to gain a general understanding of Wannier-obstructed Mott insulators, which could facilitate future study of correlated insulating phases in Moiré superlattice systems.

5.2. Nonorthogonal Localized Orbitals in Wannier Obstructed Bands

We follow the method described in Section 4 to construct the localized but nonorthogonal Wannier orbitals. If we isolate the middle flat band and apply on-site repulsive interaction $U_{ij}=U_0{\delta }_{ij}$, the orbital ${\varphi }_I\left(i\right)$ that minimizes the energy $H_{\left(\right)}$ defined in Equation (32) is found to be strictly localized around the hexagon as shown in Figure 3a. This orbital is similar symmetry-wise to the localized orbitals proposed in Ref. [65] for a different model, where the strict localization in both cases comes from the destructive interference between orbitals.

If we focus on the bottom Chern band with $C=-1$ and again apply the on-site repulsion $U_0$, we find two degenerated solutions of ${\varphi }_I\left(i\right)$ as shown in Figure 3c,d. These orbitals have similar features as the Wannier orbitals in twisted bilayer graphene (tBLG) [34,66]. They have three major peaks around the triangular lattice site and are equipped with non-zero angular momentum. To make a connection to the tBLG system, we follow the notation in Ref. [36] to label these orbitals by the Wyckoff positions of their orbital centers and the angular momenta with respect to their orbital centers. The hexagon and the triangle center Wyckoff positions are denoted by a and b, respectively, as in Figure 2a, and angular momentum $0,\pm 1,\pm 2$ are denoted by s, $p_{\pm }$ and $d_{\pm }$. The orbital types are summarized in the table of Figure 3.

In the Mott limit, electrons will spontaneously choose one of the $(b,p_{-})$ orbitals in Figure 3c,d in which to reside, which spontaneously breaks the $C_6$ rotation symmetry to the $C_3$ subgroup and results in the $C_3$ nematic phases. However, if we include longer range interactions, the energetically most favorable orbital can be different. For example, if we apply third neighbor repulsion $U_{\langle \langle \langle ij\rangle \rangle \rangle }=U_3$ to the bottom Chern band, we obtain a $(a,d_{+})$ orbital as shown in Figure 3b. This orbital preserves the $C_6$ rotation symmetry, so the corresponding Mott phases are not nematic. As shown in Figure 4, all the orbitals we constructed are indeed exponentially localized. As a result, the tensors $g_{IJ}$, $t_{IJ}$ and $U_{IJKL}$ should all decay exponentially with the inter-orbital distance, so we expect the resulting effective spin model to exhibit well-controlled locality.

5.3. Effective Spin Models and Possible Phases

In the Mott limit, the spin dynamics for any of these orbitals can be described by the effective spin Hamiltonian on a triangular lattice, as shown in Figure 5. The spin-rotational symmetric Hamiltonian takes the following form

$$H=J_1\sum _{\langle IJ\rangle }{\mathit{S}}_I·{\mathit{S}}_J+\sum _{\langle IJK\rangle \in △/▽}{K}_{△/▽}{\mathit{S}}_I·({\mathit{S}}_J\times {\mathit{S}}_K)+\dots$$

(34)

where three sites $IJK$ surrounding both up and down triangular plaquettes are arranged in the counterclockwise order. Again we focus on the bottom Chern band. The coupling strengths of the nearest-neighbor Heisenberg interaction $J_1$ and the chiral spin interactions $K_{△/▽}$ are plotted in Figure 6 as a function of $t/U_0$ or $t/U_3$.

Now, we briefly discuss the symmetries of this effective Hamiltonian and direct readers to Appendix D for more details. The electron Hamiltonian has translation symmetries $T_1$ and $T_2$ along two different directions, six-fold rotation symmetry $C_6$, and an anti-unitary mirror symmetry $\mathcal{M}$. The optimal localized orbitals ${\varphi }_I$ can spontaneously break some of the symmetries. For example, the $(b,p_{-})$ orbital breaks $C_6$ to $C_3$ (see Figure 5). Given the symmetries of the orbitals, we can infer the symmetry of tensors $g_{IJ}$, $t_{IJ}$ and $U_{IJKL}$. It turns out that between nearest-neighboring sites I and J, $t_{IJ}$ ($g_{IJ}$) can be generated by a single parameter $t\equiv |t_{\langle IJ\rangle }|$ ($g\equiv |g_{\langle IJ\rangle }|$) given $T_{1,2}$, $C_3$ and $\mathcal{M}$. Then, the chiral spin interaction $K_{▽}/K_{△}$ result from $t_{IJ}$ and $g_{IJ}$ must be opposite on neighboring triangular plaquettes, i.e., $K_{△}=-K_{▽}$. The interaction tensor $U_{IJKL}$ breaks this pattern, but $K_{▽}/K_{△}$ result from $U_{IJKL}$ is much smaller than others in this model, so we still have a good approximate symmetry $K_{△}\simeq -K_{▽}$. Furthermore, if we have $C_6$ symmetry like in the case of $(a,d_{+})$ orbital, $t_{IJ}$ and $g_{IJ}$ will be restricted to real numbers and the chiral spin interaction is restricted to be uniform $K_{△}=K_{▽}$. This symmetry analysis is in agreement with a previous study on a similar model [20].

For the $(b,p_{-})$ orbital favored by the onsite interaction $U_0$, the Heisenberg interaction changes from ferromagnetic (FM) to anti-ferromagnetic (AFM) as t increases (Figure 6a). In this case, the nonorthogonality enabled channel $t_{IJ}{g}_{JI}$ favoring AFM. Thus, AFM interaction starts to dominate after this new channel takes over at large t. The same physics happens for the $(a,d_{+})$ orbital favored by the third neighbor interaction $U_3$, while the transition occurs at a much smaller t (Figure 6b). Close to the transition, there can be intermediate phases that require taking the chiral spin interaction into account. It turns out that near the FM-AFM transition regime, the chiral spin interaction is dominated by the $t{g}^2$ channel in Equation (29) (see Appendix D). The $(a,d_{+})$ orbital respects the $C_6$ symmetry, which constrains the chiral spin interaction to be vanishing small (Figure 6b). On the other hand, the $(b,p_{-})$ orbital breaks the $C_6$ symmetry, so $K_{△}$ and $K_{▽}$ are approximately related by the staggered pattern (Figure 6a).

Now, we compute the coupling strengths perturbatively with two control parameters t and g to obtain insight for their behavior in more general orbitals. We approximate the orbitals by only major and secondary peaks, and the orbital wave function is determined by the angular momentum and the amplitude ratio between major and secondary peaks tunable by g. Most importantly, the nearest orbital overlap g parameterizes the nonorthogonality of the orbitals, and controls the competition between different magnetic phases. Since g only slightly depends on the magnitude of U, we approximate it by a constant in the following analysis. For the $(b,p_{-})$ orbital, to the leading order in the hopping t and the orbital overlap g, we have

$$J_1=\frac{t^2}{3U_0}+\frac{2}{3}tg-U_0{g}^2,$$

(35)

where new channels in the theory of nonorthogonal basis contribute to those terms that contain g. The Heisenberg coupling $J_1$ changes sign at ${(U_0/t)}^{*}=1/g$ as shown in Figure 6a. In the flat band limit $U_0/t\gg 1/g$, the FM exchange dominates. In Appendix E, we provide a rigorous proof of the ferromagnetism for Wannier-obstructed Mott insulators in the flat band limit. As the band dispersion becomes larger (but still on the strong coupling side) $1/g\gg U_0/t\gg 1$, the AFM interaction takes over. Due to the geometric frustration on the triangular lattice (especially when higher order spin interactions are also taken into account), several candidate orders may compete for the ground state, which we will leave for later discussion.

From the above analysis, we expect the FM phase to become unstable toward AFM-like phases around $U_0/t\simeq 1/g$. However, around this point, the exchange interaction $J_1$ tends to vanish, so we need to consider higher order interactions, e.g., the chiral spin interaction,

$$K_{▽}=-K_{△}=\frac{t^3}{2U_0^2}-\frac{t^{2}g}{U_0}-\frac{5}{3}t{g}^2.$$

(36)

Again, terms that contain g arise from nonorthogonality-enabled channels. The chiral spin interaction dominates the spin model $|K_{△,▽}|>|J_1|$ in a narrow window of $\left|U_0/t-1/g\right|<13/8$. A similar analysis can be repeated for the $(a,d_{+})$ orbital under third neighbor repulsion $U_3$. Combining this information, we obtain a schematic phase diagram in Figure 7.

As a reminder, the coefficients in Equations (35) and (36) are specific to the localized orbital in our model. However, we expect the Heisenberg interaction $J_1$ to be quadratic in $t/U$ and g for generic systems, with different order-one coefficients, such that the FM-AFM cross over happens at $t/U\sim 1/g$ scale. A similar analysis can be generalized to the chiral spin interaction which is cubic in $t/U$ and g.

The spin model in Equation (34) can give rise to different phases. We first consider the case when the on-site interaction $U_0$ dominates, which favors the $(b,p_{-})$ orbital. The $(b,p_{-})$ orbital breaks the $C_6$ symmetry to $C_3$ spontaneously, resulting in nematic phases. In this case, the chiral spin interaction is approximately staggered $K_{△}\simeq -K_{▽}$. We present a classical picture of possible phases in the following. Away from the $U_0/t\sim 1/g$ transition regime, the nearest Heisenberg interaction $J_1$ dominates, which leads to a Heisenberg FM state when $J_1<0$ and a 120° AFM state when $J_1>0$. At the classical level, the in-plane 120° AFM state can be tuned towards the z-axis FM state by canting the 120° spin configuration in the x-y plane toward the z-axis, as illustrated in the inset of Figure 7. The canted 120° AFM configuration happens to have opposite spin chirality between up and down triangles, which is indeed favored by the staggered chiral spin interaction. We denote this intermediate state as the canted state in Figure 6 and Figure 7, which carries both non-zero magnetization and staggered scalar spin chirality. This spin configuration has been previously studied as an umbrella-type noncoplanar phase in Ref. [67] within a Kondo system under different settings. Since both the canted state and the 120° AFM state spontaneously break the spin $\mathrm{U}\left(1\right)$ symmetry, they actually belong to the same phase. In contrast, there has to be a phase transition between the canted AFM phase and the Heisenberg FM phase which preserves the spin $\mathrm{U}\left(1\right)$ symmetry (Figure 7). However, this classical picture might be modified under quantum fluctuations and longer-range geometric frustrations. We then comment on the other case when the longer-range interaction becomes important. For example, the third-neighboring interaction $U_3$ favors the $(a,d_{+})$ orbital, which preserves all the lattice symmetries. In this case, the chiral spin term is parametrically small. The sign change of $J_1$ still happens when $U_3/t$ is around the order of $1/g$, which drives the transition between the FM and AFM phases. Such a transition is likely first-order (Figure 7). Finally, when $U/t\sim 1$, the system is no longer captured by the strong coupling theory presented in this work. We simply denote the weak coupling phase as the Fermi liquid (FL) phase, whose instability should be further analyzed using weak coupling approaches. The phase diagram presented here agrees qualitatively with various theoretical and numerical studies of the Hubbard model in topological bands [20,23,68,69,70,71,72,73,74,75].

Let us further remark on some previous studies on the spin model in Equation (34) regarding more exotic phases due to quantum fluctuation. Though it is believed that uniform flux $K_{△}\simeq K_{▽}$ can drive the system toward a chiral spin liquid (CSL) phase [76,77], it is not clear what happens in the staggered limit $K_{△}\simeq -K_{▽}$. Some numerical study [78] and parton construction [79] on related models suggest a possibility of gapless spin liquid. When we further add a next-nearest-neighbor AFM coupling $J_2$ to the model [80,81,82,83], it can drive the system toward a Dirac spin liquid (DSL) phase before the system fully develops a stripe order. These are all possible phases of the effective spin model we construct through nonorthogonal projection and perturbation. We will leave these rich possibilities for future numerical investigations.

6. Conclusions

In this work, we present a different approach to understanding Mott physics in Wannier-obstructed systems including Chern insulators and fragile topological systems like twisted bilayer graphene. To avoid the obstruction, we sacrifice the orthogonality of the Wannier basis and develop a method to construct the trial nonorthogonal Wannier orbitals by numerically optimizing the Hartree energy of the system. In the Mott limit, we fill these trial orbitals with one electron per orbital. To study the low energy spin dynamics, we systematically project the Hamiltonian to a nonorthogonal spin basis and further study perturbative corrections. This new procedure concerning nonorthogonal Wannier basis gives rise to new channels to spin interactions. For example, at the level of direct projection, we find new channels that contribute to ferromagnetic, antiferromagnetic, or spin liquid phases. We demonstrate our approach with a toy model that carries Chern bands and fragile topological bands. In this model, new channels widen the antiferromagnetic phase and enhance chiral spin interactions that may lead to rich magnetic phases.

Our result may shed light on the magnetism in Moiré superlattice systems, which often host Wannier-obstructed bands. For example, the two middle bands near the charge neutrality in twisted bilayer graphene (tBLG) are identified to be fragile topological bands. [24,36,37] Aligning the tBLG with hexagonal boron nitride (hBN) substrate, the fragile topological bands further develop into separate Chern bands within each valley, which is analogous to the Chern bands in our toy model. The observation of ferromagnetic hysteresis [3] suggests that the three-quarter-filling insulating state in such a system exhibits ferromagnetism. As the tBLG band structure only preserves the $C_3$ rotation symmetry within each valley, the scenario is similar to the $U_0$ dominated case in our toy model, where the localized $(b,p_{-})$ orbital exhibits the famous fidget spinner structure [24,34,66].

Our analysis shows a non-vanishing inter-site ferromagnetic coupling from the Fock term due to finite overlap g between nonorthognal orbitals even in the limit when the bandwidth W approaches zero. This ferromagnetism becomes unstable when the bandwidth increases up to $Ug$ set by the interaction strength U and the orbital nonorthogonality g, which is inconsistent with previous studies on narrow Chern bands [16]. When the system is close to the ferromagnet-antiferromagnet crossover, our approach provides a systematic framework to write down an explicit effective spin model that enables further numerical investigation of intermediate phases. Meanwhile, new channels from our framework also open up the possibility of new and richer magnetic phases close to the crossover.