Looking beyond the Standard Model with Third Generation Quarks at the LHC

Abstract

The Large Hadron Collider (LHC) is at the frontier of collider physics today, probing new physics at unprecedented energy scales. Many theories of physics beyond the Standard Model seek to elucidate the underlying mechanism of electroweak symmetry breaking. Given their large Yukawa couplings to the Higgs boson, third generations quarks of the Standard Model, and especially the top quark, play a key role in such theories. Therefore, new particles predicted by these theories often couple preferentially to top and bottom quarks. The favoured coupling to third generation can also be used to explain recently observed flavour physics anomalies in the LHCb, Babar or Belle experiments. This article will review recent searches for new physics performed by the ATLAS and CMS experiments at the LHC, in final states containing top and bottom quarks. In particular, searches for vector-like quarks, leptoquarks, and heavy scalar and gauge bosons will be discussed.

1. Introduction

The study of third generation quarks is one of the main focuses of research at the LHC [1]. Their high mass, especially in the case of the top-quark, makes them of particular interest the mechanism of electroweak symmetry breaking and the Higgs-boson properties. Many theories beyond the Standard Model (SM) like composite Higgs [2,3,4,5], little Higgs [6] or those with extra dimensions [7] predict new heavy particles that can decay into third-generation quarks, some of them with preferential or exclusive couplings [8,9,10]. Searches with third generation quarks in the final state are a crucial element of any programme that looks for these exotics particles and are the topic of a wide array of searches performed by both the ATLAS [11] and CMS [12] Collaborations.

This review presents a collection of those searches, all of them using $pp$ collision data at a centre-of-mass energy of 13 TeV. It is divided into three different sections: Vector-Like-Quark (VLQ) searches, Leptoquark (LQ) searches, and other resonant searches that are not aiming to specific models but rather look for resonant behaviour in final states with at least one top-quark. Searches specifically aimed at supersymmetric particles are not included.

1.1. The Phenomenology of Vector-like Quarks

The origin of the large hierarchy between the electroweak mass scale (and the Higgs boson mass along with it) and the Planck scale is an open question in the SM. Naturalness arguments [13] require that quadratic divergences that arise from radiative corrections to the Higgs boson mass are cancelled out by some new mechanism in order to avoid fine-tuning. Several theories beyond the SM (BSM) attempt to provide a solution to this hierarchy problem.

The fine tuning can be resolved by a new strongly interacting sector, in which the Higgs boson would be a pseudo–Nambu–Goldstone boson (pNGB) [14] of a spontaneously broken global symmetry. The Composite Higgs [2,3,4] model is a particular realisation of this scenario, which also addresses other open questions in the SM, including the hierarchy in the mass spectrum of the SM particles. In the typical Composite Higgs scenario, the top quark would be a mostly composite particle [15], while all other SM fermions would be mostly elementary. New fermionic resonances called vector-like quarks (VLQs) naturally arise in this scenario, as they do in many other BSM models. Vector-like quarks are defined as colour-triplet spin-1/2 fermions whose left- and right-handed chiral components have the same transformation properties under the weak-isospin SU(2) gauge group [16,17]. Renormalisable and gauge complete models restrict the SU(2) representation of the vector-like quarks to seven possible multiplets consisting of: singlets ($T^{2/3}$) or ($B^{-1/3}$), doublets ($X^{5/3}{T}^{2/3}$) or ($T^{2/3}{B}^{-1/3}$) or triplets ($X^{5/3}{T}^{2/3}{B}^{-1/3}$) or ($T^{2/3}{B}^{-1/3}{Y}^{-4/3}$). As can be seen from the above, the $T^{2/3}$ and $B^{-1/3}$ quarks, carrying the same quantum numbers as the SM t and b quarks respectively, are required to be present in all possible multiplets. Additionally, the multiplets can contain the up-type $X^{5/3}$ quark and the down-type $Y^{-4/3}$ quarks with non-standard isospins. The $X^{5/3}$ quark commonly arises in Composite Higgs models as a result of custodial symmetries [18], and is often the lightest vector-like quark state in these models.

The vector-like quarks in these models couple preferentially to third-generation quarks and can have flavour-changing neutral-current decays in addition to the charged-current decays characteristic of chiral quarks [16,19]. Thus, the up-type T quark can decay into a W boson and a b-quark, and also to a top quark and a Z or Higgs boson. The relative couplings for these interactions are determined by the gauge representation of the vector-like quarks. The relative couplings to the W, Z, and Higgs bosons can be expressed in terms of the ${\xi }_W$, ${\xi }_Z$ and ${\xi }_H$ parameters, respectively [20]. For a T or B singlet, ${\xi }_W=0.5$ and ${\xi }_Z={\xi }_H=0.25$. For T quarks that are in an ($XT$) doublet, or in a ($TB$) doublet with mixing only to up-type SM singlets [19,20], ${\xi }_W=$0, and ${\xi }_Z={\xi }_H=$0.5. The coupling to W bosons is similarly suppressed for B quarks in ($TB$) doublets. On the other hand, B quarks in (B Y) doublets have vanishing coupling to Higgs bosons, such that ${\xi }_H=$0, and ${\xi }_W={\xi }_Z=$0.5. In all gauge multiplets, $X^{5/3}$ quarks couple uniquely to W bosons, and only $Wb$ decays are allowed for $Y^{-4/3}$ quarks.

Vector-like quarks with masses below ∼1 TeV would be predominantly produced in pairs via the strong interaction at the LHC. The process is illustrated for T quarks in the left plot of Figure 1. However, this process becomes kinematically disfavoured at larger masses, where electroweak single production may dominate instead depending on the VLQ coupling strength [17]. Since vector-like quarks are assumed to have the same colour charges as SM quarks, the cross section for VLQ pair production is only dependent on the assumed mass of the VLQ. Thus, relatively model-independent constraints on the VLQ masses can be obtained by VLQ pair production searches. Conversely, the single production channel for vector-like quarks allows the universal coupling constant ($\kappa$) to be constrained. The universal coupling strength controls both the production cross section and the resonance width of the VLQ. For a VLQ with mass M, the resonance width $\Gamma$ scales as $\Gamma \propto {\kappa }^2{M}^3$ [20]. Thus, the relative width ($\Gamma$/M) of the VLQ resonance scales quadratically with both $\kappa$ and with M, and is independent of the multiplet representation.

The dominant channel for resonant production of a single vector-like quark is t-channel production mediated by a gauge boson. In the four-flavour scheme at leading order, and assuming couplings only to third generation SM quarks, this process requires an initial-state gluon to split into a $b\overline{b}$ or $t\overline{t}$ pair. Due to the mass hierarchy of the top and bottom quarks, b-associated (or W-mediated) T-quark production is kinematically favoured over t-associated (or Z-mediated) production. However, in gauge representations where the coupling to W bosons vanishes (such as the ($XT$) and ($TB$) doublets mentioned earlier), the t-associated mode is the only allowed production channel. An example process involving the W-mediated production of T quarks is shown in the right plot of Figure 1. Physical realisations of Composite Higgs models require the presence of additional scalar [23] and vector bosons [24,25,26] for UV-completeness, thus opening up new production channels. The new charged and neutral vector bosons, $W^{\prime \pm }$ and $Z^{\prime 0}$, can decay via $W^{\prime }\to tB/bT$ or $Z^{\prime }\to tT/bB$ final states, so long as the decay is kinematically allowed by the mass hierarchy.

1.2. The Phenomenology of Leptoquarks

Quarks and leptons in the SM have many similarities, including their transformations under the electroweak gauge groups. Both types of particles have the same number of generations, formed of one up-type and one down-type particle. These and other parallels between the quark and lepton sectors raise the possibility of a fundamental symmetry that connects the two. Such symmetries are predicted by many BSM models, such as Grand Unified Theories [27], technicolour models [28], or other models in of quark and lepton compositeness [29]. These models predict the existence of “leptoquarks”, which are bosons that carry both lepton and baryon quantum numbers, and can therefore couple to both leptons and quarks simultaneously. Leptoquarks transform as triplets under the $S{U}_C\left(3\right)$ strong gauge group, and carry fractional electric charge. They can exist as either scalar (spin-0) or vector (spin-1) bosons. In the minimal Buchmuller-Ruckl-Wyler (BRW) model [30], leptoquarks are assumed to couple only to leptons and quarks from the same generation. The Yukawa-like interactions of scalar leptoquarks to quark-lepton pairs in this model can be expressed in terms of two parameters: a coupling strength $\lambda$, and a model parameter $\beta$ that determines the relative coupling of the leptoquark to charged and neutral leptons. The coupling to charged leptons is given by $\sqrt{\beta \lambda }$, while the coupling to neutral leptons is $\sqrt{\beta (1-\lambda )}$. For vector leptoquarks, the coupling strength additionally depends on the anomalous magnetic moment ($\kappa$) of the leptoquark. The $\kappa =$ 0 and $\kappa =$ 1 limits correspond to a Yang-Mills coupling and a minimal coupling scenario, respectively [31]. In general, cross-generational couplings of the leptoquark are also possible, and have been probed by searches from the ATLAS and CMS collaborations.

Since the coupling strengths can vary for leptoquarks of different generations, models with leptoquarks can generate lepton flavour universality-violating (LFUV) interactions. Leptoquarks have been proposed as a solution to the observed flavour anomalies in B-meson decays [32,33,34,35,36]. They have also gained particular interest recently [37,38] in light of the recent measurements of the anomalous muon magnetic dipole moment [39,40].

At the LHC, leptoquarks can be produced singly or in pairs. Analogous to the case of vector-like quarks, searches for the strong pair production process can provide relatively model-independent constraints on the leptoquark mass, while the single production process can be used to directly probe the leptoquark interactions and to constrain $\lambda$.

1.3. Other Relevant Phenomenology

As mentioned at the beginning of this section, there is a long list of models that predict heavy resonances that can decay into third generation quarks. Some of them have gained recent scrutiny as possible sources of LFUV interactions, used to explain recent experimental results in B physics [41,42]. Models with additional neutral heavy vector bosons (Z’) or charged heavy vector bosons (W’) mainly coupled to third-generation lepton and quarks have been used to offer an explanation to those flavour anomalies. [43,44].

In spite of the large number of available models, ATLAS and CMS searches interpret their results only in a handful of benchmarks that capture the relevant phenomenology of the process being studied. In this section those models are briefly described.

Top assisted technicolour (TC2) [10] models are commonly used as benchmark for new Z’ decaying into $t\overline{t}$. In this type of model the large top-quark mass is obtained through the formation of a dynamical $t\overline{t}$ condensate, generated by a new strong gauge force coupling preferentially to the third generation. In order to allow for the existence of this condensate, a leptophobic Z’ is usually introduced. For most models the Z’ is narrow and has a strong coupling to $t\overline{t}$ while still coupling with lighter quarks. The production of Z’ is dominated by the diagram shown in Figure 2 (left). This type of model and production mechanism provides a very general phenomenology that can be easily interpreted in other models such as simplified dark matter models, in which the Z’ acts as a mediator [45].

Spin-1 Kaluza-Klein gluons ($g_{kk}$) [46] are also produced following the same mechanism. They appear in Randall-Sundrum models with a single warped extra dimension [47] and are used as a useful benchmark for larger $t\overline{t}$ widths. Another interesting $t\overline{t}$ resonance that appears in Randall-Sundrum models is the Kaluza-Klein excitation of the graviton ($G_{kk}$). In this case, being a Spin-2 colour singlet, the dominant production mechanism differs from the previous two cases, and is shown in Figure 2 (right).

Figure 2. Feynman diagram for the dominant production of a Z’ boson in a TC2 model decaying into $t\overline{t}$ (left) and for a Kaluza-Klein graviton (right). Both from [48].

Figure 2. Feynman diagram for the dominant production of a Z’ boson in a TC2 model decaying into $t\overline{t}$ (left) and for a Kaluza-Klein graviton (right). Both from [48].

Many searches for scalar and pseudoscalar particles decaying into third generation quarks are interpreted in the context of two-Higgs-doublet models (2HDM) [49]. This large category of models is characterised by the presence of additional Higgs bosons. These models provide solutions to several challenges of the SM such as necessary sources of CP-violation [50] and a mechanism to explain baryon assymmetry [51]. They are also an integral part of any supersymmetric extension of the SM as the additional Higgs doublets are neccesary to provide mass and cancel anomalies [52]. The additional particles may include charged Higgs bosons ($H^{\pm }$), scalar neutral H and h bosons (where h denotes the lighter of the two states), and a pseudoscalar neutral A boson. These particles have a Yukawa-like coupling with the top-quark and due to its large mass, couple strongly to it. There are several particular realizations of 2HDM models used as benchmark for ATLAS and CMS searches such as the minimal supersymmetric standard model (hMSSM) [53] or the $M_h^{125}\left(\tilde{\chi }\right)$ benchmark scenario [54].

When searches have to deal with heavy pseudo-scalar or scalar particles with strong coupling to top quarks such as in 2HDM models, the $t\overline{t}$ final state presents an additional challenge. Interference effects become very relevant between the dominant production diagram, through gluon-gluon fusion and a top-loop, and standard model $t\overline{t}$ production. These diagrams are shown in Figure 3. Depending on the phase-space region and specific signal model, the interference may result in a peak-dip or dip-peak structure in the $m_{t\overline{t}}$ distribution that requires special care.

Within models with dominant or preferential couplings to third generation quarks a different production mechanism, known as associated production, is also important. The study of associated production, with additional third generation quarks in the final state, increases the complexity of the final state but avoids interference effects. Example diagrams for the associated production of a heavy neutral and charged Higgs bosons are shown in Figure 4.

A new W’ that can decay into a top quark and a bottom quark appears in many extensions of the SM. Due to the large mass of the top quark, its interactions decouple from the rest of the phenomenology in many of those theories. ATLAS and CMS W’ searches take advantage of this fact and use an effective Lagrangian to represent the phenomenology of the W’$\to tb$ in the Sequential Standard Model (SSM) [58], where the W’ boson has the same coupling strength to the fermions as the SM W boson. Such an effective lagrangian allows for a large flexibility in terms of couplings values and W’ chiralities. The dominant production mode of W’$\to tb$ is shown in Figure 5.

Excited heavy quarks searches focus on models in which the quarks are composite [59]. Such quarks would have an internal structure that, excited, can produce a state with higher mass able to decay into a final state containing top-quarks.

Finally, monotop searches consider specific dark matter models in which the mediator is a coloured charge-2/3 scalar ($\varphi$) [60]. In this scenario the resonant production of a top-quark and a spin-1/2 dark matter particle ($\chi$) becomes possible, allowing for the top-quark and $E_T^{miss}$ final state. The production diagram is shown in Figure 6.

1.4. Common Objects and Methods

Most of the analyses described in this review use similar definitions for objects and use similar methods and tools. In this section a brief description is provided. Special cases and deviations are described when necessary in the individual analyses. No attempt is made at describing the specific reconstruction algorithms that each detector or specific analysis uses as they are covered in each of the publications referenced in the following sections.

• Leptons.

When referring to leptons in this review, it is always meant as charged leptons only, specifically electrons or muons. Tau leptons are generally not included and dealt with separately. Prompt electrons and muons are reconstructed by dedicated algorithms using tracking information and energy deposits in the calorimeters and muon systems. In most cases leptons are required to be isolated in order to reduce the contamination from non-prompt or fake leptons, i.e., other objects misidentified as leptons.

• Jets.

Jets are the experimental signature of a quark or gluon from the hard scattering. They are collimated jets of high-energy hadrons which are the result of the fragmentation and hadronisation of the outgoing quark or gluon. They are reconstructed in the detector using algorithms that look to identify such structures by clustering together calorimeter deposits, tracking information or more complex structures. A popular option, used in all CMS analyses and several of the ATLAS ones is to run jet algorithms on Particle Flow objects [62,63]. These objects combine information from different parts of the detector to reconstruct the individual particles such as the already mentioned electrons and muons and other such as photons or hadrons.

The most commonly used jet algorithm is the anti-$k_t$ algorithm [64] as implemented in the Fastjet package [65] but others are used for specific purposes. Generally jets are divided into two categories, with slightly different definitions for both ATLAS and CMS: small-R jets aiming at the identification of light-quarks or gluons, using a distance parameter of 0.4, and large-R jets used for the identification of hadronically decaying top quarks or bosons, using a distance parameter of 0.8, common in CMS or 1.0, common in ATLAS.

• Missing transverse momentum.

Missing transverse momentum (${\overrightarrow{P}}_T^{miss}$) and its magnitude, missing transverse energy ($E_T^{miss}$), refer to a 4-momentum imbalance in the transverse plane of a particular event. Such an imbalance indicates the presence of a weakly interacting particle that has escaped detection by any part of the detector. This is used to identify neutrinos and other possible weakly interacting particles, such as dark matter candidates. It is defined as the projection on the plane perpendicular to the beams of the negative vector sum of the momenta of all identified objects in an event.

• Jet tagging.

Jet tagging refers to a series of techniques used to assign a specific flavour to a jet, i.e., identify which particle they originate from. Different techniques are used depending on the target flavour and the type of jet. In this review the terms top-tagging, b-tagging, W-tagging or Higgs tagging are used to refer to jets identified as coming from an hadronically decaying top quark, a b-quark, an hadronically decaying W boson or an hadronically decaying Higgs respectively. They are particularly relevant in the context of high transverse momentum ($p_T$), Lorentz boosted (commonly abbreviated to “boosted”), hadronically decaying particles. The decay products of these boosted particles are very close together due to the Lorentz boost experienced by the parent particle and are difficult to disentangle. They are usually reconstructed as a single jet and its internal structure is used by the aforementioned tagging techniques.

• Monte Carlo samples.

Most of the analyses described in this review use Monte Carlo (MC) samples to study and estimate the contribution of different background and signal processes. Simulated events are produced with a large variety of generators and processed through detailed models of the ATLAS or CMS apparatus based on Geant4 [66] that include detector response.

• Statistical analysis.

In order to test for the presence of new physics, templates of both background and signal, obtained through MC simulations or data-driven methods, are compared to the data. Unless indicated otherwise a binned maximum-likelihood fit [67] is used while the variable, or variables, being fitted depends on the specific analysis. Several regions may be simultaneously fitted, including regions sensitive to the signal, also called signal regions, and other regions used to help estimate the background modelling. Systematic uncertainties are included as nuisance parameters [68] with either log-normal or gaussian constraints. In the absence of any significant excess upper limits at the 95% confidence level are obtained. Most analyses use the asymptotic approximation of the $C{L}_s$ method [69] to estimate limits while some use a Bayesian approach [70].

2. Vector-like Quark Searches

Both ATLAS and CMS have a rich search programme for VLQs, covering a wide range of parameter space. The searches are generally separated into three categories:

• pair production through strong interaction;
• single production through electroweak interaction;
• “exotic” production involving other BSM particles.

Individual analyses in each category usually focus on a limited set of final state signatures, targeting a particular area of the VLQ decay branching ratio space. Most of the searches discussed here target VLQ decays to third generation SM quarks, although both ATLAS [71,72] and CMS [73] had performed searches for VLQs decaying into light-flavour quarks with data from Run 1 of the LHC.

2.1. Searches for the Pair Production of VLQs

The ATLAS and CMS collaborations have performed a large number of searches for the pair production of T and B-type VLQs decaying to third generation SM quarks. Searches are performed in different final states according to the target decay modes of the VLQs. The signal events are characterised by the presence of b-quarks and heavy resonances (t quarks, W/Z/Higgs bosons) produced at high transverse momentum. Therefore, signal discrimination in these searches often relies on the identification of b-jets, and of boosted hadronically-decaying resonances. Charged leptons in the event can also be expected from leptonic decays of the W and Z bosons (including those arising from upstream $t\to Wb$ and $H\to WW/ZZ$ decays). Observables such as $H_T$ (scalar sum of the $p_T$ of jets and leptons in the event) or $S_T$ (scalar sum of $H_T$ and $E_T^{miss}$) (also known as effective mass or $m_{eff}$) are powerful discriminants of VLQ pair production signal, given the large multiplicity of high-$p_T$ jets and leptons in the final state. Most of the VLQ pair production searches discussed in this section were performed with the LHC proton–proton collision data collected during the 2015–2016 run period of the LHC. Therefore, unless otherwise stated, the ATLAS (CMS) searches discussed here correspond to an integrated luminosity of 36.1 (35.9) fb${}^{-1}$.

Searches in the single-lepton channel, with one or more b-jets, target $T\to Wb$ and $B\to Wt$ decays, with subsequent leptonic decays of the W boson, as well as $T\to Ht/Zt$ decays with semi-leptonically decaying t quarks. The dominant background in these searches comes from SM $t\overline{t}$ pair production. Both ATLAS [74] and CMS [75] published results on searches targeting $TT\to WbWb$ processes, with data collected during the 2015–2016 run period. Both searches rely on fitting the reconstructed T quark mass for final signal discrimination. The ATLAS search excludes T quarks with masses up to 1350 GeV, assuming $BR(T\to Wb)=$ 100%, as shown in Figure 7 (left), while the CMS search excludes T quark masses up to 1295 GeV in the same scenario, as shown in the right plot of the same figure. In addition, the ATLAS search sets exclusion limits on the T quark mass across the branching ratio plane spanned by $BR(T\to Wb)$ and $BR(T\to Ht)$ (Figure 8 (top left)). In particular, masses below 1170 GeV are excluded at the singlet branching ratio point.

A similar search for B quarks was performed by ATLAS, targeting $B\to Wt$ decays [76]. This search employs two signal regions, one utilising the reconstruction of the $TT$ system by minimising the difference between the reconstructed mass of the two T quarks in the event, and the other instead using a Boosted Decision Tree (BDT) to discriminate the signal from background processes. A combined analysis on both signal regions results in B quark masses being excluded up to 1350 GeV for 100% branching ratio to $Wt$, and 1170 GeV for a singlet B quark. As was done for the T quark, limits on the B quark mass are additionally set across the two-dimensional $BR(B\to Wt)$–$BR(B\to Hb)$ plane (Figure 9 (left)).

Figure 8. Observed 95% lower limits on the T quark mass in the $BR(T\to Wb)$–$BR(T\to Ht)$ from various ATLAS searches. Clockwise from top left, the results are from [74,77,78,79].

Figure 8. Observed 95% lower limits on the T quark mass in the $BR(T\to Wb)$–$BR(T\to Ht)$ from various ATLAS searches. Clockwise from top left, the results are from [74,77,78,79].

ATLAS also performed a search in the single-lepton channel using events with large $E_T^{miss}$, targeting the $TT\to Z\left(\nu \nu \right)t+X$ process [77]. Signal discrimination relies on selections on the multiplicity of b-jets and massive large-radius jets, as well as several transverse mass variables. The observed number of events in the signal region is used as the final discriminant in a maximum likelihood fit. Masses below 1.16 TeV are excluded for T quarks with 100% branching ratio to $Zt$ final states. The mass exclusion limits across the $BR(T\to Wb)$–$BR(T\to Ht)$ plane are shown in Figure 8 (top right). As can be seen, the search sensitivity extends across a large portion of the branching ratio plane. For doublet T quarks, masses below 1.05 TeV are excluded. This is a conservative limit, since it excludes contributions from the other B or $X^{5/3}$ quark of the doublet, both of which can decay to $Wt$ final states.

A broad-band search was performed by ATLAS using a combination of a single-lepton channel, and a channel with zero leptons and large $E_T^{miss}$ [79]. The single-lepton channel is designed to be sensitive to $TT\to H\left(bb\right)t+X$ processes, with a leptonically-decaying top quark. Conversely, the zero-lepton channel targets $TT\to Z\left(\nu \nu \right)t+X$ processes where the top quark would decay hadronically. The analysis exploits the presence of multiple boosted hadronically-decaying resonances and b-jets in the event to effectively discriminate the signal from the SM background. In the zero-lepton channel, the multijet background could be reduced to very small levels by requiring the direction of $E_T^{miss}$ to point away from the four leading jets in the event, thus ensuring that the $E_T^{miss}$ was not a result of the mis-measured jet transverse momenta. As a result, the dominant background in both the zero-lepton and one-lepton channel was from SM $t\overline{t}$ production. A combined fit on the $m_{eff}$ variable across signal regions in both channels is used to set exclusion limits across a wide region of the branching ratio plane Figure 8 (bottom left). The mass exclusion is strongest in the $BR(T\to Ht)=$100% scenario, where it reaches 1.43 GeV. Singlet(doublet) T quarks with mass below 1.19 (1.31) TeV are also excluded.

Both $TT$ and $BB$ processes can give rise to multi-lepton signatures in a variety of ways.In particular, $TT\to Z\left(ll\right)t+X$ and $BB\to Z\left(ll\right)b+X$ events will contain at least one pair of opposite-sign and same-flavour (OSSF) leptons. A pair of same-sign leptons can occur in $TT\to H\left(WW\right)t+X$ and $BB\to H\left(WW\right)b+X$ events since they always contain at least two W bosons with the same sign, including the ones from top quark decays. The trilepton channel is sensitive to signal events with at least one $T\to Zt$, $B\to Zb$ or $B\to Wt$ decay. The main background in the OSSF dilepton channel comes from Z+jets production. Vector boson pair production and $t\overline{t}V$ (where $V=W/Z$) processes are the important backgrounds in the trilepton channel, but $t\overline{t}V$ processes dominate once b-jets are required in the event. Both the same-sign (SS) dilepton and the three-lepton channels benefit from very small background from SM processes. However, background from fake/non-prompt leptons is a dominant background in the SS dilepton channel, and events containing prompt leptons with misidentified electric charge is also an important source of background. Since these reducible backgrounds are mostly instrumental in nature, they are usually estimated using data-driven techniques to avoid over-reliance on the accuracy of the detector simulation.

The SS dilepton channel is uniquely sensitive to both single and pair production of $X^{5/3}$ quarks. Since $X^{5/3}$ quarks decay to $Wt$ final states 100% of the time, pair-produced $X^{5/3}{X}^{5/3}$ events always contain a pair of SS lepton pairs [78] if both W bosons decay leptonically. This is also true of singly-produced $X^{5/3}$ events, since both the production and decay of the $X^{5/3}$ quark are mediated by the same charge current coupling to t quarks and W bosons [78].

CMS performed a broad-band search for $TT$ and $BB$ production in final states containing at least one lepton [80]. The search is divided into separate channels with one lepton, two same-sign leptons, or three leptons, each channel being sensitive to different decays of the T and B quarks. A variety of signal discriminants is used in the different channels, such as $H_T$ and $S_T$ variables, or, in the case of the single lepton channel, the minimum invariant mass between the lepton and any b-tagged jet in the event. A statistical combination is performed across all channels to improve sensitivity across T quark and B quark decay branching ratios (Figure 10). T(B) quarks were excluded at 95% CL for masses below 1200 (1170) GeV in the singlet scenario and 1280 (940) GeV in the doublet scenario. A search using similar techniques, in the single lepton and SS dilepton channel, is interpreted for the pair production of $X^{5/3}$ quarks [81]. The lower limit on the mass of the $X^{5/3}$ quark was 1.33 (1.30) TeV assuming right-handed (left-handed) couplings to the W boson (Figure 11).

The ATLAS search in the same-sign dilepton channel [78] is designed to be sensitive to a large number of BSM signals, including the pair production of T, B and $X^{5/3}$ vector-like quarks. This search additionally includes signal regions with three leptons. Events in the signal regions are selected to have large $H_T$ and $E_T^{miss}$, and the final signal discrimination relies on a simultaneous fit on the number of observed events in all signal regions in both channels. This search excludes singlet B quarks with mass below 1.00 TeV and singlet T quarks with mass below 0.98 TeV. Mass exclusion limits are also set across the usual branching ratio plane, and get strongest for the $BR(B\to Wt)=$100% (Figure 8 (bottom right)) and $BR(T\to Zt)=$100% (Figure 9 (right)) scenarios. $X^{5/3}$ quarks are excluded for masses below 1.19 TeV.

CMS performed a search targeting $TT\to Z\left(ll\right)t+X$ and $BB\to Z\left(ll\right)b+X$ signals [82] in final states containing OS lepton pairs. The lepton pair is required to have a invariant mass consistent with that of the Z boson, and events were categorised according to the multiplicity of b-jets and hadronically-decaying top quarks and $W/Z/H$ bosons in the event, identified using different techniques in the boosted and resolved regimes respectively. The final signal discriminant used in the maximum likelihood fit is the $S_T$ observable. The results were interpreted across the full branching ratio plane (Figure 12). Masses of the T quark below 1280 GeV and 1185 GeV were excluded for the $BR(T\to Zt)=$ 100% and the singlet branching ratio scenarios, respectively. For B quarks with $BR(B\to Zb)=$ 100%, masses below 1130 GeV are excluded. ATLAS has performed a more recent search in this channel with the full Run-2 dataset, corresponding to an integrated luminosity of 139 fb${}^{-1}$ [83]. The search is performed in a similar way to the CMS analysis, but utilises separate dilepton and trilepton channels. Both channels are required to have an OS lepton pair with invariant mass consistent with the Z boson mass, and events are categorised according to the multiplicity of boosted hadronic resonances in the event, as identified by a multi-class neural network-based tagger. The reconstructed mass of the VLQ is used as the signal discriminant in the OS dilepton channel, while an $H_T$ variable is used in the trilepton channel. This search significantly extends the exclusion limits for T/B quarks decaying to $Zt$/$Zb$ final states (Figure 13). For pure $T\to Zt$/$B\to Zb$ decays, this search excludes T quark masses below 1.6 TeV and B quark masses below 1.42 TeV. T quark masses below 1.46 TeV and B quark masses below 1.32 TeV are excluded for doublet branching ratio scenarios.

The final category of searches to be discussed in the context of VLQ pair production are those that are performed in the all-hadronic final state, where QCD multijet production is the dominant background process. These searches typically try to discriminate the signal from background processes by utilising the expected presence of multiple boosted hadronic resonances, and b-tagged jets from $t\to qb$ or $H\to bb$ decays in the signal events. The multijet background is estimated with data-driven techniques, usually with some version of the so-called “ABCD” or two-dimensional sideband method [84]. Alternatively, the b-tagging and boosted object tagging rates for multijet events can be estimated in data and used in the background estimation. The ATLAS search for $TT$ and $BB$ production in the all-hadronic channel [85] was built on these techniques, and additionally uses the matrix element method to construct the final discriminant. This search is interpreted for all possible third generation decays of the T and B quarks, but is most sensitive to $T\to Ht$ and $B\to Hb$ decays (Figure 14), for which masses below 1010 GeV are excluded. The large number of b-jets expected from $H\to bb$ decays in these channels can be exploited to effectively suppress SM background.

The CMS search in the all-hadronic final state utilises a combination of cut-based and neural network-based techniques to similarly probe T and B decays across the branching ratio planes [86]. The search excludes T quark masses between 740–1370 GeV for pure $T\to tH$ decays, and below 1040 GeV for T quarks decaying solely to $Wb$ final states (Figure 15 (left)). For B quarks decaying purely to $tW$ final states, masses up to 1230 GeV are excluded.

The exclusion limits for B quarks decaying to $bH$/$bZ$ final states were improved for a more recent CMS search utilising the full LHC Run-2 dataset, corresponding to an integrated luminosity of 137 fb${}^{-1}$ [87]. The search uses a ${\chi }^2$ technique to reconstruct the mass of the B quark, and the signal region is constructed by choosing the subset of well-reconstructed events corresponding to low-${\chi }^2$ values. The shape of the reconstructed B mass is fitted for final signal discrimination. This search excludes B quark masses below 1.57 TeV for pure $T\to Ht$ decays and 1.39 TeV for pure $T\to Zt$ decays. The lower limit on B quark masses for the doublet branching ratio scenario from this search is 1.45 TeV (Figure 15 (right)).

Figure 15. Observed 95% lower limits on the T quark mass in the $BR(T\to Wb)$–$BR(T\to Ht)$ from [86] (left) and on the B quark mass in the $BR(B\to Wt)$–$BR(T\to Hb)$ from [87] (right).

Figure 15. Observed 95% lower limits on the T quark mass in the $BR(T\to Wb)$–$BR(T\to Ht)$ from [86] (left) and on the B quark mass in the $BR(B\to Wt)$–$BR(T\to Hb)$ from [87] (right).

The ATLAS collaboration performed a statistical combination of all $TT$ and $BB$ searches with 36.1 fb${}^{-1}$ data, which greatly improved the sensitivity reach across all T and B decay branching ratios. The left and right plots in Figure 16 show the excluded branching ratios for different mass hypotheses for for T quarks and B quarks, respectively, and illustrate the complementarity of the individual searches. The searches in leptonic final states tend to be especially sensitive, especially at high VLQ masses. In particular, the single-lepton search in [79] for $H\left(bb\right)t+X$ final states, and the search in the OSSF channel [83] for $Z\left(ll\right)t/b+X$ decays, had leading sensitivities in the pure $T\to Ht$ and pure $T\to Zt$ scenarios, respectively. Both searches could exploit striking event signatures to effectively suppress SM background. In addition to its special sensitivity to $X^{5/3}$ signals, the SS dilepton search [78] had competing sensitivity to $B\to Wt$ decay scenarios, especially at low mass. This channel suffers from problems of low statistics at higher signal masses.

Singlet T quarks with masses below 1.31 TeV and singlet B quarks with masses below 1.22 TeV are excluded by the full combination. T and B quarks with masses below 1.37 TeV are excluded for a (TB) doublet scenario. The observed lower limits on the T and B quark mass as a function of the decay branching ratios can be seen in Figure 17 (left) and (right) respectively.

Many of the pair production searches discussed in this section were performed with partial Run-2 datasets from 2015 and 2016. The reach of these searches would be significantly extended with the full Run-2 dataset, which corresponds to almost a threefold increase in integrated luminosity. Furthermore, new and more complex analysis techniques, such as those involving machine learning, can improve the signal discrimination for these searches. Even at higher VLQ masses, pair production searches remain important to probe the narrow-width regime, which corresponds to small VLQ electroweak couplings.

2.2. Searches for the Single Production of VLQs

Many of the discriminating features of VLQ pair production events, such as the presence of b-jets and boosted hadronic resonances at large transverse momenta, also hold for single VLQ production signals. However, in comparison to pair production processes, the expected overall multiplicity of objects is smaller in this production channel. As discussed in Section 1.1, the dominant channel for resonant production of a single vector-like quark is t-channel production mediated by a gauge boson. The initial quark recoiling off from the gauge boson often emerges at high pseudorapidity. Thus, the presence of jets in the forward region is an important discriminating characteristic of these signals often used in the single production searches. Searches for particular decays of the VLQ are designed to probe the different relative couplings to the W, Z and Higgs bosons. The searches described below use some different but equivalent parameterisations of these couplings. The model in [88] uses the left- and right-handed mixing angles ${\theta }_{L,R}$ to encode the VLQ couplings in the up- and down-sectors. The gauge representation of the vector-like quark determines the chirality. For example, only the left-handed component contributes for singlet T quarks, while right-handed component is dominant for T quarks in (T B) doublet models. Equivalently, the couplings are expressed in terms of the parameters $c_{L,R}^{W,Z,H}$ in [89]. Furthermore, the individual couplings $c_{L,R}^{W,Z,H}$ are re-parameterised in terms of a global coupling parameter $\kappa$ and relative couplings ${\xi }_{W,Z,H}$ in [20].

A search for singly-produced T/Y quarks decaying to $Wb$ final states was published by ATLAS in the single lepton channel with 36.1 fb${}^{-1}$ integrated luminosity [90]. Only the dominant W-mediated production channel was considered in this search. A similar search had been published by CMS with the data collected only in the year 2015, corresponding to 2.3 fb${}^{-1}$ integrated luminosity. Both searches use the reconstructed mass of the VLQ as a signal discriminant. The CMS search excludes Y quarks with masses between 0.85–1.40 TeV, assuming a coupling of 0.5. The ATLAS search, by comparison, excludes Y quarks with masses up to ∼1.64 TeV for the same benchmark. In addition, the ATLAS search sets exclusion limits in the coupling–mass plane for several benchmarks. The exclusion limits for singlet T quarks are presented in Figure 18. The limits are shown on the mixing angle sin(${\theta }_L$) and the coupling parameter $c_L^{Wb}$. For a singlet T quark, these two parameters are related by the equation $c_L^{Wb}=\sqrt{2}\sin \left({\theta }_L\right)$. A distinguishing feature of this search is the treatment of interference effects between the signal and SM background, which can have significant impact on the signal distributions at moderate and large resonance widths. (Figure 19). The signal mass spectrum with interference is closer in shape to the background, which in turn degrades the search sensitivity. This degradation is illustrated in Figure 20, showing exclusion limits on the production cross section of a right-handed Y quark and the comparison to a no-interference case.

A complementary search to the one described above was performed by the CMS collaboration for singly-produced B/$X^{5/3}$ quarks decaying to $Wt$ in single-lepton final states [91]. The search strategy is similar to the $T/Y\to Wb$ search, but additionally uses boosted top quark and W-boson tagging to enhance the signal purity. Cross section limits as a function of the VLQ mass are set for a range of assumed decay widths (Figure 21). As mentioned earlier in Section 1.1, the resonance width is directly related to the mass and global coupling parameters, and larger width signals are correlated with stronger couplings and higher production cross sections. B quarks with left-handed couplings and mass below 1490 GeV are excluded for 10% relative width and above. The mass exclusion limits are stronger at higher width benchmarks. $X^{5/3}$ quarks with left-handed couplings and width and mass below 920 GeV are excluded for 10% widths and above. The lower limit exclusion on $X^{5/3}$ quark mass increases to 1450 GeV for 30% relative width.

As discussed above, the SS dilepton channel is uniquely sensitive to both pair production and single production of $X^{5/3}$ quarks. The aforementioned ATLAS search in the SS dilepton channel was also used to constrain the plane spanned by the $X^{5/3}tW$ coupling strength and the $X^{5/3}$ quark mass (Figure 22). Both single and pair production signals are considered, since the single production cross section is very small at low coupling values.

Both ATLAS and CMS have searched for $T\to Z\left(\nu \nu \right)t$ single production in events containing a boosted hadronically-decaying top quark and large $E_T^{miss}$ [61,92]. While the ATLAS search was performed on the 36.1 fb${}^{-1}$ dataset, the CMS search is more recent and utilises the full 136 fb${}^{-1}$ dataset from LHC Run 2. The main sources of background are from $t\overline{t}$, single t production, and $V+$ jets processes. The transverse mass of the top-tagged jet and the missing transverse momentum is the signal discriminant used in both searches. The ATLAS search sets constraints on the coupling parameter $c_W=\sqrt{c_{W,L}^2+c_{W,R}^2}$ and the mass (Figure 23 (left)), while the CMS search is interpreted in the plane of mass and relative width (Figure 23 (right)). Since the resonance width of the VLQ is directly related to its mass and coupling strength [20], these two interpretations are interchangeable.

Events with leptonic $Z\to ll$ decays are also studied by both experiments in the $T\to Zt$ single production search. The ATLAS search uses data from the 2015–2016 run period, and the search is performed across an OS dilepton and a trilepton channel [93]. The CMS search uses data collected in 2016 only, and the search is restricted to the OS dilepton channel [94]. Both searches required the invariant mass of the OS dilepton pair to be consistent with the Z boson mass. The reconstructed VLQ mass is used as the signal discriminant in the OS dilepton channel in both searches, while $S_T$ is used for the trilepton channel in the ATLAS search. The CMS search is interpreted for T quarks in both singlet and doublet scenarios (Figure 24). While b-associated production is dominant for the singlet benchmark, right-handed T quarks in the doublet scenario do not couple to W bosons, and therefore the t-channel production mode must be considered for it. A variety of interpretations in terms of different model parameterisations are included in the ATLAS publication. Figure 25 shows the exclusion limits in the $c_W$–mass plane for the singlet T benchmark.

A recent search by ATLAS with 139 fb${}^{-1}$ integrated luminosity probed single T production in the single lepton channel, targeting $T\to Ht/Zt$ decays [22]. Events are categorised by the multiplicity of boosted hadronic resonances, reconstructed semileptonically decaying top quarks and b-tagged jets into several signal regions, and a simultaneous fit of the $m_{eff}$ distribution across all signal regions was used for signal discrimination. The search is interpreted for singlet T quarks to constrain the coupling–mass parameter space (Figure 26 (left)). A generalised interpretation was also included, showing mass exclusion limits in the plane spanned by the relative decay width and $BR(T\to Wb)$ (Figure 26 (right)).

The CMS search for $T\to Zt/Ht$ single production [95] was done in the all-hadronic final state, with a 35.9 fb${}^{-1}$ integrated luminosity dataset. The search employs different strategies for the resolved topology expected for low-mass signals, and the boosted topology expected for high-mass signals. Both b-associated and t-associated production modes are considered in this search. The relative sensitivities in the two channels can be seen from the mass exclusion limits, as shown in Figure 27 for a 30% decay width assumption.

In the B quark sector, two recent searches by ATLAS have been performed targeting $B\to bH$ decays, with subsequent $H\to \gamma \gamma$ [96] and $H\to b\overline{b}$ [97] decays. The first of these searches was performed with a partial dataset of 78.8 fb${}^{-1}$ integrated luminosity, while the latter used the full 139 fb${}^{-1}$ dataset. The search in the $H\left(\gamma \gamma \right)$ uses a sideband technique to estimate the continuum diphoton background in the signal region, and the signal hypothesis is tested with a fit on the reconstructed VLQ mass distribution. The search excludes B quark masses below in the (B, Y) doublet scenario, for a $\kappa =$ 0.5 benchmark (Figure 28 (left)). The all-hadronic $B\to bH\left(bb\right)$ search also uses the reconstructed B mass as a signal discriminant, and the dominant multijet background is estimated with the ABCD technique. It excludes masses in the 1.0–2.0 TeV mass range for coupling points as low as $\kappa =$ 0.3. The areas of the $\kappa$–mass plane excluded by this search can be seen in Figure 28 (right). The earlier $B\to bH\left(bb\right)$ search [98] published by CMS used very similar techniques, but was performed using the smaller 35.9 fb${}^{-1}$ dataset. The search excludes B quark masses below ∼1150 GeV in the doublet scenario for 30% decay width (Figure 29).

The current search programme for single VLQ production in both ATLAS and CMS is dominated by searches for T quarks. Publications with searches in B sector using the full Run-2 data, especially in $Zb$ and $Wt$ final states, can be expected in the near future. There are opportunities to extend the reach of single VLQ searches with a statistical combination. Looking forward, one can envision that statistical combinations between pair production and single production searches could be interesting in order to constrain the phase space with intermediate values of VLQ couplings, where both production modes have comparable cross sections.

2.3. Searches for Exotic Production of VLQs

The searches discussed in the previous section considered only VLQ couplings to SM particles. However, more novel production processes involving additional BSM particles are possible in non-minimal and UV-complete models. The CMS collaboration has conducted searches for two such processes, involving the resonant production of a heavy gauge boson that decays to a pair consisting of one vector-like quark, and one third generation SM quark [99,100]. The search for $Z^{\prime }\to Tt$ targets events with $T\to Zt$ and $T\to Ht$ decays [101]. The search set exclusion limits on the cross section for this process for benchmark values of the $Z^{\prime }$ boson and T quark masses (Figure 30). The search for $W^{\prime }\to Tb/bT$ was conducted in the all-hadronic final state with 137 fb${}^{-1}$ integrated luminosity [100]. Both $T\to Ht$ and $B\to Hb$ modes are considered, with boosted hadronic top quarks and Higgs bosons in the final state. The results are interpreted for different benchmarks of mass hierarchies between the $W^{\prime }$ boson and VLQ. For the benchmark scenario where the VLQ mass is two-thirds the mass of the $W^{\prime }$ boson, the latter is excluded up to 3.2 TeV.

Along with their decays to SM particles, VLQs can decay to new exotic modes in many BSM models. For example, composite Higgs models predict additional scalar particles into which the VLQs can decay [23]. Current searches are not typically sensitive to these decays, and so future dedicated searches for them will be important to constrain the full parameter space of physical models.

3. Leptoquark Searches

The landscape of leptoquark searches in both the ATLAS and CMS experiments is dominated by the search of pair production of scalar leptoquarks. The difference in kinematics between scalar and vector leptoquarks of the same generation are small however, thus making these results re-interpretable for vector leptoquarks. The CMS collaboration has published vector leptoquark interpretations in many of their searches, and have performed dedicated searches for single leptoquark searches as well. The pair production and decay of third generation leptoquarks can result in very similar final state signatures to the production of stop and sbottom quarks in supersymmetry models, and searches can often be designed to be sensitive to both processes.

A summary of the results from the recent searches for scalar leptoquarks from the ATLAS collaboration with the 139 fb${}^{-1}$ dataset is presented in Figure 31. These searches assume leptoquark decays to quarks and leptons from the same family. Limits on up-type leptoquarks ${\mathrm{LQ}}_3^u$ come from a search for stop quarks in the all-hadronic $t\overline{t}+E_T^{miss}$ final state [102], and from a search in the $b\tau +E_T^{miss}$ final state, which also targets stop quark pair production with decays via stau leptons [103]. The former is especially sensitive to pure ${\mathrm{LQ}}_3^u\to t\nu$ decays. Events in this search are categorised into several signal regions that are constructed with selection requirements on several different observables, including transverse masses, angular variables and large-R jet masses. The latter search is sensitive to both mixed decays such as ${\mathrm{LQ}}_3^u{\mathrm{LQ}}_3^u\to t\nu b\tau$ and the pure decay final state ${\mathrm{LQ}}_3^u{\mathrm{LQ}}_3^u\to b\tau b\tau$. A variety of transverse mass observables are used for signal discrimination in this search. Together, the searches exclude up-type leptoquarks with masses below around 1.24–1.25 TeV for most of the branching ratio range (Figure 31 (left)).

The $b\tau +E_T^{miss}$ search additionally included interpretations for vector leptoquarks in the minimal-coupling (Figure 32 (middle)) and the Yang-Mills scenarios (Figure 32 (right)). Since the event kinematics for vector leptoquarks is very similar to the scalar leptoquarks, the experimental acceptances and efficiencies are also similar and thus the shape of the exclusion contour remains similar between the two cases. However, the larger theoretical cross section for the vector leptoquarks leads to a stronger exclusion limit on the leptoquark mass. These limits extend to 1.5 TeV in the minimal-coupling scenario and 1.8 TeV in the Yang-Mills scenario for intermediate values of $BR({\mathrm{LQ}}_V^u\to b\tau )$.

In addition to the $b\tau +E_T^{miss}$ search mentioned above, the relevant searches for down-type leptoquarks ${\mathrm{LQ}}_3^d$ include a search in the $bb+E_T^{miss}$ final state [104] that also targets the pair production of sbottom quarks, and a direct search for ${\mathrm{LQ}}_3^d$ pair production with ${\mathrm{LQ}}_3^d\to t\tau$ decays [105]. Each of these covers a complementary range of decay branching ratios. The $bb+E_T^{miss}$ search applies requirement on a combination of kinematic observables, the output score of a BDT and a dedicated secondary-vertex finding algorithm called TC-LVT to construct several regions that are used for signal extraction. This search excludes ${\mathrm{LQ}}_3^d$ masses up to 1.26 TeV for the $BR({\mathrm{LQ}}_3^d\to t\tau )=0$ scenario, while masses up to 1.43 TeV are excluded for $BR({\mathrm{LQ}}_3^d\to t\tau )=1$ in the $t\tau t\tau$ search. In conjunction with the limits obtained from the $b\tau +E_T^{miss}$ search, masses below 1.2 TeV are excluded over the full range of decay branching ratios considered (Figure 31 (right)).

Both scalar and vector leptoquark interpretations were provided in the CMS search for the pair production of leptoquarks in the $t\tau \nu b$ final state [106], with 137 fb${}^{-1}$ integrated luminosity. Only scalar down-type leptoquarks (${\mathrm{LQ}}_S^d$), with decays to $t\tau$ and $b\nu$, and only vector up-type leptoquarks ((${\mathrm{LQ}}_V^u$)), with decays to $t\nu$ and $b\tau$, were considered however, since they are particularly relevant to certain new models that address the recently observed B-physics anomalies. Both the Yang-Mills coupling (k = 1) and the minimal coupling (k = 0) scenarios were considered for vector leptoquarks. Events were categorised into two regions, based on the number of reconstructed hadronically-decaying $\tau$ leptons (${\tau }_h$). The first category, with one ${\tau }_h$ candidate, is required to have a reconstructed hadronically-decaying top quark, and the $p_T$ of the top quark candidate is used as the signal discriminant. The observed number of events in the signal region with two ${\tau }_h$ candidates is used directly in a likelihood fit to extract signal. The search excluded leptoquark masses up to 0.95 TeV for scalar leptoquarks, and upto 1.29 TeV and 1.65 TeV for the k = 0 and k = 1 cases of the vector leptoquark, respectively (Figure 33).

A search for single production of leptoquarks in association with a $\tau$ lepton is also presented in [106]. Since the both the production cross section and the resonance width of the leptoquark depends on the coupling parameter $\lambda$, the exclusion limits obtained for single production are coupling-dependent and can be used to constrain $\lambda$. The single and pair production searches are combined to constrain the $\lambda$–mass plane in the range $\lambda <0.25$ (Figure 34). The search constrains a significant portion of the parameter space region preferred by B-physics anomalies (shaded in grey).

A dedicated search for single production of scalar up-type leptoquarks in association with a $\tau$ lepton was earlier performed by the CMS collaboration with the 35.9 fb${}^{-1}$ dataset [107]. The events were categorised according to the multiplicity of reconstructed electrons, muons, and hadronically-decaying $\tau$ leptons, and a combined fit on the $S_T$ variable was used for signal extraction. Exclusion limits were set in the $\lambda$–mass plane, assuming pure ${\mathrm{LQ}}_V^u\to b\tau$ decays (Figure 35). A significant portion of the parameter space is constrained, and masses below 740 GeV are excluded for $\lambda =$ 1. However, most of the parameter space preferred by the B-physics anomalies is still allowed by the results.

Both ATLAS and CMS have performed searches for cross generational decays for down-type leptoquarks to $te$ and $t\mu$ final states with the full Run-2 dataset. The ATLAS search targeted scalar leptoquark pair production with subsequent decays to either $te$ or $t\mu$ final states [108], where both top quarks in the event decay hadronically. The output score of a BDT trained on several kinematic observables is used for final signal discrimination in this search. Simultaneous couplings to both first and second generation leptons are disfavoured by current limits from lepton flavour violation (LFV) searches. This search excludes leptoquark masses below 1480 GeV for pure $te$ decays, and below 1470 GeV for pure $t\mu$ decays. Limits were also set on the allowed branching ratios for $\mathrm{LQ}\to te$ (Figure 36 (left)) and $\mathrm{LQ}\to t\mu$ decays (Figure 36 (right)).

The CMS search, also for scalar leptoquark production, was performed in the multilepton final state, targeting events in which at least one of the top quark daughters of either leptoquark decays semi-leptonically [109]. The events are categorised by the multiplicity of leptons, and signal discrimination is achieved with a BDT trained on observables including $H_T$, $p_T^{miss}$ and transverse masses. This search also considered $\mathrm{LQ}\to t\tau$ decays. Masses below 1120 GeV are excluded for leptoquarks decaying exclusively to a top quark and a lepton of any flavour. The limits are weakest in the $\tau$ channel, and strongest in the muon channel, where masses below approximately 1.4 TeV are excluded (Figure 37).

While many of the current leptoquark bounds come from re-interpretations of supersymmetric searches, one can expect to see more dedicated searches for these processes in the future. While CMS has already produced some results on the search for single production of leptoquarks, this channel is still relatively uncovered by ATLAS searches. The constraints on leptoquark pair production would also greatly benefit from a statistical combination of the existing searches, in a similar manner to what has been seen for VLQ pair production [21].

4. Other Resonance Searches

ATLAS and CMS have performed a wide range of searches of new heavy particles decaying into at least one top quark that do not form part of the VLQ, LQ, or supersymmetry search programmes. They are usually referred simply as ‘resonance searches’. Results of many of these searches are interpreted in the context of simplified models that predict the presence of new heavy gauge bosons, several of them considering benchmarks in which the new resonance couples preferentially or exclusively to third generation quarks. A description of the benchmark models used in the interpretation and the relevant phenomenology is found in Section 1.3.

They are generally characterised by the attempt to partially or fully reconstruct the target resonance and look for excesses in the invariant mass or similar variable above the standard model background. They can be classified looking at the type of resonance decays:

• resonances decaying into a $t\overline{t}$ pair;
• resonances decaying into $tb$ (Used to refer both to both $t\overline{b}$ and $\overline{t}b$);
• resonances decaying into any other final state including at least one top quark.

There is another large category of searches in which the top quarks are present in the final state but do not originate from the resonance, this is the case, for example, of dark matter searches produced in association with $tW$ [110] or $t\overline{t}$ [111]. These searches are not described in detail in this review.

4.1. Searches for Resonances Decaying into Two Top Quarks

Both collaborations have a long history of $t\overline{t}$ analyses in different channels targeting mainly heavy vector resonances [48,112,113,114,115,116,117,118,119,120,121,122,123]. ATLAS has recent results in the all-hadronic [122] and lepton-plus-jets final states [48] while CMS has recent results on all possible final sates in a combined paper [123].

The lepton-plus-jets ATLAS search uses an integrated luminosity of 36.1 fb${}^{-1}$. Events with exactly one isolated lepton and additional jets, some of them b-tagged are selected. The analysis distinguishes between a boosted and a resolved regime and uses distinct approaches to reconstruct the top-quark candidates in each case and obtain the invariant mass of the $t\overline{t}$ pair which is used to perform the search. The boosted selection relies on top-tagged large-R jets to reconstruct the hadronic top while the resolved selection relies on a ${\chi }^2$ algorithm to find the best assignment of jets to the top candidates. Upper limits are set on the production of new heavy particles for a variety of scenarios including new Z’ bosons arising in TC2 models with narrow relative widths, Z’ from simplified models of dark matter, spin-2 Kaluza-Klein gravitons and spin-1 Kaluza-Klein gluons. Narrow Z’ TC2 bosons (with a 1% relative width) are excluded for masses up to 3 TeV.

The combined CMS search paper, using an integrated luminosity of 35.9 fb${}^{-1}$, follows a similar approach. The search is performed in the invariant mass of the $t\overline{t}$ system for the lepton-plus-jets and the all-hadronic channels. The boosted lepton-plus-jets and all-hadronic channels use top-tagging large-R jets to identify the hadronic tops while the resolved lepton-plus-jets channel uses a similar ${\chi }^2$ method as the ATLAS search. For the dilepton channel, however, the scalar sum of all of the leptons, jets and $E_T^{miss}$ of the events is considered instead due to the difficulty of properly reconstructing the $t\overline{t}$ system. The lepton-plus-jets channel introduces a BDT that rejects additional W-plus-jets background, improving the sensitivity. The statistical combination of the three channels allow for a 1% relative-width TC2 Z’ boson to be excluded for masses up to 3.8 TeV. The search also includes interpretations for additional Z’ widths, up to 30% relative width, and a Kaluza-Klein gluon.

ATLAS has recently released a all-hadronic search with an integrated luminosity of 139 fb${}^{-1}$. The search uses a Deep Neural Network (DNN) based approach to identify two boosted top quarks in events with two large-R jets, reducing the multijet background contamination, and a functional-form fit to estimate the background in the most sensitive regions as a function of $m_{t\overline{t}}$. Two signal regions are constructed with different number of b-tagged jets. The search provides an exclusion of narrow TC2 Z’ masses up to 3.9 TeV.

A comparison between the three aforementioned results for narrow Z’ resonances is shown in Figure 38.

As described in Section 1.3, searches for scalar or pseudo-scalar particles in the $t\overline{t}$ final state need to consider interference effects. CMS published a search of this kind using 35.9 fb${}^{-1}$ of integrated luminosity [55]. In addition of dealing properly with the interference effect in the statistical analysis, the search uses a series of angular variables to differentiate between scalar and pseudo-scalar hypotheses and discriminate between signal and background. In the single-lepton channel the variable $|cos{\theta }_{tl}^{*}|$ is used, where ${\theta }_{tl}^{*}$ denotes the angle between the momentum of the semileptonically decaying top quark in the $t\overline{t}$ rest frame and the momentum of the $t\overline{t}$ system. In the dilepton channel the chosen variable is the cosine of the angle between the lepton momenta in their respective helicity frames. Both channels are combined in a simultaneous fit using both the aforementioned angular variables and the reconstructed $m_{t\overline{t}}$. Using this combined fit the search sets model independent constraints on the coupling strength of the new particle to top quarks for several relative widths. A moderate signal-like deviation is observed, compatible with a pseudo-scalar boson with a global significance of 1.9 standard deviations. Upper limits on the production cross section showing this excess are shown in Figure 39 (left). Model dependent limits are also set using the hMSSM model. The latest ATLAS result with a similar target was published with a centre-of-mass energy of 8 TeV [115], providing an interpretation based on a 2HDM model with different parameters scenarios.

Another alternative when considering models with exclusive, or very dominant, couplings to top quarks is to perform searches for associated production of $t\overline{t}$, in which the resonance is produced accompanied with additional top quarks. Two searches have been recently published dealing with this type of production.

CMS published an analysis on a 4-top final state using 137 fb${}^{-1}$ of integrated luminosity [57], including an interpretation of scalar and pseudoscalars in a 2HDM model. Events are selected with 2 same-sign leptons as well as those with more than 3 leptons, in addition to a large number of jets. In order to separate 4-top-quark events from the sum of SM backgrounds a BDT discriminant output is used. Different regions are defined using the number of jets and how many of them are b-tagged and a binned likelihood is constructed using the yields of the different processes from those regions. Limits are set on the production of new scalar or pseudoscalar particles decaying to $t\overline{t}$ and on the coupling of the top quark and light new scalar or vector heavy particles. Such limits in the context of a pseudoscalar 2HDM model are shown in Figure 39 (right).

ATLAS has recently published preliminary results on the associated production of a Z’ boson with exclusive couplings to top quarks in the all-hadronic channel [124]. Events with a large number of jets are selected and categorised using the number of jets and whether they are b-tagged or not. A background method combining MC predictions and a functional form is used to estimate the background in the most sensitive region, with 4 or more b-tagged jets. The binned likelihood fit is performed in the mass of the two leading large-R jets which are used as proxies for the two highly boosted top quarks assumed to originate from the Z’ resonance. Limits are set on the production cross-section of a Z’ but no exclusion is obtained for the considered model in the explored mass range between 1 and 3 TeV.

Inclusive $t\overline{t}$ searches have improved recently in their high mass reach, taking advantage of the developments in top and b-tagging, and relying mainly on hadronic channels. Leptonic channels provide necessary complementarity at lower Z’ masses (below 1.5 TeV). The introduction of machine learning classification methods and dedicated treatment of highly boosted leptonic tops have enhanced CMS sensitivity, even with smaller integrated luminosities, and provide a blueprint for ATLAS searches to improve. Associated production searches have so far been mainly focused on the low mass regime, relevant for scalar and pseudo-scalar new particles in 2HDM models and rely on leptonic final states. Fully hadronic associated production is completely uncovered and represent a future challenge. Additional associated production analyses aiming at higher masses and a revision of the interpretation strategy, for example including models with explicit LFUV implications are also interesting extensions of the programme. Finally, only a mild excess has been observed, for a low mass pseudoscalar particle. This excess should increase the scrutiny in the low mass region for other searches considering pseudo-scalar particles and future iterations of similar analyses.

4.2. Searches for Resonances Decaying into One Top Quark and a b-Quark

As for the $t\overline{t}$ case, searches in a final state of $tb$ have been performed with several datasets by both ATLAS and CMS [125,126,127,128,129,130,131,132,133]. Recent results has been published by both ATLAS and CMS in the lepton-plus-jets and all-hadronic channels and a combination of both channels was published by ATLAS with 36 fb${}^{-1}$.

The latest lepton+jets results were published by ATLAS [130] and CMS [131] with 36 fb${}^{-1}$ of integrated luminosity. Both analyses follow a similar pattern: events with exactly one isolated lepton, missing transverse energy and additional jets are selected and additional kinematic cuts are used to separate signal and background. The top quark candidate is identified using a method that uses the W boson and top-quark mass to choose the most likely combination of objects and the search is performed in the reconstructed $m_{\mathrm{tb}}$ distribution. The CMS search optimises the search separately for two different $m_{\mathrm{tb}}$ regimes which allows to exclude masses up to 3.6 TeV for right handed heavy W bosons (W’). Exclusion limits are also obtained for W’ bosons as a function of their coupling strength to left and right-handed fermions. The ATLAS search only excludes right-handed W’ masses up to 3.15 TeV but includes a combination with a previously published all-hadronic analysis [127], extending the exclusion up to 3.25 TeV and additional interpretation for right-handed W’ bosons with larger and smaller couplings, which translates to different relative widths. The ATLAS search assumes that the right handed W’ is leptophobic while the CMS search considers two distinct scenarios. A comparison of the limits obtained in the two searches is shown in Figure 40.

The latest all-hadronic results were published by CMS using 137 fb${}^{-1}$ [132] while preliminary results were released by ATLAS using 139 fb ${}^{-1}$ [133]. Both searches select events based on the number of large-R jets and small-R jets and use DNN based top-taggers and b-taggers to identify the top quark and b-quark candidates. The dominant background, from multijet production, is obtained using slightly different two-dimensional sideband methods. Data in control regions failing top-tagging or b-tagging requirements are used to estimate the background in the most sensitive regions. The CMS search sets limits on right handed and left handed W’ bosons hypotheses, excluding masses up to 3.4 TeV in models in which the W’ couples to the same particles as the SM W boson. The left-handed hypothesis has a substantial interference with single-top production in the s-channel that is properly taken into account. The ATLAS preliminary result sets limits only considering right handed W’ bosons, excluding masses up to 4.4 TeV for models in which the W’ is leptophobic. A comparison between the right handed W’ limits obtained by both analyses is shown in Figure 41.

Associated production searches also take place for $tb$ resonances, gaining importance for searches with exclusive or dominant couplings of the new particle to third generation quarks. ATLAS has recently published a result for a charged Higgs decaying into $tb$ with 139 fb${}^{-1}$ of integrated luminosity [134] while the latest CMS result targeting the same process was published with 35.9 fb${}^{-1}$ in the all-hadronic channel [56]. The CMS search selects events with a large number of jets, some of them b-tagged and distinguishes between a boosted and a resolved approach aiming at different $m_{tb}$ ranges. Several signal categories of different jet multiplicities are defined. In the resolved analysis a BDT classifier is used to increase the sensitivity. The search is performed using both the reconstructed $m_{\mathit{tb}}$ mass and the scalar sum of the momentum of all selected jets in each event ($H_T$). Model independent limits are set on the production of a heavy charged higgs and model dependent ones are set for several interpretations such as the hMSSM or the $M_h^{125}\left(\tilde{\chi }\right)$ benchmark scenario. The ATLAS search considers events with one isolated lepton in the final state with several additional jets, some of them b-tagged. Different categories are defined for different jet and b-jet multiplicities. A Neural Network (NN) algorithm is used to enhance the separation between signal and background and is used in the likelihood fit. Limits are set in the context of the hMSSM and several $M_h^{125}$ scenarios. The comparison of the limits obtained for both searches in the tan-$\beta$ vs $m_{H^{\pm }}$ plane for an hMSSM model is shown in Figure 42.

As in the $t\overline{t}$ case the high mass results for inclusive searches have recently been improved largely by taking advantage of top and b-tagging developments. With fairly similar strategies CMS leptonic-channel analyses have improved reach at high mass through better treatment of boosted leptons but are comparable at low mass (below 1.5 TeV). The opposite can be said for the hadronic channel, where an improved strategy has enhanced the sensitivity to masses beyond 3 TeV for ATLAS searches. Associated production has only been studied in the context of a charged Higgs and using leptonic channels. Other interpretations and final states remain uncovered.

4.3. Other Resonances Decaying into at Least One Top Quark

In addition to $t\overline{t}$ and $tb$ resonant decays there are few other final states considered by ATLAS and CMS involving top quarks. CMS has published three searches on excited bottom and top quarks, considering the decay $b^{*}\to tW$ [135,136] and one on excited top quarks, considering the decay $t^{*}\to tg$ [137]. ATLAS has published one additional search in a monotop final state, in which the heavy resonance decays into one top and a dark matter particle, yielding a final state of a single top quark and missing transverse energy [61].

The two excited b-quark searches performed by CMS with an integrated luminosity of 137 fb${}^{-1}$ are complementary, a publication on the fully hadronic final state [136] and another one in the lepton plus jets final state [135]. In the all-hadronic search events with large-R jets are selected and W-tagging and top-tagging are used to identify events with one top-candidate and one W candidate. The search is performed on the invariant mass of the tW system. A two-dimensional sideband method is used to estimate the dominant multijet background using CR in which the candidate jets fail the top or W-tagging. Limits are set on different types of excited b-quark, with left-handed, right-handed and vector-like chiralities, excluding masses up to 2.6, 2.8 and 3.1 TeV, respectively. Upper limits for the vector-like case are shown in Figure 43. The preliminary lepton-plus-jets results utilises a selection based on an isolated lepton, missing transverse energy and additional jets. The lepton is assumed to originate from the W boson and the top quark is reconstructed using a variable-R jet algorithm [138] that aims to identify top quarks in both the resolved and boosted regimes. Events are further categorised using the number of b-jets. Backgrounds from miss-identified top quarks in the 1 b-jet and 2 b-jet regions are identified using the data in the 0 b-jet regions and a transfer factor from simulated events. Limits are set on the same interpretations as the all-hadronic results, excluding masses up to 2.95, 3.03 and 3.22 TeV respectively.

The excited top quark search was published by CMS using 35.9 $f{b}^{-1}$ of integrated luminosity [137], and it looks for the pair production of excited top quarks in a lepton-plus-jets final state. Events are required to have an isolated lepton, missing transverse energy and exactly six jets, two of them b-tagged. The top-candidates are reconstructed by minimising a metric based on the masses of the W boson and the top quarks, as recorded by the particle data group [139], and the reconstructed detector resolution of simulated particles. The background is estimated using a functional form directly in the signal regions as a function of the mass of the excited top-quark candidate. Upper limits are set on an excited top-quark pair production, excluding masses up to 1.2 TeV for a model with 100% branching ratio into a top quark and a gluon. They are shown in Figure 43.

The monotop search published by ATLAS uses 36.1 fb${}^{-1}$ of integrated luminosity [61]. The search considers three distinct interpretations with similar final states: non-resonant dark-matter produced in associated with a top quark, a VLT decaying into a Z-boson and a top quark and a coloured scalar decaying into a top quark and a dark matter particle. The heavy scalar search focuses only on the all-hadronic channel, using final states with a top-tagged large-R jet and high missing transverse energy. The search is performed on the distribution of the transverse mass of the top-tagged large-R jet and the $E_T^{miss}$ system. Additional kinematic variables, such as requiring both objects to be back-to-back, are used to separate signal and background. A two-dimensional sideband method is used to estimate the multijet background, using regions with zero top-tagged large-R jets and failing back-to-back requirement. Limits are set on the production of the new scalar, excluding masses up to 3.4 TeV for a 10 GeV dark matter particle mass and a specific set of coupling values. Limits are also set on the coupling vs mass plane for a lighter dark particle of 1 GeV and are shown in Figure 44.

As in $t\overline{t}$ or $tb$ searches, the exotic quarks searches present a complementarity between leptonic and hadronic channels, where the former target low mass regime while the latter extend the high mass reach. They have only been tackled by the CMS experiment and there are plenty of decay modes and channels left uncovered. In the case of monotop searches, only tackled by ATLAS, the resonant production of dark-matter considers only the all-hadronic channel, given the low sensitivity of the leptonic one. Future iterations should profit from similar improvements as the latest versions of $t\overline{t}$ and $tb$ searches, allowing for high mass enhancements without significant changes in strategy.

5. Summary and Outlook

In this article a comprehensive review of the most recent results from the ATLAS and CMS collaborations on searches with third generation quarks in the final state has been presented. It includes the latest VLQ and LQ results and a selection of other resonance searches with at least one top-quark in the final state.

ATLAS and CMS are gearing up for the upcoming start of Run-3 data taking. As Run-3 data become available, they can be combined to the existing Run-2 dataset to increase the available integrated luminosity for searches. In the meantime, one way of improving the sensitivity of searches without having access to additional statistics or experimental improvements is to combine the existing searches under a common underlying model. In addition to the sensitivity increase, it allows for a more complete interpretation of realistic models that may have a varied phenomenology. This approach has been successfully implemented for VLQ searches in ATLAS, as discussed in Section 2 and in heavy resonances decaying into leptonic and bosonic final states [140,141], among others. The extension of these efforts for singly produced VLQs, LQ and W’ or Z’ bosons decaying into third generation quarks represents an interesting prospect.

Both collaborations have focused their attention on the single production of vector-like quarks for the first round of publications with Run-2 data. This was a reasonable approach, given that the excluded mass ranges for VLQs from pair production searches were already around 1.2–1.3 TeV for most benchmark scenarios. However, single and pair production searches provide complementary sensitivities to different ranges of couplings and a future combination involving both types of searches would be beneficial in maximising the search potential across the parameter space. Searches for the exotic production and decay of vector-like quarks are beginning to appear in the literature and more work in this direction can be anticipated.

The interest in leptoquarks searches has grown in recent years, in light of observed anomalies in the flavour sector. A large portion of the parameter space for these searches still remain unexplored, especially in the context of single production searches. Future searches can improve the parameter space coverage, and make way for a statistical combination to further improve the reach.

The inclusive searches for heavy resonances decaying into $t\overline{t}$ and $tb$ final states included in Section 4 have recently obtained large sensitivity enhancements at high masses, taking advantage of new methods of top and b-quark tagging. The high mass sensitivity of these searches is unlikely to be improved significantly in the short term once they are updated to the full Run-2 dataset. However, improvements in resolved top-quark identification, boosted leptonic top treatment and the introduction of new background techniques, can help in the low and intermediate mass ranges where fully hadronic final states are not dominant.

Although much of the same can be said of searches for associated production, the field is still to be fully explored. Many channels and specific final states remain fully or partially uncovered. New ideas include high-mass dedicated analyses of $tttt$ and $ttbb$ final sates or the introduction of more inclusive searches such as $t\overline{t}bX$, where X refers to the inclusiveness of the process. The latter case has been shown to be the most sensitive choice in an important fraction of parameters space for several BSM models [142]. As mentioned in Section 1.3, associated production is particularly relevant for models with preferential, or exclusive, couplings with third generation leptons and quarks. Additional searches may provide insight to help build models with LFUV interactions that offer an explanation to the flavour anomalies observed in recent experimental results in B physics.

Other searches with top quarks in the final state may also play an important role in the near future as both collaborations try to find final states that have not been explored yet. In addition, new iterations of the monotop or excited top-quark searches, with the full Run-2 dataset, are also expected to improve their sensitivity in a similar manner as the latest iteration of $t\overline{t}$ and $tb$ searches.

Looking further into the future, to the High-Luminosity upgrade of the LHC [143], a large increase of integrated luminosity is expected that will increase the high mass reach of most of the searches discussed. The higher centre-of-mass energy may also allow access to new resonances that were suppressed before. However, the challenging conditions of the HL-LHC, with a substantial increase in the average number of interactions per bunch crossing, represent a new environment that the collaborations will need to navigate. Finally, new developments in the trigger system of both detectors [144,145] are in preparation and may provide new capabilities to trigger on the complex final states that characterise many of the analyses discussed in this review.